From 5c6558c1e50c6dd4554f43c9ff8c0b4f9e6aa573 Mon Sep 17 00:00:00 2001 From: luizhsuperti <62964489+luizhsuperti@users.noreply.github.com> Date: Sat, 8 Feb 2025 17:03:05 +0100 Subject: [PATCH 1/4] Update .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index fd1a6a21..e40cd467 100644 --- a/.gitignore +++ b/.gitignore @@ -170,3 +170,4 @@ todos.txt # experiments/ +cluster-experiments.code-workspace From b6babccdf976ae1f353e64dd6a4f12ef52285e7e Mon Sep 17 00:00:00 2001 From: luizhsuperti Date: Tue, 25 Feb 2025 20:25:57 +0100 Subject: [PATCH 2/4] Revamp documentation --- README.md | 281 ++++----------------------------- docs/quickstart.md | 134 ++++++++++++++++ docs/stylesheets/overrides.css | 13 ++ docs/stylesheets/style.css | 9 ++ mkdocs.yml | 74 ++++++--- 5 files changed, 237 insertions(+), 274 deletions(-) create mode 100644 docs/quickstart.md create mode 100644 docs/stylesheets/overrides.css create mode 100644 docs/stylesheets/style.css diff --git a/README.md b/README.md index 6b35f64b..30ae29c7 100644 --- a/README.md +++ b/README.md @@ -11,265 +11,46 @@ https://codecov.io/gh/david26694/cluster-experiments/branch/main/graph/badge.svg ![License](https://img.shields.io/github/license/david26694/cluster-experiments) [![Pypi version](https://img.shields.io/pypi/pyversions/cluster-experiments.svg)](https://pypi.python.org/pypi/cluster-experiments) -A Python library for end-to-end A/B testing workflows, featuring: -- Experiment analysis and scorecards -- Power analysis (simulation-based and normal approximation) -- Variance reduction techniques (CUPED, CUPAC) -- Support for complex experimental designs (cluster randomization, switchback experiments) +**`cluster experiments`** is a comprehensive Python library for end-to-end A/B testing workflows, designed for seamless integration with Pandas in production environments. -## Key Features +--- -### 1. Power Analysis -- **Simulation-based**: Run Monte Carlo simulations to estimate power -- **Normal approximation**: Fast power estimation using CLT -- **Minimum Detectable Effect**: Calculate required effect sizes -- **Multiple designs**: Support for: - - Simple randomization - - Variance reduction techniques in power analysis - - Cluster randomization - - Switchback experiments -- **Dict config**: Easy to configure power analysis with a dictionary +## πŸš€ Key Features -### 2. Experiment Analysis -- **Analysis Plans**: Define structured analysis plans -- **Metrics**: - - Simple metrics - - Ratio metrics -- **Dimensions**: Slice results by dimensions -- **Statistical Methods**: - - GEE - - Mixed Linear Models - - Clustered / regular OLS - - T-tests - - Synthetic Control -- **Dict config**: Easy to define analysis plans with a dictionary +### πŸ“Œ **Experiment Design & Planning** +- **Power analysis** and **Minimal Detectable Effect (MDE)** estimation +- Support for **complex experimental designs**, including: + - 🏒 **Cluster randomization** + - πŸ”„ **Switchback experiments** -### 3. Variance Reduction -- **CUPED** (Controlled-experiment Using Pre-Experiment Data): - - Use historical outcome data to reduce variance, choose any granularity - - Support for several covariates -- **CUPAC** (Control Using Predictors as Covariates): - - Use any scikit-learn compatible estimator to predict the outcome with pre-experiment data +### πŸ›  **Data Preprocessing** +- Tools for **efficient data preparation** +- Seamlessly integrates with **Pandas** for streamlined workflows -## Quick Start +### πŸ“Š **Comprehensive Experiment Analysis** +##### **βœ… Metrics** +- Simple and **ratio-based metrics** for evaluating experiment outcomes -### Power Analysis Example +##### **πŸ“ˆ Statistical Methods** +- πŸ“Œ **Generalized Estimating Equations (GEE)** +- πŸ“Œ **Mixed Linear Models** for robust inference +- πŸ“Œ **Ordinary Least Squares (OLS)** and **Clustered OLS** with covariates +- πŸ“Œ **T-tests** with variance reduction techniques (**CUPED, CUPAC**) +- πŸ“Œ **Synthetic control methods** for causal inference in observational studies -```python -import numpy as np -import pandas as pd -from cluster_experiments import PowerAnalysis, NormalPowerAnalysis +--- -# Create sample data -N = 1_000 -df = pd.DataFrame({ - "target": np.random.normal(0, 1, size=N), - "date": pd.to_datetime( - np.random.randint( - pd.Timestamp("2024-01-01").value, - pd.Timestamp("2024-01-31").value, - size=N, - ) - ), -}) +### ⚑ Why Use `cluster experiments`? +βœ… **Production-ready** – built for real-world applications +βœ… **Data-driven decision-making** – designed for rigorous statistical analysis +βœ… **Easy to work** – integrates effortlessly with Pandas -# Simulation-based power analysis with CUPED -config = { - "analysis": "ols", - "perturbator": "constant", - "splitter": "non_clustered", - "n_simulations": 50, -} -pw = PowerAnalysis.from_dict(config) -power = pw.power_analysis(df, average_effect=0.1) +--- -# Normal approximation (faster) -npw = NormalPowerAnalysis.from_dict({ - "analysis": "ols", - "splitter": "non_clustered", - "n_simulations": 5, - "time_col": "date", -}) -power_normal = npw.power_analysis(df, average_effect=0.1) -power_line_normal = npw.power_line(df, average_effects=[0.1, 0.2, 0.3]) +`cluster experiments` empowers analysts and data scientists with **scalable, reproducible, and statistically robust** A/B testing workflows. +πŸ”— **Get Started:** [Documentation Link] -# MDE calculation -mde = npw.mde(df, power=0.8) - -# MDE line with length -mde_timeline = npw.mde_time_line( - df, - powers=[0.8], - experiment_length=[7, 14, 21] -) - -print(power, power_line_normal, power_normal, mde, mde_timeline) -``` - -### Experiment Analysis Example - -```python -import numpy as np -import pandas as pd -from cluster_experiments import AnalysisPlan - -N = 1_000 -experiment_data = pd.DataFrame({ - "order_value": np.random.normal(100, 10, size=N), - "delivery_time": np.random.normal(10, 1, size=N), - "experiment_group": np.random.choice(["control", "treatment"], size=N), - "city": np.random.choice(["NYC", "LA"], size=N), - "customer_id": np.random.randint(1, 100, size=N), - "customer_age": np.random.randint(20, 60, size=N), -}) - -# Create analysis plan -plan = AnalysisPlan.from_metrics_dict({ - "metrics": [ - {"alias": "AOV", "name": "order_value"}, - {"alias": "delivery_time", "name": "delivery_time"}, - ], - "variants": [ - {"name": "control", "is_control": True}, - {"name": "treatment", "is_control": False}, - ], - "variant_col": "experiment_group", - "alpha": 0.05, - "dimensions": [ - {"name": "city", "values": ["NYC", "LA"]}, - ], - "analysis_type": "clustered_ols", - "analysis_config": {"cluster_cols": ["customer_id"]}, -}) -# Run analysis -print(plan.analyze(experiment_data).to_dataframe()) -``` - -### Variance Reduction Example - -```python -import numpy as np -import pandas as pd -from cluster_experiments import ( - AnalysisPlan, - SimpleMetric, - Variant, - Dimension, - TargetAggregation, - HypothesisTest -) - -N = 1000 - -experiment_data = pd.DataFrame({ - "order_value": np.random.normal(100, 10, size=N), - "delivery_time": np.random.normal(10, 1, size=N), - "experiment_group": np.random.choice(["control", "treatment"], size=N), - "city": np.random.choice(["NYC", "LA"], size=N), - "customer_id": np.random.randint(1, 100, size=N), - "customer_age": np.random.randint(20, 60, size=N), -}) - -pre_experiment_data = pd.DataFrame({ - "order_value": np.random.normal(100, 10, size=N), - "customer_id": np.random.randint(1, 100, size=N), -}) - -# Define test -cupac_model = TargetAggregation( - agg_col="customer_id", - target_col="order_value" -) - -hypothesis_test = HypothesisTest( - metric=SimpleMetric(alias="AOV", name="order_value"), - analysis_type="clustered_ols", - analysis_config={ - "cluster_cols": ["customer_id"], - "covariates": ["customer_age", "estimate_order_value"], - }, - cupac_config={ - "cupac_model": cupac_model, - "target_col": "order_value", - }, -) - -# Create analysis plan -plan = AnalysisPlan( - tests=[hypothesis_test], - variants=[ - Variant("control", is_control=True), - Variant("treatment", is_control=False), - ], - variant_col="experiment_group", -) - -# Run analysis -results = plan.analyze(experiment_data, pre_experiment_data) -print(results.to_dataframe()) -``` - -## Installation - -You can install this package via `pip`. - -```bash -pip install cluster-experiments -``` - -For detailed documentation and examples, visit our [documentation site](https://david26694.github.io/cluster-experiments/). - -## Features - -The library offers the following classes: - -* Regarding power analysis: - * `PowerAnalysis`: to run power analysis on any experiment design, using simulation - * `PowerAnalysisWithPreExperimentData`: to run power analysis on a clustered/switchback design, but adding pre-experiment df during split and perturbation (especially useful for Synthetic Control) - * `NormalPowerAnalysis`: to run power analysis on any experiment design using the central limit theorem for the distribution of the estimator. It can be used to compute the minimum detectable effect (MDE) for a given power level. - * `ConstantPerturbator`: to artificially perturb treated group with constant perturbations - * `BinaryPerturbator`: to artificially perturb treated group for binary outcomes - * `RelativePositivePerturbator`: to artificially perturb treated group with relative positive perturbations - * `RelativeMixedPerturbator`: to artificially perturb treated group with relative perturbations for positive and negative targets - * `NormalPerturbator`: to artificially perturb treated group with normal distribution perturbations - * `BetaRelativePositivePerturbator`: to artificially perturb treated group with relative positive beta distribution perturbations - * `BetaRelativePerturbator`: to artificially perturb treated group with relative beta distribution perturbations in a specified support interval - * `SegmentedBetaRelativePerturbator`: to artificially perturb treated group with relative beta distribution perturbations in a specified support interval, but using clusters -* Regarding splitting data: - * `ClusteredSplitter`: to split data based on clusters - * `FixedSizeClusteredSplitter`: to split data based on clusters with a fixed size (example: only 1 treatment cluster and the rest in control) - * `BalancedClusteredSplitter`: to split data based on clusters in a balanced way - * `NonClusteredSplitter`: Regular data splitting, no clusters - * `StratifiedClusteredSplitter`: to split based on clusters and strata, balancing the number of clusters in each stratus - * `RepeatedSampler`: for backtests where we have access to counterfactuals, does not split the data, just duplicates the data for all groups - * Switchback splitters (the same can be done with clustered splitters, but there is a convenient way to define switchback splitters using switch frequency): - * `SwitchbackSplitter`: to split data based on clusters and dates, for switchback experiments - * `BalancedSwitchbackSplitter`: to split data based on clusters and dates, for switchback experiments, balancing treatment and control among all clusters - * `StratifiedSwitchbackSplitter`: to split data based on clusters and dates, for switchback experiments, balancing the number of clusters in each stratus - * Washover for switchback experiments: - * `EmptyWashover`: no washover done at all. - * `ConstantWashover`: accepts a timedelta parameter and removes the data when we switch from A to B for the timedelta interval. -* Regarding analysis methods: - * `GeeExperimentAnalysis`: to run GEE analysis on the results of a clustered design - * `MLMExperimentAnalysis`: to run Mixed Linear Model analysis on the results of a clustered design - * `TTestClusteredAnalysis`: to run a t-test on aggregated data for clusters - * `PairedTTestClusteredAnalysis`: to run a paired t-test on aggregated data for clusters - * `ClusteredOLSAnalysis`: to run OLS analysis on the results of a clustered design - * `OLSAnalysis`: to run OLS analysis for non-clustered data - * `DeltaMethodAnalysis`: to run Delta Method Analysis for clustered designs - * `TargetAggregation`: to add pre-experimental data of the outcome to reduce variance - * `SyntheticControlAnalysis`: to run synthetic control analysis -* Regarding experiment analysis workflow: - * `Metric`: abstract class to define a metric to be used in the analysis - * `SimpleMetric`: to create a metric defined at the same level of the data used for the analysis - * `RatioMetric`: to create a metric defined at a lower level than the data used for the analysis - * `Variant`: to define a variant of the experiment - * `Dimension`: to define a dimension to slice the results of the experiment - * `HypothesisTest`: to define a Hypothesis Test with a metric, analysis method, optional analysis configuration, and optional dimensions - * `AnalysisPlan`: to define a plan of analysis with a list of Hypothesis Tests for a dataset and the experiment variants. The `analyze()` method runs the analysis and returns the results - * `AnalysisResults`: to store the results of the analysis -* Other: - * `PowerConfig`: to conveniently configure `PowerAnalysis` class - * `ConfidenceInterval`: to store the data representation of a confidence interval - * `InferenceResults`: to store the structure of complete statistical analysis results +πŸ“¦ **Installation:** +```sh +pip install cluster-experiments \ No newline at end of file diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 00000000..7abf05c4 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,134 @@ +# Quickstart + +## Installation + +You can install **Cluster Experiments** via pip: + +```bash +pip install cluster-experiments +``` + +!!! info "Python Version Support" + **Cluster Experiments** requires **Python 3.9 or higher**. Make sure your environment meets this requirement before proceeding with the installation. + +--- + +## Usage + +Designing and analyzing experiments can feel overwhelming at times. After formulating a testable hypothesis, +you're faced with a series of routine tasks. From collecting and transforming raw data to measuring the statistical significance of your experiment results and constructing confidence intervals, +it can quickly become a repetitive and error-prone process. +*Cluster Experiments* is here to change that. Built on top of well-known packages like `pandas`, `numpy`, `scipy` and `statsmodels`, it automates the core steps of an experiment, streamlining your workflow, saving you time and effort, while maintaining statistical rigor. +## Key Features +- **Modular Design**: Each componentβ€”`Splitter`, `Perturbator`, and `Analysis`β€”is independent, reusable, and can be combined in any way you need. +- **Flexibility**: Whether you're conducting a simple A/B test or a complex clustered experiment, Cluster Experiments adapts to your needs. +- **Statistical Rigor**: Built-in support for advanced statistical methods ensures that your experiments maintain high standards, including clustered standard errors and variance reduction techniques like CUPED and CUPAC. + +The core functionality of *Cluster Experiments* revolves around several intuitive, self-contained classes and methods: + +- **Splitter**: Define how your control and treatment groups are split. +- **Perturbator**: Specify the type of effect you want to test. +- **Analysis**: Perform statistical inference to measure the impact of your experiment. + + +--- + +### `Splitter`: Defining Control and Treatment Groups + +The `Splitter` classes are responsible for dividing your data into control and treatment groups. The way you split your data depends on the **metric** (e.g., simple, ratio) you want to observe and the unit of observation (e.g., users, sessions, time periods). + +#### Features: + +- **Randomized Splits**: Simple random assignment of units to control and treatment groups. +- **Stratified Splits**: Ensure balanced representation of key segments (e.g., geographic regions, user cohorts). +- **Time-Based Splits**: Useful for switchback experiments or time-series data. + +```python +from cluster_experiments import RandomSplitter + +splitter = RandomSplitter( + cluster_cols=["cluster_id"], # Split by clusters + treatment_col="treatment", # Name of the treatment column +) +``` + +--- + +### `Perturbator`: Simulating the Treatment Effect + +The `Perturbator` classes define the type of effect you want to test. It simulates the treatment effect on your data, allowing you to evaluate the impact of your experiment. + +#### Features: + +- **Absolute Effects**: Add a fixed uplift to the treatment group. +- **Relative Effects**: Apply a percentage-based uplift to the treatment group. +- **Custom Effects**: Define your own effect size or distribution. + +```python +from cluster_experiments import ConstantPerturbator + +perturbator = ConstantPerturbator( + average_effect=5.0 # Simulate a nominal 5% uplift +) +``` + +--- + +### `Analysis`: Measuring the Impact + +Once your data is split and the treatment effect is applied, the `Analysis` component helps you measure the statistical significance of the experiment results. It provides tools for calculating effects, confidence intervals, and p-values. + +You can use it for both **experiment design** (pre-experiment phase) and **analysis** (post-experiment phase). + +#### Features: + +- **Statistical Tests**: Perform t-tests, OLS regression, and other hypothesis tests. +- **Effect Size**: Calculate both absolute and relative effects. +- **Confidence Intervals**: Construct confidence intervals for your results. + +Example: + +```python +from cluster_experiments import TTestClusteredAnalysis + +analysis = TTestClusteredAnalysis( + cluster_cols=["cluster_id"], # Cluster-level analysis + treatment_col="treatment", # Name of the treatment column + target_col="outcome" # Metric to analyze +) +``` + +--- + +### Putting It All Together for Experiment Design + +You can combine all classes as inputs in the `PowerAnalysis` class, where you can analyze different experiment settings, power lines, and Minimal Detectable Effects (MDEs). + +```python +from cluster_experiments import PowerAnalysis +from cluster_experiments import RandomSplitter, ConstantPerturbator, TTestClusteredAnalysis + +# Define the components +splitter = RandomSplitter(cluster_cols=["cluster_id"], treatment_col="treatment") +perturbator = ConstantPerturbator(average_effect=0.1) +analysis = TTestClusteredAnalysis(cluster_cols=["cluster_id"], treatment_col="treatment", target_col="outcome") + +# Create the experiment +experiment = PowerAnalysis( + perturbator=perturbator, + splitter=splitter, + analysis=analysis, + target_col="outcome", + treatment_col="treatment" +) + +# Run the experiment +results = experiment.power_analysis() +``` + +--- + +## Next Steps + +- Explore the **Core Documentation** for detailed explanations of each component. +- Check out the **Usage Examples** for practical applications of the package. diff --git a/docs/stylesheets/overrides.css b/docs/stylesheets/overrides.css new file mode 100644 index 00000000..9b029d3c --- /dev/null +++ b/docs/stylesheets/overrides.css @@ -0,0 +1,13 @@ +/* Custom admonition styling */ +.md-typeset .admonition { + border-radius: 8px; + border-left: 4px solid var(--md-primary-fg-color); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); +} + +/* Code block styling */ +.md-typeset pre { + border-radius: 8px; + background-color: var(--md-code-bg-color); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); +} \ No newline at end of file diff --git a/docs/stylesheets/style.css b/docs/stylesheets/style.css new file mode 100644 index 00000000..c22863bb --- /dev/null +++ b/docs/stylesheets/style.css @@ -0,0 +1,9 @@ +/* Apply text justification to all paragraphs in the documentation */ +.md-content p { + text-align: justify; +} + +/* Optionally, justify lists or other specific elements */ +.md-content ul, .md-content ol { + text-align: justify; +} diff --git a/mkdocs.yml b/mkdocs.yml index 6ded671c..a329eb30 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,32 +1,17 @@ site_name: Cluster Experiments Docs -extra_css: [style.css] repo_url: https://github.com/david26694/cluster-experiments site_url: https://david26694.github.io/cluster-experiments/ site_description: Functions to design and run clustered experiments site_author: David Masip use_directory_urls: false edit_uri: blob/main/docs/ +docs_dir: docs +site_dir: site + nav: - - Home: - - Index: index.md - - Cupac example: cupac_example.ipynb - - Custom classes: create_custom_classes.ipynb - - Switchback: - - Stratified switchback: switchback.ipynb - - Switchback calendar visualization: plot_calendars.ipynb - - Visualization - 4-hour switches: plot_calendars_hours.ipynb - - Multiple treatments: multivariate.ipynb - - AA test clustered: aa_test.ipynb - - Paired T test: paired_ttest.ipynb - - Different hypotheses tests: analysis_with_different_hypotheses.ipynb - - Washover: washover_example.ipynb - - Normal Power: - - Compare with simulation: normal_power.ipynb - - Time-lines: normal_power_lines.ipynb - - Synthetic control: synthetic_control.ipynb - - Experiment analysis workflow: experiment_analysis.ipynb - - Delta Method Analysis: delta_method.ipynb - - API: + - Home: index.md + - Quickstart: quickstart.md + - Core Documentation: - Experiment analysis methods: api/experiment_analysis.md - Perturbators: api/perturbator.md - Splitter: api/random_splitter.md @@ -39,24 +24,65 @@ nav: - Dimension: api/dimension.md - Hypothesis Test: api/hypothesis_test.md - Analysis Plan: api/analysis_plan.md + - Usage Examples: + - CUPAC: cupac_example.ipynb + - Switchback: + - Stratified switchback: switchback.ipynb + - Switchback calendar visualization: plot_calendars.ipynb + - Visualization - 4-hour switches: plot_calendars_hours.ipynb + - Multiple treatments: multivariate.ipynb + - AA test clustered: aa_test.ipynb + - Paired T test: paired_ttest.ipynb + - Different hypotheses tests: analysis_with_different_hypotheses.ipynb + - Washover: washover_example.ipynb + - Normal Power: + - Compare with simulation: normal_power.ipynb + - Time-lines: normal_power_lines.ipynb + - Synthetic control: synthetic_control.ipynb + - Experiment analysis workflow: experiment_analysis.ipynb + - Delta Method Analysis: delta_method.ipynb + - Contribute: + - Contributing Guidelines: development/contributing.md + - Code Structure: development/code_structure.md + - Testing: development/testing.md + - Building Documentation: development/building_docs.md + +extra: + social: + - icon: fontawesome/brands/github + link: https://github.com/david26694/cluster-experiments + plugins: - mkdocstrings: watch: - cluster_experiments - mkdocs-jupyter - search + +extra_css: + - stylesheets/overrides.css + - stylesheets/style.css + copyright: Copyright © 2022 Maintained by David Masip. + theme: name: material font: text: Ubuntu code: Ubuntu Mono - feature: - tabs: true + features: + - content.tabs + - content.code.annotate + - navigation.instant + - navigation.tracking + - navigation.sections + - navigation.top palette: primary: indigo accent: blue + markdown_extensions: + - admonition - codehilite - pymdownx.inlinehilite - pymdownx.superfences @@ -66,4 +92,4 @@ markdown_extensions: - pymdownx.highlight: use_pygments: true - toc: - permalink: true + permalink: true \ No newline at end of file From 6363d80370679cd334ac04f85de66932f56caafe Mon Sep 17 00:00:00 2001 From: luizhsuperti Date: Wed, 26 Feb 2025 17:28:44 +0100 Subject: [PATCH 3/4] Add API subfolder --- mkdocs.yml | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index a329eb30..8dfe50c9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -12,18 +12,19 @@ nav: - Home: index.md - Quickstart: quickstart.md - Core Documentation: - - Experiment analysis methods: api/experiment_analysis.md - - Perturbators: api/perturbator.md - - Splitter: api/random_splitter.md - - Pre experiment outcome model: api/cupac_model.md - - Power config: api/power_config.md - - Power analysis: api/power_analysis.md - - Washover: api/washover.md - - Metric: api/metric.md - - Variant: api/variant.md - - Dimension: api/dimension.md - - Hypothesis Test: api/hypothesis_test.md - - Analysis Plan: api/analysis_plan.md + - API: + - Experiment analysis methods: api/experiment_analysis.md + - Perturbators: api/perturbator.md + - Splitter: api/random_splitter.md + - Pre experiment outcome model: api/cupac_model.md + - Power config: api/power_config.md + - Power analysis: api/power_analysis.md + - Washover: api/washover.md + - Metric: api/metric.md + - Variant: api/variant.md + - Dimension: api/dimension.md + - Hypothesis Test: api/hypothesis_test.md + - Analysis Plan: api/analysis_plan.md - Usage Examples: - CUPAC: cupac_example.ipynb - Switchback: From ba44999d35b0bc464c180e1ec0c9855a8b9ec1f5 Mon Sep 17 00:00:00 2001 From: luizhsuperti Date: Mon, 16 Jun 2025 12:20:29 +0200 Subject: [PATCH 4/4] Fix typos, white spaces and clarified README as per reviewer feedback --- README.md | 53 ++++++++++++++-------------------- docs/quickstart.md | 2 +- docs/stylesheets/overrides.css | 2 +- mkdocs.yml | 29 ++++++++++--------- 4 files changed, 38 insertions(+), 48 deletions(-) diff --git a/README.md b/README.md index 38cf2eb9..aad07e43 100644 --- a/README.md +++ b/README.md @@ -11,47 +11,37 @@ https://codecov.io/gh/david26694/cluster-experiments/branch/main/graph/badge.svg ![License](https://img.shields.io/github/license/david26694/cluster-experiments) [![Pypi version](https://img.shields.io/pypi/pyversions/cluster-experiments.svg)](https://pypi.python.org/pypi/cluster-experiments) -**`cluster experiments`** is a comprehensive Python library for end-to-end A/B testing workflows, designed for seamless integration with Pandas in production environments. +**`cluster experiments`** is a comprehensive Python library for end-to-end A/B testing workflows. --- -## πŸš€ Key Features +## πŸš€ Key Features -### πŸ“Œ **Experiment Design & Planning** -- **Power analysis** and **Minimal Detectable Effect (MDE)** estimation -- Support for **complex experimental designs**, including: - - 🏒 **Cluster randomization** - - πŸ”„ **Switchback experiments** +### πŸ“Œ Experiment Design & Planning +- **Power analysis** and **Minimal Detectable Effect (MDE)** estimation + - **Normal Approximation (CLT-based)**: Fast, analytical formulas assuming approximate normality + - Best for large sample sizes and standard A/B tests + - **Monte Carlo Simulation**: Empirically estimate power or MDE by simulating many experiments + - Ideal for complex or non-standard designs (e.g., clustering, non-normal outcomes) -### πŸ›  **Data Preprocessing** -- Tools for **efficient data preparation** -- Seamlessly integrates with **Pandas** for streamlined workflows +- Supports complex **experimental designs**, including: + - 🏒 **Cluster randomization** + - πŸ”„ **Switchback experiments** + - πŸ“Š **Observational studies**, including **synthetic control** -### πŸ“Š **Comprehensive Experiment Analysis** -##### **βœ… Metrics** -- Simple and **ratio-based metrics** for evaluating experiment outcomes +### πŸ§ͺ Statistical Methods for Analysis +- πŸ“Œ **Ordinary Least Squares (OLS)** and **Clustered OLS**, with support for covariates +- 🎯 **Variance Reduction Techniques**: **CUPED** and **CUPAC** -##### **πŸ“ˆ Statistical Methods** -- πŸ“Œ **Generalized Estimating Equations (GEE)** -- πŸ“Œ **Mixed Linear Models** for robust inference -- πŸ“Œ **Ordinary Least Squares (OLS)** and **Clustered OLS** with covariates -- πŸ“Œ **T-tests** with variance reduction techniques (**CUPED, CUPAC**) -- πŸ“Œ **Synthetic control methods** for causal inference in observational studies +### πŸ“ˆ Scalable Experiment Analysis with Scorecards +- Generate **Scorecards** to summarize experiment results, allowing analysis for multiple metrics +- Include **confidence intervals, relative and absolute effect sizes, p-values**, ---- - -### ⚑ Why Use `cluster experiments`? -βœ… **Production-ready** – built for real-world applications -βœ… **Data-driven decision-making** – designed for rigorous statistical analysis -βœ… **Easy to work** – integrates effortlessly with Pandas - ---- - -`cluster experiments` empowers analysts and data scientists with **scalable, reproducible, and statistically robust** A/B testing workflows. +`cluster experiments` empowers analysts and data scientists with **scalable, reproducible, and statistically robust** A/B testing workflows. -πŸ”— **Get Started:** [Documentation Link] +πŸ”— **Get Started:** [Documentation Link] -πŸ“¦ **Installation:** +πŸ“¦ **Installation:** ```sh pip install cluster-experiments ======= @@ -67,7 +57,6 @@ mde_timeline = npw.mde_time_line( print(power, power_line_normal, power_normal, mde, mde_timeline) ``` - ### Experiment Analysis Example ```python diff --git a/docs/quickstart.md b/docs/quickstart.md index 7abf05c4..2b10e458 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -68,7 +68,7 @@ The `Perturbator` classes define the type of effect you want to test. It simulat from cluster_experiments import ConstantPerturbator perturbator = ConstantPerturbator( - average_effect=5.0 # Simulate a nominal 5% uplift + average_effect=5.0 # Simulate a nominal 5% uplift ) ``` diff --git a/docs/stylesheets/overrides.css b/docs/stylesheets/overrides.css index 9b029d3c..bf676b78 100644 --- a/docs/stylesheets/overrides.css +++ b/docs/stylesheets/overrides.css @@ -10,4 +10,4 @@ border-radius: 8px; background-color: var(--md-code-bg-color); box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); -} \ No newline at end of file +} diff --git a/mkdocs.yml b/mkdocs.yml index 8dfe50c9..db41ed48 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,7 +9,7 @@ docs_dir: docs site_dir: site nav: - - Home: index.md + - Home: ../README.md - Quickstart: quickstart.md - Core Documentation: - API: @@ -26,22 +26,23 @@ nav: - Hypothesis Test: api/hypothesis_test.md - Analysis Plan: api/analysis_plan.md - Usage Examples: - - CUPAC: cupac_example.ipynb + - Variance Reduction: + - CUPAC: cupac_example.ipynb - Switchback: - Stratified switchback: switchback.ipynb - Switchback calendar visualization: plot_calendars.ipynb - Visualization - 4-hour switches: plot_calendars_hours.ipynb - - Multiple treatments: multivariate.ipynb - - AA test clustered: aa_test.ipynb - - Paired T test: paired_ttest.ipynb - - Different hypotheses tests: analysis_with_different_hypotheses.ipynb - - Washover: washover_example.ipynb - - Normal Power: - - Compare with simulation: normal_power.ipynb - - Time-lines: normal_power_lines.ipynb - - Synthetic control: synthetic_control.ipynb - - Experiment analysis workflow: experiment_analysis.ipynb - - Delta Method Analysis: delta_method.ipynb + - Multiple treatments: multivariate.ipynb + - AA test clustered: aa_test.ipynb + - Paired T test: paired_ttest.ipynb + - Different hypotheses tests: analysis_with_different_hypotheses.ipynb + - Washover: washover_example.ipynb + - Normal Power: + - Compare with simulation: normal_power.ipynb + - Time-lines: normal_power_lines.ipynb + - Synthetic control: synthetic_control.ipynb + - Delta Method Analysis: delta_method.ipynb + - Experiment analysis workflow: experiment_analysis.ipynb - Contribute: - Contributing Guidelines: development/contributing.md - Code Structure: development/code_structure.md @@ -93,4 +94,4 @@ markdown_extensions: - pymdownx.highlight: use_pygments: true - toc: - permalink: true \ No newline at end of file + permalink: true