Claude Data Analysis advanced

A/B Test Design and Results Analyzer

Added Apr 1, 2026

You are a statistician and experimentation specialist. I want to run an A/B test for [TEST_PURPOSE] on [PLATFORM_OR_PRODUCT]. My primary metric is [PRIMARY_METRIC] and secondary metrics are [SECONDARY_METRICS]. Current baseline for the primary metric is [BASELINE]. Help me with: 1) Proper hypothesis formulation (null and alternative), 2) Sample size calculation with assumptions (significance level, power, minimum detectable effect), 3) Test duration estimation based on [DAILY_TRAFFIC] daily visitors, 4) Randomization strategy and potential confounders to control for, 5) Segmentation recommendations for deeper analysis, 6) A pre-analysis plan that specifies what decisions we will make based on each possible outcome. If I provide test results, also analyze them for statistical significance and provide a clear recommendation with confidence intervals.

Try in Claude

About This Prompt

Most A/B tests fail not because of bad ideas but because of poor experimental design: underpowered tests, peeking at results too early, or no pre-defined success criteria. This prompt walks you through rigorous test design that would satisfy a statistician while remaining practical for product and marketing teams. The pre-analysis plan is especially important because it prevents the common trap of moving goalposts after seeing results. Whether you are testing landing page variations, pricing experiments, or feature changes, this structured approach dramatically increases the reliability of your conclusions.

Variables to Customize


[TEST_PURPOSE]

What you want to test

Example: whether adding customer testimonials above the fold increases trial sign-ups


[PLATFORM_OR_PRODUCT]

Where the test runs

Example: our SaaS landing page


[PRIMARY_METRIC]

Main success metric

Example: free trial conversion rate


[SECONDARY_METRICS]

Additional metrics to track

Example: bounce rate, time on page, and scroll depth


[BASELINE]

Current performance

Example: 3.2% conversion rate


[DAILY_TRAFFIC]

Daily visitor volume

Example: approximately 2,500 unique visitors per day

Tips for Best Results

Always set your minimum detectable effect before running the test, not after
Avoid peeking at results before reaching full sample size
Run the test for full weeks to account for day-of-week effects

Example Output

## Hypothesis
**Null (H0):** Adding customer testimonials above the fold has no effect on the free trial conversion rate (Control = Variant).
**Alternative (H1):** Adding customer testimonials above the fold increases the free trial conversion rate (Variant > Control).

## Sample Size Calculation
Assumptions: Significance level (alpha) = 0.05, Power (1-beta) = 0.80, Minimum detectable effect = 20% relative lift (3.2% to 3.84%)
**Required sample size: 12,400 per variation (24,800 total)**

## Duration Estimate
At 2,500 daily visitors with a 50/50 split: **20 days minimum** (rounded up to 21 days to complete 3 full weeks).

AB-testing experimentation statistics CRO hypothesis-testing