Statsig Calculator

Estimate Feature Flag Impact and Experiment Performance

Statsig Impact & Experiment Estimator

Estimated Experiment Metrics

Required Sample Size (per variant): —

Total Sample Size: —

Minimum Detectible Lift: —

Key Assumptions:

Traffic Allocation: —

Baseline Conversion Rate: —

Statistical Significance (Alpha): —

Statistical Power (1-Beta): —

Formula Used: This calculator estimates sample size based on the normal approximation to the binomial distribution. It considers traffic allocation, baseline conversion rate, minimum detectible effect (relative lift), desired statistical significance (alpha), and statistical power (1-beta). The formula for sample size per variant (n) for comparing two proportions is roughly:

n = [(Z_α/2 + Z_β)² * (p₁(1-p₁) + p₂(1-p₂))] / (p₁ – p₂)²

Where: Z_α/2 is the Z-score for significance level, Z_β is the Z-score for power, p₁ is the baseline conversion rate, and p₂ is the target conversion rate (baseline * (1 + MDE)).

Sample Size vs. MDE

Experiment Performance Summary

Metric	Control Group	Variant Group
Conversion Rate (%)	—	—
Sample Size	—	—
Conversions	—	—
Observed Lift (%)	—

Results copied to clipboard!

What is a Statsig Calculator?

A Statsig calculator is a specialized tool designed to help product managers, data scientists, and engineers estimate the necessary parameters for running effective A/B tests or feature flag experiments using the Statsig platform. It bridges the gap between ideation and execution by providing data-driven insights into experiment design. Instead of launching an experiment blindly, a Statsig calculator allows you to pre-determine crucial metrics like the required sample size, expected lift, and the statistical power needed to confidently detect meaningful changes in user behavior or key performance indicators (KPIs).

Essentially, it's a forward-looking tool that helps you answer questions like: "How many users do I need to show this new feature to in order to be sure if it's better than the old one?" or "What's the smallest improvement I can reliably measure with my current traffic?" This proactive approach minimizes wasted resources, reduces the risk of inconclusive results, and maximizes the chances of making statistically sound product decisions.

Who Should Use a Statsig Calculator?

• Product Managers: To validate new features, understand potential impact, and prioritize roadmap items based on data.
• Data Scientists & Analysts: To design robust experiments, determine optimal sample sizes, and ensure statistical validity.
• Engineers: To understand the implications of rolling out feature flags and experiments, and to estimate resource needs.
• Marketing Teams: To test different campaign creatives, landing pages, or user flows to optimize conversion rates.
• Anyone Implementing A/B Testing: To ensure experiments are well-designed and have a high probability of yielding actionable insights.

Common Misconceptions about Statsig Calculators

• "It guarantees a successful experiment." Misconception: Calculators estimate requirements based on inputs. Actual results depend on the experiment's quality, execution, and real-world user behavior.
• "More traffic always means better results." Misconception: While larger sample sizes increase power, the key is having *enough* relevant traffic for the desired MDE and significance. Over-sampling can be inefficient.
• "It replaces statistical expertise." Misconception: Calculators simplify calculations, but understanding the underlying statistical principles (like p-values, confidence intervals, and bias) is still crucial for proper interpretation.
• "Results are fixed once calculated." Misconception: The calculator provides estimates for a *specific* set of assumptions. If your traffic patterns change, or you decide to target a different MDE, the required sample size will change.

Statsig Calculator Formula and Mathematical Explanation

The core of a Statsig calculator often revolves around determining the required sample size for an A/B test or feature flag experiment. This is typically achieved using formulas derived from statistical principles for comparing two proportions (or means, depending on the metric). A common approach uses the normal approximation to the binomial distribution.

Step-by-Step Derivation (Sample Size for Proportions)

The goal is to find the sample size per variant (n) needed to detect a specific Minimum Detectible Effect (MDE) with a certain level of statistical significance (alpha) and statistical power (1 – beta).

Let:

• p1 = Baseline Conversion Rate (Control Group)
• p2 = Target Conversion Rate (Variant Group) = p1 * (1 + Relative MDE)
• α = Significance Level (e.g., 0.05 for 95% significance)
• β = Type II Error Rate (e.g., 0.20 for 80% power)
• Zα/2 = Z-score corresponding to the significance level (for a two-tailed test)
• Zβ = Z-score corresponding to the statistical power

The standard formula for sample size per group (n) when comparing two proportions is:

n = [ Zα/2 * sqrt(ppooled * (1 - ppooled) * 2) + Zβ * sqrt(p1*(1-p1) + p2*(1-p2)) ]² / (p1 - p2)²

A simplified, commonly used approximation (especially when p1 and p2 are close) focuses on the variance:

n ≈ ( (Zα/2 + Zβ)² * (p1*(1-p1) + p2*(1-p2)) ) / (p1 - p2)²

This formula balances the need for confidence (Zα/2) against the risk of missing a real effect (Zβ), considering the inherent variability in the conversion rates (p1*(1-p1) and p2*(1-p2)), and the size of the effect we want to detect ((p1 - p2)²).

Variable Explanations

• Traffic Allocation (%): The proportion of your total user base that will be exposed to the experiment (either control or variant). Higher allocation allows for faster results.
• Baseline Conversion Rate (%): The existing conversion rate of the metric you are trying to improve. This is a critical input for calculating expected outcomes and required sample sizes.
• Minimum Detectible Effect (MDE) (%): The smallest *relative* change (lift or decrease) in your target metric that you want your experiment to be sensitive enough to detect. A smaller MDE requires a larger sample size.
• Statistical Significance (Alpha) (%): The probability of rejecting the null hypothesis when it is actually true (a Type I error, or false positive). Commonly set at 95% confidence (α = 0.05). Lower alpha requires larger sample size.
• Statistical Power (1-Beta) (%): The probability of correctly rejecting the null hypothesis when it is false (i.e., detecting a real effect if it exists). Commonly set at 80% (β = 0.20). Higher power requires larger sample size.

Variables Table

Statsig Calculator Variables

Variable	Meaning	Unit	Typical Range
Traffic Allocation	Percentage of users exposed to the experiment.	%	1-100%
Baseline Conversion Rate	Current conversion rate of the control group.	%	0.1% – 50%+ (highly variable by industry/metric)
Minimum Detectible Effect (MDE)	Smallest relative change to detect.	%	0.5% – 10%+
Statistical Significance (Alpha)	Probability of false positive (Type I error).	%	90% – 99% (commonly 95%)
Statistical Power (1-Beta)	Probability of detecting a true effect (avoiding Type II error).	%	70% – 95% (commonly 80%)
Sample Size (per variant)	Number of users needed in each group (control/variant).	Users	Thousands to Millions (highly dependent on other factors)
Total Sample Size	Total users needed for the entire experiment.	Users	2 * Sample Size (per variant)
Target Conversion Rate	Expected conversion rate in the variant group.	%	Calculated: Baseline * (1 + Relative MDE)

Practical Examples (Real-World Use Cases)

Example 1: Optimizing a Sign-up Button Color

A SaaS company wants to test a new 'Sign Up' button color (e.g., changing from blue to green) to increase user registrations.

• Current Sign-up Rate (Baseline Conversion Rate): 3%
• Desired Improvement (MDE): They want to detect at least a 10% relative increase (i.e., reach 3.3% conversion).
• Traffic Allocation: They can allocate 50% of their website traffic to this experiment.
• Statistical Significance: 95% (Alpha = 0.05)
• Statistical Power: 80% (Beta = 0.20)

Using the Statsig calculator with these inputs:

• Input Values: Traffic Allocation = 50%, Baseline CR = 3%, MDE = 10%, Significance = 95%, Power = 80%.
• Calculator Output (Estimated):
  • Required Sample Size per Variant: ~10,600 users
  • Total Sample Size: ~21,200 users
  • Target Conversion Rate: 3.3%

Interpretation: To be reasonably confident (95% chance of being correct) that a 10% increase in sign-ups is real and not just random chance, and have an 80% chance of detecting it if it occurs, they need to expose approximately 10,600 users to the green button and 10,600 users to the blue button. If their average daily unique visitors are 1,000, this experiment would take about 21 days to complete (21,200 users / 1,000 users/day).

Example 2: Testing a New Checkout Flow

An e-commerce platform is considering a redesigned checkout process to reduce cart abandonment.

• Baseline Cart Abandonment Rate: 40% (This means a 'Conversion Rate' of 60% to complete checkout)
• Desired Reduction in Abandonment (MDE): They want to detect a 5% *absolute* reduction in abandonment (i.e., from 40% down to 35%), which is a relative reduction of 12.5%. The calculator typically uses relative MDE for conversion rates. So, a 12.5% relative reduction on a 60% completion rate means the new rate would need to be 60% * (1 – 0.125) = 52.5% completion rate. MDE in % terms = (60-52.5)/60 = 12.5%.
• Traffic Allocation: They can route 100% of traffic through the experiment.
• Statistical Significance: 95% (Alpha = 0.05)
• Statistical Power: 90% (Beta = 0.10) – They want higher confidence due to the significant change.

Using the Statsig calculator:

• Input Values: Traffic Allocation = 100%, Baseline CR = 60%, MDE = 12.5% (relative reduction), Significance = 95%, Power = 90%.
• Calculator Output (Estimated):
  • Required Sample Size per Variant: ~36,500 users
  • Total Sample Size: ~73,000 users
  • Target Conversion Rate: 52.5%

Interpretation: Testing a significant change like a checkout flow requires a substantial number of users, especially when aiming for high power. They need over 73,000 users to participate in the experiment to reliably detect if the new flow reduces abandonment by the targeted amount. The calculator helps justify the time and user volume needed for such a critical test.

How to Use This Statsig Calculator

This Statsig calculator is designed to be intuitive and provide actionable insights for experiment design. Follow these steps to get the most out of it:

1. Define Your Metric: Clearly identify the key performance indicator (KPI) you want to impact (e.g., conversion rate, click-through rate, sign-up rate, retention rate). This will be your 'Baseline Conversion Rate'.
2. Set Realistic Expectations (MDE): Determine the smallest *relative* improvement or change you consider meaningful for your business. A smaller MDE requires a larger sample size and longer experiment duration. Start with a reasonable value (e.g., 5-10% relative lift) and adjust if necessary.
3. Allocate Traffic: Decide what percentage of your user base can be included in the experiment. A higher percentage means faster data collection. Consider if you need to run the experiment year-round or during specific periods.
4. Choose Statistical Confidence Levels:
   • Significance (Alpha): Typically set at 95%. This means you accept a 5% chance of a false positive (concluding there's an effect when there isn't).
   • Power (1-Beta): Typically set at 80%. This means you have an 80% chance of detecting a real effect of your chosen MDE size if it truly exists. Increasing power requires a larger sample size.
5. Input Values: Enter the determined values into the corresponding fields in the calculator. Ensure you use percentages correctly (e.g., 5% should be entered as 5, not 0.05, unless specified by the helper text).
6. Click 'Calculate': The calculator will instantly provide:
   • Primary Result: The minimum required sample size per variant.
   • Intermediate Values: Total sample size needed, and the target conversion rate you aim to achieve.
   • Key Assumptions: A summary of the inputs you used.
7. Interpret the Results:
   • Sample Size: This is your primary target. Work backward to estimate the duration needed based on your daily/weekly traffic.
   • Total Sample Size: Double the per-variant size to get the total users required for the experiment.
   • Target Conversion Rate: This is the performance level your variant needs to hit to be considered a success under your chosen MDE.
8. Use the 'Copy Results' Button: Easily copy all calculated metrics and assumptions for documentation or sharing with your team.
9. Use the 'Reset' Button: Start over with default or previous values if you need to adjust parameters.

Decision-Making Guidance

• Is the required sample size feasible? If the calculated sample size is extremely large, consider if your traffic can support it within a reasonable timeframe. You might need to increase your MDE or accept lower power/significance.
• Is the MDE realistic? Aiming for tiny improvements with limited traffic might lead to inconclusive results. Ensure your MDE aligns with business goals and expected impact.
• Does the target conversion rate make sense? Compare the target CR with your baseline. Is the expected uplift ambitious yet achievable?

Key Factors That Affect Statsig Calculator Results

Several factors significantly influence the outputs of a Statsig calculator, particularly the required sample size. Understanding these can help you refine your experiment design and interpret results more effectively.

1. Baseline Conversion Rate (BCR): This is arguably the most impactful input.
   • Effect: Lower BCRs generally require larger sample sizes to detect the same *relative* lift. For example, detecting a 10% relative lift from 1% to 1.1% requires a much larger sample than detecting a 10% relative lift from 10% to 11%.
   • Reasoning: The absolute difference between the groups is smaller at lower rates, meaning more data is needed to confidently distinguish the signal from noise.
2. Minimum Detectible Effect (MDE): The size of the change you want to detect.
   • Effect: Smaller MDEs require significantly larger sample sizes.
   • Reasoning: Detecting subtle improvements is inherently harder and requires more data points to be sure the observed difference isn't random fluctuation. If you only care about large impacts, you need less data.
3. Statistical Significance (Alpha): The acceptable risk of a false positive.
   • Effect: Lowering the significance level (e.g., moving from 95% to 99% confidence) increases the required sample size.
   • Reasoning: To be *more* certain that a result is not a fluke, you need to see a stronger signal relative to the noise, which demands more data.
4. Statistical Power (1-Beta): The probability of detecting a true effect.
   • Effect: Increasing statistical power (e.g., from 80% to 90%) increases the required sample size.
   • Reasoning: Having a higher chance of finding a real effect means you need to be more sensitive to smaller signals, necessitating more data.
5. Traffic Allocation: The percentage of your user base exposed to the experiment.
   • Effect: While it doesn't change the *per-variant* sample size calculation, it dramatically affects the *duration* of the experiment. Higher allocation means faster completion.
   • Reasoning: The calculation determines how many users need to *see* the variation. If only a fraction of your users are exposed daily, it takes longer to reach the target number.
6. Nature of the Metric (Variance): While the calculator often uses proportions, the underlying variance matters.
   • Effect: Metrics with higher inherent variance (e.g., revenue per user, which can fluctuate widely) often require larger sample sizes than binary metrics (e.g., click/no-click).
   • Reasoning: High variance means individual data points are less predictable, making it harder to discern a consistent effect of the change across the population.
7. Experiment Duration: Related to traffic allocation but also seasonality or business cycles.
   • Effect: Running experiments for too short a duration (less than a full business cycle, e.g., a week) can lead to inaccurate results due to temporal biases.
   • Reasoning: User behavior can vary significantly based on the day of the week, time of year, or specific promotions. A sufficient duration helps average out these effects.

Frequently Asked Questions (FAQ)

What is the difference between relative and absolute MDE?

Relative MDE is expressed as a percentage change compared to the baseline (e.g., a 10% relative lift on a 5% baseline means the target is 5.5%). Absolute MDE is a direct difference (e.g., reducing abandonment from 40% to 35% is a 5% absolute difference). Most sample size calculators, including this one, are designed for relative MDE on conversion rates.

Can I use this calculator for metrics other than conversion rates?

This specific calculator is optimized for conversion rates (proportions). For metrics like average order value or user engagement time (continuous variables), you would need a calculator designed for comparing means, which uses a different statistical formula involving standard deviation. Statsig itself supports various metric types.

What does it mean if my experiment result is not statistically significant?

It means that the observed difference between your control and variant groups could plausibly be due to random chance alone. You cannot confidently conclude that your change had a real impact. It doesn't necessarily mean the change had *no* effect, just that you didn't gather enough evidence to prove it did.

How long should I run my A/B test?

Run your test until you reach the required sample size calculated by the tool, AND ensure it covers at least one full business cycle (typically a week) to account for daily variations in user behavior. Avoid running tests for too short a period or stopping them prematurely just because you see a seemingly significant result early on.

What happens if I change my feature flag after the experiment?

If you decide to roll out the winning variant, you would typically configure your feature flag in Statsig to serve that variant to the relevant user segment. If you decide against the change, you revert the flag. The calculator helps you make the *decision* based on data before committing to a change.

Can I run multiple experiments at once?

Yes, but be cautious. Running multiple experiments simultaneously can lead to interference if they affect the same user journey or metrics. Ensure experiments are independent or consider statistical methods to account for potential interactions if they overlap significantly. Always check the sample size requirements for each.

What is the "p-value" and how does it relate to significance?

The p-value is the probability of observing a test statistic as extreme as, or more extreme than, the one actually observed, assuming the null hypothesis is true. Statistical significance (e.g., 95%) means your p-value must be less than your alpha level (e.g., p < 0.05) to reject the null hypothesis and conclude there's a statistically significant effect.

Does the calculator account for network latency or server load?

This calculator focuses on the statistical requirements for experiment design. It does not directly model technical performance metrics like latency or server load. While experiment results might indirectly reflect performance impacts (e.g., slower loading leads to lower conversion), you'd need separate performance testing and monitoring tools for those specific aspects.

Related Tools and Internal Resources

• A/B Testing Best Practices – Learn how to design, run, and analyze experiments effectively.
• Feature Flag Implementation Guide – Understand how to use Statsig for controlled rollouts.
• Conversion Rate Optimization (CRO) Strategies – Discover techniques to improve user experience and conversions.
• Statsig SDK Integration – Resources for developers implementing Statsig in their applications.
• Experiment Analysis Framework – Deeper dives into interpreting statistical results.
• Performance Monitoring Tools – Tools to track application speed and stability.