Statistical Significance Calculator

by

Statistical Significance Calculator

Control Group (A)

Variant Group (B)

Conversion Rate A
Conversion Rate B
Observed Lift
Confidence Level:
P-Value:
Z-Score:

Understanding Statistical Significance

Statistical significance is a measure used in A/B testing to determine whether the difference in results between two versions (like a web page or an ad) is due to a real change in behavior or just random chance. In data analysis, we usually aim for a confidence level of 95% or higher before making a decision.

Key Metrics Explained

  • Conversion Rate: The percentage of visitors who took the desired action (e.g., clicked a button or made a purchase).
  • Lift: The percentage increase or decrease in the conversion rate of the Variant compared to the Control.
  • P-Value: The probability that the observed difference happened by random chance. A p-value of less than 0.05 usually indicates statistical significance.
  • Confidence Level: Represents how sure you are that your results are not due to chance. A 95% confidence level means there is only a 5% chance the results are a "false positive."

A Practical Example

Suppose you run an A/B test on a "Buy Now" button color:

Control (Blue Button): 10,000 visitors, 500 conversions (5.0% conversion rate).
Variant (Red Button): 10,000 visitors, 580 conversions (5.8% conversion rate).

In this scenario, the "Lift" is 16%. However, you need to calculate the Z-score and P-value to ensure that this 16% increase wasn't just a lucky week of traffic. This calculator performs those complex calculations for you instantly.

Why Sample Size Matters

If you only have 10 visitors and 2 conversions, your conversion rate is 20%. If another variant has 10 visitors and 3 conversions, that's 30%. While that looks like a 50% lift, the sample size is far too small to be significant. The larger your sample size (number of visitors), the more reliable your statistical findings become.

function calculateSignificance() { var cV = parseFloat(document.getElementById('control_visitors').value); var cC = parseFloat(document.getElementById('control_conversions').value); var vV = parseFloat(document.getElementById('variant_visitors').value); var vC = parseFloat(document.getElementById('variant_conversions').value); if (isNaN(cV) || isNaN(cC) || isNaN(vV) || isNaN(vC) || cV <= 0 || vV cV || vC > vV) { alert("Conversions cannot be greater than visitors."); return; } var p1 = cC / cV; var p2 = vC / vV; // Observed Lift var lift = ((p2 – p1) / p1) * 100; // Pooled Probability var p_pooled = (cC + vC) / (cV + vV); // Standard Error var se = Math.sqrt(p_pooled * (1 – p_pooled) * (1/cV + 1/vV)); // Z-Score var zScore = (p2 – p1) / se; // P-Value approximation (Normal Distribution CDF) var pValue = 1 – Math.abs(getNormalCDF(zScore) – getNormalCDF(-zScore)); // For one-tailed or two-tailed, standard A/B is usually two-tailed var twoTailedP = 2 * (1 – getNormalCDF(Math.abs(zScore))); var confidence = (1 – twoTailedP) * 100; // UI Updates var resultsDiv = document.getElementById('stat-results'); resultsDiv.style.display = 'block'; document.getElementById('rate-a').innerText = (p1 * 100).toFixed(2) + "%"; document.getElementById('rate-b').innerText = (p2 * 100).toFixed(2) + "%"; document.getElementById('observed-lift').innerText = (lift > 0 ? "+" : "") + lift.toFixed(2) + "%"; document.getElementById('confidence-val').innerText = confidence.toFixed(2) + "%"; document.getElementById('p-value-val').innerText = twoTailedP.toFixed(4); document.getElementById('z-score-val').innerText = zScore.toFixed(4); var verdictTitle = document.getElementById('verdict-title'); var verdictText = document.getElementById('verdict-text'); if (confidence >= 95) { resultsDiv.style.backgroundColor = '#e6f4ea'; verdictTitle.style.color = '#137333'; verdictTitle.innerText = "Statistically Significant!"; verdictText.innerText = "There is a " + confidence.toFixed(2) + "% chance that the difference is not due to chance."; } else { resultsDiv.style.backgroundColor = '#fef7e0'; verdictTitle.style.color = '#b05e00'; verdictTitle.innerText = "Not Significant Yet"; verdictText.innerText = "The confidence level (" + confidence.toFixed(2) + "%) is below the 95% threshold. You may need more data."; } } // Helper function: Standard Normal Cumulative Distribution Function function getNormalCDF(z) { var t = 1 / (1 + 0.2316419 * Math.abs(z)); var d = 0.3989423 * Math.exp(-z * z / 2); var p = d * t * (0.3193815 + t * (-0.3565638 + t * (1.781478 + t * (-1.821256 + t * 1.330274)))); if (z > 0) return 1 – p; return p; }

Leave a Comment