A/B Testi ve Hipotez Testi Arasındaki İlişki

Are A/B testing and hypothesis testing the same thing?

An A/B test is an experimental design; a hypothesis test is one of the methods used to evaluate the data that experiment produces. In an A/B test, users are assigned to two different conditions and a predefined metric is compared.

An e-commerce site might compare its current product page (A: control) with a new design (B: treatment). The core question is whether design B produces a change in conversion rate that is both statistically meaningful and worth something to the business.

Control and treatment groups

When groups are properly randomised, the aim is for known and unknown user characteristics to balance out on average. That makes it credible to attribute the difference in outcome to the change being tested. Without randomisation, a difference between groups may come from the user profiles rather than the design.

What to define before the experiment starts

• Primary metric: a single main measure, such as purchase conversion rate or revenue per user.
• Unit of experimentation: user, account, session or store — whatever assignment happens on.
• Hypothesis and direction: are you looking for any difference, or specifically for B to beat A?
• Sample size and duration: how many observations are needed and when the experiment ends.
• Guardrail metrics: returns, error rate, page speed or complaints — the possible negative effects.

H₀ and H₁: how are hypotheses set up?

A hypothesis test evaluates two claims defined before looking at the data.

A two-sided test evaluates change in both directions without restricting it in advance. A one-sided test examines evidence only in a predetermined direction. A one-sided test must not be chosen after seeing the result in order to shrink the p-value — that is statistical sleight of hand.

Type I and Type II error

Power is the probability of detecting a true effect of a given size. Raising it requires a larger sample, less measurement noise or a bigger effect. An underpowered experiment does not merely fail to find things; it also tends to exaggerate the effects it does find.

What does a p-value mean?

The p-value is the probability of obtaining a test statistic as extreme as, or more extreme than, the one observed — assuming the null hypothesis and the test's assumptions hold.

Finding p = 0.03 means there is a 3% chance of seeing a result this extreme under the null. It does not mean “there is a 3% chance H₀ is true” or “B has a 97% chance of succeeding”. Both are the most common misreadings.

How to interpret the result

• If p ≤ α, there is statistical evidence against H₀ under the test's assumptions.
• If p > α, H₀ cannot be rejected; that does not mean the groups are definitely identical.
• Always report the p-value alongside effect size, confidence interval, sample size and experimental design.

Effect size and confidence intervals

In an A/B test, “is there a difference?” is not the only question — “how big is it, and which values are plausible?” matters just as much. In conversion rates, absolute difference and relative lift mean different things and are frequently confused in reports.

A confidence interval gives a range of uncertainty for the parameter under the long-run repeated-sampling properties of the method used. If the conversion difference is +1 percentage point with a 95% interval of +0.2 to +1.8 points, zero difference lies outside that range. The interpretation still depends on the model and design assumptions.

It helps to set the minimum detectable effect (MDE) before the experiment. A difference of +0.1 percentage points may be statistically distinguishable and still fail to cover the development and maintenance cost.

How many users do you need?

Sample size depends on the baseline conversion rate, the minimum effect you want to detect, the significance level and the statistical power. These values must be fixed before the experiment starts.

Detecting smaller effects reliably requires a markedly larger sample. The calculation must follow the test and assumptions you will actually use; there is no universal “this many users is enough” number.

A simple A/B test in Python

The scenario: 1,000 of 10,000 users in group A convert, and 1,100 of 10,000 in group B — 10% and 11%. The code below runs a two-sided z-test for two proportions, a classic approach under assumptions of independent users and appropriate sampling.

from statsmodels.stats.proportion import proportions_ztest

basarili = [1000, 1100]
toplam   = [10000, 10000]

z_istatistigi, p_degeri = proportions_ztest(
    count=basarili,
    nobs=toplam,
    alternative="two-sided"
)

oran_a = basarili[0] / toplam[0]
oran_b = basarili[1] / toplam[1]
mutlak_fark  = oran_b - oran_a
goreli_artis = mutlak_fark / oran_a

print(f"A conversion rate: {oran_a:.1%}")
print(f"B conversion rate: {oran_b:.1%}")
print(f"Absolute difference: {mutlak_fark:.1%}")
print(f"Relative lift: {goreli_artis:.1%}")
print(f"z: {z_istatistigi:.3f}, p: {p_degeri:.4f}")

Here B's observed conversion is one percentage point above A's. The two-proportion z-test computes the statistic using a pooled rate under the null hypothesis.

Reporting the result

A report should contain: users per group, each group's conversion rate, the absolute difference, the relative change, the p-value, a confidence interval for the difference, and whether the experiment followed its predefined rules. The decision should not rest on the p-value alone.

Common mistakes and sources of bias

• Early stopping: checking the p-value daily and ending at the first significant result inflates false positive risk.
• Multiple comparisons: testing many metrics, segments or variants raises the chance of a significant result by luck. Fix the primary metric and a correction strategy in advance.
• Sample ratio mismatch (SRM): if the observed allocation differs from the planned one without explanation, investigate the experiment infrastructure before interpreting anything.
• Picking segments afterwards: finding interesting subgroups after seeing the result is useful for exploration but is not confirmatory evidence.
• Measurement problems: conversion events under-recorded in one group corrupt the whole result.
• Spillover: when users influence each other, or the same person enters both groups, the independence assumption collapses.

Statistical significance or business significance?

Statistical significance provides evidence about how compatible the observed data is with the null hypothesis. Business significance asks what the change is worth in revenue, cost, user experience or strategic terms. They are not the same thing, and they often do not point the same way.

Do not present “not statistically significant” as “the two designs are equal”. A claim of equivalence requires a predefined acceptable range of difference and a test approach suited to it. Absence of evidence is not evidence of absence.

A checklist before the experiment starts

• Are the main hypothesis and the primary metric defined in advance?
• Is the unit of randomisation clear, along with the rule keeping users in their group?
• Are the sample size, α, power and MDE justified?
• Are the duration and the stopping rule defined beforehand?
• Have tracking and guardrail metrics been chosen?
• Will sample ratio mismatch and event-logging issues be checked?
• Will effect size and confidence intervals be reported alongside the p-value?
• Is there a plan for multiple tests and subgroup analyses?

An A/B test answers the right question with the right experiment

A/B testing supports product and marketing decisions with experimental evidence. Hypothesis testing helps interpret the observed difference between control and treatment within statistical uncertainty. But a p-value alone is not enough for a trustworthy decision.

A good A/B test brings together a clear hypothesis, randomisation, the right metric, a pre-planned sample size, confidence intervals, effect size and experiment-quality checks. Results should be assessed in both statistical and business terms.