Why Your A/B Tests Are Misleading: The Omitted Variable Bias

A/B testing promises a clean answer: show version A to one group, version B to another, then attribute the difference to the design change. In practice, the result can be distorted by factors your experiment did not measure, control or even notice.

This is the problem behind many misleading experiments. An omitted variable may affect both who receives an experience and how they respond to it, making a weak idea appear successful or hiding the value of a genuinely useful change.

For Australian organisations, the risk is especially relevant across fragmented markets, seasonal demand and diverse customer groups. A test run in Sydney during a major sporting event may behave very differently from one run in regional Queensland, while “no worries” feedback can conceal serious friction in a service journey.

What Omitted Variable Bias Looks Like

Omitted variable bias occurs when an unmeasured factor influences both the supposed cause and the outcome. Imagine a retailer testing free delivery. Customers who see the offer may also be the customers shopping during a payday weekend, when purchase rates are naturally higher.

The experiment then credits free delivery for behaviour partly caused by timing. The same issue appears in product analytics, conversion research and employee experience studies when motivation, urgency, prior knowledge or customer value is left out of the analysis.

When Randomisation Is Not Enough

A properly randomised A/B test protects against confounding at the point of assignment. If people are genuinely allocated at random, an omitted variable should be balanced between groups on average, even when researchers never record it.

Real-world tests often depart from that ideal. Users may switch devices, encounter different channels, leave before exposure, receive overlapping campaigns or influence one another. A test can also suffer from selective eligibility, uneven traffic sources, novelty effects and changes during the test window. Randomisation remains powerful, but implementation determines whether its promise holds.

The Australian Context Changes The Result

Australian businesses often test across a wide geographic footprint. A national bank, supermarket or healthcare provider may combine metropolitan customers with people in remote communities, where connectivity, travel time and access to services differ substantially. A small uplift in the aggregate can conceal a poor experience for a high-friction segment.

Calendar effects matter too. School holidays, tax time, end-of-financial-year purchasing, public holidays and major events such as the Australian Open can alter intent. A test conducted around Black Friday may capture promotional urgency rather than a durable improvement in the customer experience.

Read The Mechanism, Not Just The Metric

A conversion lift is an outcome, not an explanation. Examine intermediate measures such as page comprehension, search refinement, error rates, time to decision, cancellation, repeat use and customer support contacts. These signals can show whether the intervention changed behaviour through the mechanism you expected.

Suppose a shorter form increases bookings but also increases no-shows. The omitted variable may be the urgency of the people willing to complete the form quickly, or the absence of information that helps customers judge whether the appointment suits them. For a practical perspective on reducing booking friction, see appointment procrastination.

Design Better Behavioural Comparisons

A useful experiment starts with a behavioural hypothesis: what decision is difficult, what friction interrupts it and what cue should change behaviour? Mapping the service journey can reveal variables that a standard dashboard misses, including trust, perceived effort, ambiguity and social proof.

This is where choice architecture becomes valuable. The order of options, default settings, labels and prompts can influence behaviour, but their effect depends on context. A default that helps customers in a self-service portal may frustrate people who need reassurance from a staff member.

Account For People And Situations

Segment analysis should be planned before results arrive. Consider new and returning customers, mobile and desktop users, high- and low-frequency customers, language needs, accessibility requirements and different service channels. Avoid slicing data until a preferred story appears; define meaningful comparisons in advance and check whether sample sizes support them.

Human factors also shape test execution. Staff may explain a new process differently, managers may promote it more enthusiastically or customers may respond to perceived warmth rather than the interface itself. Research on emotional intelligence can help teams recognise how interpersonal behaviour affects supposedly standardised experiences.

Practical Safeguards For More Reliable Tests

Treat an A/B test as a designed piece of research rather than a button comparison. Document the hypothesis, assignment rule, exposure conditions, primary outcome, guardrail metrics and likely sources of bias before launch. Then monitor implementation, not just statistical significance.

Useful safeguards include:

• Randomise at the right unit, such as customer, account, household or store.
• Check balance between groups across traffic source, device, location and customer history.
• Record exposure, non-compliance, switching and repeated contact across channels.
• Use pre-specified primary metrics alongside safety measures such as complaints or cancellations.
• Run tests long enough to cover weekday, weekend and seasonal behaviour.
• Analyse meaningful segments without treating every subgroup result as definitive.
• Replicate important findings in a different period, channel or customer population.

These steps do not eliminate uncertainty. They make it easier to distinguish a genuine behavioural effect from a temporary association created by timing, selection or implementation.

A statistically significant result can still be operationally insignificant, ethically questionable or impossible to generalise. The strongest teams combine experimentation with qualitative research, service observation and frontline knowledge before scaling a change.

When an A/B test produces an unexpected result, investigate what else changed. Review the customer journey, speak with staff and examine the people and situations hidden behind the average. If you need support applying behavioural science to a service, product or employee experience, explore Nick Naumof’s consulting and research work to turn noisy evidence into better decisions.