Practice : A/B or Multivariate Testing Infrastructure

Purpose and Strategic Importance

A/B or Multivariate Testing Infrastructure reduces uncertainty and delivery risk by enabling teams to validate product, system, or algorithm changes in production with real users before making them the default. By providing controlled experimentation capabilities, teams gain evidence-based insights, make informed decisions, and optimise system behaviour or user experience safely.

Without robust testing infrastructure, changes are shipped based on assumptions rather than data, increasing the risk of regressions, poor user experience, and missed improvement opportunities.

Description of the Practice

  • A/B and multivariate testing tools allow for safe, targeted experimentation with different system behaviours, features, or algorithms.
  • Changes are rolled out to a defined subset of users, with controlled comparison groups to assess impact.
  • Test outcomes inform decisions about full rollout, rollback, or further iteration.
  • Testing infrastructure integrates with observability and analytics tools to measure results accurately.

How to Practise It (Playbook)

1. Getting Started

  • Select or implement a testing platform that aligns with your system architecture (e.g. LaunchDarkly, Optimizely, custom tools).
  • Define clear success metrics and evaluation criteria for experiments.
  • Educate teams on designing valid experiments and interpreting results.
  • Start with low-risk tests to build confidence and technical capability.

2. Scaling and Maturing

  • Expand testing infrastructure to support more complex experiments (e.g. algorithm performance, UI changes, system behaviours).
  • Integrate with feature flags and deployment pipelines to enable progressive rollouts.
  • Establish experimentation as a routine part of system and product development.
  • Automate result collection, analysis, and decision support where possible.

3. Team Behaviours to Encourage

  • Use experimentation to reduce uncertainty, not as an afterthought.
  • Design experiments with clear hypotheses and measurable outcomes.
  • Collaborate with product, data, and engineering teams to ensure valid, useful tests.
  • Learn from both successful and unsuccessful experiments to inform system evolution.

4. Watch Out For…

  • Poorly designed experiments producing misleading results.
  • Lack of control groups undermining result validity.
  • Overuse of experiments slowing delivery or creating technical complexity.
  • Teams making decisions based on assumptions rather than test outcomes.

5. Signals of Success

  • Experiments reduce uncertainty and guide system or product decisions.
  • Changes are validated with real user impact before full rollout.
  • Experimentation is integrated into delivery and improvement processes.
  • Teams report increased confidence in system changes and user experience optimisation.