Practice : Value Hypothesis Testing

Purpose and Strategic Importance

Every AI system is built on an implicit hypothesis: that it will produce some meaningful improvement in a business or user outcome. Value hypothesis testing makes this hypothesis explicit and tests it empirically before full-scale investment is committed. Without this discipline, AI teams build systems based on untested assumptions about impact, discover months or years later that the impact did not materialise, and struggle to understand why or what to do differently.

This practice also creates accountability for outcomes, not just outputs. An AI team whose success metric is "deployed a model" has a fundamentally different incentive structure from one whose success metric is "improved customer resolution rate by 15%". Value hypothesis testing shifts the team's orientation from technical delivery to business impact, which is ultimately what justifies AI investment and builds the credibility needed to sustain an AI programme over time.

Description of the Practice

  • Articulates a specific, measurable value hypothesis for every AI use case before development: "if we build X, then Y metric will change by Z, because W mechanism".
  • Designs measurement approaches for testing the hypothesis — including A/B tests, pre/post comparisons, and holdout groups — that can produce reliable causal evidence of impact.
  • Establishes a minimum viable signal that would confirm or disconfirm the hypothesis, enabling a clear go/no-go decision point before full deployment.
  • Tests the hypothesis on a limited user group or in a controlled deployment before scaling, using evidence from the test to refine the hypothesis and inform the decision to proceed.
  • Tracks value realisation as an ongoing responsibility after deployment, not just as a pre-deployment exercise, to confirm that impact is sustained and evolves as expected.

How to Practise It (Playbook)

1. Getting Started

  • Require a documented value hypothesis for every AI use case before development work begins — make it a standard artefact alongside the use case brief and feasibility assessment.
  • Work with business stakeholders to identify the metrics that matter most — the outcomes the business cares about — and build hypotheses around those rather than proxy technical metrics.
  • Design the measurement approach for the hypothesis before building the system, not after — the ability to measure impact must be an explicit design input, not an afterthought.
  • Define the decision criteria upfront: what evidence of impact is needed to scale, what evidence would cause the team to pivot or stop, and who is accountable for making those decisions.

2. Scaling and Maturing

  • Build measurement infrastructure into the AI deployment pipeline — logging, event tracking, and outcome data collection — as a standard component, not a project-specific addition.
  • Develop an experimentation capability that supports rapid hypothesis testing through randomised experiments, enabling teams to test multiple value hypotheses efficiently.
  • Create a value portfolio view that tracks the status of value hypotheses across all AI use cases, giving leadership visibility into what has been validated, what is under test, and what has been closed.
  • Use hypothesis testing data to build predictive models of AI value — understanding which types of use cases and interventions tend to produce the most impact in your specific context.

3. Team Behaviours to Encourage

  • Be specific and ambitious with value hypotheses — vague hypotheses like "improve customer experience" cannot be tested; specific hypotheses like "reduce time-to-resolution by 20% for tier-2 support tickets" can.
  • Share hypothesis test results openly, including null results — use cases where the AI did not produce the expected impact — as these are as important for organisational learning as confirmed successes.
  • Treat value hypothesis testing as a team responsibility, not a handover to a separate analytics function — the team that builds the AI is accountable for demonstrating its impact.
  • Update hypotheses as testing reveals new understanding, treating them as living statements of the team's beliefs about value rather than fixed commitments.

4. Watch Out For…

  • Hypotheses that are technically testable but not actually causal — measuring correlation between AI deployment and a business metric without establishing that the AI caused the change.
  • Measurement approaches that are designed to confirm the hypothesis rather than test it honestly — selecting metrics after deployment based on which ones look good is a common form of outcome cherry-picking.
  • Teams that focus on the volume of AI deployed rather than the value generated, losing sight of the purpose of the AI programme and building a portfolio of technically functional but practically ineffective systems.
  • Value hypotheses that are never formally closed — either confirmed or disconfirmed — leaving the question of impact perpetually open and preventing learning from accumulating.

5. Signals of Success

  • Every AI use case has a documented, specific value hypothesis with defined measurement methodology before development begins.
  • At least one AI use case hypothesis has been formally disconfirmed and the use case deprioritised or redesigned in response, demonstrating that the testing process operates with genuine rigour.
  • Value realisation data for deployed AI systems is reviewed regularly by the team and by leadership, maintaining accountability for outcomes rather than just delivery.
  • The organisation can point to specific, measured business impact from AI investments — not general claims about efficiency or innovation, but specific, validated outcomes tied to individual systems.
  • Value hypothesis testing has informed the prioritisation of the AI use case backlog, with use cases showing stronger early evidence of impact receiving greater investment.