• Home
  • BVSSH
  • C4E
  • Playbooks
  • Frameworks
  • Good Reads
Search

What are you looking for?

Standard : Experiments are run before major commitments

Purpose and Strategic Importance

This standard ensures that teams test assumptions early by running small, focused experiments before making significant product, architectural, or operational commitments. In complex environments, evidence-based learning is essential for reducing uncertainty, validating value, and avoiding waste.

It supports the policy “Embrace Iteration over Perfection” by embedding fast feedback loops and lightweight exploration into delivery workflows. Without this standard, teams risk investing heavily in unproven ideas, leading to rework, missed opportunities, or customer dissatisfaction.

Strategic Impact

  • Reduces risk by validating assumptions early
  • Accelerates learning cycles and product-market fit
  • Prevents over-investment in the wrong solutions
  • Encourages incremental progress and iterative value delivery
  • Builds a culture of curiosity, exploration, and evidence-based decision-making

Risks of Not Having This Standard

  • Teams build full solutions before validating the problem
  • Large-scale rework due to incorrect assumptions
  • Valuable insights are discovered too late or not at all
  • Planning is based on opinion rather than data
  • Confidence in delivery decisions erodes over time

CMMI Maturity Model

Level 1 – Initial

Category Description
People & Culture - Experiments are rare and usually informal.
- Decisions are made by opinion or seniority.
Process & Governance - No formal mechanisms to support exploratory work.
Technology & Tools - Limited or no tooling for testing, toggling, or measuring experiments.
Measurement & Metrics - No tracking of experimental outcomes.

Level 2 – Managed

Category Description
People & Culture - Teams occasionally run experiments, but often after significant planning.
- Learning happens reactively, not proactively.
Process & Governance - Early-stage testing encouraged but inconsistently supported.
Technology & Tools - Feature toggles or A/B testing capabilities exist in parts of the stack.
Measurement & Metrics - Some experiments tracked, but outcomes not always used to guide decisions.

Level 3 – Defined

Category Description
People & Culture - Teams proactively frame risky work as hypotheses to test.
- Failure is seen as part of learning.
Process & Governance - Experiments are scoped, prioritised, and tracked before major work starts.
- Discovery and delivery are intentionally linked.
Technology & Tools - Tools support safe, measurable experimentation (e.g., toggles, canaries, user cohorts).
Measurement & Metrics - % of high-risk items preceded by experiments; ratio of validated to invalidated assumptions.

Level 4 – Quantitatively Managed

Category Description
People & Culture - Experimentation is the default for complex or ambiguous work.
- Teams share learnings openly and refine hypotheses collaboratively.
Process & Governance - Experimentation is baked into planning, refinement, and delivery rituals.
- Success criteria are defined and revisited.
Technology & Tools - Platforms provide experiment analytics and link results to backlog decisions.
Measurement & Metrics - Time-to-learn from experiment; improvement in value delivery precision.

Level 5 – Optimising

Category Description
People & Culture - Teams continually explore, test, and refine through live experiments.
- Experimentation is celebrated as a core delivery competency.
Process & Governance - Strategic decisions and portfolio investments are informed by experimental evidence.
- Organisational learning loops feed continuous innovation.
Technology & Tools - Real-time experimentation dashboards; insight pipelines into product and delivery tooling.
Measurement & Metrics - Learning velocity; % of roadmap decisions backed by experiments; reduced investment in failed initiatives.

Key Measures

  • % of roadmap items preceded by an experiment
  • Number of validated vs invalidated hypotheses per quarter
  • Average time from experiment start to insight
  • % of product decisions informed by experimental data
  • Reduction in rework due to validated discovery
Associated Policies
Associated Practices
  • Rapid Prototyping
  • Hypothesis-Driven Development

Technical debt is like junk food - easy now, painful later.

Awesome Blogs
  • LinkedIn Engineering
  • Github Engineering
  • Uber Engineering
  • Code as Craft
  • Medium.engineering