Standard : AI prototypes reach real users before full-scale build begins
Purpose and Strategic Importance
This standard requires that AI capabilities are validated with real users in a meaningful operational context before the team invests in full-scale engineering, infrastructure, and operational build-out. It supports the policy of prototyping and validating before building at scale by creating a mandatory checkpoint where user feedback, adoption signals, and practical performance data inform the decision to proceed. Building AI at scale before validating user value is one of the most expensive and common failure modes in AI programme delivery.
Strategic Impact
- Reduces the risk of large-scale AI build-outs that deliver technically sound systems that users do not adopt or trust
- Generates real-world performance data in weeks rather than months, enabling informed scaling decisions
- Creates early evidence of user value that strengthens the business case for full investment
- Surfaces UX, integration, and performance issues in a lower-cost, lower-risk context before they become expensive to fix
- Builds user involvement and co-design habits that produce AI systems better aligned with actual needs
Risks of Not Having This Standard
- Teams spend months building production-grade AI infrastructure before discovering that users do not trust or use the AI output
- Full-scale build commitments are made based on internal demos rather than real-world user evidence
- AI systems are technically correct but operationally impractical because real-world workflows were not considered in the design
- Large sunk costs make it difficult to pivot or abandon an approach even when user evidence suggests it will not succeed
- The organisation develops a pattern of AI projects that are "finished" but not adopted, creating portfolio-level value destruction
CMMI Maturity Model
Level 1 – Initial
| Category |
Description |
| People & Culture |
- Teams move directly from idea to full build; user validation is deferred to after the system is complete |
| Process & Governance |
- No requirement for user validation before build; project plans proceed from discovery to delivery without a prototype gate |
| Technology & Tools |
- No lightweight prototyping infrastructure; demonstrating to users requires near-production-level build effort |
| Measurement & Metrics |
- No user validation metrics; adoption issues are discovered post-launch |
Level 2 – Managed
| Category |
Description |
| People & Culture |
- Teams create demos or proof-of-concept builds and seek informal feedback from users or domain experts before committing to full build |
| Process & Governance |
- A prototype review is included in the project plan; stakeholder sign-off is required before proceeding to full-scale development |
| Technology & Tools |
- Simple prototyping tooling (notebooks, Streamlit, hosted demo environments) enables rapid demonstration to real users |
| Measurement & Metrics |
- Qualitative user feedback from prototype reviews is documented and influences the full-build specification |
Level 3 – Defined
| Category |
Description |
| People & Culture |
- User prototype validation is a mandatory project lifecycle gate; teams cannot enter full-scale build without documented evidence of user validation |
| Process & Governance |
- A prototype validation standard defines minimum user sample size, feedback collection method, and success criteria |
| Technology & Tools |
- Prototype deployment infrastructure enables real users to interact with AI capabilities in a controlled environment with usage telemetry |
| Measurement & Metrics |
- User engagement, task completion rate, and qualitative feedback from prototype interactions are reported at the build decision gate |
Level 4 – Quantitatively Managed
| Category |
Description |
| People & Culture |
- Teams treat prototype validation as an investment in de-risking full-scale build; prototype findings are shared at portfolio reviews |
| Process & Governance |
- Build decisions are informed by quantitative prototype outcomes; minimum adoption and satisfaction thresholds gate full-scale commitment |
| Technology & Tools |
- A/B prototype testing infrastructure enables comparison of multiple AI approaches with real users before the full-build architecture decision is made |
| Measurement & Metrics |
- Prototype-to-production adoption rate, prototype feedback score, and build decision accuracy (projects that reached their stated value post-build) are tracked |
Level 5 – Optimising
| Category |
Description |
| People & Culture |
- Prototype validation is deeply embedded in AI delivery culture; teams compete to learn from users faster, not to build faster |
| Process & Governance |
- Prototype standards are continuously refined based on correlation between prototype signals and post-build adoption outcomes |
| Technology & Tools |
- Self-service prototype infrastructure enables teams to deploy AI experiments to real users within hours without engineering overhead |
| Measurement & Metrics |
- Predictive models use prototype metrics to forecast post-build adoption and value, improving the accuracy of build investment decisions |
Key Measures
- Percentage of AI projects that conducted user prototype validation before committing to full-scale build
- Mean time from AI concept to first real-user prototype interaction
- User engagement rate and task completion rate in prototype environments
- Proportion of full-scale builds where prototype findings materially changed the product scope or architecture
- Post-deployment adoption rate correlated with prototype validation score (to calibrate the predictive value of prototype metrics)