Practice : Experiment Tracking and Management

Purpose and Strategic Importance

Machine learning is an experimental discipline. Teams run hundreds of experiments varying architecture, hyperparameters, training data, and pre-processing choices before arriving at a production-quality model. Without systematic tracking, this process becomes a black box — teams lose the ability to compare approaches, reproduce past results, or understand which decisions actually drove improvements. Time and compute are wasted re-running experiments whose results have been forgotten, and teams cannot confidently build on past work.

Experiment tracking transforms the iterative nature of ML from a liability into an asset. When every experiment is recorded — with its configuration, data version, metrics, and artefacts — teams accumulate a navigable history of learning that accelerates future work. This is also a governance requirement: being able to explain how a model was developed and which alternatives were considered is increasingly expected by regulators and demanded by responsible AI practice.

Description of the Practice

  • Logs every training run with its full configuration (hyperparameters, data version, random seeds, code version) and resulting metrics, stored in a queryable experiment tracking system.
  • Links experiments to the exact code commit, dataset version, and environment specification used, ensuring full reproducibility of any recorded result.
  • Organises experiments into structured projects with consistent naming conventions and tagging, enabling efficient search and comparison across large experiment histories.
  • Enables side-by-side comparison of experiments across metrics, artefacts, and configurations through visualisation tooling integrated into the team's workflow.
  • Establishes a process for graduating experiments from exploration to candidate models, with explicit criteria for what constitutes a promising result worth pursuing further.

How to Practise It (Playbook)

1. Getting Started

  • Adopt an experiment tracking tool — MLflow, Weights and Biases, Neptune, or similar — and integrate it with your existing training scripts through a minimal instrumentation layer.
  • Define a standard set of metrics that every experiment must log, ensuring comparability across runs and experiments conducted by different team members.
  • Establish naming conventions for experiments and runs that make the experiment history navigable — include the model type, key hypothesis, and date in experiment names.
  • Run a retrospective on a recent model to record its development history in the tracking system, building a baseline dataset of experiments and establishing team familiarity with the tooling.

2. Scaling and Maturing

  • Integrate experiment tracking with CI/CD so that training pipeline runs are automatically logged, reducing the manual effort required to maintain complete records.
  • Build experiment comparison workflows into model review processes, requiring teams to show the experiment history that supports their recommendation for a candidate model.
  • Implement automated baseline comparison — every new experiment is automatically benchmarked against the current production model and previously best-performing experiment.
  • Use experiment data to drive team-level learning: regularly review experiment histories to identify which types of changes tend to produce improvements in your domain and why.

3. Team Behaviours to Encourage

  • Log every run, not just the successful ones — failed experiments are valuable data that prevents the same dead ends from being explored repeatedly.
  • Record hypotheses before running experiments, not just results after — the discipline of stating what you expect to learn and why builds scientific rigour into the ML development process.
  • Share experiment results with the full team regularly, not just when a candidate model is ready for review — building collective understanding of the development trajectory.
  • Treat the experiment tracking system as a shared team asset and invest in keeping it organised and searchable, not as a personal notebook for individual data scientists.

4. Watch Out For…

  • Logging metrics without logging the configuration that produced them, making results unreproducible and comparisons meaningless.
  • Experiment tracking that records what happened without supporting understanding of why — pair quantitative metrics with qualitative notes on hypotheses and observations.
  • Over-reliance on a single aggregate metric that can be gamed or does not capture the full picture of model quality, leading to optimisation of the wrong thing.
  • Letting the experiment registry become disorganised and unsearchable, destroying its value as institutional memory of the team's learning.

5. Signals of Success

  • Every training run is automatically recorded with full configuration and metrics, with no manual logging steps that can be skipped under time pressure.
  • Teams can reproduce any recorded experiment from the tracking system alone, without reliance on individual knowledge or undocumented local environments.
  • Experiment history is used actively in model review discussions — teams present the development trajectory, not just the final candidate model.
  • The experiment tracking system enables teams to answer questions like "what is the best result we have ever achieved on this dataset?" and "what configurations have we already tried?" in minutes.
  • Experiment cycle time — from hypothesis to logged result — is tracked and is demonstrably decreasing as the team matures its workflow.