ML experiment notebook template (tracking & reporting)

ML experiment notebook template — tracking & reporting

Purpose: Capture everything needed to reproduce, compare, and understand an ML experiment. Paste this structure into a Jupyter/Colab notebook or adapt it to your experiment tracking tool. Keep the top cells small and machine-readable so automated tools can extract metadata.

How to use

1. Fill the Experiment Metadata cell first. Use a stable experiment ID and tag releases.
2. Record dataset/version hashes before any preprocessing.
3. Keep preprocessing, feature engineering, and training each in separate notebook sections or scripts with clear refs and hashes.
4. Log metrics and artifacts to an experiment tracker (MLflow, W&B, or the platform's submission endpoint) where available.

Top — Experiment metadata (machine-readable)

Place a single JSON / dict here so it can be parsed automatically.

{
  "experiment_id": "projX_expt_2026-08-15_v01",
  "title": "Predictive model for Y",
  "author": "Name / ORCID / team",
  "date": "2026-08-15T10:30:00Z",
  "project": "Project X",
  "tags": ["baseline", "feature-set-A", "regression"],
  "purpose": "Compare baseline model with feature-set-A",
  "notebook_commit": "git-commit-hash",
  "code_repo": "https://git.example.com/org/projX",
  "environment_snapshot": "env-2026-08-15.txt",
  "hardware": "gpu:V100 x1",
  "random_seed": 42
}

Dataset provenance

• Source name and path (S3/GCS/local), access instructions if private.
• Dataset version identifier or snapshot timestamp.
• Checksum or hash of the raw dataset files (md5/sha256) and a note on how it was computed.
• Preprocessing input: which raw snapshot was used and why (link to data catalogue entry).

Preprocessing & data splits

Describe preprocessing steps concisely and include code cell references.

• List transformations in order (e.g., filter rows, impute missing, normalize features, encode categoricals).
• Record exact parameters (e.g., imputation strategy=median, scaler=robust).
• Data split procedure and random seed; share indices or split hashes where feasible.

Feature engineering (Feature catalogue)

Table or list of features used, derived features, and transformation logic.

features = [
  {"name": "age", "source": "demographics.csv", "transformation": "as-is"},
  {"name": "income_log", "source": "finance.csv", "transformation": "log(x+1)"},
  {"name": "interaction_a_b", "source": "derived", "transformation": "a*b"}
]

Training configuration & hyperparameters

• Model family and architecture (including library and version, e.g., scikit-learn 1.3.0, PyTorch 2.1.0).
• Hyperparameters (learning rate, epochs, batch size, regularization, etc.) — list as a dict.
• Training script / command, git commit, and container image or environment hash.
• Compute resources and wall-clock training time.

Evaluation & metrics

Define exactly how each metric is computed (including thresholds, averaging method, and the dataset split used).

• Metrics to report: primary metric (e.g., AUROC on test set), secondary metrics, and calibration checks.
• Confidence intervals or bootstrap procedure, if used.
• Confusion matrix and class-wise performance for classification.

Model artifacts & provenance

• Artifact storage path (URL) and artifact hash.
• Serialization format (pickle, TorchScript, ONNX) and conversion notes.
• Postprocessing required for inference (scalers, vocabulary files).
• Access & licensing notes for produced artifacts.

Results summary (human-readable)

Short paragraph: what happened, whether this meets the experiment purpose, and next steps.

Reproducibility checklist

1. Experiment metadata cell filled with experiment_id, tags, and notebook_commit.
2. Dataset snapshot and checksum recorded; raw data not overwritten.
3. Random seeds set for preprocessing, feature engineering, and model training.
4. Environment capture: pip freeze / conda env export saved (env-YYYYMMDD.txt).
5. Code and notebook committed; commit hash included in metadata.
6. Model artifact stored with hash and linked in notebook.
7. Metrics and evaluation code included; metric definitions unambiguous.
8. Short reproducibility run instructions included (commands to re-run end-to-end).

Lightweight logging examples

Example: Log experiment metadata and a few metrics to an experiment tracker or to a JSON file.

# pseudo-code (adapt to your tracker)
tracker.log_metadata(metadata)
trainer.train()
metrics = evaluator.compute_metrics()
tracker.log_metrics(metrics)
tracker.log_artifact(model_path)

Suggested naming conventions

• Experiment ID: project_shortname_YYYYMMDD_vNN (keeps chronological ordering).
• Artifact name: {experiment_id}__model__{git_commit_hash}.pt
• Dataset snapshot: {dataset_name}__YYYYMMDD__sha256.txt

Example metadata JSON schema (for automation)

{
  "type": "object",
  "properties": {
    "experiment_id": {"type":"string"},
    "title": {"type":"string"},
    "author": {"type":"string"},
    "date": {"type":"string", "format":"date-time"},
    "project": {"type":"string"},
    "tags": {"type":"array", "items":{"type":"string"}},
    "notebook_commit": {"type":"string"},
    "data_snapshot": {"type":"string"},
    "env_snapshot": {"type":"string"},
    "random_seed": {"type":"integer"}
  },
  "required": ["experiment_id","date","notebook_commit","data_snapshot"]
}

Reporting & handoff

Include a short Results slide or markdown summary for stakeholders with key metric, baseline comparison, risks, and next experiments.

Common mistakes to avoid

• Failing to record the exact data snapshot used.
• Not saving the environment or library versions.
• Allowing notebooks to run with non-deterministic defaults (no seeds).
• Mixing exploratory code and production training code without recording the exact training script.

Extensions & integrations

This template is intentionally tool-agnostic. Where available, integrate with your experiment tracker (MLflow, Weights & Biases, or an organizational tracking endpoint) and CI/CD for model promotion. Prefer structured logging (JSON) so metadata is queryable.

Quick checklist to include at commit

• Top metadata cell complete and valid JSON.
• Code committed and commit hash recorded.
• Env snapshot file attached.
• Dataset snapshot location and checksum recorded.
• Artifacts uploaded with hash and linked.

— End of template —