Reproducible compute starter kit — containers, CI, provenance

Purpose

This starter kit gives researchers and teams a compact, practical set of patterns and templates to make compute pipelines reproducible, auditable, and cost-aware. It focuses on containerized environments, CI for automated runs, workflow engine skeletons with checkpointing, and explicit provenance capture so results can be reproduced and audited later.

When to use this kit

• You're running multi-step analyses that must be repeatable by others (or by you months later).
• You want a minimal, portable environment that can run on laptop, cluster, or cloud.
• You need provenance (what code, inputs, parameters, environment) recorded for audit or publication.

1) Container strategy & environment locking

Prefer immutable containers for compute code and a separate dependency lockfile for reproducible environments. Choose based on your environment:

• Docker — best for cloud and many CI systems.
• Singularity/Apptainer — useful on HPC where Docker isn't allowed.
• Keep small base images and install only required runtime packages; pin package versions and include hashes when possible.

Example Dockerfile (starter)

<!-- Dockerfile -->
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y python3 python3-venv build-essential && rm -rf /var/lib/apt/lists/*
COPY environment.lock.txt /opt/env-lock/
RUN python3 -m venv /opt/venv && /opt/venv/bin/pip install --upgrade pip && /opt/venv/bin/pip install -r /opt/env-lock/environment.txt
WORKDIR /workspace
COPY src/ /workspace/src/
CMD ["/opt/venv/bin/python", "src/run_pipeline.py"]

Example Singularity definition (starter)

%post
    apt-get update && apt-get install -y python3 python3-venv
    mkdir /opt/venv
    python3 -m venv /opt/venv
    /opt/venv/bin/pip install -r /workspace/environment.txt
%runscript
    exec /opt/venv/bin/python /workspace/src/run_pipeline.py "$@"

Include a separate environment lockfile (pip/conda/renv/packrat) and commit it. For R or Python, include explicit package versions with hashes where supported.

2) Workflow engine skeleton & checkpointing

Use a workflow engine (Nextflow, Snakemake, CWL, Airflow, or similar) to encode steps, dataflow, resource requests, and checkpoints. Key practices:

• Make each step idempotent and side-effect-minimizing.
• Write outputs to explicit, versioned artifact locations (not ephemeral temp). Include a manifest file for produced artifacts.
• Enable resume/retry behavior so interrupted runs can continue from last successful checkpoint.
• Record engine runtime metadata (start/end times, exit codes, node/VM IDs).

Skeleton: Nextflow example

// nextflow.config
process { executor = 'local' }
workflow.onComplete { workflow -> println "Workflow ${workflow}") }

// main.nf
process stepA { 
  container 'ghcr.io/your-org/analysis:1.0'
  input:
    path sample
  output:
    path 'outA'
  script:
  """
  python /workspace/src/stepA.py --in $sample --out outA
  """
}

3) CI / automation for testable runs

Automate small test runs in CI so changes to code/environment are validated. Use GitHub Actions, GitLab CI, or your preferred system to:

• Build and tag container images (include image digest/hash in release notes).
• Run a minimal end-to-end test using a small test dataset.
• On successful test, optionally push artifacts and container images to registries.

CI snippet (high-level)

• build-image: build container → run unit tests
• test-pipeline: spin up test environment → run workflow with small dataset → upload logs/artifacts
• publish: on tag, push container with immutable tag + digest to registry

4) Provenance capture checklist

Capture enough metadata so another engineer can reproduce the run deterministically. Record:

• Code: Git repo URL + commit SHA.
• Container: image name + digest (sha256) and Dockerfile/Singularity def contents.
• Environment: lockfile(s) for language dependencies, OS package list if relevant.
• Inputs: dataset identifiers, checksums (sha256), and storage locations plus any query/filter parameters used to subset data.
• Parameters: full parameter file or command line used (exact values, random seeds, and config files).
• Workflow run metadata: engine name/version, run id, start/end times, exit codes, hardware used (node ids, instance types).
• Outputs: checksum manifest for all published outputs and a RO-crate/Research Object or simple manifest.json describing produced artifacts.
• Logs: capture stdout/stderr and engine logs; store alongside artifacts.

5) Artifact registries & storage patterns

• Containers → container registry (GitHub Container Registry, Docker Hub, private registry). Pin releases by digest.
• Large datasets → object storage with versioning (S3 with versioning, institutional data repository), and provide stable identifiers (DOI) when publishing.
• Small artifacts and metadata → attach to releases in GitHub/GitLab or push to Zenodo/figshare for archival DOI.

6) Cost & performance considerations

• Use caching where possible (workflow-level caches, container layer caches, data caches).
• Prefer spot/preemptible instances for non-critical batch jobs; design for graceful restart.
• Limit parallelism during exploratory runs; scale up for large production runs after validation.
• Tag cloud resources and link run ids to billing tags so cost per run is traceable.
• Instrument with lightweight metrics (runtime, memory, I/O) and export to Prometheus/Grafana or central logs for bottleneck analysis.

7) Quick reproducibility checklist (practical)

1. Commit code and push a tag (record commit SHA).
2. Build container and capture image digest; push to registry.
3. Run workflow on a small test dataset via CI; collect output checksums and logs.
4. Publish artifacts: container tag + digest, manifest.json with inputs/parameters/checksums, and run metadata.
5. Create a minimal README that describes exact reproduction steps and required credentials/access.

8) Next steps & templates

This toolkit should be paired with a small repo of templates: Dockerfile & Singularity def, Nextflow/Snakemake skeleton, GitHub Action workflows for build/test/publish, and a provenance manifest template (manifest.json/ro-crate). If you adopt this kit for a team or project, create a standard release checklist that includes an archival step (e.g., push artifacts to Zenodo) for archival reproducibility evidence.

9) Common pitfalls to avoid

• Committing large raw data to Git – instead store checksums and dataset pointers.
• Relying on unpinned base images or floating package versions.
• Not recording random seeds or non-deterministic configuration options.
• Assuming local developer environments match cluster production without testing in containerized runtimes.

Use this starter kit as a practical baseline: keep templates small, test often, and record provenance at every run. That combination makes compute work reproducible, auditable, and easier to scale.