04 Volume Reconstruction Infrastructure

Session outcomes (60 minutes)

• Design a reproducible ingest-to-serving architecture.
• Define lineage, release, and rollback requirements.
• Select reliability metrics (SLOs) aligned to scientific quality.

Pedagogical arc

• Hook: how infrastructure failure becomes scientific failure.
• Model: architecture and data contracts.
• Practice: failure-mode tabletop exercise.
• Check: release-gate design defense.

Reference architecture

Ingest -> Transform -> Inference -> Post-process -> Serving

Reliability and lineage are first-class scientific requirements.

Visual context: pipeline overview

• Prompt: where would you place mandatory quality gates?

Real data: raw EM vs automated segmentation

• Flood-filling network output: 197 distinct objects in this one field.
• Every colour is an object the pipeline asserts is continuous in 3D.

Data contracts and lineage (minimum fields)

• input artifact IDs
• code/model version
• parameter hash
• timestamp + executor identity
• output artifact IDs

Without this, outputs are not auditable science.

Orchestration essentials

• Idempotent jobs.
• Retry policy with bounded backoff.
• Region-scoped reprocessing.
• Deterministic build environment.

Reliability SLOs tied to science

• Throughput (volume/day)
• Failure rate and MTTR
• Quality gate pass rate
• Cost envelope per released volume

Failure modes and containment

• Non-deterministic rebuilds -> irreproducible claims.
• Provenance drift -> untraceable figures/tables.
• Hotspot bottlenecks -> stale releases and biased datasets.

Tabletop exercise (10 min)

Given a failed post-process stage:

1. decide rollback scope,
2. identify required lineage fields,
3. define release hold criteria.

Activity deliverable

One pipeline diagram including:

• gate locations,
• rollback triggers,
• mandatory lineage schema,
• on-call decision path.

Rubric checkpoint

• Pass: rollback and lineage are explicit and feasible.
• Strong: failure detection links directly to release policy.
• Flag: architecture diagram without decision logic.

External paper figure slots

• Januszewski et al. flood-filling network + infrastructure context figure.
• MICrONS pipeline/infrastructure overview figure.
• Cloud-scale segmentation/proofreading system architecture figure.

Bridge

Next unit: ultrastructure interpretation on top of trusted reconstruction output.