Reconstruction Pipeline

End-to-end connectome reconstruction architecture — from raw image ingest through alignment, segmentation, agglomeration, synapse detection, and serving. Full instructor script with references.

Stylized vector art: pipeline stages running above a chunk grid.

Overview

Connectome reconstruction is a production data-engineering problem as much as a neuroscience problem. Converting petabytes of raw EM images into a queryable graph of neurons and synapses requires a pipeline of interdependent computational stages, each with its own failure modes, quality metrics, and scaling challenges. This document walks through the canonical reconstruction pipeline used in modern connectomics projects.

Instructor script: pipeline architecture

The five-layer model

Think of the reconstruction pipeline as five layers, each transforming the data toward a higher-level representation:

Raw images → Aligned volume → Segmentation → Agglomerated objects → Connectome graph
   (L1)          (L2)            (L3)              (L4)                 (L5)

Each layer depends on the previous one, errors propagate forward, and reprocessing may require re-running everything downstream of the change.

Layer 1: Ingest

What happens: Raw image tiles arrive from the microscope. Each tile is a 2D image, typically 4K×4K to 8K×8K pixels at 4-8 nm/pixel. A single section may contain hundreds to thousands of tiles. A full dataset may have thousands to tens of thousands of sections.

Key operations:

Scale context: The MICrONS dataset (1 mm³ mouse cortex at 4 nm XY, 40 nm Z resolution) is approximately 2 petabytes of raw image data. The H01 human cortex fragment is approximately 1.4 petabytes. Storage and I/O bandwidth are first-order constraints.

Layer 2: Alignment

What happens: Individual tiles are stitched into section mosaics, and consecutive sections are registered to produce a coherent 3D volume.

Tile stitching: Adjacent tiles overlap by 5-15%. Cross-correlation of overlapping regions determines the precise offset. Intensity normalization across tiles corrects for illumination non-uniformity.

Section registration: Consecutive sections are aligned using feature matching or cross-correlation. This is conceptually similar to video stabilization but with unique challenges:

Methods: Saalfeld et al. (2012) developed TrakEM2’s elastic alignment for serial-section datasets. More recent approaches use deep-learning-based feature matching (Mitchell et al. 2019). The key metric is registration residual — the remaining misalignment after correction, typically targeting <1 pixel (4-8 nm).

Critical failure mode: Accumulated alignment drift. If each section has a small residual error (~0.5 pixel), over 10,000 sections this accumulates to ~70 pixels (~560 nm) of drift. Mitigation: anchor alignment to known structures (blood vessels, soma boundaries) and apply global optimization.

Layer 3: Segmentation

What happens: Every voxel in the aligned volume is assigned to a specific object (neuron, glia, blood vessel, extracellular space, etc.). This is an instance segmentation problem — not just “this is neural tissue” but “this is neuron #47,293.”

Modern approach — two-stage pipeline:

  1. Affinity/boundary prediction: A convolutional neural network (typically a 3D U-Net or similar encoder-decoder architecture) predicts, for each voxel, the probability that it belongs to the same object as each of its neighbors (affinity map) or the probability that it sits on an object boundary (boundary map). Trained on manually annotated ground-truth regions.

  2. Watershed + agglomeration: Initial over-segmentation via watershed transform on the affinity/boundary maps produces millions of small “supervoxels” — fragments that are almost certainly part of a single neuron. These supervoxels are then agglomerated (merged) based on affinity scores between adjacent supervoxels.

Alternative approach — Flood-Filling Networks (FFN): Januszewski et al. (2018) introduced an iterative approach where a neural network “grows” each segment by predicting which neighboring voxels belong to the same object, starting from a seed point and expanding outward (like flood-fill). FFN was used for the FlyWire and other Google-based reconstructions.

Scale challenges: Running inference on a 1 mm³ volume at 4 nm resolution requires processing ~10^13 voxels. This is distributed across hundreds to thousands of GPUs. Typical compute time: weeks to months. Cost: hundreds of thousands to millions of GPU-hours.

Quality: State-of-the-art methods achieve “superhuman” accuracy on benchmarks (Lee et al. 2019), meaning they make fewer errors per unit volume than individual human annotators. However, error rates of even 0.1% per supervoxel accumulate rapidly across a volume containing millions of supervoxels.

Layer 4: Post-processing

Agglomeration refinement: The initial agglomeration (Layer 3) produces objects that are mostly correct but contain merge and split errors. Post-processing refines these:

Synapse detection: A separate neural network identifies synapses in the aligned volume:

Synapse detection is critical because the connectome graph depends on it — edges without accurate synapse detection are meaningless.

Layer 5: Serving

What happens: The reconstructed volume, segmentation, synapses, and graph are made available for proofreading and analysis through web APIs and visualization tools.

Key components:

What it costs, in the units that bind

The single fact trainees most need about reconstruction is missing from most descriptions of it: what running the pipeline actually consumes. The binding constraints are not the same at each stage, which is why “it’s expensive” is not useful planning information.

Stage Binding constraint Roughly what dominates
Ingest and alignment Wall clock and I/O Reading and rewriting petabytes; alignment itself is cheap by comparison
Boundary / affinity prediction GPU time Every voxel passes through a network. This is where a mm³ reconstruction’s compute bill is
Supervoxel and agglomeration CPU and memory Graph operations over billions of fragments; memory, not FLOPs
Meshing and skeletonization CPU, embarrassingly parallel Cheap per object, expensive because there are millions
Proofreading Human hours The most expensive input, the hardest to scale, and the one you cannot buy more of at short notice
Serving Storage and egress Ongoing rather than one-off; egress is what surprises people

Two things follow. First, GPU time and human hours are the two large numbers, and they trade against each other: a better segmentation costs more compute and less proofreading. That trade is the actual design decision behind a reconstruction pipeline, and the agglomeration threshold is the dial.

Second, only one of those two scales by spending money. You can rent more GPUs this week. You cannot rent more trained proofreaders this week, and a proofreader who has not worked through something like Units 05-07 will generate errors faster than they fix them. That asymmetry is why acquisition QA and segmentation quality matter disproportionately: they are the levers that reduce demand on the input you cannot scale.

Do the arithmetic before you commit. Take the voxel count from Unit 03’s cost arithmetic, your model’s throughput in voxels per GPU-hour, and your pilot’s measured error rate against your per-neuron proofreading time. The result is a GPU-hour figure and a person-hour figure. If the second is larger than your program can staff, the answer is not to proofread faster — it is to change the segmentation or the acquisition, upstream, where it is still cheap.

Where the pipeline is not a pipeline

The five-layer model above is a batch pipeline: data flows one way and each stage completes before the next begins. That is true right up until the first edit, and then it stops being true.

Proofreading edits do not re-run the pipeline. They change which supervoxels are grouped together, and that grouping lives in the ChunkedGraph, which sits beside the batch pipeline rather than inside it. The consequences are the ones that confuse people most often:

If you take one thing from this page into your own work: the batch pipeline produced the segmentation, but the ChunkedGraph is what you are actually querying, and the version you pin is a property of the second, not the first.

Provenance and reproducibility

Every stage must record:

Provenance field Purpose
Input data version/hash Exactly which data was processed
Code revision (git hash) Which software version ran
Model artifact ID Which trained model (for ML stages)
Parameter configuration All hyperparameters and thresholds
Runtime environment Hardware, OS, library versions
Output data version/hash Fingerprint of results

Why this matters: If a downstream analysis produces unexpected results, you need to trace back through the pipeline to determine whether it’s a biological finding or a processing artifact. Without provenance, this is impossible.

Worked example: diagnosing a connectivity anomaly

Scenario: An analysis reveals that neurons in one corner of the volume have 30% fewer synaptic connections than neurons in the center.

Diagnostic pipeline trace:

  1. L5 (Graph): Verify the connectivity difference is real in the graph database, not a query bug.
  2. L4 (Synapse detection): Check synapse detection confidence scores in the two regions. Finding: synapse confidence is 15% lower in the corner.
  3. L3 (Segmentation): Check segmentation quality. Finding: more split errors in the corner.
  4. L2 (Alignment): Check alignment residuals. Finding: normal.
  5. L1 (Raw images): Inspect raw image quality. Finding: membrane contrast is reduced in the corner — staining gradient from incomplete osmium penetration.
  6. Root cause: Staining artifact → reduced membrane detection → more split errors → missed synapses → apparent connectivity deficit.
  7. Resolution: (a) Flag region in metadata. (b) Re-run segmentation with adjusted model threshold. (c) Prioritize proofreading in that region. (d) Report the spatial quality gradient in any publication using this data.

Common misconceptions

Misconception Reality Teaching note
“Segmentation is the hard part” Every stage matters equally — alignment errors can be just as damaging as segmentation errors Quality is a chain; the weakest link dominates
“Once segmentation is done, we have a connectome” Synapse detection, proofreading, and graph construction are separate, critical stages Segmentation alone gives you objects, not connections
“Reprocessing means starting over” Good pipeline design supports partial reprocessing — e.g., re-segment one region without re-aligning the whole volume Design for regional rollback from the start
“More GPUs = faster results” I/O bandwidth and data staging often bottleneck before compute Profile your pipeline for I/O vs compute balance

References