Module 13: Machine Learning in Neuroscience

Apply machine-learning workflows to connectomics features, with explicit controls for data leakage, bias, and interpretability.

Stylized vector art: a layered neural network with one weighted path lit.

Lesson Flow

Learn

Goals and Concepts

Start with the capability target and concept set for this module.

Practice

Studio Activity

Apply the ideas in a guided activity tied to realistic outputs.

Check

Assessment Rubric

Use the rubric to verify competency and identify improvement targets.

Interactive Lab

Practice in short loops: checkpoint quiz, microtask decision, and competency progress tracking.

ML Pipeline Checkpoint

Q1. Your fragment classifier scores 0.94 on a random row split and 0.71 when split by parent neuron. What is the honest reading?

A random row split blocks no leakage channel, so it is an upper bound rather than an estimate. Grouping by parent neuron costs effective sample size and needs reliable parent IDs at a pinned version, which is the price of a number you can defend. The 23-point gap is a measurement of how much leakage was present and is worth reporting.

Q2. Your model ranks segments for a team that can inspect 500 per week. Which metric matches that decision?

The decision is which 500 segments get opened, so the only errors that matter are those above the cutoff. Precision at k forces you to know reviewer capacity before evaluating, which is what makes the number actionable. Accuracy and macro F1 average over segments nobody will ever look at.

Q3. Model and expert label disagree on 12% of a rare class. Before calling that model error, what must you measure?

Your ceiling is set by how well two experts agree, not by 100%. If annotators themselves disagree on 15% of that class, a 12% model-label mismatch sits inside the label noise and calling it model error misdirects the next month of work. Prevalence and calibration answer other questions and neither tells you whether the target is trustworthy.

Feature Audit Microtask

You are about to add raw image intensity statistics to the feature set. What must the design sheet record?

Progress Tracker

State is saved locally in your browser for this module.

0% complete

Capability target

Design and critique an ML analysis pipeline for connectomics that includes feature rationale, evaluation plan, leakage controls, and interpretation limits. Concretely: choose a split strategy from the leakage channels present in your data rather than from convention, pick metrics from the decision the model will support, quantify how much of your reported performance survives a harder split, and write a limitation statement specific enough that a reader knows which uses of your model you would refuse.

Why this module matters

ML accelerates connectomics analysis, and naive workflows produce misleading biological claims at the same speed. The characteristic failure is not a model that performs badly — it is a model that performs suspiciously well because the split leaked, and whose reported number then propagates into a paper as though it described generalization.

Connectomics is unusually leaky. Fragments of one neuron appear in many rows. Neighboring neurons share staining, imaging conditions, and section artifacts. Cell-type labels are often derived from connectivity, so a model predicting connectivity from cell type may be reading its own answer. The proofread subset is not a random sample of the volume: neurons get proofread because someone wanted them, which usually means they were large, central, or interesting. Every one of these is a channel by which test data informs training, and none of them is visible in a learning curve.

Concept set

1) Feature engineering defines the hypothesis space

2) Evaluation must match the decision the model supports

3) Leakage and distribution shift are endemic

4) Base rates decide what accuracy means

5) Labels are a hypothesis, not ground truth

Split strategy: what each choice blocks and what it costs

Split strategy Leakage channel it blocks What it costs you
Random row split None Optimistic by a wide and unknown margin; useful only as an upper bound
Group by parent neuron Fragments of one neuron on both sides Fewer effective samples; requires reliable parent IDs at a pinned version
Spatial block (e.g. 100 µm tiles) Shared staining, alignment, local artifacts Fewer, noisier folds; block size is a judgment call you must defend
By proofreading batch or annotator Annotator style and QC-round provenance May confound with region if batches were assigned region by region
Held-out dataset or species Domain shift Smallest and hardest test; under-states in-domain performance
Decision the model supports Metric that matches Metric that misleads What the right metric costs
Ranking a proofreading queue Precision at k, with k = weekly reviewer capacity Overall accuracy You must know reviewer capacity before evaluating
Finding rare cell types Per-class recall, precision at fixed recall Macro accuracy Confidence intervals are wide at small N
Gating automated merges False-positive rate at a high threshold F1 Throughput falls sharply as the threshold rises
Population statistics from predicted labels Calibration curve, label-noise correction Accuracy Requires a separately labeled calibration set

Worked example: the classifier that scored 0.92 and taught nothing

The numbers below are illustrative — they show the shape of the reasoning, not results from a specific published dataset.

You have 4,000 labeled neurite fragments in five coarse classes and train a gradient-boosted model on the feature set from Concept 1. A random 80/20 split gives macro-F1 = 0.92. That number is the first thing to distrust, because 0.92 on a five-class morphological problem is better than trained human annotators typically agree with each other.

First question: where did the rows come from? The 4,000 fragments came from 600 neurons, so on average nearly seven fragments per neuron, and a random split puts fragments of the same neuron on both sides. The model can memorize a neuron’s caliber and branching signature and recognize its other fragments. Re-splitting grouped by parent neuron gives macro-F1 = 0.71. Twenty-one points of the original score were fragment duplication.

Second question: what else do neighbors share? Fragments from the same cortical column share staining, section artifacts, and alignment residual. Re-splitting into 100 µm spatial blocks gives 0.66. The assumption here is explicit and worth stating in the same sentence as the number: 100 µm blocks only block adjacency leakage if no relevant structure spans a block. Apical dendrites routinely span more than 100 µm, so this estimate is still slightly optimistic for dendrite-derived features.

Third question: does it transfer? Evaluated on fragments from a second dataset with different staining, macro-F1 = 0.41, and the drop is not uniform — one class falls to near chance while three barely move. This is the finding, not a disappointment: the model has learned three classes robustly and two only in-domain.

Fourth question: does the aggregate hide the use case? The intended use is ranking segments for proofreading. Per-class recall for the rarest class is 0.18 at the operating threshold, and that class is 4% of the sample, so it contributes almost nothing to macro-F1 either way. The metric that matters is precision at k = 500, the weekly review capacity — measured directly at 0.62, meaning roughly 310 of the 500 flagged segments contain a real error.

What gets reported. Not 0.92. Report 0.66 as the in-dataset generalization estimate, 0.41 as the cross-dataset estimate, precision@500 = 0.62 as the deployment metric, and per-class recall as a table so the weak classes are visible. The limitation statement says: supported use is prioritizing a review queue within this dataset; unsupported uses are assigning final labels, comparing class proportions across datasets, and any use on tissue with different staining without re-measurement.

What this example does not establish. It does not show that 0.66 is the true generalization performance — only that each successive control removed leakage the previous one missed, and the sequence had not yet converged. A fifth control, blocking by proofreading batch, might drop it again.

Hidden curriculum scaffold

Core workflow: connectomics ML protocol

  1. Write the biological decision the model will support, naming who acts on the output and what they do differently as a result.
  2. Enumerate leakage channels present in your data — fragment duplication, spatial adjacency, annotator provenance, label circularity — and choose the split that blocks the strongest one.
  3. Construct the feature set with a one-line rationale per feature, and record the segmentation version the features were computed from.
  4. Fit all preprocessing (scaling, imputation, feature selection) inside the training fold only.
  5. Train a trivial baseline first — majority class, or a single-feature threshold — and report it alongside every later model.
  6. Evaluate with the metric that matches the decision from step 1, plus per-class recall and prevalence.
  7. Run error analysis on the failures: sample 20-30 misclassified examples and classify the failure reason by hand.
  8. Write the model card: intended use, unsupported uses, evaluation splits, metrics with intervals, and the domain in which the numbers hold.

Pre-class preparation

60-minute tutorial run-of-show

  1. 00:00-08:00 | Task framing and leakage examples Present the 0.92 result and ask the room to explain it. Collect hypotheses on the board before revealing the group-split number.
  2. 08:00-20:00 | Feature rationale workshop Each learner writes a one-line rationale per feature and marks any feature that could encode dataset identity rather than biology.
  3. 20:00-34:00 | Split strategy and baseline modeling Teams implement two splits — random and grouped — on the same data and report both scores. The gap is the exercise.
  4. 34:00-46:00 | Error analysis and biologically relevant metrics Learners compute per-class recall and prevalence, then precision at a k set by a stated review capacity.
  5. 46:00-56:00 | Model-card limitation writing Each learner drafts three unsupported uses for their own model and has a neighbor try to break them.
  6. 56:00-60:00 | Competency checkpoint Each learner names the leakage channel they consider most likely still present in their own work.

Studio activity: leakage-resistant ML mini-pipeline

Scenario: You must classify neurite fragments into coarse categories to prioritize a proofreading queue. You have roughly 4,000 labeled fragments drawn from about 600 neurons in one dataset, five classes with prevalences of approximately 38%, 27%, 19%, 12%, and 4%, and a reviewer team that can inspect 500 segments per week. A second, differently stained dataset is available as a held-out domain.

Tasks

  1. Propose a feature set with a one-line rationale per feature, and flag any feature that could encode dataset identity.
  2. Design the split, naming the leakage channel each choice blocks and the cost you accept for it.
  3. Train one baseline and one improved model, or write the pseudocode plan if compute is unavailable.
  4. Report two standard metrics, one biologically targeted metric tied to the 500-segment review capacity, and per-class recall with prevalence.
  5. Sample 20 misclassified fragments, classify the failure reason by hand, and propose the one data improvement that would fix the largest group.
  6. Draft a model limitation statement naming at least three unsupported uses.

Expected outputs

Assessment rubric

ML in the connectomics pipeline

Machine learning is embedded at every stage of the reconstruction pipeline:

Stage ML task Key methods Training data source
Segmentation Predict voxel affinities/boundaries U-Net, FFN (Januszewski et al. 2018) Manual ground-truth annotations
Synapse detection Identify cleft locations + pre/post partners 3D CNN on local patches Expert-annotated synapse sets
Cell-type classification Assign neuron type from morphology/connectivity Random forest, GNN, clustering Morphologically typed neurons
Error detection Flag likely merge/split errors for proofreading Classifier on segment features Proofreading correction logs
Automated proofreading Suggest corrections Reinforcement learning, heuristic models Before/after correction pairs

Domain shift is the central challenge: A model trained on well-stained MICrONS data may fail on under-stained H01 regions, on different species (mouse → fly), or on tissue with pathology (near epileptic foci). Always evaluate on held-out data from the target domain, not just the training domain.

Common errors and how to recover

What this module does not cover

Content library references

Teaching resources

References

Quick practice prompt

For one candidate model, write:

  1. one plausible leakage pathway,
  2. one metric blind spot,
  3. one limitation you would report publicly.

Teaching Materials

Activity Worksheet

Learner worksheet aligned to the studio activity and rubric.

Open worksheet

Slide Source

Marp source file for editing and rendering.

course/decks/marp/modules/module13.marp.md

Related Content