One embryo resource, two modalities

Embryo collections across eleven stages (A), single-cell measurement (B), 3D MERFISH from serial sections (C), and the conditional knock-outs (D).

Download

The training stages are released to registered participants. Validation and test targets come back as a score rather than as answers, until the final phase releases the validation answers for every task.

To look at these stages before downloading them. Cells in 3D, coloured by type, with gene expression. They are in the atlas: browse the challenge training data ↗

Gene panel, per board

A submission’s var_names is compared to the board’s panel element by element: same genes, same order. The panel is not the same on every board, and it is not simply “the panel the target file was measured on”.

It is the ordered intersection of the genes present in every stage that board actually loads: the training stages visible before the target, and the target itself. Where a stage was measured on fewer genes, the intersection shrinks. In the embryo setting, Casp4 and Pnliprp1 were not measured at E6.75 or E7.25, so those two are absent from every embryo board. Including the one whose target, E7.5, does carry them. A prediction cannot be scored against a gene the reference stage never observed.

Each file is one gene name per line, in the order the scorer expects. Take the panel from here rather than from a data file, no released stage is guaranteed to match a board, and a submission whose genes are right but ordered differently is rejected just the same. Machine-readable index, including cell limits and the required obsm keys: index.json.

How many cells to submit

Anywhere in the range above. Cell count is not a scored quantity on any task, and there is no advantage to either end of it. The number of cells in a submission is a sample size. How much evidence you give the metrics to estimate from, and nothing else. It is not a prediction of how large an embryo is, and it is not compared to anything.

The upper bound is operational. A Task 1 answer densifies to n_cells × 32,285 float32 on the scoring host, so the cap is what keeps one submission inside its memory. It is set at three times the larger of the board's truth half and its reference stage, with a floor of 5,000. Sized so that the floor baseline, which resubmits an observed stage verbatim, can satisfy it. The lower bound is statistical: every primary metric estimates a distribution, and below it the estimate is dominated by its own sampling noise rather than by your model.

Going high buys nothing. The three heaviest metrics take their own subsample whatever you send: the unbiased MMD draws 2,000 cells, the energy distance and the variogram 1,500 each, so cells beyond roughly 2,000 are discarded before they are measured. The remaining terms are pseudobulk and proportion statistics, where more cells reduce noise slightly and change nothing systematically. A few thousand cells is a good submission at every board.

One thing that surprises people, because it makes the numbers above look far smaller than the files you downloaded: the scorer subsamples every stage to 10% before comparing anything. E9.5 as released holds 17,057 cells; the scorer's reference stage is 1,706. The target is downsampled the same way and then split in half, one half to score against and one to estimate the ceiling. So truth_cells and ref_cells in index.json are counts of our working copy, not of an embryo and not of the release. Your submission is not expected to mirror them.

What the Task 1 dissection covers

The Task 1 stages are a heart-centred dissection, not whole embryos and not isolated hearts. How much surrounding tissue comes with the heart differs by stage, because what can be dissected cleanly changes as the embryo grows. Populations that are plainly not cardiac are therefore present at some stages and absent at others: Neural Tube is 4.6% of E8.5 and 0% of E9.5, and Paraxial Mesoderm and extra-embryonic mesoderm behave the same way.

This matters for how you read the task. The metrics compare your submitted population to the target population, so the target carries both the biology and the dissection. A model is not being asked to predict our dissection protocol, but it is being scored against a sample that reflects it.

So that the observation process is visible rather than something each team reconstructs, the cell-type composition of the released stages is published: t1_composition.json. Released stages only. Nothing there describes E10.5 or E12.5.

One caution when you read it. The annotation vocabulary is not harmonised across stages: Endocardium at E9.5 appears as Endocardium-1 and Endocardium-2 later, and a label missing from a stage may be present under a different name rather than biologically absent. Comparing label sets across stages will overstate how much has changed. We are working on a harmonised mapping.

Modalities

Single-cell

Task 1

Single-cell RNA

Real dissociated cells profiled across the whole transcriptome, 32,285 genes rather than a panel. Cell-type labels ship with the training stages. This is what Task 1 predicts.

Spatial

Tasks 2 and 3

3D MERFISH, 500-gene panel

A whole heart or whole embryo is sliced continuously into serial 2D spatial-transcriptomic sections, which are then reassembled into a 3D volume. Every cell therefore carries both a 3D position and measured RNA, across several developmental stages, with cell-type labels. Coordinates are per-embryo local and are not registered across time points. The conditional knock-outs (Mab21l2, Gata4 and β-catenin, each with a matched wild type) are profiled the same way.

Where the data comes from

The 3D MERFISH resource, the conditional knock-out embryos and the single-cell Multiome data were all generated by Qingquan Zhang and Neil Chi at UC San Diego. That covers the data behind all three tasks. The benchmark is built on that resource; the tasks, splits and scoring are the competition's.

Competition datasets are provided for research and educational purposes. Any use, reuse, publication, presentation, or redistribution of the data outside the Challenge must appropriately acknowledge the Virtual Embryo Challenge and credit Dr. Neil Chi’s group at the University of California, San Diego (UCSD), which generated the dataset. As the dataset has not yet been published, any use of the data in a publication or public presentation must receive prior approval from Dr. Neil Chi and the data-generating team. Any publication using the dataset must also appropriately credit Dr. Qingquan Zhang and Dr. Neil Chi.

Access and reproducibility

Training data is released for method development. Validation and test ground truth are withheld while they can still affect the board: a validation submission comes back as a leaderboard score, not the answers. At the start of the final phase the validation answers are released for every task. ranking has moved to the hidden test split by then, and the test leaderboard opens. Task 1 additionally permits training on external public single-cell data; only the evaluation stages are fixed, provided any external source is disclosed with the submission.

You may train on external data, public or your own, and for Task 1 this is expressly encouraged. What you may not use is measured data from a held-out stage or genotype, by any route: not directly, not through a model pre-trained on it, and not through a public dataset that contains it. Those are E10.5 and E12.5 for Task 1; E7.5 and E7.75 in the embryo setting and E8.5, E10.5 and E12.5 in the heart setting for Task 2; and the Gata4 and β-catenin knockouts at E8.75 for Task 3. Nearness counts, and the stage label is not the test: external data is treated as a held-out stage when it sits closer to that stage than to the nearest stage we released for the same task, the midpoint being the boundary. Training on an external E9.45 embryo while E9.5 is held out is using the answer under a different name, and so is a dataset labelled by somite count or Theiler stage that lands in the same window. For the Task 1 extrapolation targets the window is stated outright rather than derived, and it relaxes the midpoint rule rather than following it: external data is excluded from after E9.5 up to and including E13.5, E10.5 and E12.5 are excluded absolutely, and anything after E13.5 may be used provided its source and stages are disclosed. The two ends of that window differ because one of them is ours to publish and the other is not. Data measured at exactly E9.5 is permitted, since E9.5 is a released training stage and data at a stage everyone already holds tells nobody anything they were not given. Data measured at exactly E13.5 is not, since E13.5 is the far edge of the protected window rather than the first point outside it, and it sits one day from the test target. For interpolation targets the boundary is the released stage on either side of the held-out one rather than a midpoint: external data may not be used if it stages strictly between the two released stages that bracket a held-out one. In Task 2 that means heart data strictly between E8.25 and E8.75, and embryo data strictly between E7.25 and E8.0. The bracketing stages themselves are permitted, being released training stages. The same applies to genotype: data from the held-out knockouts, from another allele of the same gene at a comparable stage, or from a perturbation that phenocopies them, is treated as the held-out condition. Every external source must be disclosed with the submission; an undisclosed one is a violation whether or not it changed the result, because the disclosure is what makes this checkable.

Each task's loader in the starter kit reads a documented path list, and DATA_SOURCES.md records which file every task reads, its verified size, and its status. Including two dependencies of the richer reference baselines that are not yet on shared storage. The floor and the simple baseline need neither.

A related resource

This site also hosts the Virtual Embryo atlas. It is not the benchmark: the challenge is scored against the releases listed above. but it covers the same organism over the same window, and it is browsable without downloading anything. Useful for orienting yourself in mouse development, and for Task 1, where external public single-cell data may be trained on as long as it is disclosed with the submission.

With one limit that applies to every external source, this atlas included: no measured data from a held-out stage or condition may be used, not for training, not for fine-tuning, not transferred in under another name. The atlas spans the same window as the benchmark, so some of what it contains sits at target time points; using it there is using the answer, however it is routed. The held-out stages are named for each task above, so what is off limits is knowable before you start.