OmiCairn

Campaign run · Parkinson's disease

parkinson-foundation-embedding-r4

PDCompleted the pipelineAttempt 4 of 4 at this topic

Do zero-shot Geneformer (95M-cell variant) and scGPT embeddings of Fullard 2024 SNpc dopaminergic-neuron nuclei separate PD from control donors better than a baseline log-CPM-PCA-LR pipeline at the donor level — and which foundation model wins?

  • Not supported 1
  • Inconclusive 1
  • Not testable (data) 2
  • Not tested (run did not execute) 1
Read the final reportThis run in the hypothesis explorer

Status

How far it got
Completed the pipeline

Tested at least one hypothesis and reached the multi-agent review stage. This shows the run went through the whole pipeline, not that its work passed a quality bar.

Hypotheses

5 nominated; 4 evaluated; 1 not tested because the run did not execute them.

Recorded status

Halted. The run recorded a final status of halted: it was closed without being marked done. A halted run can still have tested hypotheses and written a final report.

Census class

Success. Census class is a label from an earlier, process-level census of the campaign. It does not map one-to-one onto the stages above. How the classes compare.

Research question

The sentence under the run id is quoted from this brief.

Full research brief given to the run

Do zero-shot Geneformer (95M-cell variant) and scGPT embeddings of Fullard 2024 SNpc dopaminergic-neuron nuclei separate PD from control donors better than a baseline log-CPM-PCA-LR pipeline at the donor level — and which foundation model wins?

Cui et al. 2024 (scGPT, Nat Methods 21:1470) and Theodoris et al. 2023+2024 (Geneformer, Nature 618:616 + 95M update) made strong claims about disease-stratification utility of foundation-model embeddings on cell atlases. Boiarsky 2024 and Kedzierska 2024 have begun to publish benchmarks showing zero-shot foundation models are often worse than baseline log-CPM-PCA-LR in disease contexts. PD has not been placed on that benchmark map. This OmiCairn run does so.

Hypotheses (pre-registered):

H1 (primary — donor-level classification, foundation model vs. baseline): Donor-level leave-one-out (LOO) AUC for PD-vs-control classification using donor-mean foundation-model embedding + logistic regression is > 0.75 for at least one of Geneformer or scGPT, AND that AUC is > the log-CPM-PCA-LR baseline AUC by ΔAUC ≥ 0.05, with DeLong test p < 0.05.

H2 (secondary — AGTR1+ subcluster recovery without supervision): Embedding-derived nearest-neighbour graph (Leiden γ = 0.6) in PD donors recovers an AGTR1+ vs. AGTR1- subcluster partition with Adjusted Rand Index ≥ 0.4 against the v1 parkinson-da-vulnerability workspace labels.

H3 (secondary — transfer-learning benefit): Geneformer fine-tuned for 1 epoch on Kamath A1 PD-vs-ctrl labels (GSE178265) generalises to Wang (GSE184950) and Adams (GSE193688) atlases with held-out AUC ≥ 0.7 in each, vs. ≤ 0.6 for the same architecture trained from scratch on the same fine-tune set.

H4 (secondary — embedding-space robustness): Repeat H1 with three subsampling levels (5K / 25K / 50K nuclei per donor average); the ΔAUC vs. baseline must be stable (within ±0.05) across the three subsamples, otherwise the foundation-model advantage is sample-size dependent and that must be reported.

H5 (secondary — cross-atlas zero-shot generalisation): Foundation-model embeddings trained on Fullard alone achieve PD-vs-ctrl AUC ≥ 0.65 when scored zero-shot on Wang and Adams (no fine-tuning). Below this floor, the embedding is Fullard-specific rather than generalising.

Null payoff. If neither Geneformer nor scGPT beats log-CPM-PCA-LR (ΔAUC < 0.05), the field's optimism about single-cell foundation models for neurodegenerative disease is overstated for PD — matching the Boiarsky 2024 + Kedzierska 2024 findings in other diseases, and placing PD on the benchmark map.

The research question comes from the topic's current research brief. All attempts at a topic share it; earlier attempts may have started from an earlier wording.

Hypotheses and verdicts

Each hypothesis was written with its decision rule before any analysis (or, for on-the-fly hypotheses, recorded during execution). AI adjudication agents applied that rule to the run's latest outputs and quoted the evidence verbatim. Where the data did not meet the rule, the verdict follows the data, even if the run's own label or report claimed support.

Each hypothesis as pre-registered (a condensed statement with its decision rule), the adjudicated verdict, the verbatim evidence and the adjudicator's note.
IDHypothesis and decision ruleVerdict
H1Pre-registeredGeneformer-V2 or scGPT donor-level LOO AUC>0.75 AND dAUC>=0.05 over log-CPM-PCA-LR with DeLong p<0.05 (Fullard multi-region)Not supportedIn the explorer

Evidence (verbatim from the run outputs)

H1 decision: NOT SUPPORTED.

Adjudication note

GF AUC 0.775 vs baseline 0.749; dAUC +0.026 (CI -0.066 to +0.118, p=0.290). scGPT never run (package/weights missing, HF 401). Report itself notes ~19% power and 'more precisely described as inconclusive'.

H2Pre-registeredAGTR1+ subcluster recovery in FM embeddingsNot testableIn the explorer

Evidence (verbatim from the run outputs)

Inapplicable: no dopaminergic neurons in the Fullard atlas.

Adjudication note

Declared inapplicable already at plan stage.

H3Pre-registeredGeneformer fine-tuned on Kamath SNpc generalises to Wang and Adams (AUC>=0.70) and beats from-scratch by >=0.10Not testableIn the explorer

Evidence (verbatim from the run outputs)

Deferred: GSE178265 (Kamath) count matrix absent. H3 inapplicable in this iteration.

Adjudication note

None recorded.

H4Pre-registereddAUC stable within +/-0.05 across 5K/25K/50K nuclei-per-donor subsamplesNot runIn the explorer

Evidence (verbatim from the run outputs)

Pre-registered subsampling levels (5K/25K/50K cells/donor) were not feasible on CPU/MPS hardware. Evaluated at 25/50/100 cells/donor

Adjudication note

Compute-limited proxy at 25/50/100 cells (50-500x below plan); run/report declare H4 'STABLE' (range 0.044) on the proxy levels, all dAUC CIs span 0. Could be read as SUPPORTED-on-proxy.

H5Pre-registeredFullard-trained FM embeddings zero-shot AUC>=0.65 with CI lower>=0.50 on BOTH Wang and AdamsInconclusiveIn the explorer

Evidence (verbatim from the run outputs)

| Geneformer (zero-shot) | 0.694 | [0.324, 1.000] | MET |

Adjudication note

DISCREPANCY: run/report say threshold MET, but pre-registered rule also requires CI lower bound >=0.50 (0.324 fails) on both atlases; Wang arm deferred (tar extraction pending); n=13 Adams. Baseline zero-shot AUC 0.944 exceeds GF. Strict reading would be NOT_SUPPORTED.