Can frontier LLMs accurately interpret spaceflight biology? RR-10 benchmark & interactive 3D viewer

AWG Discussions AI/ML AWG Topics
,
1

Hi everyone, @AIMLawg

I wanted to share an open-science project I put together using NASA OSDR data from the Rodent Research-10 (RR-10) mission called Orbiting Minds.

The project had two main goals: First, to test how well modern frontier AI models can reason through real spaceflight transcriptomics without hallucinating or over-interpreting the data. Second, to make the underlying biology and mission telemetry more intuitive to explore through an interactive 3D WebGL viewer and a slide deck.

I recorded a short 45-second walkthrough showing the benchmark, the data metrics, and the 3D viewer: https://youtu.be/nmOXcB6kAaQ

The Datasets (OSD-563 and OSD-564)

I focused on two RNA-seq accessions from female mice flown on SpaceX-21 (30 days in low Earth orbit) processed through the NASA GeneLab standard pipeline:

  1. OSD-563: Cerebellum tissue, where PC1 accounts for 25.6% of variance and PC2 explains 18.5%.
  2. OSD-564: Hippocampus tissue, where PC1 accounts for 28.3% of variance and PC2 explains 15.7%.
  3. Quality Control Concordance: Count-level reprocessing confirmed a 1.000 correlation with published variance-stabilized tables across the ribosomal RNA-depleted sets.

What the AI Benchmark Found

I set up a 5-part evaluation task on Kaggle testing frontier model families across biological comprehension, statistical rigor (handling DESeq2 vs EdgeR and FDR thresholds), code reproducibility, and epistemic caution.

The most interesting finding across all models was Epistemic Caution. While frontier models handle basic biological mechanisms well, their primary failure mode is overclaiming: repeatedly extrapolating 30-day rodent low Earth orbit microgravity into deep-space or Mars radiation scenarios without stating the real experimental constraints.

Interactive 3D WebGL Explorer

Alongside the benchmark, I built a lightweight WebGL simulation viewer to make exploring the data interactive:

  • Anatomical region pins linking directly to dataset readouts.
  • An Analytic Evidence drawer showing the PCA plots and QC concordance distributions.
  • Commander Pip, an orbital companion designed to help translate the space biology concepts for students and the public.

Project Links:

Looking Ahead & Feedback

I would love to get thoughts from the working groups on extending this framework:

  1. Testing additional central nervous system accessions (such as OSD-102 or OSD-254) to see if model reasoning holds up across longer mission durations.
  2. Exploring multi-omics integration (connecting transcriptomic shifts with proteomics or methylation data).

Any suggestions, critique, or discussion are very welcome! Thanks.

4 Likes
2

Gaston — valuable work, and I think the epistemic-caution finding is the most important result in it. Offering a mechanistic explanation, a resource, and two dataset corrections.

Why the models overclaim direction

I’ve been running a fragility/collapse-detection sweep across the OSDR corpus (637 datasets identified, 302 processed to date) under an open standard I published as λ-SPEC / FI-SPEC (v1.1, DOI 10.5281/zenodo.20724294; validated at r = 0.661, n = 117, p = 2×10⁻⁴ across 117 OSDR datasets).

One asymmetry recurs. Demarcation by co-expression structure reaches r = 0.992 across cohorts, while directional agreement collapses to r ≈ 0.15. The corpus robustly supports which genes move together; it does not robustly support which way they move.

If that holds for your accessions, model overclaiming isn’t purely a reasoning defect — models are being asked to commit to a quantity the corpus does not constrain, and they fill the gap. That turns “epistemic caution” from a behavioural trait into a measurable property of the evidence base. Worth noting your two accessions carry n = 10 each, so low directional stability is the expected regime there rather than an anomaly.

A concrete contribution: 992 contradictory genes

The sweep surfaced 992 genes whose direction of change is contradictory between OSDR datasets. That’s a ready-made overclaim trap with ground truth: ask a model for the direction of one of those genes and the correct answer is an explicit “not determinable from the available evidence.” Your rubric currently scores overclaiming qualitatively; this would let you score it against a known key. Happy to prepare the list with provenance per gene — which datasets disagree, and by how much — and contribute it to the Kaggle suite.

Two notes on the accessions

First, OSD-563 and OSD-564 each carry two assays, not one: bulk RNA-seq and spatially resolved transcriptional profiling on Illumina. The spatial layer is a second, independent handle on the same tissue — a strong test of whether a model’s claims survive when regional structure is available rather than averaged away.

Second, on your proposed extensions: per current OSDR metadata, OSD-102 is Rodent Research-1 mouse kidney and OSD-254 is RR-7 dorsal skin — neither is a CNS accession. But each fits one of your two questions, effectively swapped from how you framed them. OSD-102 carries four modalities (transcriptomics, proteomics, whole-genome bisulfite methylation, RNA methylation, n = 16), which makes it a natural substrate for your multi-omics integration question. OSD-254 carries Duration as an explicit experimental factor at n = 80, which is a much better-powered test of whether model reasoning degrades with mission length. For CNS specifically at longer durations, the accession set would need a separate pass.

One caveat on my own side, in the spirit of the thread: I have not yet verified whether OSD-563 and OSD-564 fall inside my 302 processed datasets. I’ll confirm before claiming anything specific about your two accessions.

— Omar Rafael Pérez Gallardo, OMROS LAB (Querétaro, MX) · ORCID 0009-0008-8328-6978

1 Like
3

One addendum to my note above, since it sharpens the power point rather than softening it.

Checking the OSDR metadata directly: OSD-563 and OSD-564 each carry 10 samples split 5 Space Flight / 5 Ground Control. So the operative figure is n = 5 per arm, not n = 10 per accession as I wrote.

That matters for the epistemic-caution result. At five replicates per arm, DESeq2 and EdgeR have limited power to fix a sign, and directional instability is the expected regime rather than an anomaly. It also means a model asked to commit to a direction from these two accessions is being asked for something the design does not support — which is arguably the cleanest possible demonstration of your finding, since the overclaim can be shown to exceed the experiment rather than merely the literature.

Both assays are present at that same split (bulk RNA-seq and spatial), so the spatial layer is a within-design replicate of the same five animals per arm rather than additional biological replication — worth stating explicitly if the benchmark is extended to it.

— Omar Rafael Pérez Gallardo, OMROS LAB

1 Like
4

Hi Omar,

Thank you for the detailed analysis and the clear context on the sample sizes ($n = 5$ per arm). That $n=5$ constraint really sharpens the core finding: expecting a model to state a definitive direction from these accessions asks it to commit to a quantity that the experimental design itself does not support.

I would love to accept your offer to integrate the 992 contradictory gene list with provenance into the Kaggle evaluation suite. Having a quantitative ground-truth key for directional conflict will make the overclaiming rubric much more objective, and it provides a grounded key to validate model outputs across the Orbiting Minds build.

Expanding across the broader Rodent Research payload series (RR-1 to RR-21+) to evaluate multi-omics and mission duration was the long-term vision I had when starting the Orbiting Minds build, so your callouts on OSD-102 (kidney multi-omics) and OSD-254 (skin duration series) align seamlessly with where this build is headed.

How do you feel about this plan to blend both approaches into a unified architecture:

  1. Kaggle Evaluation Key: Use your 992-gene dataset as an explicit overclaim test module for frontier models.
  2. Scaled Corpus Tasks: Incorporate the expanded RR payload series to evaluate cross-modality reasoning and duration decay.
  3. Simulator Guardrails: Connect the expanded datasets and the 992-gene lookup table back into the 3D WebGL viewer and companion assistant logic.

I’m wrapping up a couple of other project deliverables over the next week or two, maybe even month, but let’s connect on a quick call this weekend regarding data formatting for the CSV/JSON so I can prep it for integration into the benchmark pipeline and my development cycle.

- Gaston D.