Large Models for Single-Cell Omics and Drug Discovery: Data, Pretraining, and Closed-Loop Environment by Haotian Cui

TL;DR

In this talk, Haotian Cui presents lessons from scGPT, a foundation model for single-cell biology trained on tens of millions of cells, at the Foundation Models for Biology Seminar Series by GenBio AI. He outlines a roadmap for state-conditioned perturbation modeling and dynamics-aware pretraining, introducing scGPT Spatial and a foundation-model-driven self-driving lab (LUMI Lab). The central idea is to first capture the cell state, then model the perturbation effect given that state. This separation, combined with closed-loop experimentation, improves interpretability, robustness on diverse datasets, and the discovery of biological insights, including a novel brominated lipid motif for mRNA delivery.

Introduction

Imagine you profile a mix of stem cells, immune cells, and tumor cells. The same CRISPR edit or small molecule can have completely different effects depending on the cell’s state. Some cells barely respond, while others activate unexpected pathways. What looks like noise at first often reflects genuine biological context.

With large public datasets now available, foundation models such as scGPT can learn structure across genes and cell states at scale and then transfer to tasks such as annotation, mapping, and perturbation prediction. The data volume is now sufficient to make pretraining effective, and the benefits are clear when the model captures state meaningfully.

His message is clear: treat cell state and perturbation effect as separate components, align training objectives with real biological use, and connect models and experiments in a closed loop so each round of results improves the next. The goal is stronger generalization and clearer value in real-world settings, not just higher scores on benchmarks.

Full Talk

Watch the full talk below:

The Problem

• Entanglement of state and effect: Standard approaches conflate baseline identity with causal intervention, reducing interpretability.
• Heterogeneity and batch effects: Cross-lab integration can introduce spurious signals; preprocessing (e.g., rank or value binning) and robust pretraining mitigate this but are not sufficient.
• Sparse labels and rare subtypes: Long-tailed data make few-shot generalization difficult.
• Metric mismatch: Masked token losses often diverge from real task performance, such as predicting post-perturbation expression.
• Baselines: Methods such as GEARS and CPA set a high bar for perturbation prediction and should serve as minimum benchmarks for comparison.

What is State-Conditioned Perturbation Modeling?

State-conditioned perturbation modeling separates baseline state from intervention effect. First, learn a clean representation of the baseline cell state. Then, encode the effect, such as a gene edit, compound, or dose, and model how that effect changes the state. Because the outcome depends on the starting condition, this approach captures nonadditive interactions and aligns training with real biological dynamics. Instead of static masked reconstruction, the model predicts transitions from time t to t + Δ.

Architecture Overview

• State encoder: Learns cell embeddings (e.g., scGPT’s CLS token) from expression data with optional covariates. Rank or value binning stabilizes cross-study variation.
• Effect encoder: Represents edits, compounds, and doses.
• Interaction module: Transformer or hybrid architecture modeling state–effect interactions rather than simple addition.
• Task-aligned heads: Predict post-perturbation expression or identify the applied perturbation (reverse task).

Benchmarking and Evaluation

• Datasets and baselines: Compare against strong perturbation predictors like GEARS (multi-gene) and CPA, plus simpler centroids/means. Evaluate on held-out cell types/states for extrapolation.
• Metrics: Beyond loss: effect-size fidelity, rank correlation for differential expression, enrichment of known targets, and retrieval accuracy for reverse-identification tasks.
• Zero-/few-shot: Emphasize few-shot and out-of-distribution performance on rare subtypes.

Use Cases in Modern Genomic Research

• CRISPR screens (Perturb-seq): Predict transcriptional responses to specific edits or knockdowns.
• Drug response modeling: Model dose-aware responses across diverse cell states.
• Experimental design: Prioritize perturbations for rare or clinically relevant cell types.
• Cross-study integration: Use robust embeddings for transfer across protocols and labs.

Why Dynamics Modeling?

Larger models do not automatically yield better performance. In both single-cell and protein modeling, pretraining loss can fall while downstream utility remains flat, as seen with ProGen3. Aligning training objectives with real biological dynamics, such as predicting state transitions or differentiation trajectories, produces more meaningful improvements.

LUMI-lab (Foundation-Model-Driven Self-Driving Lab)

LUMI-lab, is an autonomous platform linking molecular foundation models with robotics and active learning to explore ionizable lipid space for mRNA delivery. Across ten iterations and about 1,700 tested lipid nanoparticles, the system identified brominated lipid tails as a high performing motif, validated both in vitro and in vivo. Closed loop experimentation with balanced exploration, synthesizable search spaces, and rapid readouts accelerates model improvement beyond passive scaling.

Future Directions

1. Task-aligned pretraining: Add temporal and perturbational objectives alongside reconstruction.
2. Unified prompts and tokens: Guide multi-omics, multi-task sequence-to-sequence learning (static, dynamic, and perturbation modes).
3. Robust evaluation: Separate seen and unseen perturbations and use task-aware metrics against strong baselines.
4. Model and data co-evolution: Extend closed loop strategies (as in LUMI-lab) to single-cell perturbation campaigns.

Final Thoughts

State-conditioned perturbation modeling addresses a central question: what happens in this specific cell when we intervene? By separating state from effect, aligning training with real biological dynamics, and using closed loop data collection, we move toward foundation models that are more accurate, interpretable, and useful for single cell biology.

Read more about Haotian’s research:

1. scGPT: toward building a foundation model for single-cell multi-omics using generative AI, Nature Methods
2. Towards multimodal foundation models in molecular cell biology, Nature
3. LUMI-lab: a Foundation Model-Driven Autonomous Platform Enabling Discovery of New Ionizable Lipid Designs for mRNA Delivery, bioRxiv

---

Join us in our mission to push the frontiers of AI-driven biology and strive to make a lasting impact on medicine, biotechnology, and human health. We are hiring across teams. Visit our Careers page to learn more and apply. Follow us on X, YouTube, and LinkedIn.