DNA sequence–based foundation models have advanced the readout of regulatory signals—spanning TF binding, chromatin accessibility, and transcriptional output. Yet two gaps remain: attaining high accuracy and efficiency at scale and robust multi-species generalization. We address these by combining sparse-capacity scaling with multi-species training.
AIDO.DNA2 is a multi-species, bidirectional self-supervised DNA language model built on a Mixture-of-Experts (MoE) transformer. During pretraining, the model learns from raw DNA using a masked-token recovery objective (no assay labels), while sparse expert routing increases effective capacity without raising per-token FLOPs and long-context attention expands the receptive field. After pretraining, we adapt the backbone to tasks via fine-tuning (or linear probes), following the standard pretrain then fine-tune paradigm. We evaluate across variant-level prediction (ClinVar) and genomic readouts—TF and histone ChIP-seq, DNase-seq, CAGE activity, and RNA-seq coverage.
We developed three distinct variants of AIDO.DNA2, each with a different scale of total sparse parameters: small (235M), medium (470M), and large (1B). The models were pretrained on OpenGenome2 and evaluated across three categories: (1) embedding analyses; (2) zero-shot ClinVar variant-effect prediction; and (3) supervised prediction tasks for five genomic readouts—transcription factor ChIP-seq, DNase-seq, histone mark ChIP-seq, CAGE activity, and RNA-seq coverage.
From AIDO.DNA to AIDO.DNA2: What’s new
Architecture
AIDO.DNA2 is an encoder-only transformer with the following upgrades over the vanilla encoder: (i) pre-LayerNorm; (ii) Rotary Positional Embeddings (RoPE); (iii) RMSNorm with zero-centered gamma; (iv) SwiGLU activations; and (v) replacement of the standard FFN with a Mixture-of-Experts (MoE) block, while AIDO.DNA is a dense transformer model.
We use 8 routed experts + 1 shared expert. For each token, a lightweight router computes expert scores (Linear(D, 8) → Sigmoid), adds a learnable expert bias, and selects Top-2 routed experts. Tokens are dispatched to the selected experts, processed independently, and combined using the normalized gate weights. To improve utilization and stability, we apply a load-balancing loss, router z-loss, and token dropping when capacity is exceeded at initial stage.
Pretraining Data
OpenGenome2 is, to our knowledge, the largest open-source DNA corpus curated for large-scale pretraining. It spans eukaryotic genomes, prokaryotic genomes, metagenomes, and augmented functional elements (mRNA, ncRNA, promoters, organelle DNA, splice sites), totaling 8.84 trillion nucleotides. We follow its official preprocessing used for the Evo2 series. Base on this pretraining data, we find it much sample efficient than previous AIDO.DNA pretraining, which adopt NT data.
Long-context curriculum learning
We adopt a multi-stage schedule that progressively increases context length 2K → 8K → 64K → 256K, which improves optimization stability and efficiency for long-range genomic dependencies. We finally get the medium and large model with 64K context length, and small model with 256K. While AIDO.DNA’s context length is fixed 4K.
Results
AIDO.DNA2 Captures Global Structure of rRNAs and Human Genomic Elements
We applied the AIDO.DNA2-large-8K-4T（Consumed 4T tokens with 8K context length) model to generate embeddings for 119,745 small subunit (SSU) rRNAs, 95,279 large subunit (LSU) rRNAs, and 21,680 human genomic element sequences. The embeddings were projected into two dimensions using UMAP (min_dist = 0.5, n_jobs = 25).
AIDO.DNA2 embeddings clearly resolve biological structure across multiple scales. SSU and LSU embeddings separate the three kingdoms of life (eukaryotes, bacteria, archaea), with major phyla forming distinct clusters within each kingdom. For human sequences, coding regions (CDS) and untranslated regions (5′UTR, 3′UTR) cluster apart from non-coding RNAs (lncRNA, miRNA, snRNA, snoRNA, rRNA), reflecting differences in sequence composition and function.
These results highlight the model’s ability to capture both phylogenetic relationships across species and functional distinctions within the human genome, demonstrating the broad representational power of AIDO.DNA2.
Zero-shot ClinVar Variant Effect Prediction
To evaluate the ability of AIDO.DNA2 to capture clinically relevant signals, we benchmarked it on the ClinVar variant effect prediction task, where the goal is to discriminate between Pathogenic and Benign mutations. We constructed a high-quality evaluation dataset from the ClinVar database (version 20240301), retaining only variants with multiple submitters, practice guidelines, or expert review. Variants labeled as Pathogenic, Likely_pathogenic, or Pathogenic/Likely_pathogenic were treated as positive samples, while those annotated as Benign, Likely_benign, or Benign/Likely_benign were used as negative samples. This resulted in 40,899 positive and 151,159 negative SNVs, and 8,538 positive and 5,057 negative non-SNVs.
For evaluation, the model was given the reference and alternative allele sequences centered on the mutation site, each flanked by 4,096 nucleotides. Embeddings were generated, and the L2 distance was computed between the two representations, considering only the 128 nucleotides adjacent to the mutation site. These distances, together with the ClinVar annotations, were used to calculate the area under the precision-recall curve (AUPRC).
As shown in Table 1, AIDO.DNA2 models significantly outperform the earlier AIDO.DNA versions and also surpass the similarly scaled Evo2-1B baseline. Several trends are evident: (i) longer sequence context improves predictive power—8K-context models consistently outperform 2K counterparts, confirming the benefit of modeling extended genomic context; (ii) larger models deliver better accuracy—AIDO.DNA2-1B achieves higher scores than the 470M and 235M variants; and (iii) training with more tokens enhances generalization—the AIDO.DNA2-1B-8K-4T model, trained with 4T tokens and interleaved attention, achieves better performance across both SNV and non-SNV tasks.
These results demonstrate that AIDO.DNA2 not only captures fine-grained sequence signals around mutation sites but also scales effectively with model size, context length, and training data, making it a powerful tool for variant interpretation in clinical genomics.
Supervised Coverage Prediction
We utilized the original Borzoi training data, converting TFRecord files to compressed .npz, and kept the 8-fold split unchanged (train: folds 0–2, 5–7; val: 4; test: 3). During training, inputs are randomly reverse-complemented and shifted by 0–3 nt (labels unchanged). For RNA-seq coverage, we use a frozen AIDO.DNA2 backbone on 524 kb sequences and a lightweight Flashzoi (An enhance Borzoi model) adapter, replacing Flashzoi’s initial convolutional DNA block with a linear projection to 512 channels to consume AIDO.DNA2 representations. To improve efficiency we apply sliding-window attention (±8,192 tokens), bfloat16 mixed precision, and train on 32×H100 (global batch 32) with LR warmup to 2×10⁻⁵, weight decay 1×10⁻⁸, and grad-clip 0.15.
We observe scaling behavior on RNA-seq and CAGE: DNA2-medium and DNA2-large adapters outperform smaller models and the baseline Flashzoi single run. On DNase-seq and ChIP (TF and histone) coverage, the three DNA2 sizes achieve similar correlations to one another, yet consistently match or exceed Flashzoi; These gains are obtained with a frozen backbone and a lightweight adapter, underscoring the efficiency–accuracy advantage of AIDO.DNA2.
We built four-model ensembles for both DNA2-medium and DNA2-large by averaging predictions from independently trained runs. The two ensembles are very close overall: DNA2-large edges DNA2-medium (notably on CAGE), while the remaining tracks are effectively tied. Both DNA2 ensembles substantially outperform the official Flashzoi ensemble across all five coverage-prediction tasks.
To quantify model improvements, we compared per-track scores of our DNA2-large four-model ensemble against the official Flashzoi four-model ensemble across the five coverage-prediction tasks. Per-track comparison of the DNA2-large ensemble vs. Flashzoi official (4-model ensemble). Each point is a track; the dashed line is y = x. Points above the line indicate a DNA2 advantage, with absolute gains of roughly +0.6% to +1.7% across DNase, CAGE, RNA, and histone-ChIP; TF-ChIP also shows a small positive margin.
Acknowledgements
We thank the GenBio AI pre-training team for their foundational contributions to this work, especially Pan Li, Tianhua Tao, William de Vazelhes, Ning Sun, Jing Gong and Xingyi Cheng. We are especially grateful to our CTO, Le Song, for his guidance and mentorship throughout this work. We also thank our colleagues across GenBio AI for their support and helpful discussions.
