GENERator: A Long-Context Generative Genomic Foundation Model
Submitted in February 2025. GENERator is a generative genomic foundation model with a context length of 98k nucleotides, pre-trained on 386 billion nucleotides of eukaryotic DNA. Without task-specific fine-tuning, it achieves zero-shot variant effect prediction comparable to alignment-based methods. With fine-tuning, it achieves leading performance on multiple genomic benchmarks. It can generate protein-coding DNA sequences and design cis-regulatory elements via prompting, including synthetic super enhancers validated by UMI-STARR-seq.
GENERator applies long-context generative pre-training to genomics, achieving zero-shot variant effect prediction at 98k nucleotide length for the first time without sequence alignment. Its generative capability can design functional DNA sequences (e.g., super enhancers), providing new tools for synthetic biology and gene therapy. This marks a shift from discriminative to generative genomic foundation models, potentially accelerating gene editing and drug discovery.
GENERator is based on the Transformer architecture with a causal language modeling objective, pre-trained on 386 billion nucleotides covering species such as human, mouse, and Arabidopsis. The 98k context length is achieved via sparse attention. Zero-shot variant effect prediction: on datasets like ClinVar, AUC reaches 0.85-0.90, comparable to alignment-based methods (e.g., CADD). After fine-tuning, it achieves SOTA on tasks such as ENCODE cCRE classification and splice site prediction. Generation tasks: designed promoter sequences, validated by UMI-STARR-seq, with synthetic enhancer activity 2-3 times higher than natural enhancers. Model parameters are approximately 1.2B, trained using 256 TPUs.
GENERator will transform genomic interpretation and synthetic biology. Pharmaceutical companies can use it to predict pathogenic mutations, accelerating target discovery. Synthetic biology companies can design customized regulatory elements for gene therapy and cell engineering. Its generative capability may reduce trial-and-error costs in DNA synthesis.
Recommend that genomics teams evaluate GENERator's performance in variant pathogenicity prediction and promoter design. Gene therapy companies can try using it to design tissue-specific enhancers. Consider collaborating with the model development team for fine-tuning on specific species or diseases.
Focus on the expansion of GENERator to whole-genome variant effect prediction in humans, and whether it will be used to design guide RNAs for gene editing (e.g., CRISPR). Need to observe the success rate of its generated sequences in cellular experiments and whether the model is open-sourced. It may promote the integration of DNA language models with CRISPR screening.