DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome
DNABERT-2 replaces k-mer tokenization with BPE, achieving performance comparable to SOTA on the GUE benchmark (36 datasets, 9 tasks) with 21x fewer parameters and 92x less pre-training GPU time.
The paper points out that k-mer tokenization leads to computational and sample inefficiency, proposes using BPE tokenization to build DNABERT-2, and creates a multi-species genome understanding evaluation benchmark GUE, significantly reducing model size and training cost.
DNABERT-2 adopts BPE tokenization, which generates non-overlapping tokens by statistically merging high-frequency co-occurring substrings, overcoming the fixed length and overlap issues of k-mers. The model uses attention mechanisms and positional encoding to handle long sequences, evaluated on the GUE benchmark (36 datasets, lengths 70-10000), achieving performance comparable to Nucleotide Transformer but with 21x fewer parameters and 92x less pre-training GPU time.
This work reduces the training and deployment cost of genome foundation models, enabling small and medium-sized research institutions to participate in genome AI research, accelerating applications of genomics in precision medicine, agricultural breeding, and other fields.
It is recommended that bioinformatics teams evaluate DNABERT-2 as a genome analysis tool to replace existing k-mer models, which can reduce computing costs and improve processing speed, suitable for scenarios such as gene variant detection and species classification.
Future work needs to verify the generalization ability of BPE tokenization on larger-scale genomic data, as well as the actual inference efficiency of the model on long sequences (e.g., the human genome).