Same model, better performance: the impact of shuffling on DNA Language Models benchmarking

Large Language Models are increasingly popular in genomics due to their potential to decode complex biological sequences. Hence, researchers require a standardized benchmark to evaluate DNA Language Models (DNA LMs) capabilities. However, evaluating DNA LMs is a complex task that intersects genomic’s domain-specific challenges and machine learning methodologies, where seemingly minor implementation details can significantly compromise benchmark validity. We demonstrate this through BEND (Benchmarking DNA Language Models), where hardware-dependent hyperparameters – number of data loading workers and buffer sizes – create spurious performance variations of up to 4% for identical models. The problem stems from inadequate data shuffling interacting with domain specific data characteristics. Experiments with three DNA language models (HyenaDNA, DNABERT-2, ResNet-LM) show these artifacts affect both absolute performance and relative model rankings. We propose a simple solution: pre-shuffling data before storage eliminates hardware dependencies while maintaining efficiency. This work highlights how standard ML practices can interact unexpectedly with domain-specific data characteristics, with broader implications for benchmark design in specialized domains.

💡 Research Summary

As Large Language Models (LLMs) increasingly penetrate the field of genomics, the demand for standardized benchmarks to evaluate DNA Language Models (DNA LMs) has become critical. However, this paper identifies a significant flaw in current evaluation methodologies: the reliability of DNA LM benchmarks is compromised by hardware-dependent implementation details. Through the BEND (Benchmarking DNA Language Models) framework, the authors demonstrate that seemingly minor hyperparameters, such as the number of data loading workers and buffer sizes, induce spurious performance variations of up to 4% even when using identical models.

The core of the problem lies in the interaction between inadequate data shuffling during runtime and the unique structural characteristics of genomic sequences. Genomic data often contains long-range dependencies and specific patterns that require thorough randomization to ensure unbiased evaluation. When the data loading buffer is insufficient or the number of workers is configured in a way that disrupts the shuffling process, the model is exposed to biased data batches. This leads to “sputaneous performance variations” that do not reflect the true capability of the model.

The impact of this phenomenon is profound, as it affects not only the absolute performance metrics but also the relative rankings of models. The study specifically examines three prominent DNA LMs—HyenaDNA, DNABERT-2, and ResNet-LM—and reveals that these implementation artifacts can flip the perceived hierarchy of model excellence. This undermines the validity of comparative studies in the genomics AI community, where researchers rely on benchmarks to identify state-of-the-art architectures.

To mitigate this issue, the authors propose a straightforward yet effective solution: pre-shuffling the data before storage. By performing the shuffling process at the data preparation stage rather than during the training/evaluation runtime, the randomness becomes a fixed property of the dataset itself. This approach renders the benchmark results independent of hardware-dependent hyperparameters like worker counts or buffer sizes, ensuring a deterministic and reproducible evaluation process. Furthermore, pre-shuffling maintains computational efficiency by reducing the overhead of shuffling during the data loading pipeline.

In conclusion, this research serves as a vital warning for the “AI for Science” community. It highlights that standard machine learning practices, when applied to highly specialized and structured biological data, can interact unexpectedly with domain-specific characteristics to produce misleading results. The paper emphasizes the necessity of rigorous, hardware-agnostic benchmark designs to ensure that the progress in genomic AI is measured by true algorithmic advancement rather than implementation-induced artifacts.