OpenDDI: A Comprehensive Benchmark for DDI Prediction

Drug-Drug Interactions (DDIs) significantly influence therapeutic efficacy and patient safety. As experimental discovery is resource-intensive and time-consuming, efficient computational methodologies have become essential. The predominant paradigm formulates DDI prediction as a drug graph-based link prediction task. However, further progress is hindered by two fundamental challenges: (1) lack of high-quality data: most studies rely on small-scale DDI datasets and single-modal drug representations; (2) lack of standardized evaluation: inconsistent scenarios, varied metrics, and diverse baselines. To address the above issues, we propose OpenDDI, a comprehensive benchmark for DDI prediction. Specifically, (1) from the data perspective, OpenDDI unifies 6 widely used DDI datasets and 2 existing forms of drug representation, while additionally contributing 3 new large-scale LLM-augmented datasets and a new multimodal drug representation covering 5 modalities. (2) From the evaluation perspective, OpenDDI unifies 20 SOTA model baselines across 3 downstream tasks, with standardized protocols for data quality, effectiveness, generalization, robustness, and efficiency. Based on OpenDDI, we conduct a comprehensive evaluation and derive 10 valuable insights for DDI prediction while exposing current limitations to provide critical guidance for this rapidly evolving field. Our code is available at https://github.com/xiaoriwuguang/OpenDDI

💡 Research Summary

The paper introduces OpenDDI, a comprehensive benchmark designed to address two fundamental bottlenecks in drug‑drug interaction (DDI) prediction: the scarcity of high‑quality, large‑scale data and the lack of standardized evaluation protocols. From the data perspective, OpenDDI aggregates six widely used public DDI datasets and augments them with three newly curated, large‑scale datasets built from twelve authoritative biomedical resources (DrugBank, ChEMBL, BindingDB, KEGG, etc.). These new datasets contain over 5.3 million interaction records spanning 34 000 unique drugs, and they are enriched using large language models (LLMs) to fill missing or uncertain labels, thereby improving completeness and reliability.

In addition to expanding the quantity of data, OpenDDI introduces a multimodal drug representation that integrates five complementary modalities: (1) SMILES strings for chemical structure, (2) graph‑based path embeddings that capture relational information from knowledge graphs, (3) three‑dimensional conformations, (4) protein amino‑acid sequences, and (5) textual descriptions (e.g., pharmacological annotations, indications). Each modality is encoded separately using state‑of‑the‑art encoders (e.g., transformer for text, 3D‑GNN for conformations) and then fused via a learnable aggregation function, yielding a unified feature vector for each drug node. This design aims to capture semantic, structural, biological, and relational aspects of drugs that single‑modal approaches typically overlook.

From the evaluation perspective, OpenDDI implements a unified pipeline that standardizes data loading, preprocessing, and model interfacing. Twenty recent DDI prediction methods are incorporated, covering three methodological families: similarity‑based, graph‑neural‑network‑based, and integration‑based approaches. All models are wrapped in a common API, enabling fair, reproducible comparisons. The benchmark defines three downstream prediction tasks—binary interaction classification, multiclass interaction type classification, and multilabel interaction mechanism classification—mirroring real‑world use cases. Moreover, five evaluation dimensions are systematically measured:

1. Data Quality – assessed by dataset scale, coverage, and the performance variance of models across datasets.
2. Effectiveness – measured using standard metrics (Accuracy, F1‑score, AUROC) on each task.
3. Generalization – evaluated by testing models on held‑out drugs that never appear in training graphs, thereby probing zero‑shot capability.
4. Efficiency – quantified in terms of wall‑clock training/inference time and peak memory consumption.
5. Robustness – examined under two stress conditions: (a) injected label noise to simulate erroneous DDI records, and (b) artificially sparsified graphs to mimic rare interaction scenarios.

Experimental results reveal several key insights. First, multimodal representations consistently outperform single‑modal baselines (SMILES or path alone) by 8–16 percentage points in accuracy, with the most pronounced gains observed in multilabel tasks where capturing diverse mechanistic cues is crucial. Second, the large‑scale LLM‑augmented datasets enable models to achieve higher absolute performance and better stability compared to older small‑scale benchmarks, confirming the importance of data volume and label completeness. Third, despite overall improvements, generalization to unseen drugs remains a challenge: performance drops of 10–20 % are observed when evaluating on novel drug nodes, indicating that current architectures still rely heavily on memorized structural patterns rather than truly abstract interaction principles. Fourth, efficiency analyses show a clear trade‑off: graph‑based deep models (e.g., GNN variants) deliver top accuracy but incur substantial memory footprints, whereas similarity‑based methods scale gracefully to millions of edges with modest resource demands, making them attractive for real‑time deployment. Fifth, robustness experiments demonstrate that modest label noise (≤5 %) has limited impact on most models, yet severe sparsity dramatically degrades performance, highlighting the need for regularization techniques or data‑augmentation strategies tailored to sparse interaction graphs.

The authors synthesize these findings into ten actionable recommendations for the DDI community, emphasizing (i) the adoption of multimodal drug encodings, (ii) the construction of even larger, more diverse interaction corpora, (iii) the development of zero‑shot or meta‑learning frameworks to improve generalization, (iv) the design of lightweight yet accurate architectures for resource‑constrained settings, and (v) systematic robustness testing as a standard part of model validation. OpenDDI’s codebase and curated resources are publicly released, inviting the community to extend the benchmark, add new baselines, or explore novel tasks such as drug‑target interaction prediction or adverse event forecasting. In sum, OpenDDI provides a unified, scalable, and reproducible platform that substantially raises the bar for methodological rigor and comparability in DDI prediction research.