BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

This paper introduces BioAgent Bench, a benchmark dataset and an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The benchmark contains curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) with prompts that specify concrete output artifacts to support automated assessment, including stress testing under controlled perturbations. We evaluate frontier closed-source and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that frontier agents can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, because bioinformatics workflows may involve sensitive patient data, proprietary references, or unpublished IP, closed-source models can be unsuitable under strict privacy constraints; in such settings, open-weight models may be preferable despite lower completion rates. We release the dataset and evaluation suite publicly.

💡 Research Summary

BioAgent Bench introduces a comprehensive benchmark suite designed to evaluate large‑language‑model (LLM) agents on realistic, end‑to‑end bioinformatics workflows. Unlike prior LLM benchmarks that reduce scientific tasks to single‑question or code‑generation problems, BioAgent Bench frames each task as a multi‑step pipeline (e.g., RNA‑seq differential expression with DESeq2, germline variant calling with GATK or DeepVariant, metagenomic community profiling with Kraken2, comparative genomics, experimental evolution) that requires tool orchestration, file handling, and generation of concrete output artifacts such as CSV tables. The benchmark provides a natural‑language prompt, the necessary input files, and any reference data (genomes, annotations) for each task, and explicitly specifies the expected final artifact format, enabling fully automated assessment.

The evaluation infrastructure consists of three main components: (1) an agent harness that connects a chosen LLM to a sandboxed execution environment (Claude Code, Codex CLI, OpenCode), providing a system prompt that directs the model to invoke command‑line tools, produce intermediate files, and signal completion; (2) a grader implemented as an LLM (GPT‑5.1) that receives the ground‑truth artifact, the agent’s artifact, and the execution trace, then scores (a) the number of pipeline steps actually completed, (b) an estimate of total steps required for the task, and (c) whether the final result was correctly produced. This grading strategy prioritizes evidence of pipeline completion over strict numerical accuracy, reflecting the fact that many bioinformatics analyses admit multiple valid solutions.

Ten representative tasks were curated, each constrained to run in under four hours and within 48 GB RAM to reflect typical research‑lab resources. This design deliberately excludes large‑scale human genome analyses, focusing instead on smaller organisms and simulated datasets. The tasks span three programming environments (Python, R, Bash) and a variety of domain‑specific tools, ensuring that agents must handle diverse command syntaxes and data formats.

For the model comparison, the authors evaluated five state‑of‑the‑art closed‑source LLMs (e.g., Claude‑Opus‑4‑5, GPT‑5‑1‑Codex) and five open‑weight models (e.g., Claude‑Sonnet‑4‑5, Mini‑Max‑M2.1) across the three harnesses, always using the “high” reasoning effort setting. Closed‑source agents completed more than 80 % of the tasks, often delivering the correct final artifact without custom scaffolding. Open‑source agents achieved roughly a 55 % success rate, showing more frequent step omissions or incorrect file handling.

Robustness was probed with three controlled perturbations: (i) random corruption of input files, (ii) insertion of decoy files that share the same format as legitimate inputs, and (iii) “prompt bloat” where extraneous sentences are added to the task description. Closed‑source models remained functional under input corruption in about 70 % of cases but suffered sharp drops in performance when faced with decoy files or bloated prompts, indicating fragile file‑identification logic. Open‑source models displayed lower resilience across all perturbations, especially failing to discriminate decoy files.

A key discussion point concerns data privacy and intellectual‑property constraints. Closed‑source models typically operate via cloud APIs, meaning that patient‑derived sequencing data or proprietary reference genomes must be transmitted to external servers—a potential violation of institutional policies. Open‑weight models can be deployed entirely on‑premises, making them suitable for regulated environments despite their lower completion rates. The authors argue that, for clinical or pharmaceutical settings where privacy is paramount, open‑source agents may be preferable, provided their performance gaps are addressed.

Limitations of the current work include the exclusion of large‑scale human genomics pipelines, reliance on an LLM grader (which may inherit biases or inconsistencies), and the lack of quantitative accuracy metrics such as F1‑score or ROC‑AUC for downstream biological results. Future directions suggested are (a) expanding the benchmark to include multi‑omics integration and human‑scale datasets, (b) refining grading rubrics to incorporate statistical validation of results, and (c) improving open‑source model capabilities through fine‑tuning and more efficient tool‑use strategies.

In summary, BioAgent Bench provides a novel, reproducible framework for measuring the end‑to‑end competence and robustness of AI agents in bioinformatics. By combining concrete artifact verification, stress‑testing, and privacy‑aware model comparison, it offers a practical yardstick for researchers and industry practitioners seeking to adopt LLM‑driven automation in computational biology.