AIRS-Bench

Description

AIRS-Bench is an environment for evaluating LLM agents' ability to perform end-to-end AI research. Given a problem description and dataset, the agent must build a model or solution and produce predictions (submission.csv). Tasks span NLP, code generation, math, molecular property prediction, graph ML, and time series forecasting.

Capabilities

• End-to-end AI research: problem understanding, data exploration, model building, and prediction
• Multi-domain evaluation across 6 categories and 20 tasks
• Server-side evaluation with task-specific metrics preventing label leakage

Compute Requirements

Each agent sandbox runs with 1 CPU and 2GB RAM. Network access is limited to PyPI, Hugging Face models, evaluate's metric scripts, PyG's wheel index (data.pyg.org/whl) and nltk_data (Hub datasets and other Spaces are blocked). No GPU required for the sandbox (agents work with CPU-friendly approaches or pre-trained models).

License

CC BY-NC 4.0

Tasks

20 tasks across 6 categories, available in both train and test splits:

TimeSeriesForecastingKaggleWebTrafficMASE is not served: a submission is 145,063 full-length series (about 1 GB of CSV), too large to grade in the environment server. list_tasks returns the other 19.

check_ground_truth.py checks each task's ground truth against its grader: the columns it reads exist, a submission reproducing the labels scores the optimum, the agent's test split has one row per label, and each task's worst score is the score of its best trivial model or constant answer.

Reward Structure

The raw metric is normalised to a reward in [0, 1]: (worst - raw) / (worst - optimal), clipped, where optimal is the metric's best value and worst is the task's estimated_worst_score. For the regression tasks worst is the score of the best trivial model, so a trivial model earns 0:

• QM9 G and U_0 (MAE, meV): the per-element reference model, a sum of per-element energies over each molecule's H, C, N, O and F counts, fitted on train by least squares. Total energies are almost linear in composition, so a constant is not a meaningful baseline for them; the reward measures how much of the per-element model's error a submission removes.
• QM9 c_v and R_2_Abs, and ZINC (MAE): a constant prediction of the train-set median, the constant that minimises absolute error.
• Rideshare and Solar (MAE): the stronger of the naive forecast (each series' last observed value) and each series' historical mean.
• SICK relatedness (Spearman): 0, the correlation of a constant prediction.

For the Accuracy tasks (SVAMP, Winogrande, SuperGLUE WSC, Yelp, SICK classification) worst is the accuracy of the best constant answer, the majority label of the ground truth, and for DuoRC it is the accuracy of answering "no answer" to every question. A constant answer earns 0.

Predictions must be finite: a submission with NaN or infinite predictions is not graded and can be fixed and resubmitted. The raw metric, the reward, metric and lower_is_better are returned in the metadata.

MRR (CodeXGLUE) scores the first 100 ids of each query's rankings list: a correct id ranked lower counts as not found (reciprocal rank 0), so ranking the whole 19,210-item corpus adds nothing. The prompt states this.

A submission.csv larger than 64 MiB is not graded: submit checks the file's size before reading it and reports the size and the limit, and the agent can shrink the file and submit again. The largest legitimate submissions, MRR at 100 ids per query and Pass@5, are typically about 14 MB.

Pass@5 (APPS) runs the submitted programs against the hidden test cases in a separate, network-blocked grading sandbox that mounts the task's ground truth; the agent's sandbox never holds the hidden tests. Each program may allocate up to 1 GiB; past that an allocation fails inside the program, which fails that test case.

Data

• Source: 16 HuggingFace datasets
• Format: HuggingFace datasets format, mounted at /home/ubuntu/data/{train,test}/
• Test labels: Stripped from agent-visible data; held server-side for evaluation

Tools

Time Horizon

Multi-turn. Agents typically need 20-100+ tool calls to explore data, write code, train models, and produce predictions.

Environment Difficulty

Varies by task. NLP tasks with pre-trained models are easier; molecular property prediction and time series forecasting are harder. Each task's reference SOTA score, from its AIRS-Bench task metadata, is sota_score in task_config.py; the reward does not use it.

Other Environment Requirements

• OpenReward API key (for sandbox access)
• No other external API keys required

Safety

Sandboxed execution environment. Its network reaches PyPI, Hugging Face models, evaluate's metric scripts, PyG's wheel index and nltk_data only; everything else, including Hub datasets, other Spaces, PyG's dataset mirror and the rest of GitHub, gets HTTP 403 from the egress proxy.

Citations

@article{lupidi2026airsbenchsuitetasksfrontier,
      title={AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents},
      author={Alisia Lupidi and Bhavul Gauri and Thomas Simon Foster and Bassel Al Omari and Despoina Magka and Alberto Pepe and Alexis Audran-Reiss and Muna Aghamelu and Nicolas Baldwin and Lucia Cipolina-Kun and Jean-Christophe Gagnon-Audet and Chee Hau Leow and Sandra Lefdal and Hossam Mossalam and Abhinav Moudgil and Saba Nazir and Emanuel Tewolde and Isabel Urrego and Jordi Armengol Estape and Amar Budhiraja and Gaurav Chaurasia and Abhishek Charnalia and Derek Dunfield and Karen Hambardzumyan and Daniel Izcovich and Martin Josifoski and Ishita Mediratta and Kelvin Niu and Parth Pathak and Michael Shvartsman and Edan Toledo and Anton Protopopov and Roberta Raileanu and Alexander Miller and Tatiana Shavrina and Jakob Foerster and Yoram Bachrach},
      year={2026},
      eprint={2602.06855},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.06855},
}