RE-Bench

Description

RE-Bench (Research Engineering Benchmark) is an ORS environment for evaluating language model agents on challenging, open-ended ML research engineering tasks. Based on the RE-Bench benchmark from METR, agents are given a research engineering problem and must iteratively develop, test, and optimize solutions using bash and a scoring tool. Tasks span diverse ML research skills including model debugging, constrained model design, API-based code generation, hyperparameter scaling law prediction, and GPU kernel optimization.

Capabilities

• Debugging and repairing pre-trained neural network models
• Designing ML models under strict architectural constraints
• Building LLM-powered code generation pipelines with budget constraints
• Running small-scale experiments to predict large-scale training outcomes
• Writing optimized GPU kernels using Triton
• Long-horizon multi-turn iteration with bash and scoring tools

Compute Requirements

Agents in RE-Bench are given a sandbox with an NVIDIA L4 GPU, with internet access enabled. Each task uses a dedicated Docker image with pre-installed dependencies (PyTorch, Triton, etc.) specific to that task.

License

MIT.

Tasks

There is 1 split with 5 tasks total, each corresponding to a distinct ML research engineering challenge:

The 5 tasks are:

Each task provides the agent with starter code, data files, a scoring script, and detailed instructions. The agent iterates freely using bash and calls the answer tool to evaluate its current solution at any time.

Reward Structure

This is a dense, verifiable reward environment with continuous scoring. The answer tool runs the task's scoring script and returns a reward after each call, while the bash tool returns a reward of 0.

Raw scores are normalized to a 0-1 scale using:

$\text{reward} = \frac{\text{raw\_score} - \text{starting\_score}}{\text{reference\_score} - \text{starting\_score}}$

This maps the baseline (no improvement) to 0 and the reference solution to 1. Agents can exceed 1 by outperforming the reference.

Task-specific score aggregation determines the final reward:

We do not use LLM graders for this environment.

Data

Each task includes pre-configured data within its Docker image and/or mounted from a read-only bucket:

• Fix Embedding: Corrupted model weights (large_model.pth), reference model (small_correct_model.pth), and OpenWebText training/validation data
• Restricted MLM: Baseline model weights (basic_model.pt) and OpenWebText training/validation data
• Rust CodeContests: Training and validation JSONL datasets of competitive programming problems; a held-out test set in a protected directory
• Scaling Law: OpenWebText training/validation data for running small-scale transformer training experiments
• Triton Cumsum: No external data; task is purely computational

Tools

Agents are given two tools:

• bash: Execute shell commands in the sandbox environment
• answer: Run the scoring script and receive a reward (can be called multiple times; get_final_score=True returns the best/final aggregated score)

Time Horizon

RE-Bench is a long-horizon, multi-turn environment. In the original paper, human experts were given 8 hours per task. There is no hard limit on the number of tool calls; the agent decides when to stop iterating and can call answer multiple times to check progress.

Environment Difficulty

Results from the original RE-Bench paper (normalized scores, 2-hour time limit for agents):

Human ML experts achieved a median normalized score of approximately 68% after 8 hours.

Other Environment Requirements

RE-Bench requires the following secrets to be passed via the session:

• openai_api_key -- Required for the ai_rd_rust_codecontests_inference task, which uses GPT-3.5-turbo to generate code solutions. Passed into the sandbox as an environment variable. Not needed if only running other tasks.

Safety

Agents in RE-Bench operate within isolated sandboxes with GPU access and internet connectivity. Each task uses a dedicated Docker image with pre-installed dependencies, and the sandbox is destroyed after the session ends. Held-out test data for the Rust CodeContests task is placed in a protected directory that the agent cannot read. The primary safety consideration is that agents execute arbitrary code with GPU access and network connectivity, which is contained by the sandbox.

Citations

@article{wijk2024rebench,
  title={RE-Bench: Evaluating Frontier {AI} {R\&D} Capabilities of Language Model Agents Against Human Experts},
  author={Wijk, Hjalmar and Lin, Tao and Becker, Joel and Jawhar, Sami and Parikh, Neev and Broadley, Thomas and Chan, Lawrence and Chen, Michael and Clymer, Josh and Dhyani, Jai and Ericheva, Elena and Garcia, Katharyn and Goodrich, Brian and Jurkovic, Nikola and Kinniment, Megan and Lajko, Aron and Nix, Seraphina and Sato, Lucas and Saunders, William and Taran, Maksym and West, Ben and Barnes, Elizabeth},
  journal={arXiv preprint arXiv:2411.15114},
  year={2024}
}