ReactBench

Description

ReactBench is an agentic React coding benchmark, ported from millionco/reactbench. Each task gives the agent a real React (TypeScript/JavaScript) repository checked out at a base commit plus a GitHub-issue-style instruction — either implement a feature from a merged pull request ("Write React") or refactor code to remove code-quality findings ("Fix React"). The agent edits the source under /app; on submit, a clean-room verifier restores a pristine tree over the edits and grades. Reward is binary: 1 if and only if the hidden behavioural tests pass and the change introduces no new React Doctor code-quality issues.

Capabilities

• Realistic React feature implementation and bug fixing in real open-source repositories
• Editing TypeScript/JavaScript, building with Vite, and reasoning about component behaviour
• Writing code that is not just correct but healthy — no broken effects, unnecessary renders, or accessibility/maintainability regressions
• Multi-file changes under a hidden acceptance test suite (Vitest + Testing Library, and Playwright/Chromium end-to-end where applicable)

Compute Requirements

Each task runs in a sandbox booted from its own public base image (ghcr.io/millionco/react-bench-*), which ships Node, pnpm, Playwright with Chromium, and the scaffold. Sizes come from each task's metadata (1–4 CPU, 2–4 GB). The task's clean-room verifier is baked into the sandbox at session start (network is used only during setup); grading then runs offline.

License

MIT.

Tasks

A single test split. This port ships a pilot of 2 tasks — hello-react (a smoke task: Vitest + React Doctor) and same-recursive-file-drop (a real repo exercising Vitest + Playwright/Chromium + React Doctor) — with the pipeline structured to scale to the upstream suite (~52 tasks across 50+ repositories) by vendoring more task directories. Each task_spec is just {"id": "<task-id>"}.

Reward Structure

Sparse, binary reward delivered once when the agent calls submit. The task's shared grader.mjs runs in a clean room — it restores pristine git metadata, node_modules, and protected configs over the agent's edits, re-applies the hidden tests from the baked verifier state, then evaluates two gates:

$R = \mathbb{1}[\text{tests pass}] \cdot \mathbb{1}[\text{no new react-doctor footguns vs. the baked baseline}] \in \{0, 1\}$

The test gate runs the task's config.test.cmd (e.g. pnpm exec vitest run && pnpm exec playwright test) and requires zero failures. The code-quality gate runs a pinned react-doctor scan of the task's scan_root and fails if any new "footgun" diagnostic (rule × file) appears relative to the baked base-commit baseline. Both tests and react_doctor sub-results are returned in the tool metadata.

Data

Task assets (Harbor format: task.toml, instruction.md, environment/, tests/, solution/) are vendored from the upstream repository. The verifier state (pristine node_modules, base git, the pinned React Doctor baseline, and the hidden test harness) is baked into the sandbox at setup by replaying the task's tests/Dockerfile on its public base image — no custom images are built. React Doctor is the open-source react-doctor npm package (pinned to 0.7.8, oxlint-based). The hidden tests and reference solutions are never exposed to the agent. Upstream files carry a reactbench-canary header — this is an evaluation environment and its task data must be kept out of training corpora.

Tools

• bash — run a shell command (cwd defaults to /app)
• view — view a file, optionally a line range
• str_replace / insert / create — edit and create files
• submit — grade the current tree in the clean-room verifier and end the episode

Time Horizon

Multi-turn agentic coding episodes — read the issue, explore and edit the repo, build/verify, then submit once. We have not yet measured the average number of tool calls per task on this environment.

Environment Difficulty

Deliberately harder than test-only React benchmarks: passing the behavioural tests is necessary but not sufficient — a solution that regresses code health (effects, renders, accessibility, maintainability) scores 0. Control checks per task verify the reference solution scores 1 and the unchanged repository scores 0. A full model sweep has not yet been run.

Other Environment Requirements

• api_key — an OpenReward API key for the sandbox.

Grading is deterministic and runs entirely inside the sandbox (hidden tests + the local react-doctor binary), so no external LLM or search API keys are required.

Safety

The agent edits code in an isolated, offline-graded sandbox and takes no real-world actions. Task repositories are third-party open-source projects used under their own licenses; this environment redistributes only the small task harness and instructions, and bakes the repositories from their public sources at setup.

Citations

@misc{reactbench2026,
  title        = {ReactBench: Evaluating Coding Agents on Realistic React Work},
  author       = {Million.co},
  year         = {2026},
  howpublished = {\url{https://github.com/millionco/reactbench}},
  note         = {OpenReward port by the General Reasoning Inc. Team; https://reactbench.com}
}