SWE-rebench-V2

Description

SWE-rebench-V2 is an OpenReward port of the SWE-rebench V2 dataset by Badertdinov et al. (Nebius AI). It evaluates agents on real-world software engineering tasks across multiple programming languages. Agents are given a repository checked out to a specific commit and a problem statement, and must modify the source code so that previously-failing tests pass without breaking existing tests. The dataset covers 32K+ instances across Python, JavaScript, Go, Rust, Java, Ruby, and many more languages.

Community

You can reach out with any questions in Discord: https://discord.gg/V8FqXQ4CgU

Capabilities

• Multi-language codebase navigation and understanding
• Bug diagnosis from problem statements and test failures
• Source code editing to fix defects
• Reasoning about test expectations and code behavior

Compute Requirements

Agents are given a sandboxed Docker environment with a pre-built instance image for each task. Default sandbox size is 1 CPU and 2 GB RAM.

License

MIT. The underlying SWE-rebench V2 dataset is subject to its own license terms.

Tasks

There is one split in this environment:

• Train: 18,403 software engineering tasks, 2,025 of them Go

The split serves the dataset rows whose quality label (meta.llm_metadata.code) is A (solvable: the problem is clearly specified and the tests match it), in dataset order. Rows with any other label are left out: B1-B6 flag tests coupled to unrelated code, names or strings the problem statement does not give, external or implicit context, or unrelated patch changes; combined labels such as A|B5 and unlabelled rows are left out too. The sandbox has no network, so tasks whose build or test steps download dependencies are left out as well. Rust tasks (language is rust), and tasks in other languages whose test command runs go test or cargo, all fetch crates or modules and are dropped. A Go task is served only if it is listed in go_offline_tasks.json: its reference fix passed the held-out tests twice in the task image with the network blocked and the go command restricted to the image's module cache or the repository's vendor/ directory (GOPROXY=off, GOSUMDB=off, GOTOOLCHAIN=local, and GOFLAGS=-mod=vendor for vendored repositories), finishing within 400 seconds both times, and the image's module cache holds no copy of the task's own repository. The other Go tasks need modules or a toolchain the image does not hold.

The sandbox of a listed Go task is not cut off from the network but filtered: it reaches only the Go module proxy, its checksum database and the proxy's download storage (the platform's go-modules preset), with GOPROXY=https://proxy.golang.org and GOTOOLCHAIN=local (and GOFLAGS=-mod=vendor for vendored repositories). The proxy serves every version of every public module, so the policy denies, on every proxy route, the task's own code: each module path its go.mod files declare at the base commit and on the default branch, the repository's GitHub path (and its current one if renamed), forks with the same repository name, and a few known successor projects. go_offline_tasks.json holds those module paths; fetch_go_modules.py regenerates them. Every other host, including github.com, is refused. The held-out tests still run offline (GOPROXY=off, no egress proxy), as when the tasks were admitted.

Each task provides a repository, base commit, problem statement, the new or changed interfaces the held-out tests call (the dataset's interface field), and a set of tests that should transition from failing to passing after the agent's fix. Tasks span issue-based and PR-based scenarios across dozens of programming languages and frameworks.

Reward Structure

This is a multi-turn environment with binary reward:

• 1.0 — All FAIL_TO_PASS tests now pass and all PASS_TO_PASS tests remain passing
• 0.0 — Any required test fails or regresses

On submission, the environment applies the held-out test patch, runs the task's test command, and parses the output using a language/framework-specific log parser to determine per-test pass/fail status. If the test command does not finish within 600 seconds, nothing is graded: the held-out tests are removed again and the agent can submit again. Other tool calls wait while a submission is being graded.

Data

Data is loaded from parquet files uploaded to the environment's data directory. Each row contains the instance ID, repository, base commit, test patch, problem statement, Docker image name, language, test expectations (FAIL_TO_PASS and PASS_TO_PASS lists), and install/test configuration. The dataset is derived from the SWE-rebench V2 collection on HuggingFace (nebius/SWE-rebench-V2).

Tools

Time Horizon

SWE-rebench-V2 is a multi-turn environment. Agents explore the repository, read code, diagnose the issue, make edits, and optionally run tests before submitting. A typical task may involve 10-50+ tool calls depending on complexity.

Environment Difficulty

SWE-rebench V2 is a challenging benchmark spanning many languages and difficulty levels. Tasks are annotated with difficulty codes. Performance varies significantly by language, framework, and problem complexity. As of the paper's publication, frontier models solve a modest fraction of tasks, with Python tasks being the most commonly attempted.

Safety

Agents operate within sandboxed Docker containers with no network access to external services, except that Go tasks can reach the Go module proxy, minus the task's own code. The environment does not involve private data or production systems. Agents can only modify files within the repository checkout; the test patch is applied automatically at submission time and cannot be tampered with.

Citations

@misc{badertdinov2026swerebenchv2languageagnosticswe,
      title={SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale},
      author={Ibragim Badertdinov and Maksim Nekrashevich and Anton Shevtsov and Alexander Golubev},
      year={2026},
      eprint={2602.23866},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2602.23866},
}