OpenSWE

Description

OpenSWE is a software-engineering environment built on GAIR's OpenSWE dataset (Fu et al., 2026), a SWE-bench-style corpus of executable bug-fixing tasks synthesised from real GitHub pull requests across 12.8k Python repositories. The agent is dropped into a repository checked out at a pre-fix commit, given the PR/issue description, and must edit the source code so that the instance's evaluation tests pass.

This environment exposes GAIR's difficulty-filtered subset (filtered_ids.csv): 8,876 quality-curated instances with unsolvable and trivial tasks removed, which is the subset behind the paper's headline results.

Capabilities

• Repository-level bug fixing and feature implementation from PR descriptions
• Multi-step code exploration, editing, and test-driven iteration
• Real-world Python codebases spanning 12.8k repositories and Python 2.7-3.14

Compute Requirements

Each task runs in a sandbox with 2 CPUs and 4GB of RAM. Because GAIR publishes per-instance Dockerfiles rather than prebuilt images, the sandbox is provisioned at session start: the repository is cloned at the task's base commit and the instance Dockerfile's ENV/RUN steps are replayed inside a shared base image (ghcr.io/generalreasoning/openswe-python:<version>, one per Python version, rebuilt from GAIR's recipe). Session setup therefore takes a few minutes depending on the instance's dependencies. The sandbox has network access, which setup requires.

License

The upstream dataset is released under GAIR's mixed license (permissive licenses for redistributable repositories plus CC-BY-4.0; see the dataset LICENSE). Each task record carries its source repository's license in the license field.

Tasks

There are two training splits:

• filtered: 5,670 instances — GAIR's 8,876 quality-curated instances minus 3,206 excluded after full-corpus verification (see below).
• lite: a fixed 29-instance subset for quick evaluation and testing, drawn from a seed-42 random sample of 50 and reduced to the instances that passed per-instance verification.

data/bad_instances.txt records the excluded instances and data/verification.json is the per-instance ledger of every check performed. A GitHub existence scan of all instances found 49 whose source repository or base commit no longer exists (guaranteed setup failure). Execution sweeps of two 50-instance random samples (the lite sample and a second seed-43 sample) found three further failure classes, each confirmed by at least two independent runs: every instance in the corpus was executed in OpenReward sandboxes and verified in both directions: an empty patch must score 0 and, where a gold patch exists, applying it must score 1. Any check that failed was immediately re-run and only counted when it failed twice. Exclusions: repositories or commits that no longer exist (49), deterministic setup failures (~1,060), instances whose evaluation already passes at the base commit so an empty patch earns full reward (~990), instances whose gold patch does not make the evaluation pass (~1,100), and a handful of unclassifiable errors. The full per-instance ledger is in data/verification.json. One systematic failure mode was recovered rather than excluded: the sandbox network layer breaks TLS verification for direct wget/curl downloads, so the generated setup script disables certificate checks for those tools inside the disposable sandbox. Of the surviving instances, 4,393 are verified in both directions; 1,277 have no gold patch in the dataset and are verified on the empty-patch direction only. Instances outside the executed samples (100 of 8,876) may exhibit the execution-class issues at similar rates; running the same golden/xfail sweep over the full split is the recommended hardening step before large training runs.

Task data is trimmed from the upstream dataset to the fields the environment needs (problem statement, Dockerfile, evaluation script, patches, commit metadata) by prepare_data.py. The upstream dataset is gated on HuggingFace, so data preparation requires an approved HF token; the environment itself ships with the prepared data and needs no HuggingFace access at runtime.

7,151 instances come from repositories whose licenses allow redistribution (openswe_oss.jsonl) and include the reference gold patch; the remaining 1,725 (openswe_other.jsonl) ship without a gold patch upstream, which does not affect task solvability or grading — only gold-patch-based verification.

Reward Structure

Binary, fully verifiable, and faithful to GAIR's official grading (from their SWE-bench fork): when the agent calls answer, the environment

1. captures the agent's patch (git add -A && git diff --cached <base_commit> --binary),
2. restores a pristine tree at the base commit and re-applies the patch (fresh-container semantics),
3. reverts any agent edits to files touched by the instance's test patch (agents cannot tamper with tests),
4. runs the instance's self-contained eval_script, which applies the test patch and runs the tests.

Reward is 1.0 iff the eval log contains OPENSWE_EXIT_CODE=0, else 0.0. No LLM graders are used.

Tools

• bash: run a command in the repository sandbox (default working directory /testbed).
• view, str_replace, insert, create: file viewing and editing tools.
• answer: submit the work and receive the final reward (ends the session).

Other Environment Requirements

No external API keys are required. Sandboxes need network access at setup time to clone the task repository and install dependencies.

Because instance environments are built at session start from unpinned upstream package indexes, a small fraction of instances may fail setup as the Python packaging ecosystem drifts from the dataset's construction date; such sessions raise an error at setup rather than returning a spurious reward.

Safety

Agents operate on real open-source codebases inside isolated, disposable sandboxes with network access. The tasks themselves are ordinary bug fixes with verifiable outcomes and present no direct safety risks. As with any networked coding sandbox, agents could in principle fetch information about the original fix from the internet; the reward remains verifiable regardless.

Citation

@misc{fu2026davincienv,
      title={daVinci-Env: Open SWE Environment Synthesis at Scale},
      author={Dayuan Fu and Shenyu Wu and Yunze Wu and Zerui Peng and Yaxing Huang and Jie Sun and Ji Zeng and Mohan Jiang and Lin Zhang and Yukun Li and Jiarui Hu and Liming Liu and Jinlong Hou and Pengfei Liu},
      year={2026},
      eprint={2603.13023},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2603.13023},
}