SWE-Bench-Pro

OpenReward port of ScaleAI's SWE-bench Pro. The agent is given a repository and an
issue, works in a sandbox, and calls answer to be graded: the env captures its
work as a patch, re-applies it on a pristine base_commit, restores the gold test
files, runs the hidden tests, and returns reward 1.0 iff every FAIL_TO_PASS and
PASS_TO_PASS test passes.

Upstream

Pinned to ca10a60
(2026-05-18) of scaleapi/SWE-bench_Pro-os. Both halves of the port pin to that
one commit:

• Grader logic — upstream_port.py, byte-identical to upstream apart from a
  single recorded two-line delta.
• Task data — 731 instances x 4 files (the two Dockerfiles, the run script,
  the parser), fetched by vendor_upstream.sh. Not committed: it is served from
  the platform's /orwd_data mount in production and ./upstream locally, and
  verified on every read against the committed upstream_manifest.txt. See
  UPSTREAM.txt and DATA_UPLOAD.md.

Task text and test lists come from the HuggingFace dataset
ScaleAI/SWE-bench_Pro
(731 rows, the public split). That dataset does not carry the four grading
artifacts above, which is why they are vendored from the git repo instead.

Everything specific to this port lives outside the vendored boundary — server.py
and this file. Keeping that boundary clean is deliberate: an upstream fix arrives
as a re-vendor rather than a merge.

Deliberate divergences

Three, all in server.py, none in the vendored code:

1. Patch capture uses git add -A + git diff --cached, mirroring SWE-agent's
   tools/submit/bin/submit — which is what produces the model_patch upstream
   actually grades. A plain git diff omits untracked files, so a task solved by
   adding a new module produced a patch without it.
2. Binary hunks are stripped from the captured patch before it is applied, via
   upstream's own strip_binary_hunks. Upstream applies it in its patch-assembly
   step; this port had no equivalent, so one touched binary asset failed the whole
   git apply.
3. A grader that cannot grade raises instead of returning reward 0.0. Upstream
   scores an exception as False, which is right for an offline batch scorer whose
   warnings a human reads — but here a fabricated 0.0 is indistinguishable from a
   genuine agent failure and feeds training as one. This is a divergence on purpose;
   do not "restore parity" by reverting it.

The boundary is drawn so the agent can never cause a raise: a wrong fix, a broken
repo, a patch that will not apply and a test timeout all score 0.0 with a reason
in metadata. See the GradingError docstring.

What must not be "fixed"

create_entryscript uses only the last line of before_repo_set_cmd. That is
verbatim upstream and load-bearing. The full block is git reset --hard /
git clean -fd / git checkout <base> / git checkout <gold> -- <test files>,
and it runs after git apply. Running all four would reset and clean away the
agent's patch and then test a pristine tree, scoring every task 0. Only the final
line, which restores the gold test files, may run.
test_upstream_port.py locks this in.

Setup

pip install -r requirements.txt

Running the server

python server.py

Tests

Hermetic — no sandbox, no API key, no dataset:

pytest test_upstream_port.py
python check_upstream_drift.py            # our copy vs the pin
python check_upstream_drift.py --online   # the pin vs upstream (opt-in, never hard-fails)
python check_upstream_drift.py --data     # local task data vs the manifest
python check_upstream_drift.py --remote   # uploaded task data vs the manifest (needs a key)

A fresh clone needs ./vendor_upstream.sh before anything that touches task data.

Live, needs OPENREWARD_API_KEY and a sandbox; parametrized over all 731 tasks, so
select a subset:

pytest tests.py -k <instance_id>

Re-pinning to a newer upstream

See DATA_UPLOAD.md — re-vendoring now means re-uploading too, and the manifest,
the pinned hash and the upload all have to move together.