BackfillGym

Long-horizon "re-implement a removed subsystem" tasks mined automatically from real
Python repositories, graded by the repo's own test suite with the create-swe-env
telescoping reward. The RepoLaunch idea (auto-build an executable environment per repo)
applied to subsystem backfill instead of issue↔PR pairs.

A task

tasks/<instance_id>/ — one per (repo, commit, target test file):

The agent works in /testbed (editable install, conda env testbed). Stubs keep
signatures, decorators, class attributes and (LLM-paraphrased) docstrings; bodies are
raise NotImplementedError.

Reward

The target test file on disk is the visible half of the target tests (the agent's spec,
runnable freely, unscored). The other half is hidden: it lives only in the task dir on the
server and never enters the agent's sandbox. run_tests starts a fresh grader sandbox from the
same image, copies in the agent's changed, added and deleted files under /testbed (never graded
test files, conftest.py, pytest config, .pth or sitecustomize/usercustomize, and nothing outside
/testbed), writes the full target test files there, runs hidden + visible + P2P with
--junitxml (xunit1, exact node ids), discards the grader, and pays
Δ = scored_passes_now − scored_passes_previous over the hidden ∪ P2P set (+1 newly passing,
−1 regressed). The baseline is seeded with the P2P set (they pass on the stubbed tree by
construction), so Σ rewards = hidden target tests passing at the end − net regressions; a first
call on the untouched tree scores 0. Visible passes are reported but pay nothing, so building to
the visible tests earns nothing unless it generalises (the reason for this design: with visible
tests scored, a special-cased solution earned the same return as a correct one and a trainer
truncating at max_turns never saw the finish gate). Failing hidden tests are reported by function name only (the
prompt lists every hidden test name up front), never by id or body.

The agent's code runs next to the hidden tests in the grader, so nothing from the grader comes back
but per-test status: the grader's pytest log is never shown. A collection error is explained from the
same run against the visible tests in the agent's own sandbox; a timeout shows no log. Both score 0
and leave the baseline unchanged, as does a change set over 8 MB compressed (not graded). The next
grader starts in the background as soon as a grade ends, so a grade does not wait for a sandbox.

Editing graded tests, conftest.py or pytest config → BLOCKED (sha256 baseline from setup()).
Adding a conftest.py or pytest config anywhere in the repo, sitecustomize/usercustomize, a .pth
file or a pytest11 plugin → BLOCKED (inventory baseline from setup()). The grader would not
copy any of them, so these blocks are feedback that such a change cannot count. Grading runs without
pytest's cache or PYTEST_ADDOPTS/PYTEST_PLUGINS. When every hidden + P2P test passes, the
no-op audit re-stubs the subsystem files in the same grader and reruns the visible target tests;
if they still pass the "solution" lives outside the subsystem → flagged, not finished. Otherwise
finished.

Pipeline (in ../../backfill/)

1. vendor/FeatureBench fb data with backfill/specs/spike.py (repo specs; glm-5.2 via
   vendor/FeatureBench/config.toml) → traced, LLM-selected, masked, validated level-1 cases.
2. backfill/make_task.py <run> <SPECS_NAME> <case> <repo_dir> envs/BackfillGym/tasks
   → stubs (backfill/stubber.py), exact F2P/P2P sets re-derived in the instance image,
   50% hidden/visible split at test-function level, leak fingerprints, task dir.
3. Task sandbox images are built by CI: .github/workflows/build-task-images.yml runs
   build/build_task_image.py tasks/<id> --push per task (matrix), pushing
   ghcr.io/generalreasoning/backfillgym:<id>-<content sha> (linux/amd64, GitHub's GITHUB_TOKEN).
   The env server computes the identical tag from the task dir (tasklib.image_tag), so nothing is
   committed back. Each task dir is self-contained (image/ holds the recorded base/instance
   Dockerfiles + setup script; the repo is fetched from GitHub at base_commit). Build one locally:
   python3 build/build_task_image.py tasks/<id> (add --platform linux/arm64 on a Mac for speed).

Test

cd /Users/anthonyhartshorn/gr/envs && BACKFILL_TASKS_DIR=swe_lib_env_creator/envs/BackfillGym/tasks \
  uv run python swe_lib_env_creator/envs/BackfillGym/golden_tests.py     # fake sandbox, reward contract
# real env code against a LOCAL docker image (no platform): stub → gold → finished, tamper, syntax error
BACKFILL_TASKS_DIR=swe_lib_env_creator/envs/BackfillGym/tasks uv run python \
  swe_lib_env_creator/envs/BackfillGym/test_local.py <task_id> <local_image> <repo_dir>
uv run python swe_lib_env_creator/envs/BackfillGym/test_live.py <task_id> <repo_dir>  # same, through the platform sandbox

Known limitations

• Code under test can still read the hidden tests inside the grader, and the per-test results
  that come back are a narrow channel: code that read them could choose which hidden tests fail, and
  the failing names are reported. Nothing it writes in the grader survives, and the telescoping reward
  pays nothing for passing a test it first failed on purpose; closing the channel fully means reporting
  counts only.
• A forgery inside /testbed that detects the no-op audit would pass (e.g. the package patching
  pytest's reporting when imported). Startup hooks, installed packages and the interpreter are outside
  the copied tree, so only in-repo code can try this.
• FeatureBench cannot remove a library's central core (anything a regression test touches is
  protected), so tasks skew to peripheral subsystems; a subsystem-first selector is future work.
• Only Python / pytest for now.