env-lodestar-harbor

938 execution-validated, real-world Issue -> PR SWE tasks in Harbor format, for an
OpenReward harbor environment (isHarbor, subdirectory = tasks).

Produced by Lodestar (~/lodestar, quality-curation layer for SWE RL envs): mined GitHub
issue->PR pairs -> per-repo environment build (cached image) -> gold-patch differential
(test_patch only = before, test_patch + gold = after) -> FAIL_TO_PASS / PASS_TO_PASS.
Grading reuses RepoLaunchGym's verifier (grade.py / _gradelib.py, verbatim).

Contents

train holds the tasks with measured training signal for rl-r3-step170: step170 fails them at
least once and a model solves them at least once. pool holds the rest. Splits are set from
data/solvability.csv by tools/set_split.py. Re-run it after regenerating tasks/, because
harborize_validated.py writes split = "train" for every task.

Task layout (tasks/<instance_id>/)

No task sets [environment].docker_image: OpenReward builds each environment/Dockerfile with
Cloud Build into openreward/sandbox-images/<envId>:<task> (the only *.pkg.dev path the
sandbox proxy accepts).

Git history

The agent sees only the base commit. Each Dockerfile writes .git/shallow = HEAD before
git gc --prune=now, so ancestors are dropped along with later commits, and the build fails if any
history survives (git rev-list --count HEAD must be 1 and the parent commit object must be gone).

Ancestors used to be kept, and that leaked one task. In sooperset__mcp-atlassian-1076, upstream
landed the fix (#1072), reverted it, and re-landed it in the task's PR, so the gold fix sat two
commits below base with the gold patch's exact post-image blob. RL rollouts found it with
git log/git show. A scan of all 938 tasks found no other case: it checked revert commits in the
100 commits below each base for a reverted commit that wrote a gold post-image blob. 46 near-misses
touched a gold file, but with at most 8% of its added lines (changelogs, setup.py). Cutting history
for every task also covers leaks that scan can't see, at the cost of git log/git blame for the
agent. The solvability measurements below were taken with history present.

Grading

Binary reward (SWE-bench resolved). grade.py injects the gold tests, rebuilds, runs
test_cmds, parses with the task's parser, and requires every FAIL_TO_PASS to pass and no
PASS_TO_PASS to break. Failures are re-run twice (flaky filter), then checked against a fresh
gold run: a test gold also fails today counts as rotted and is ignored. A task where gold
passes no FAIL_TO_PASS is ungradable (reward 0.0, gradable:false in reward.json).

test_cmds / log_parser are exactly what produced the labels (tools/build_validate.py:
targeted_cmd = only the test_patch's files, parser_for = per-language parser, embedded as a
def parser(log) script so _gradelib does not fall back to the pytest parser).

tests/test.sh pins the image PATH into /etc/profile.d: _gradelib runs commands with
bash -lc, and Debian's /etc/profile resets root's PATH, which drops /usr/local/go/bin and
/usr/local/cargo/bin, so every Go and Rust task would otherwise grade 0.

Specification check (tools/hidden_interface.py)

Some tasks can only be passed by guessing a name. The hidden tests call something the gold fix
introduces (a function, class, method or CLI command), and the issue never names it, so a correct
fix with a different name fails. A test file that imports the missing name fails every
FAIL_TO_PASS test in it. Example: PyThaiNLP-1058's tests import
pythainlp.tokenize.display_cell_tokenize, while the issue only says "Thai character splitter
to display cell".

The narrow check only covers names the fix defines: def/class/func/method/CLI command, used by
the tests, absent from instruction.md. It misses parameter and field names. --broad adds every
new identifier the tests use, which catches mcp-atlassian-252's content_type, but it is noisy:
it flags a quarter of easy tasks that were solved. Names that already existed before the fix are
never flagged; the agent can see them. The fix for a flagged task is to state the interface in the
issue text, not to drop the task.

Measured solvability (rl-r3-step170 vs GLM-5.3-Flash, 16 Oct 2026)

Both models ran in the agentorg harness at budget 0, with the per-task network policy enforced.
step170 ran every task 3 times. GLM-5.3-Flash ran only the tasks step170 never solved, because only
there does a GLM pass add information. It covered all of those in the first 200-task batch and the
easy, no-hidden-interface ones after that. Failures count only if graded, and passes only if the
agent did not fetch its own project upstream. 158 trials that ran the wrong task during a
mid-run task-list change are excluded (data/excluded_sessions.txt). Per task:
data/solvability.csv (tools/solvability_export.py); this section is generated by
tools/solvability_summary.py.

GLM's per-task coverage differs by group, so compare GLM-certified counts within a difficulty
bucket, not across buckets. glm_not_run tasks (medium/hard, or with a hidden interface, outside
the first batch) may still hold a few candidates.

Oracle sweep (every task: gold must score 1, empty submission must score 0)

All tasks went through tools/oracle_sweep/ (build the Dockerfile, run solve.sh or nothing,
bash /tests/test.sh, the same flow OpenReward runs). Every task in tasks/ passed it.
Dropped 51 of the original 981: the empty patch already scored 1 on 31 (no signal), gold did not
score 1 on the rest. The cause was a validation bug: F2P was "passes after gold − passes before",
so tests that merely didn't run before (crashed or cold run) became F2P. Fixed in
tools/build_validate.py (a warm recheck run without gold; F2P must fail there). 9 tasks were
re-validated with the fix, re-harborized and passed the sweep; they replace their old versions
(new F2P lists). Grade time on passing tasks: p50 20 s, p90 88 s, p99 474 s.

Smoke test (oracle must be 1, empty submission must be 0)

Emulated the OpenReward flow on a dev VM (build Dockerfile, run solve.sh or nothing, upload tests/,
bash /tests/test.sh). Oracle=1 / empty=0 on python x3, go x2, rust x1, ts x1 (subliminal-1201, marimo-7770, haystack-10421, casbin-1632, templ-484, maturin-2165, virtual-967). Not yet run on OpenReward itself.

Known caveats

1. 33% of tasks have empty PASS_TO_PASS. Validation ran only the test_patch's own files, so
   regression guarding is weak on those tasks; they grade on FAIL_TO_PASS only.
   (~/lodestar/scripts/backfill_p2p.py widens scope to the test directory; not applied here.)
2. Validation ran once per task (no N-run flakiness gate at label time). The grader's
   flaky re-runs and gold recheck absorb some of this, not all of it.
3. JS/TS labels are per test file, e.g. foo.test.ts (3 tests) (vitest summary lines),
   not per test name. The raw labels carried run durations (... 28ms) in 95 tasks; these are
   stripped from both the labels and the grade-time parse.
4. Large label sets. Some tasks have thousands of F2P, where the before-run failed wholesale
   (for example a new test file that cannot import before the fix). These are valid
   differentials, but they behave as "make the suite pass".
5. Ruby/Java: 8 of 20 kept. The original 20 (19 ruby, 1 java) were dropped before the sweep. Their labels came from parse_generic,
   which matches "ok"/"PASS" anywhere in the output, so their "tests" were strings such as
   1.81 seconds to load) from rspec's timing line. They could not be graded. Re-validated with rspec/minitest/surefire parsers and the fixed F2P
   rule: 8 validated, 7 pass the oracle sweep (the Java one fails: its image has no python3 for
   the grader). Per-repo setup recipes (REPO_RECIPES in tools/build_validate.py: Postgres for
   theodinproject, generated test app for activeadmin, gemfile shim for sorcery) recovered
   theodinproject-4507. The rest stay out: gumroad (MySQL/Mongo/Redis stack), sorcery-371/379
   (gold removes a test / only edits CHANGELOG), activeadmin-8411 (F2P needs the test app
   regenerated after gold), view_component-2015 and theodinproject-4349 (new test file only
   loads with gold, so there is no clean failing "before").
6. Base images live in us-central1-docker.pkg.dev/dev-ran-gr-inc/lodestar-envs. That
   repo's IAM policy is empty. OpenReward's Cloud Build identity needs read access before the
   Dockerfiles can build.
7. JS/TS test commands use npx --yes jest|vitest, which may download at grade time, so the
   sandbox needs npm registry egress.

Regenerating

(tools/ holds snapshots of the generating scripts. The canonical copies are in ~/lodestar/scripts.)