ProgramGym

Description

ProgramGym is a long-horizon program-reproduction environment. Each task gives the agent a single compiled reference binary and the project's bundled documentation, and asks it to write an original codebase from scratch — plus a compile.sh that builds a new ./executable with behaviour identical to the reference. The agent may only learn what the program does by running the binary and reading its docs; copying, wrapping, decompiling or downloading the original source is disallowed. It is a port of the HuggingFace dataset ProgramGym/programgym-1648 (Yang et al., 2026) to the OpenReward sandbox contract.

Capabilities

• Reverse-engineering a program's behaviour from a black-box binary
• Writing an original implementation in C, C++, Go or Rust
• Building a reproducible, dependency-installing compile.sh
• Long-horizon, multi-turn development against a hidden behavioural test suite

Compute Requirements

Each task runs in two sandboxes sized at 8 CPUs / 16 GB RAM: an agent sandbox the model works in, and a fresh verifier sandbox created at grade time. Both are offline (no network egress).

License

MIT.

Tasks

This port preserves the 1,624 task directories in the pinned source snapshot. The upstream dataset advertises 1,648 tasks and has since changed its directory layout. The preserved tasks span Go, C++, Rust and C, drawn from real open-source repositories, with difficulty labels from "easy" to "hard". Every task shares one instruction: reproduce the reference executable's behaviour.

This is a work-in-progress release: packaging and platform verification are separate. Every packaged task has passed the local reference gate, but platform verification and further repairs remain in progress. A verified reference must score 1.0 twice through the same two-sandbox grading path used for submissions. Task IDs are retained when a platform incompatibility or a defective fixture is found, with the failure kept in the repair queue:

• Runtime repair (<!--LOADER_REPAIRED-->64<!--/--> tasks). Some upstream references are dynamically linked against a library the image has but not on the loader path — for example one the task's download step staged under /tmp, or a newer libstdc++ than the system copy. The library is located in the image and registered with the loader in both sandboxes.
• Source repair (<!--SOURCE_REPAIRED-->59<!--/--> tasks). Some references depend on the project's own shared library, or on a helper binary, that upstream never packaged, so they cannot run at all. The missing piece is rebuilt from the task's pinned upstream commit; where possible the original executable is kept and pointed at the rebuilt library.
• Calibration (<!--CALIBRATED_TOTAL-->489<!--/--> tasks). Tests the reference itself fails — for instance ones that need an external network, a display, or the original binary's exact ELF layout — are marked ignored through the dataset's own ignored_tests mechanism, so the maximum reward stays attainable. A calibrated suite must still fail a behaviour-free stub submission.

Reward Structure

Reward is the fraction of the active hidden pytest suite that the agent's rebuilt executable passes — a dense, continuous signal in [0, 1]. Suite size varies by task; tests exercise flags, stdin/stdout, files, exit codes and local servers. The environment uses the dataset's verifier (programgym_evaluator.py / test.sh) with the task's reviewed fixtures and calibration overlay. A rollout ends when the agent replies with an ordinary message (no tool call); its text is not graded — grading comes entirely from the state of its workspace. A gpt-5-mini judge reviews each scoring submission for disallowed shortcuts and can zero its reward (see Safety).

Data

Every task's reference binary, documentation and hidden test suite live inside two pre-built per-task Docker images (generalreasoning/programgym:<id>-agent and <id>-verifier), pulled by the platform at runtime and pinned by digest. The images are built from the upstream task Dockerfiles; the repository itself carries only code plus the task manifest (data/tasks.jsonl) and the split (splits.json). The hidden tests live only in the verifier image and the verifier sandbox, which is created after the agent has finished; the agent's own development sandbox never contains them.

Tools

The agent gets the standard sandboxed CLI toolset — bash, read, write, edit, multi_edit, ls, glob, grep, todo_write — rooted at its /workspace. All of them run as an unprivileged user, so the agent can run the reference binary but cannot modify it or the toolchains. There is no submit tool: the agent ends the task with a plain message.

Time Horizon

ProgramGym is an open-ended, long-horizon environment: the agent explores a binary, writes a codebase and iterates on its compile.sh over many turns within one episode.

Environment Difficulty

Difficulty varies widely with the reference program, from small CLI utilities to large libraries with thousands of tests. The source dataset labels most tasks "easy", with a minority "medium"/"hard". Because reward is a test pass-rate, partial progress is rewarded continuously rather than all-or-nothing.

Other Environment Requirements

The submission judge calls gpt-5-mini, so sessions need an OpenAI API key in the session secrets (openai_api_key). Setting PROGRAMGYM_TRAJECTORY_JUDGE=0 on the environment server turns the judge off, and then no key is needed.

Safety

The agent only reimplements open-source command-line programs in an offline sandbox; there is no direct safety risk. The task rules forbid copying, wrapping, decompiling or downloading the original. The reference binary has to be readable — the platform's sandbox runtime cannot execute a file the caller cannot read — so the rules are enforced by structure plus detection. Both sandboxes are offline, so the original source cannot be fetched, and the reference binary is not carried into the verifier. A submission whose compiled executable is byte-identical to the reference scores zero, as does one that carries an exact copy of a repaired task's rebuilt library. A gpt-5-mini judge (on by default) reads each scoring submission's source, plus an inventory of every other file it ships, and zeroes the reward for reading the hidden tests, hardcoding their outputs, wrapping, copying or embedding the original binary, or downloading its source. The judge fails open — an error never zeroes a reward.

Citations

This environment ports the ProgramGym dataset:

@misc{yang2026programgym,
  title         = {ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents},
  author        = {Chengyi Yang and Renxu Bao and Jue Hou and Zongpei Teng and Ziheng Wang and Yujie Lin and Chengsong Huang and Junnan Zhu and Delai Qiu and Yining Wang and Shengping Liu and Fei Long and Jinsong Su},
  year          = {2026},
  eprint        = {2610.13395},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2610.13395}
}

@dataset{GRProgramGym,
  author    = {General Reasoning Inc. Team},
  title     = {ProgramGym},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/ProgramGym}
}