Terminal-Lego

Description

Terminal-Lego is an agentic terminal environment: the agent is dropped into a Linux sandbox with a real shell and must solve a self-contained command-line task — recovering data, writing a script, fixing a broken setup, transforming files, and so on. Tasks are drawn from the SWE-Lego/Terminal-Lego-15k dataset, a large collection of StackOverflow-derived, Docker-verified tasks in the Terminal-Bench 2.0 / Harbor format spanning many technical domains (Linux, Python, networking, databases, Docker, Git, and more).

Each task ships its own Docker image and a hidden pytest-based verifier. The agent works in the sandbox and, when finished, replies with an ordinary message (no tool call) — that signal ends the rollout and triggers grading against the task's test suite.

Capabilities

• Executing and iterating on shell commands in a real Linux sandbox
• Reading, editing, and creating files
• Solving self-contained terminal/SWE tasks across many technical domains
• Multi-turn tool use with verifier-based, outcome-graded rewards

Compute Requirements

Each task runs in a sandbox sized from the task's own task.toml (cpus / memory), snapped up to the nearest available machine size. Most Terminal-Lego tasks are small (1 CPU, 1–2 GB). The sandbox has network access, which the verifier requires (its test.sh installs uv at grade time) and which many tasks use while the agent works.

License

MIT. The upstream dataset is distributed by its authors under Apache-2.0.

Tasks

Tasks come from the upstream Terminal-Lego dataset, where each task is a Harbor directory: an instruction.md (the prompt), a task.toml (resources and timeouts), an environment/Dockerfile (the sandbox image), a tests/ suite (the hidden verifier), and a solution/solve.sh (the reference oracle).

This environment ships only tasks that pass a two-sided verification gate, run per task before it is included:

• Gold gate: running the reference solution/solve.sh and then the verifier yields a reward of exactly 1.0.
• No-op gate: running the verifier with no changes yields a reward below 1.0 — i.e. the task is not already-passing and the reward is earned by doing the work.

Tasks whose oracle does not solve them, or that are already passing before any work, are excluded. The exact set of shipped tasks lives in splits.json; all tasks are in a single train split.

Reward Structure

The reward is the value the task's own verifier writes to /logs/verifier/reward.txt (falling back to reward.json). For the pytest-based Terminal-Lego tasks this is 1.0 when the test suite passes and 0.0 otherwise. There is no LLM grader; scoring is fully programmatic and comes entirely from the state of the sandbox after the agent's work. The agent's final message text is not used for grading.

Data

Task instructions, task.toml, and the verifier tests/ are vendored into this environment directory. The per-task sandbox images are pre-built from each task's environment/Dockerfile and hosted on a container registry; each task's sha.txt records the image digest the sandbox pulls. Source tasks are from SWE-Lego/Terminal-Lego-15k on HuggingFace.

prepare_and_verify.py reproduces the pipeline: it downloads tasks from HuggingFace, builds each image, runs the gold and no-op gates, pushes the surviving images, and vendors the task files.

Tools

The agent is given CLI tools backed by the sandbox: bash (run a command), view (read files or list directories), str_replace (edit a unique string in a file), and create_file (write a new file). The grading tool is a hidden, no-argument terminal tool — the agent never sees it and instead ends the rollout by replying with an ordinary message.

Time Horizon

Terminal-Lego is a multi-turn environment. The agent iterates in the sandbox until it decides the task is done and ends with a plain message. Per-task agent and verifier timeouts are taken from each task's task.toml.

Other Environment Requirements

This environment works with the OpenReward endpoint and requires no external API keys or secrets.

Safety

Agents run inside an isolated, disposable per-task sandbox and interact only with that container's filesystem and the task's own resources. Tasks are derived from public StackOverflow content and standard command-line workflows. As always, sandboxes should retain their network and resource isolation when running untrusted agent code.

Citations

@misc{terminallego2026,
  title         = {SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving},
  author        = {SWE-Lego Team},
  year          = {2026},
  eprint        = {2601.01426},
  archivePrefix = {arXiv},
  url           = {https://huggingface.co/datasets/SWE-Lego/Terminal-Lego-15k}
}

@misc{merrill2026terminalbench,
  title         = {Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces},
  author        = {Merrill, Mike A. and Shaw, Alexander G. and others},
  year          = {2026},
  eprint        = {2601.11868},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2601.11868}
}