CCompiler

Description

CCompiler is a long-horizon coding environment that tasks an agent with building a C compiler from scratch, in Rust, with no help from existing compiler backends. Reward is the change in how many .c tests a single build of the agent's compiler handles correctly, relative to its previous build — valid programs must produce assembly that links and runs to the same exit code and stdout as a gcc reference; invalid_* programs must be rejected with a non-zero exit and no output file. finished=True only when one build passes every test in the corpus at once.

Capabilities

• Authoring a substantial Rust crate (lexer → parser → semantic analysis → IR → x86_64 codegen)
• Multi-file refactoring across an evolving codebase
• Reward-guided iteration against a 20-chapter curriculum of C features
• Git / worktree workflow when coordinating parallel subagents
• Open-ended long-horizon execution (thousands of tool calls)

Compute Requirements

Agents are given a sandbox with 8 GB of RAM and 4 CPUs. The network is blocked. Rust toolchain, bubblewrap, the system cc (used by the harness to link emitted asm), and the cc-runner harness binary are preinstalled.

License

ORLv1 for the environment code. The underlying test corpus is sourced from nlsandler/writing-a-c-compiler-tests (MIT) and is filtered + reference-graded at image build time. See citations.

Tasks

A single mega-task: ccompiler-train — pass every .c test in the filtered Sandler corpus. The corpus is baked into the image at build time. There is no train/test split.

Reward Structure

This is a dense, verifiable reward environment.

Reward is the change in how many tests a single build of the agent's
ccompiler binary passes, relative to its previous build:

• +1 for each test the current build passes that the previous build did not.
• −1 for each test the previous build passed that the current build now fails,
  so a build that breaks more than it fixes is net negative.
• Reject cases are credited per chapter in proportion to that chapter's valid pass
  rate: credited_invalid_N = min(invalid_passed_N, ceil(valid_rate_N × invalid_total_N)).
  Rejecting a program is trivial (exit 1 rejects all 692 invalid_* cases), so without
  this an always-reject binary would score 55% of the corpus. With it, an always-reject
  binary scores 0, a compiler that fully handles chapters 1..k scores every test in those
  chapters (the same set Sandler's --chapter k runner tests), and a build that passes
  everything is credited everything.
• Valid cases expecting exit 0 with no output are credited per chapter in proportion
  to that chapter's other valid cases: credited_trivial_N = min(trivial_passed_N, ceil(nontrivial_rate_N × trivial_total_N)). 284 of the 557 valid programs exit 0 and
  print nothing, so without this a compiler that emits return 0 for every input would
  score 284. With it, that binary scores 0, while a compiler that fully handles a
  chapter is credited all of it. The reject credit above uses the resulting valid
  count.
• Each chapter is scaled by a perturbed re-grade. Every build, up to 16 of the
  programs it passes in each chapter (valid and reject cases drawn separately, with a
  fresh os.urandom seed) are graded again after perturb.py renames their
  identifiers and re-flows their whitespace and comments. Valid programs from chapter 2
  on also get token-changing rewrites that use only constructs of their own chapter
  (return (e);, a constant re-expressed with the same type and value such as (3 + 4),
  an extra ; at the start of a block; see below). The expected result is unchanged,
  so a compiler passes a copy exactly when it passes the original, while a table of
  answers keyed on the public sources (by hash, path, or the token sequence with
  identifiers, whitespace and comments erased) passes only the originals. Per chapter
  and kind,
  credited = min(passed, ceil(min(1, still_passed/sampled + 1/16) × passed)), and the
  reject credit above uses the credited valid rate. The sample comes from the build's
  own passes, so a deterministic compiler is never capped (no sampling noise); the
  1/16 slack forgives one flaky copy per sample and is all a table keeps.
• 0 for a build that fails to compile, and 0 for a grading run that
  wall-times-out before the whole corpus has run — neither changes the baseline.
• finished=True is reported only when one build passes every test in the
  corpus at once.

Because reward tracks the current binary rather than a historical union of
everything ever passed, total trajectory reward telescopes to the credited pass
count of the build the agent finishes with. Every scored run grades the full corpus,
which is what makes that delta coherent; run_tests(paths=...) only narrows
which failures are listed back to the agent and cannot change the score.

Per-test pass/fail is decided mechanically, with no LLM grader:

• valid/*.c — the agent's compiler must emit asm that, once cc -no-pie-linked and run, matches the reference's exit code and stdout (the reference is gcc run against the same source at image-build time).
• invalid_*/*.c — the agent's compiler must exit non-zero AND produce no output file. The category name (invalid_lex, invalid_parse, invalid_semantics, invalid_declarations, invalid_types, invalid_struct_tags, invalid_labels) hints at the stage that should reject.

Data

The test corpus is Nora Sandler's writing-a-c-compiler-tests suite (MIT), pinned to a specific commit at image build time. build/filter_corpus.py clones the repo, copies the single-file .c programs into /srv/ccc/train/chapter_*/, and pre-computes a (exit, stdout) reference for every valid/*.c by compiling and running it under gcc 11 on x86_64 (build/gen_expected.Dockerfile; the result matches Sandler's own expected_results.json, checked by build/check_expected.py). Multi-translation-unit tests (Sandler's helper_libs/ / client/ layouts) are excluded. Sources gcc itself refuses to compile are dropped. The (exit, stdout) reference answers live only in the env-server image (/srv/ccc/expected/expected_by_path.jsonl), never in the agent's sandbox — grading is server-side, so the sandbox holds only the read-only .c inputs.

Tools

Beyond the standard 9 CLI tools (bash, read, write, edit, multi_edit, glob, grep, ls, todo_write), the environment provides:

• run_tests(paths=None, timeout_s=1800) — builds /workspace/bin/ccompiler, runs the C-compiler harness against the full corpus (always, regardless of paths), and returns delta reward + a size-bounded sample of failures (the tool output stays under 32KB); every case's result is written to /tmp/run_tests/last.tsv in the sandbox (previous build: prev.tsv). paths (.c files or directory prefixes) only filters which failures are listed back, so a narrow call cannot change the score. timeout_s is clamped to 60–3600s. Calls are serialized per session (concurrent callers share one sandbox).
• progress() — read current pass count and cumulative reward without rebuilding.

Time Horizon

Open-ended. The agent is expected to take many turns: scaffold a Rust crate, implement features incrementally (start with chapter 1's int main(void) { return <const>; } programs — 24 free tests), use the reward delta from run_tests to decide which feature to invest in next, and use git worktrees if spawning parallel subagents.

Environment Difficulty

This environment is hard. A C compiler with enough coverage to pass Sandler's 18-chapter curriculum is thousands of lines of Rust across lexer, parser, semantic analyzer, IR, register allocator, and x86_64 codegen. The chapter ordering — returns → unary → binary → locals → control flow → functions → pointers → arrays → structs → dynamic memory → optimisations → register allocation — gives a smooth dense-reward gradient. At image-build time we kept 1,249 tests (557 valid + 692 invalid) after dropping sources the system gcc refuses to compile.

Other Environment Requirements

None. CCompiler requires no external API keys.

Anti-cheat

The task is "build a C compiler from scratch in Rust", so the artifact must be genuine
from-scratch Rust — it must not delegate to another compiler (e.g. a Rust binary
that shells out to gcc -S). "From scratch in language Y" is a property of the source and the
build, not of runtime behaviour, so it is enforced primarily at build time (a runtime sandbox
can never prove what a binary was written in — anything can be embedded). Each run_tests build is
BLOCKED (0 reward, baseline unchanged) if any rule is broken:

• Controlled build: env -i, pinned toolchain 1.85.0, fresh CARGO_HOME + target dir,
  --offline, RUSTFLAGS="-F unsafe_code -C target-feature=+crt-static -C strip=none --print link-args". -F unsafe_code is compiler-enforced (an #[allow] cannot override it) and also bans
  asm!/global_asm!/#[no_mangle]/#[link_section]; +crt-static removes runtime dynamic
  loading.
• No launching programs: the built binary is nm-scanned for execve/fork/posix_spawn/
  system/popen/… (linked in only if std::process is used) — the direct block for gcc -S.
  dlopen is not banned (static glibc always links it for NSS).
• std-only: cargo metadata must show local path crates only — no external crates, build.rs,
  proc-macros or native links; compiler-backend crates (llvm-sys, cranelift*, clang-sys,
  inkwell, bindgen, gcc-rs, the build-script cc crate, dynasm*, iced-x86,
  rustc_codegen_*) and their symbols are banned outright.
• No foreign link inputs: the rustc link line is checked — linker is cc, inputs come only from
  the fresh target dir / toolchain rlibs, and -l is limited to the std set (catches #[link],
  which is legal in safe Rust).
• Toolchain integrity: rustc, cc/gcc/as/ld, bwrap/setpriv, ld.so.preload,
  nsswitch.conf and the test corpus under /srv/ccc (the sources the harness compiles; the
  answers are keyed by path, so a source rewritten into a trivial program with the same result
  would otherwise pass) are fingerprinted (path/type/size/mode/owner/ctime, which is
  unforgeable without CAP_SYS_TIME; reading never changes a row). Under /srv/ccc only the files
  the grader reads (the case sources and manifests) are compared: the harness is handed the explicit
  case list, so files added next to them (an a.out from compiling a test in place) and directory
  ctimes are ignored, while editing, replacing, moving or deleting an input still shows. The case list and harness
  SHA-256-checksummed at session setup; any later
  change blocks all subsequent grades. The fingerprint listing is compared on the env server, a
  mismatch is re-measured 3× before it blocks (gVisor path lookups can fail transiently), and the
  block names the changed paths. The same check runs again after the harness has graded, before
  anything is credited, so a change made while a build is being graded (e.g. by a concurrent tool
  call) is not banked either. A stray-process sweep runs first: root leftovers are killed,
  a process of another uid (which root cannot kill) blocks.

Grade-time isolation (defence-in-depth). The network is blocked, and the artifact runs under
non-root bubblewrap (dropped to uid 65534 first — bwrap fails as root under the gVisor sandbox)
with no /usr/bin, so even attempting to exec cc/gcc fails. Isolation is fail-closed: if
bwrap cannot isolate on an untampered image, grading errors rather than direct-spawning the artifact
as root (the path that enabled the original hack). This is a second layer only; the build-time rules
hold regardless.

Baked answer tables. The corpus is public and readable during development, so an
agent can run gcc -S at dev time and embed the results in pure safe Rust, keyed by a
hash of the source or its path. Two layers stop this:

• the perturbed re-grade above (the main defence): such a table fails the renamed,
  re-flowed and (for valid programs from chapter 2 on) rewritten copies, so its chapters
  are credited at the 1/16 slack;
• stopgap artifact checks on every build (blocked, baseline unchanged): the binary
  must be at most 64 MiB, and neither the binary nor the asm it emits may contain gcc's
  .ident "GCC: ..." directive; the binary must not contain verbatim corpus source text
  (the longest distinctive line of each test file).

perturb.py's renaming is an injective map onto fresh names (keywords, main, names
starting with _, names used on preprocessor lines, and every symbol the C runtime and
linker provide are never renamed), and each result is checked to lex to the expected
token sequence. Valid programs from chapter 2 on are also rewritten (restructure), each
rewrite keeping the program's exact behaviour and using only constructs of its chapter:

• return e; → return (e); (parentheses arrive in chapter 2);
• a constant → (c), (-(-n)) or (~(~n)) (chapter 2), or (a + b) / (a - b) with
  positive int operands (chapter 3). The arithmetic forms are drawn only for a plain
  decimal int constant in 1..INT_MAX, so the result is an int of the same value with
  no overflow; 0, suffixed, floating and larger constants are only parenthesized, so a
  null pointer constant stays a literal 0. Constants are left alone outside function
  bodies, in static/extern declarations, in case labels and in [N], where the book's
  subset wants a literal;
• one or two null statements at the start of a block (chapter 5), never before a switch
  body's first label.
  Reject cases keep their token sequence, so they stay invalid for the same reason; their
  credit is already bounded by the chapter's credited valid rate.
  build/check_perturb.py checks all of it with gcc on the whole corpus: for 10
  perturbations of each of the 1,242 perturbable programs, valid programs produce the same
  exit status and stdout, and every program gets the same compile status and diagnostics
  (the 7 others are lexically broken invalid_lex cases, left out of the sample).

Residual risk. Chapter 1 programs are all return N;, which no chapter-1 construct
can re-express, so a table still matches their copies (a chapter-1 compiler is a few
lines anyway). A table keyed on a normalization that undoes the rewrites (folding
constant subexpressions, dropping redundant parentheses and null statements) is not
caught; it takes a C parser, and closing it fully needs held-out programs rather than
perturbed public ones.

This environment teaches compiler-construction skills and contains no inherently dual-use objective.

Citations

@book{Sandler_WritingACompiler,
  author    = {Sandler, Nora},
  title     = {Writing a C Compiler},
  year      = {2024},
  publisher = {No Starch Press},
}

@software{Sandler_WACTests,
  author = {Sandler, Nora},
  title  = {writing-a-c-compiler-tests},
  url    = {https://github.com/nlsandler/writing-a-c-compiler-tests},
  note   = {MIT license; pinned at image build time.}
}

@dataset{GRCCompiler,
  author    = {General Reasoning Inc. Team},
  title     = {CCompiler},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/ccompiler}
}