Raven

Description

Raven is a long-horizon coding environment that tasks an agent with implementing the Raven programming language from scratch in Python. Raven is a real, small, obscure language — statically inferred, WASM-targeting, with multi-dispatch, pattern matching, and macros instead of keywords. The agent gets the language spec, a worked example set and Raven's own standard library as reading material, and is graded per case against expected outputs produced by the real Raven toolchain. Reward is the per-build change in how many cases the current artifact passes, so it telescopes to the pass count of the build the agent finishes with.

The point of choosing this particular language is contamination. The family's other language-implementation environments grade against targets a model has effectively memorised: a C compiler against Nora Sandler's public suite, a SQL engine against sqllogictest. Raven's repository was created in December 2025, has 24 stars and no forks, and its semantics are documented in one 507-line file — so an agent has to actually read the spec and run the examples rather than recall an implementation. And because the spec is external and public, the environment avoids the unfairness of grading against a specification we invented ourselves.

Capabilities

• Implementing a language front-to-back: lexer, macro-aware parser, pattern matcher with multi-dispatch, evaluator, pretty-printer, type-directed value formatter
• Reading a specification carefully and implementing to it, rather than to intuition — Raven's semantics contradict Python and Rust in several load-bearing places
• Multi-file refactoring across an evolving codebase, under a regression-penalising reward
• Test-driven iteration against a local example set before spending a graded call
• Open-ended long-horizon execution (hundreds to thousands of tool calls)

Compute Requirements

Agents get a sandbox with 8 GB of RAM and 4 CPUs. The network is blocked. Python 3 and bubblewrap are preinstalled; there is deliberately no Node, npm or Raven toolchain on the image.

License

MIT. The environment grades against the upstream Raven language, which is also MIT licensed (© Mike J Innes); upstream's DOCS.md and its Raven-source standard library are shipped to the sandbox verbatim with attribution under build/vendor/.

Tasks

A single mega-task, raven-train, on the train split: implement enough of Raven to pass every case in the corpus. The curriculum lives in the reward rather than in the task list — the 3,151 cases are grouped into 32 chapters that run roughly easiest-first, so a working tokeniser plus one builtin earns reward within a few turns.

247 cases (8%) expect a runtime error; the rest expect successful output. The ch9x_upstream_* chapters are programs taken from Raven's own test suite (MIT, attributed), with expected outputs re-derived from the pinned toolchain rather than read from upstream's assertions.

Reward Structure

This is a dense, verifiable reward environment with no LLM grader.

• +1 for each case the current build passes that the previous build did not.
• −1 for each case the previous build passed and this build now fails.
• A build that fails to compile scores 0 and leaves the baseline untouched.

Reward tracks the current artifact, so the trajectory sum telescopes to the pass count of the build the agent finishes with, and churn cannot farm it. finished=True only when a single build passes every case at once.

A case passes only if all of (stdout, error kind, error class, error position) match. Grading the response content rather than the artifact's exit status is deliberate: it closes the reward leak in the sibling ccompiler environment, where 692 of 1,249 cases pass on any non-zero exit and a stub calling exit(1) collects more than half the corpus.

Data

Expected outputs are produced by the real Raven toolchain, pinned to upstream commit f58f3a49 built from source on Node 25.9.0 — deliberately not the npm release, which is byte-identical to the v0.5.0 tag, sits ~80 language-surface commits behind master, and ships a materially weaker formatter. Generation runs author-side via build/oracle.py and the corpus is committed, so no Node runtime and no upstream clone ever enters either Docker image.

Every case must clear six generation gates or it is dropped with a recorded reason: it must produce a clean Raven-level error rather than an upstream compiler crash; three runs must agree; the interpreted path and the raven build + WASM path must agree (they share only the front end, which is what makes the check a real independence test); every free identifier must appear in the published builtin manifest; a generator-predicted answer must match where the template computed one; and no expected output may contain an opaque reference or address.

Tools

Beyond the standard 9 CLI tools (bash, read, write, edit, multi_edit, glob, grep, ls, todo_write) from the SDK's CLIToolset:

• run_tests(paths=None, timeout_s=1800) — freezes /workspace/src/ into a zipapp, runs the full corpus through it under bubblewrap, grades server-side, and returns the delta plus a bounded, categorical failure listing. paths is a display filter over corpus chapters and cannot change the score.
• progress() — current pass count without rebuilding, at zero reward.

Time Horizon

Open-ended. The agent scaffolds a Python package, implements the language incrementally, and uses the run_tests delta to choose what to invest in next.

Note for anyone driving this environment: there is no terminal tool, and finished fires only when one build passes every case, so the rollout budget is the driver's to spend. A model that decides it is done after a partial implementation will stop emitting tool calls, and a driver that simply loops will watch it repeat a wrap-up summary for the rest of the budget. Inject a continuation message when a turn produces no tool call — test_agent.py shows the pattern.

Environment Difficulty

Hard. Upstream's own implementation is roughly 478 KB of TypeScript. A deliberately partial reference implementation — tokeniser, the real precedence table, evaluator, fn with literal-pattern multi-dispatch, if/while/for, lists, and the canonical show echo, but no bundles, match, @extend, packs, bit-widths, unicode views or & swap — scores 876/3151 (27.8%). Every cheat stub scores at or below 0.5%: an empty stub and a protocol-shaped do-nothing stub score exactly 0, a stub that claims one fixed error class on every input gets 3 cases, and a regex heuristic that echoes the literal out of println("…") gets 13. The analytic constant-artifact ceiling is 3/3151 = 0.10%, so no artifact that ignores its input can do better than that.

In a 100-turn rollout against the deployed environment (109 tool calls, 32 graded builds) gpt-5.2 reached 657/3151 (20.9%), climbing in chunks — 120, then −120 when a refactor broke everything it had, recovering to 318, then 358, 435, 582, 657 — with three regressions costing 185 cases in total. The partial reference scores 27.8%, so a frontier model lands just under it.

Several of Raven's semantics are specifically where memorised intuition fails: 7/2 is 3.5 because division is always floating; rem truncates toward zero rather than flooring; + binds tighter than - and / binds tighter than *; ==, ^ and .. are non-associative; 1/0 is Infinity rather than an error; and 8.0 == 8 is false because arithmetic promotes across numeric types but comparison does not.

Other Environment Requirements

None. Raven needs no external API keys.

Safety

The sandbox blocks network egress and contains no Raven implementation of any kind — no Node, no npm, no upstream TypeScript, and no expected answers, since grading happens in the env server. The agent's artifact additionally runs under bubblewrap with only the interpreter, the system libraries and its own zipapp visible, so it can reach neither the corpus nor the workspace at grade time. The producer binary's SHA-256 is pinned at session setup and rechecked on every graded call, and it reports the hash of the bytes it actually fed the artifact so a swapped input on disk is detected. This environment teaches language-implementation skills and has no dual-use objective.

Citations

@software{Raven_language,
  author = {Innes, Mike J},
  title  = {Raven: a language for sorcery},
  url    = {https://github.com/Unkindnesses/raven},
  note   = {MIT licensed. The language this environment implements, and the oracle for its expected outputs.}
}

@dataset{GRRaven,
  author    = {General Reasoning Inc. Team},
  title     = {Raven},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/Raven}
}