CommonMark

A long-horizon OpenReward environment: the agent builds commonmark from
scratch in /workspace, and the cmark-runner harness grades the built
artifact against a public conformance corpus baked into the sandbox image.
Published as GeneralReasoning/CommonMark.

See DESIGN.md for the full design and build/PROTOCOL.md for the artifact
invocation + grading contract.

Reward

Per-build delta: +1 per corpus case the current build passes
that the previous build did not, −1 per regression. Reward tracks the current
artifact, so the trajectory total telescopes to the pass count of the build the
agent finishes with. finished=True only when a single build passes every
case at once.

run_tests

• timeout_s is clamped to 60–3600s, and the producer's sandbox.run gets a
  deadline above it.
• Calls are serialized per session, so a concurrent build can't swap the binary
  under a running grade.
• The tool output is bounded (metadata included). Every case's result is written
  to /tmp/run_tests/last.tsv in the sandbox (previous build: prev.tsv),
  failures first, capped at 512KB. Failure messages carry the agent's own output,
  never the expected HTML.
• "From scratch in Go" is checked at build time: a package policy on
  go list -deps (no unsafe, os/exec, syscall, plugin, golang.org/x/sys
  imports; no toolchain cmd/... packages; no cgo/C/assembly/.syso files), a clean-environment build, and a
  go tool nm scan for process-spawning code. Violations score 0 and leave the
  baseline unchanged. The sandbox image has no Python or Perl.
• The artifact runs as nobody in a non-root bubblewrap jail (unjailed as nobody
  where bwrap can't start). The producer, bwrap and setpriv are checksummed,
  and a listing of GOROOT and the corpus input files is diffed against the
  session's baseline (a block names the changed paths; a change there blocks the
  rest of the session). New files next to the corpus inputs are ignored.
• A command the sandbox kills (out of memory) is retried, then reported as not
  graded; it is never read as tamper, a banned dependency or a build failure.

Layout

Develop / test

# No-Docker invariant tests:
uv run python -m pytest golden_tests.py -v

# Local sandbox-image build (iterate / reference-solution check). On push to main
# .github/workflows/build-sandbox.yml builds + pushes ghcr.io/generalreasoning/commonmark:latest (:latest) to GHCR;
# build + push by hand only as a fallback:
docker build --platform linux/amd64 -f sandbox.Dockerfile -t ghcr.io/generalreasoning/commonmark:latest .
docker push ghcr.io/generalreasoning/commonmark:latest

# Local server smoke test:
uv run python server.py            # in one shell
uv run python test_agent.py        # in another

Generated by swe_lib_env_creator. Search the tree for TODO(DESIGN: to find
everything still to fill from DESIGN.md (the build command, prompt,
anti-cheat regexes, corpus filter, harness grading logic, toolchain install).