CtfFlag

A CTF-solving OpenReward environment. Each task is one real, internet-sourced CTF
challenge; the agent recovers the flag in a network-blocked sandbox and submits it with
submit_flag. Grading is an exact flag-string match done server-side — the flag is
never present on the sandbox, so the agent must actually solve the challenge.

• One challenge per task (list_tasks returns N), binary reward (1.0 correct / 0.0).
• Real-sourced from raw CTF archives (CTFd/ctfcli challenge.yml format); the
  published flag is the ground truth (no solve scripts needed to grade).
• v1 = static categories (crypto / forensics / reversing / misc). Interactive pwn/web
  come in v2 via an env-mediated service channel (RL-safe). See DESIGN.md.

Layout

Corpus status

59 challenges, each with a passing per-challenge golden test (crypto 38, forensics 7,
misc 9, reversing 5), from pinned sources:

• csictf 2020 (csivitu/ctf-challenges, ctfcli challenge.yml) — 12
• CryptoHack CTF archive (cryptohack/ctf_archive, description.yml) — 16
• DownUnderCTF authored (josephsurin/my-ctf-challenges, details.yaml) — 21
• DownUnderCTF official 2021–2024 (DownUnderCTF/Challenges_*, ctfcli challenge.yml/ctfcli.yaml) — 10

Most real CTF challenges are live-service (the shipped file is the server; the flag is
its runtime secret, not in the artifact) — those are not offline-golden-testable and are
listed in EXCLUDE_IDS for a future v2 service-channel. The corpus here is the
offline-static, fully-solver-verified subset.

Every shipped challenge has a reference solver (build/corpus/challenges/<id>/solve.sh)
that recovers the recorded flag OFFLINE in the sandbox image — see "Golden tests" below.
Challenges that could not be solved offline with the toolchain are excluded
(EXCLUDE_IDS in build/ingest.py): live-service web/pwn, a corpus file whose stego
carrier was destroyed upstream, SageMath/index-calculus-grade crypto, and binary/mobile
reversing that needs executing x86-64/APK/IPA/Swift artifacts (deferred until the verify
path runs amd64/qemu).

Scaling further is additive: append a source to SOURCES in build/ingest.py (a ctfcli
challenge.yml, description.yml, or details.yaml repo, pinned by SHA), re-run ingest
(existing solvers are preserved), author a solve.sh per new challenge, and prune any
without a passing solver. Adding sagemath to sandbox.Dockerfile would unlock the
pruned Sage crypto; adding an amd64/qemu verify path would unlock the binary-rev set.

Golden tests (two layers)

1. Env-logic invariants (no Docker): golden_tests.py — exact flag grading,
   no-leak failure messages, the _setup_done gate, schemas. Plus the corpus accuracy
   gate build/ingest.py --verify — right flag extracted, files present, no plaintext
   flag leak.
2. Per-challenge solvers (the "ensure each challenge works" gate):
   build/verify_solvers.py runs each solve.sh in docker run --network none ctfflag-sandbox:dev and asserts it reproduces the recorded flag — and rejects any
   solver that hardcodes the flag (it must DERIVE it). Reference solvers live ONLY in the
   repo (CI), and are never shipped to the agent sandbox. Both layers run in CI as a
   publish gate (.github/workflows/build-sandbox.yml): the sandbox image is not
   pushed unless all 28 solvers pass.

uv run --no-project --with pyyaml python build/verify_solvers.py --jobs 2 --timeout 300
uv run --no-project --with pyyaml python build/verify_solvers.py --only <id>   # one challenge

Build / verify the corpus

cd envs/CtfFlag
python3 build/ingest.py            # clone pinned sources, verify, emit build/corpus/
python3 build/ingest.py --verify   # CI gate: re-check accuracy of the committed corpus

Add sources by appending to SOURCES in build/ingest.py (any ctfcli challenge.yml
repo, pinned by commit SHA) and re-running.

Test the env logic

cd envs/CtfFlag
uv run --no-project --with 'openreward>=0.1.125' --with pytest --with pydantic \
  python -m pytest golden_tests.py -v

Build images locally

docker build -f Dockerfile -t ctfflag-server:dev .
docker build -f sandbox.Dockerfile -t ctfflag-sandbox:dev .   # heavy (CTF toolchain)

Publishing (the user runs these)

• Env-server: connect this repo to the env via the OpenReward GitHub App, then push.
• Sandbox: .github/workflows/build-sandbox.yml pushes :latest + :sha-<gitsha> to
  GHCR on a path-filtered push to main. Repin SANDBOX_IMAGE in ctfflag.py to the
  :sha-… (the GHCR mirror serves :latest stale).

Attribution

Challenges are redistributed from the upstream archives listed in
build/corpus/ATTRIBUTION.md, each pinned by commit SHA; flags are their published values.