CTF

Description

CTF (Capture The Flag) is an environment for evaluating language model agents on Capture The Flag security challenges. Agents are given a sandboxed environment with access to challenge files and must solve security problems across domains including binary exploitation, cryptography, web security, and steganography to extract hidden flags.

Capabilities

• Binary exploitation and buffer overflow analysis
• Cryptographic puzzle solving
• Web security vulnerability identification
• Reverse engineering and code analysis
• Steganography and hidden data extraction
• Autonomous security tool usage in sandbox

Compute Requirements

Agents in CTF are given a sandbox with 0.5 CPU and 1GB of RAM. The served challenges are self-contained — their files are mounted into the sandbox and no external network connection is required to solve them.

License

MIT

Tasks

There are two splits in this environment:

• train: 103 challenges for training
• test: 5 challenges for evaluation

Two classes of challenge are excluded from both splits:

• External-dependency challenges (56): those that depend on an external
  host or URL (live netcat/web services or off-box file downloads). The
  original CTF infrastructure is largely offline, and a rollout's solvability
  should not hinge on third-party network state.
• Server-side-flag pwn challenges (8): remote binary-exploitation tasks
  whose flag lived server-side in a flag.txt that the binary reads only after
  a successful exploit, protected by privilege separation and network-only
  access. That model can't be reproduced in this single-host, root-shell,
  substring-graded sandbox: with no flag.txt staged the exploit reads
  nothing, and with a readable flag.txt the agent could simply cat it
  instead of exploiting — so they are unsolvable-or-degenerate either way.

Challenges span multiple security domains:

• Binary exploitation (buffer overflows, format strings)
• Cryptography (ciphers, encoding, hashing)
• Web security (injection, authentication bypass)
• Reverse engineering (binary analysis)
• Steganography (hidden data in images/files)

Each challenge provides a description and associated files. The agent must analyze the challenge, exploit the vulnerability or solve the puzzle, and extract the flag.

Reward Structure

This is a sparse, verifiable reward environment. Rewards are issued when the agent submits an answer:

• 1.0: Flag correctly extracted and submitted (case-insensitive substring match)
• 0.0: Incorrect flag or already submitted

No LLM grader is used. Flag validation is exact matching against known flags extracted from challenge files. The environment supports 60+ flag formats from various CTF platforms (e.g., flag{...}, CTF{...}, picoCTF{...}, RITSEC{...}).

Data

The benchmark is drawn from a corpus of 185 challenges sourced from various CTF competitions; 108 self-contained challenges are served (103 train + 5 test) after excluding those that require an external host/URL or that are remote server-side-flag pwn challenges. Each challenge contains:

• Challenge description with context and hints
• Associated files (binaries, scripts, images, etc.)
• Hidden flag to extract

Challenge files are mounted read-only at /tmp/gr-datasets/ctf/challenges/{task_id}/ in the sandbox.

Tools

Agents have access to 6 tools:

• bash: Execute bash commands in the sandbox (returns stdout/stderr)
• list_files: List directory contents with options for hidden files and recursion
• read_file: Read file contents (50KB limit)
• write_file: Write content to files in the sandbox
• submit_answer: Submit the extracted flag for verification
• todo_write: Task planning and progress tracking

Time Horizon

CTF is a multi-turn environment where agents iteratively explore challenges, run tools, and analyze results before submitting the flag.

[Statistics on average tool calls here]

Environment Difficulty

[Statistics on environment difficulty here]

Other Environment Requirements

There are no other requirements for running this environment.

Safety

CTF challenges are run in isolated sandbox environments. Agents interact only with pre-defined challenge files and cannot affect external systems. The environment is designed for educational security research and competitive CTF solving.

Challenge content is sourced from public CTF competitions and does not include novel exploit development for real-world systems. However, as a dual-use domain, models that are better at cybersecurity tasks may be directed to use these capabilities for cyberattacks. When training with multi-environment reinforcement learning, is important that rewards in these domains are complemented with rewards for alignment with ethical behaviour.

Citations

@dataset{GRCTF,
  author    = {General Reasoning Inc. Team},
  title     = {CTF Environment},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/CTF}
}