Breakthrough

Description

Breakthrough is an ORS environment for evaluating agents on playing Breakthrough, a chess-like abstract strategy game, against an LLM opponent. This environment wraps the Breakthrough implementation from TextArena, a framework for text-based game environments.

Capabilities

• Strategic pawn advancement and positioning
• Tactical capture decisions using diagonal forward movement
• Race-style endgame reasoning to reach opponent's back rank
• Competitive two-player gameplay against an LLM opponent

Compute Requirements

Breakthrough does not require a sandbox. It has minimal compute requirements.

License

MIT.

Tasks

There are two splits: train (900 tasks) and test (900 tasks). Each split contains 50 tasks across each of 18 variants:

• Breakthrough-v0
• Breakthrough-v0-blind
• Breakthrough-v0-blind-raw
• Breakthrough-v0-blind-train
• Breakthrough-v0-large
• Breakthrough-v0-large-raw
• Breakthrough-v0-large-train
• Breakthrough-v0-long
• Breakthrough-v0-long-raw
• Breakthrough-v0-long-train
• Breakthrough-v0-raw
• Breakthrough-v0-small
• Breakthrough-v0-small-raw
• Breakthrough-v0-small-train
• Breakthrough-v0-tiny
• Breakthrough-v0-tiny-raw
• Breakthrough-v0-tiny-train
• Breakthrough-v0-train

Each task is seeded for reproducibility.

Reward Structure

This is a sparse reward environment. Rewards are mapped from TextArena's native range of {-1, 0, 1} to {0.0, 0.5, 1.0} via (raw + 1) / 2.

We do not use LLM graders for this environment; reward is determined programmatically.

Data

Game state is generated procedurally by the TextArena engine using seeded randomness. No external data files are required.

Tools

Agents are given a single tool:

• move_pawn(from_square, to_square): Move a pawn from one square to another. Specify both from_square and to_square in algebraic notation (e.g., from_square='a2', to_square='a3').

Time Horizon

Breakthrough is a multi-turn environment.

Environment Difficulty

Medium

Other Environment Requirements

This environment requires an OpenAI API key (passed via secrets) to power the LLM opponent.

Safety

Agents in Breakthrough interact only with a strategic board game and have no access to external systems, the internet, or sensitive data. The environment does not present safety risks.

Citations

@software{textarena2024,
  author    = {Guertler, Leon and Banting, Wilfried and Pignatelli, Eduardo},
  title     = {TextArena},
  year      = {2024},
  publisher = {GitHub},
  url       = {https://github.com/LeonGuertler/TextArena}
}