Codenames

Description

Codenames is an ORS environment for evaluating agents on word association and collaborative team play in the Codenames board game. This environment wraps the Codenames implementation from TextArena, a framework for text-based game environments.

Capabilities

• Word association and semantic reasoning
• Strategic clue generation
• Team coordination and communication
• Risk assessment in word selection

Compute Requirements

Codenames does not require a sandbox. It has minimal compute requirements.

License

MIT.

Tasks

There are two splits: train (300 tasks) and test (300 tasks). Each split contains 50 tasks across each of 6 variants:

• Codenames-v0
• Codenames-v0-train
• Codenames-v0-raw
• Codenames-v0-hardcore
• Codenames-v0-hardcore-train
• Codenames-v0-hardcore-raw

Each task is seeded for reproducibility.

Reward Structure

This is a sparse reward environment. Rewards are mapped from TextArena's native range of {-1, 0, 1} to {0.0, 0.5, 1.0} via (raw + 1) / 2.

We do not use LLM graders for this environment; reward is determined programmatically.

Data

Game state is generated procedurally by the TextArena engine using seeded randomness. No external data files are required.

Tools

Agents are given three tools:

• give_clue(word, number): Give a one-word clue and number of related words (as Spymaster).
• guess_word(word): Guess a word on the board (as Operative).
• pass_turn(): Pass your turn without guessing.

Time Horizon

Codenames is a multi-turn environment.

Environment Difficulty

Medium-High. Agents play as the Red Spymaster in a 2v2 team game, requiring them to generate clever clues that connect multiple words while avoiding opponent words, neutral words, and the Assassin.

Other Environment Requirements

This environment requires an OpenAI API key (passed via secrets) to power the LLM opponents.

Safety

Agents in Codenames interact only with a word association game and have no access to external systems, the internet, or sensitive data. The environment does not present safety risks.

Citations

@software{textarena2024,
  author    = {Guertler, Leon and Banting, Wilfried and Pignatelli, Eduardo},
  title     = {TextArena},
  year      = {2024},
  publisher = {GitHub},
  url       = {https://github.com/LeonGuertler/TextArena}
}