GuessWho

Description

GuessWho is an ORS environment for evaluating agents on the classic Guess Who game, where agents must identify a target character by asking yes-or-no questions about their traits. This environment wraps the GuessWho implementation from TextArena, a framework for text-based game environments.

Capabilities

• Information-theoretic question generation
• Logical reasoning and constraint satisfaction
• Strategic inquiry to maximize information gain

Compute Requirements

GuessWho does not require a sandbox. It has minimal compute requirements.

License

MIT.

Tasks

There are two splits: train (150 tasks) and test (150 tasks). Each split contains 50 tasks across each of 3 variants:

• GuessWho-v0: Standard Guess Who with formatted feedback
• GuessWho-v0-train: Training variant with guidance
• GuessWho-v0-raw: Raw feedback without formatting

Each task is seeded for reproducibility.

Reward Structure

This is a sparse reward environment. Rewards are mapped from TextArena's native range of {-1, 0, 1} to {0.0, 0.5, 1.0} via (raw + 1) / 2.

We do not use LLM graders for this environment; reward is determined programmatically.

Data

Game state is generated procedurally by the TextArena engine using seeded randomness. No external data files are required.

Tools

Agents are given a single tool:

• send_message(message): Send a question or guess [Name] to the gamemaster.

Time Horizon

GuessWho is a multi-turn environment.

Environment Difficulty

Medium. The game requires efficient question generation to narrow down candidates, with optimal play requiring understanding of information theory to maximize elimination per question.

Other Environment Requirements

This environment requires an OpenAI API key to be provided via the secrets parameter. The API key is used to power the LLM gamemaster that responds to questions about character traits.

Pass the API key when creating a session:

async with env.session(task=task, secrets={"openai_api_key": "your-key"}) as session:
    # use session

Safety

Agents in GuessWho interact only with a deduction game and have no access to external systems, the internet, or sensitive data. The environment does not present safety risks.

Citations

@software{textarena2024,
  author    = {Guertler, Leon and Banting, Wilfried and Pignatelli, Eduardo},
  title     = {TextArena},
  year      = {2024},
  publisher = {GitHub},
  url       = {https://github.com/LeonGuertler/TextArena}
}