ScenarioPlanning

Description

ScenarioPlanning is an environment for evaluating agents on creative problem-solving and strategy formulation for survival scenarios. This environment wraps the ScenarioPlanning implementation from TextArena, a framework for text-based game environments.

Capabilities

• Creative strategy generation
• Survival scenario analysis
• Detailed planning and resource reasoning
• Competitive strategic thinking

Compute Requirements

ScenarioPlanning does not require a sandbox. It has minimal compute requirements.

License

MIT.

Tasks

There are two splits: train (150 tasks) and test (150 tasks). Each split contains 50 tasks across each of 3 variants:

• ScenarioPlanning-v0
• ScenarioPlanning-v0-train
• ScenarioPlanning-v0-raw

Each task is seeded for reproducibility.

Reward Structure

This is a sparse reward environment. Rewards are mapped from TextArena's native range of {-1, 0, 1} to {0.0, 0.5, 1.0} via (raw + 1) / 2.

We do not use LLM graders for this environment; reward is determined programmatically.

Data

Game state is generated procedurally by the TextArena engine using seeded randomness. No external data files are required.

Tools

Agents are given a single tool:

• send_message(message): Submit your survival strategy for the scenario.

Time Horizon

ScenarioPlanning is a multi-turn environment.

Environment Difficulty

Medium-High. Agents must propose detailed, creative survival strategies for challenging scenarios. After both players submit their strategies, a panel of LLM judges evaluates them to determine which is more effective.

Other Environment Requirements

This environment requires an OpenAI API key (passed via secrets) to power the LLM opponents and LLM jury for evaluation.

Safety

Agents in ScenarioPlanning interact only with a scenario planning game and have no access to external systems, the internet, or sensitive data. The environment does not present safety risks.

Citations

@software{textarena2024,
  author    = {Guertler, Leon and Banting, Wilfried and Pignatelli, Eduardo},
  title     = {TextArena},
  year      = {2024},
  publisher = {GitHub},
  url       = {https://github.com/LeonGuertler/TextArena}
}