Gaffer

Description

Gaffer is a single long-horizon environment in which the agent builds a complete, browser-playable Football-Manager-style game for the real Premier League 2026/27 season — sourcing every fact itself from the live web. One task contains an entire software project: research real clubs, squads, managers, fixtures, rules and finances into a database; implement a deterministic, database-generic match/season engine as headless ES modules; and ship a zero-build static-site UI that an automated player can actually play. Grading is deep rather than end-to-end-fuzzy: roughly six thousand hidden checks per grade across ten fixed-weight categories, plus LLM review exactly where design freedom lives.

Capabilities

• Long-horizon project planning and execution (data sourcing → engine → UI in one rollout)
• Live web research against official sources (fixtures, squads, rules, finances, FIFA rankings)
• Simulation engineering: seeded deterministic engines, statistical realism, db-generic rule enforcement
• Frontend engineering without a build step, judged by an automated LLM player through the rendered UI
• Game design under plausibility review (ratings, wages, market values, tactics, negotiation behavior)

Compute Requirements

Agents get a sandbox with 2 CPUs and 4 GB RAM (node 22, python3, git, curl, jq, Firefox + Playwright for self-testing) with full internet access.

License

MIT.

Tasks

One training task: train/main/gaffer_0001. The agent receives a starter workspace containing almost nothing: a one-page README stating the deliverable, JSON Schemas for the four db/*.json tables whose values are graded as facts about the real season, and audit — the grader's own probe, pointed at the agent's game. There is no engine API, no field names, no feature checklist, no scenario list and no scoring table. Knowing what a football management game is, and inferring where reward lives, is part of what is tested. Everything else — data, engine, interface — is built by the agent. The rollout ends when the agent replies with an ordinary message; the final workspace state is then graded.

Reward Structure

Reward = Σ category_weight × pooled pass-fraction over ten frozen-weight categories: db_fidelity 0.15, engine_contract 0.08, match_engine 0.12, season_loop 0.10, rules_enforcement 0.09, finances 0.06, injuries_training 0.05, internationals 0.05, ui_functional 0.20, ui_polish 0.10. The one mechanical requirement is that the workspace is a static site whose index.html a browser can open and play: everything else is graded by playing it. A mechanical driver boots the game, finds its own way in with no knowledge of the interface, drives a season forward and reads the rendered screens — identifying a league table by its arithmetic rather than its headers, so a game is measured on what it shows rather than on whether it guessed our column names. Two LLM playtesters then attempt twenty-eight goals through the interface, judges verdict the screenshot trails against host-measured evidence, and code-review judges read the agent's own source for anything presented but not really there. Facts in the database are still identity-matched (player name + date of birth, club-name aliases, fixture pairings) against a dated snapshot of official sources, with tolerances for legitimately drifting fields.

LLM grading (gpt-5-mini) is used only where design freedom lives, at final grading: a five-part review of the agent's own player model — factual accuracy of sampled raw rows, Football-Manager-style richness, and the plausibility of its ratings, wages and market values (majority-of-three per question, anchored to real 2026 market bands), an automated player that must complete sixteen gameplay scenarios through the served UI in a real browser (verified per scenario by separate judges from the screenshot trail), a code review that convicts presented-but-fake features (a cosmetic tactics panel fails; an absent one does not), and a 23-item visual-design checklist scored on the play-through's screenshots. Text addressed to graders halves the final reward (regex-nominated, judge-confirmed).

Calibration (final-grading pipeline, live judges): empty 0.000 · spec-only 0.000 · perfect database only 0.150 · reference solution 0.9609 (the environment ships a full reference engine and playable reference UI as its own satisfiability proof; the deliberately plain reference now pays real polish points for its plainness — that headroom is the design). A strong 2026 frontier model reached 0.8222 on the same pipeline from a single 63-turn rollout: its engine pools sit at 0.92–1.00 while the playable experience holds it back (functional scenarios 0.54, design review 0.52), with every loss individually diagnosable — no time-flow loop, instant-click match results, checkbox team selection, dead news world, no player pages, judge-rejected wage/value book.

Data

The agent sources all real-world data itself from the live web at rollout time. Grading compares against the environment's own dated snapshot of official and primary sources — the Fantasy Premier League API (squads, fixtures, club strengths), Wikipedia season pages (stadiums, managers, EFL club membership), football-data.co.uk historical seasons (statistical envelopes), and the FIFA rankings API (nation strength) — with a published graded-fields manifest defining exact/tolerated/excluded fields so building from current live data lands inside tolerance. Canonical data and hidden suites live on the OpenReward platform, never in the sandbox. The player model has no licensed ground truth and no published schema: its structure, attribute vocabulary, ratings, wages and market values — and club budgets — are deliberately the agent's own design. Grading discovers them in whatever shape they were written, and scores coverage of the real squads deterministically, model coherence through properties that no naming convention can fake, and football plausibility through review — never against a hidden formula.

Tools

Nine filesystem/terminal CLI tools (bash, read, write, edit, multi_edit, glob, grep, ls, todo_write), web_search (host-side search returning titles/URLs/snippets; degrades gracefully when unavailable — direct fetching from the sandbox always works), and audit (runs the agent's copy of the public smoke suite; deterministic feedback, no score, at most 40 calls).

Time Horizon

A single open-ended build rollout (driver cap 150–400 turns in our runs). Observed complete builds from strong 2026 models: 42 assistant turns (~11 minutes) and 55 turns in earlier iterations; weaker models spend more turns and plateau lower. Final grading takes 10–20 minutes wall-clock — a browser session that drives a season mechanically, two concurrent playtesters, and the judged review of what they saw.

Environment Difficulty

The environment separates sharply by capability tier: a model that ignored the live web and invented its data scored ≈0.4 (database fidelity ≈ 0 — fabrication earns nothing), while a web-sourcing frontier model scored 0.9105 with category-level diagnostics pinpointing each remaining defect. The last ~10% requires genuinely correct statistical realism, faithful rules bookkeeping across a full simulated season, an interface a non-scripted player can operate end to end, and football-informed design judgment.

Other Environment Requirements

Requires an openai_api_key secret (design-plausibility judges, the automated UI player, and visual review all use gpt-5-mini). Optionally accepts a tavily_api_key secret to enable the web_search tool; without it the tool reports unavailable and agents fall back to direct fetching.

Safety

The agent works in an internet-connected sandbox and researches public sports data; the graded deliverable is an offline static site. Risks are those inherent to open-web agentic environments (prompt injection from fetched pages), mitigated structurally: all grading assets live outside the sandbox, hidden tools are name-hidden and token-guarded, judges carry anti-injection headers, and grader-addressed text in the submission halves the reward. The environment does not involve real transactions, personal data, or actions with external side effects.

Citations

@dataset{GRGaffer,
  author    = {General Reasoning Inc. Team},
  title     = {Gaffer},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/gaffer}
}