tax1040

A long-horizon, text-only, mechanically graded environment where an LLM
agent writes a tax engine in Python and is scored by running it against
held-out synthetic U.S. federal individual income tax returns (Form 1040,
tax year 2024).

The idea, in one paragraph

The corpus is ~1,000 households across 10 chapters of increasing complexity
(single W-2 → MFJ → dependents → 1099s → itemized → cap gains → Sch C/SE
→ K-1 → IRA/credits → combined). Each household is a folder of documents
(W-2s, 1099-*, 1098, K-1, …) plus an instructions file. The agent does
NOT submit answers directly. Instead, it reads documents from a 60%
"prototyping pool" of packets visible in its sandbox, writes a tax engine
at /workspace/tax_engine.py with a single entry point — solve(packet_id, packet_dir) -> dict[str, int|float] — and calls run_engine to execute
that engine against a held-out 20% test set. The grader compares each
returned line to the precomputed expected value and rewards +1 per atom
the engine got right (-1 per regression). Reward is dense; the engine
either generalises or it doesn't.

The point is to defeat recall: hardcoding answers requires manually
computing every value first, which is the same work as building the
engine — and the engine generalises.

Splits and tasks

Two splits, each with one engine-graded task:

Pick a split → the task is implied. firehorse --split train ... runs
the train task; --split test ... runs the test task. The two held-out
sets are deterministic (seed 2026) and non-overlapping, suitable for
cross-validation (a model whose train and test scores agree is generalising;
divergent scores signal overfit-to-train).

What the agent sees

No held-out document is ever in the agent's sandbox, for either split. run_engine
copies /workspace into a fresh grader sandbox (same image and bucket mount),
uploads this split's held-out docs only there, runs solve() on them and discards
the grader. The agent's code runs beside the docs inside the grader, so nothing
comes back from it except each packet's line dict and, for an engine exception,
its type and file:line — never its message, a printed line or a file it wrote.
Before this, the engine ran in the agent's own sandbox with /workspace writable,
and agents logged held-out packets from solve() and read them back.

What the agent does

Two custom tools, on top of the standard 9 CLI tools (bash, read,
write, edit, grep, glob, ls, multi_edit, todo_write):

• run_engine(entry_module="tax_engine", entry_function="solve", packets=None, per_packet_timeout_s=10, batch_timeout_s=900) —
  copy /workspace into a fresh grader sandbox, execute solve() there against the
  split's 200 held-out packets, capture per-packet line dicts, grade, return a
  per-chapter scoreboard. The engine can read only /workspace and the read-only
  /srv/tax there. A /workspace over 16 MB compressed is not run.
• progress() — read the current atom count without grading.

There is no submit_return tool. The only path to reward is via run_engine.

The agent iterates: read a few prototyping-pool packets → write the
engine → call run_engine → inspect which chapters/atoms failed →
extend the engine → repeat.

How agents are expected to solve this

The corpus is structured to reward engine-building:

• One long episode building a single engine that handles every chapter.
  Code you write for chapter 1 keeps paying off through chapter 10.
• /workspace ships with a pre-initialised git repo — commit checkpoints
  as you go.
• Chapters are engine features, not difficulty tiers (see Curriculum
  below). A working chapter-1 engine plus an MFJ branch sweeps chapter 2;
  add a CTC/ACTC computation and you sweep chapter 3; and so on.
• Tax law is deterministic. Pub 17 plus the published TY2024 bracket /
  phaseout / credit tables are enough — derive the engine from first
  principles. The reference engines (PolicyEngine-US, Tax-Calculator) are
  blacklisted (see Anti-cheat); writing one is the point, not importing one.

The CLI toolset is the engine-building toolset. The two tax-specific tools
(run_engine, progress) are how you check your engine's output against
the grader.

Why we can grade this mechanically

Tax law is a deterministic function. At corpus build time we run two
independent open-source tax engines on every sampled household:

• PolicyEngine-US
• PSL Tax-Calculator

A household is kept in the corpus only if both engines agree to the
dollar on every gradable line. Disagreements get logged and the packet
is dropped. The surviving packets are guaranteed-correct by construction —
two completely separate implementations of the U.S. tax code arrive at the
same number.

At runtime the engines are gone. The env-server image ships only the
pre-computed (packet, line) → expected_value dictionary, in a directory
the agent's sandbox cannot read.

Curriculum

Households are organised into ten "chapters" of increasing complexity.
Each chapter is a parametric household generator that introduces one new
tax concept on top of the previous chapter's mechanics. Read the "Adds"
column as a roadmap for your engine — each row is the new piece of tax
logic your code has to grow into before that chapter's packets become
solvable:

Chapter 1 packets are ~10 gradable lines each with most values "free"
(wages, withholding, std deduction — direct from the W-2 or a constant);
chapter 10 packets are 40+ lines and require getting everything earlier
right.

The 60/20/20 split is per-chapter, so train and test each contain ~20
packets from every chapter — the held-out sets exercise every mechanic.

Anti-cheat

The agent could trivially "solve" this by importing PolicyEngine or
Tax-Calculator. So:

• Both libraries are blacklisted in the agent's sandbox. pip freeze is
  scanned before every run_engine call; the vendored pip index excludes
  them.
• Network is blocked.
• The expected JSONs are not present in the sandbox at all — they live
  only in the env-server process. The agent has no filesystem path to
  the answer keys.
• No held-out packet docs are in the agent's sandbox, for either split: the
  engine runs on them in a per-call grader sandbox (see above).
• SHA-256 checksums of the bucket manifest and the answer-key directory
  are recorded at session start and verified before each run_engine.

Reward shape (telescoping)

run_engine produces a delta vs. your previous run on the same session:

• +1 per (packet_id, line_id) atom that flipped from wrong/missing
  to currently correct in this run.
• −1 per atom that flipped from currently correct to wrong (a regression
  caused by an engine change that broke a previously-passing case).
• 0 for atoms that didn't change state.

Cumulative reward = currently-correct atom count on the split's held-out
set (out of ~2,840 total). finished=True only when every held-out atom
is currently correct on the same run. The agent must keep prior atoms
green as the engine grows — pushing a regression costs reward.

The grader applies the IRS whole-dollar rounding rule (Pub 17, "round
half away from zero") to both the engine's value and the expected value
before comparing. Engines can return floats; they're rounded server-side.

License

ORLv1 for the environment code. The
reference engines (PolicyEngine-US is AGPL-3.0, Tax-Calculator is MIT)
are used only at corpus build time and are not shipped in the runtime
image. IRS forms are public domain.