UnderwriteBench

Description

UnderwriteBench is an environment for running a year of property insurance on
the Florida + Louisiana Gulf coast — the two most hurricane-exposed US books.
The agent acts as the chief underwriter of a small mutual: it inherits a
prior-year book, quotes incoming applicants (drawn from real NFIP property
records) into a competitive market, holds reserves (case + IBNR), buys
catastrophe excess-of-loss reinsurance, and weathers a 52-week claims process
whose tail risk is driven by correlated catastrophe shocks (hurricane
footprints).

The benchmark targets a specific failure mode that language models reliably
exhibit: they can recite premium-loading formulas in isolation but misprice
tail risk when individual claims become correlated through a common
catastrophe driver. In a competitive market, crude pricing is adversely
selected against (it wins exactly the risks it underprices), overpricing loses
the book, and a correlated loss season punishes a book without reinsurance.

Capabilities

• Empirical risk scoring from real historical NFIP claims and policies and the HURDAT2 track record
• Risk-differentiated premium setting in a competitive market (price vs volume vs adverse selection)
• Reserve management (case + IBNR) after correlated catastrophe events, under two-sided regulatory tolerances
• Reinsurance treaty structuring (CAT XL layers, attachment / limit / peril) sized to the book's tail
• Long-horizon balance-sheet stewardship over a 52-week season
• Multi-turn execution combining environment tools with code authoring

Compute Requirements

Agents in UnderwriteBench are given a sandbox with 16 GB of RAM and 4 CPUs.

License

ORLv1.

Tasks

There are five training tasks in this environment:

• season-alpha-2026
• season-bravo-2026
• season-charlie-2026
• season-delta-2026
• season-echo-2026

And three test tasks:

• season-foxtrot-2026
• season-golf-2026
• season-hotel-2026

Each task simulates a single 52-week book year starting June 1, 2026 with
$50,000,000 of starting surplus and an inherited book of ~3,900 policies
written over the prior year (at the market price), which run off at their
one-year anniversaries and are not renewed. About 150 new applicants arrive
per week. Each scenario's hurricane realization is based on a historical
Atlantic season and is not disclosed to the agent — storms that occur
during the season are reported as "Storm Alpha", "Storm Bravo", etc. as they
make landfall. The agent has the same climatological information any real
underwriter has: the HURDAT2 basin record and ~10 years of NFIP claims and
policy data.

Reward Structure

This is a dense, verifiable reward environment. Rewards occur after each
weekly advance_week call. The per-week reward is the agent's log-equity
change minus the baseline pricer's log-equity change on the same scenario,
minus penalties:

$r_t \;=\; \log\!\frac{E_t^{\text{agent}}}{E_{t-1}^{\text{agent}}} \;-\; \log\!\frac{\tilde B_t}{\tilde B_{t-1}} \;-\; \text{penalties}_t, \qquad \tilde B_t = \max(B_t,\ 0.25\,E_0)$

The sum telescopes: cumulative reward is $\log(E_T / \tilde B_T)$ minus
penalties, so a positive score means ending the year with more equity than the
reference pricer. Equity is capital minus true outstanding claims (case +
IBNR). The baseline's equity is floored at 25% of starting surplus so a
baseline insolvency hands the agent a bounded bonus.

Penalties:

• Reserving (weekly): 0.5 × (deficit + S·ln(1 + redundancy / S)) /
  starting surplus with S = $1M, judged against liabilities at the time of posting. Redundancy is posted reserves above 125$250k materiality allowance; it costs the same as a deficit when small and
  grows logarithmically when large.
  Sustained material under-reserving (below 50% of true for 3 weeks) triggers
  a regulatory underwriting freeze.
• Minimum volume (week 8): up to 0.5, proportional to the shortfall below
  200 new policies written this year.

If equity goes non-positive the agent is insolvent: the task ends with a
reward of -3.0 for that week.

We do not use LLM graders for this task. Reserves, claims, premiums, and
reinsurance are all deterministic functions of the seeded simulation state.

Data

Ground-truth data consists of the per-scenario hurricane event timeline (track,
intensity, surge), the per-scenario applicant queue and inherited book sampled
from real NFIP policies, and the environment's calibrated lognormal severity
parameters. Agents do not see any of this; it lives under env_only/ in the
OpenReward bucket and is read only by the environment process. The
climatological event set (cat_event_set.json, 127 HURDAT2 hurricane
landfalls on the FL/AL/MS/LA coast, 1900–2025) ships with the environment.

Agents are given access to publicly available historical data for developing
models:

• NFIP Redacted Claims v2 (FL+LA, 2010–2023)
• NFIP Redacted Policies v2 (FL+LA sample)
• NOAA HURDAT2 Atlantic basin track database (1851–2025)
• ZIP × flood-zone × base-flood-elevation lookup

All four are sourced from FEMA OpenFEMA and NOAA NHC. Data files are stored
on the OpenReward platform. See data_prep/README.md for the build pipeline.

Tools

Agents are given access to CLI tools (bash, read, write, edit,
grep, glob, ls, multi_edit, todo_write) for fitting models and
analysing data, plus the following environment-specific tools:

• view_capital — show balance sheet (surplus on posted-reserve basis, premium, losses)
• view_applicants — list this week's quote requests with full risk features
• quote_policy — quote an individual applicant; binds only if the applicant accepts the price
• set_underwriting_rule — bulk-apply an accept/reject + pricing rule to un-quoted applicants
• view_portfolio — concentration, exposure, in-force premium
• buy_reinsurance — purchase a CAT XL layer (weeks 1–2 only; premium adjustable on the in-force book)
• set_reserves — post case + IBNR reserves
• view_reported_claims — list reported open claims with adjuster estimates
• pay_claim — settle a reported claim early (claims otherwise settle 2 weeks after report)
• advance_week — process the week (earn premium, fire claims, apply reinsurance, compute reward)

Time Horizon

UnderwriteBench is an open-ended, long-horizon environment over 52 weekly
steps per task. A representative agent makes one advance_week call per week
plus a varying number of view_* / quote_* / pay_claim calls per week
depending on whether it uses individual quotes or a bulk underwriting rule.

Environment Difficulty

The environment is unsaturated by frontier models at time of writing. In
scripted-strategy sweeps (data_prep/sweep_strategies.py, mean over the 8
scenarios), pricing at 1.1–1.2× the true-risk market price with the baseline's
reinsurance scores about +0.35; crude flat loadings score about −0.15 to 0; absurd
prices or writing nothing new score about −0.1 to −0.7; underpricing (0.7× the
market) about −0.44; and dropping reinsurance insolvency-risks the book.

Simulation and Calibration

• Market. Every applicant holds a competing quote. Its hidden
  reservation premium is its true expected annual loss / 0.65 (competitors'
  target loss ratio) times a logistic factor (spread 0.15 in log-price), fixed
  per applicant. Quotes above it are declined; premiums may not exceed total
  coverage. Acquisition expenses are 15% of earned premium.
• Wind. Storms track straight inland along their landfall bearing; peak
  wind uses a Holland (1980) profile at cross-track distance, with inland decay
  per Kaplan & DeMaria (1995) (R = 0.9, Vb = 26.7 kt, α = 0.095/h) and a
  marine-to-terrain factor of 0.65. Vulnerability is a logistic fit to the
  Florida Public Hurricane Loss Model (FPHLM V8.3) Form V-3 reference frame
  house; masonry, manufactured-home and post-2002-code factors follow FPHLM
  (concrete's 0.8 factor is an assumption). Wind-only loss costs match FPHLM
  Form A-4 county loss costs on average (10 FL counties, geo-mean ratio 1.09;
  within ~2x except Orange County, which is under-represented by a 126-year
  catalogue of straight tracks). See data_prep/check_calibration.py.
• Surge. Saltwater depth-damage from the USACE NACCS (2015) prototype 5A/5B
  curves, structure and contents separately.
• Non-cat claims. Compound Poisson with ~0.5%/yr base frequency (~1%/yr in
  the SFHA, FEMA's 1%-annual-chance zone) and NFIP-fitted lognormal severity.
• Claims. Cat claims are reported 0, 1, 2, ... weeks after the event and
  settle two weeks after report, so a storm's loss sits in IBNR first.
• Reinsurance. Layers are priced from the climatological event set on the
  current in-force book (rate on line = max(3%, multiple × expected loss /
  limit), multiples 2.0 / 2.8 / 4.0 by attachment) and re-rated weekly.
• Scenario seasons are based on historical seasons with every storm
  jittered (landfall moved up to ±75 km along the coast, ±2 weeks, category
  ±1 half the time, radius / forward speed ×0.8–1.25, bearing ±20°), so a
  landfall no longer matches its HURDAT2 original exactly and later storms
  cannot be read off the historical record.
• Known limitations. The number and rough geography of a season's storms
  still follow the historical season; the event
  set is a 126-year historical catalogue rather than a stochastic one; the
  reinsurance multiples are modelling assumptions.

Other Environment Requirements

UnderwriteBench requires no external API keys beyond the standard OpenReward
endpoint. All data sources are pre-baked into the environment bucket.

Safety

Agents in UnderwriteBench are told to maximize the log-growth of an
insurance balance sheet, which is a form of long-run wealth maximization. The
environment is purely simulated and does not interact with real markets,
counterparties, or claimants. There may, however, be indirect risks in
training agents to single-mindedly maximize a balance sheet — agents that
learn aggressive reserving or pricing here should be jointly trained on
environments that reward more general ethical and prudential reasoning.

Citations

@dataset{GRUnderwriteBench,
  author    = {General Reasoning Inc. Team},
  title     = {UnderwriteBench},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/UnderwriteBench}
}