FrontierFinance

Description

FrontierFinance is an agentic financial-research benchmark from Samaya AI. Each task is an expert-crafted investor query — asked as of a specific date — which the agent answers by researching the web (SEC filings, earnings call transcripts, company press releases, market data) and submitting a long-form answer. The answer is graded checklist-style against expert-authored rubrics.

Capabilities

• Financial research across the investor workflow: financial data/modeling, sector/industry/macro analysis, earnings/events, company research, coverage/catalyst monitoring, and screening/discovery
• Exhaustive retrieval (temporal, cross-entity, thematic) from primary sources
• Numerical reasoning over financial figures with correct units and periods
• Temporal anchoring: interpreting the query relative to its as-of date (instructed, not enforced by retrieval — see Data)
• Long-form synthesis with professional investor judgement

Compute Requirements

This environment does not require a sandbox; compute requirements are minimal. Grading and web search are API calls.

License

CC-BY-4.0, matching the source dataset.

Tasks

A single train split with 220 tasks, one per benchmark query. Each task exposes only the query and its as-of date to the agent; the rubrics stay server-side. Queries span six use cases: financial data/modeling (70), sector/industry/macro (38), earnings/events (36), company research (32), coverage/catalyst monitoring (27), and screening/discovery (17).

Reward Structure

Sparse, LLM-graded reward delivered once when the agent replies with its long-form answer as an ordinary message (no tool call), which ends the rollout. The answer is judged against the query's expert-authored rubrics — 11,543 across the benchmark, ranging from 3 to 475 per query (mean ≈ 52) — each an atomic pass/fail criterion. The reward is the query's rubric qualification rate:

$R = \frac{\text{rubrics satisfied}}{\text{total rubrics}} \in [0, 1]$

Grading reimplements the official FrontierFinance grader (judge prompt used verbatim; rubrics judged in batches of 30 per call), with one deviation: we use a single gpt-5-mini judge, whereas the official evaluation takes a majority vote over a three-judge panel. Must-have rubrics (7,487 of 11,543) do not weight the reward — as in the official metrics — but the must-have qualification rate and per-rubric verdicts with reasons are returned in the tool metadata.

Data

Sourced from the samaya-ai/FrontierFinance dataset on Hugging Face (frontier_finance_public.jsonl): 220 queries with expert-authored rubrics, each rubric annotated with must-have status, rubric type, and required data-source type. Data files are hosted on the OpenReward platform.

Queries are anchored to past dates, spanning 2024-08-08 to 2026-04-30. Retrieval is not bounded to the query date, so the agent can encounter information published after it: the prompt instructs it to answer as of that date and the judge anchors temporal interpretation there, but neither is enforced against the search index. How much later material is reachable therefore grows as wall-clock time passes, which is worth knowing before comparing scores gathered months apart.

This matches the official benchmark rather than departing from it. Upstream ships a grader only, and its paper applies a query-date retrieval cut-off to Samaya's in-house harness alone; the public harnesses are given the date in the prompt and nothing more. The paper also declines a point-in-time corpus as "much smaller and less diverse than the open web", relying instead on the dataset excluding predictive queries to stay robust "even when data postdating the query date is accessible".

Bounding retrieval is therefore an option rather than a correction, and not a free one. The default backsearch backend accepts a cutoff, and setting one (a web_as_of property returning the task's date) works as documented — but it roughly doubles the rate of failed web_fetch calls, because a URL in the corpus at today's cutoff often has no backdated snapshot. Agents answer by searching and fetching about twice as much, which lengthens rollouts enough to push some into their wall-clock budget. Any attempt to close the leak should treat that as the problem to solve first, and should use several seeds per task when measuring: run-to-run spread on a single task is wider than the effect being looked for.

Tools

Search and fetch come from the OpenReward SDK's WebToolset (toolsets = [WebToolset]) rather than being implemented in this environment.

• web_search — search the web. Takes a query and optional allowed_domains or blocked_domains; returns a Links: list of {title, url, snippet} sources
• web_fetch — fetch the readable content of a URL. Takes a url and a prompt describing what to extract

Choosing a search backend

Which provider answers those two tools is configuration on the environment server, not code here, so swapping it needs no change to this environment:

Grading runs through a hidden @terminal tool rather than a tool the agent can
call: replying with a plain message ends the rollout, and that message text is
graded against the query’s rubric checklist.

Time Horizon

Multi-turn agentic research episodes ending in a single submission. In our test rollouts with gpt-5.2, episodes used between 7 and 26 tool calls depending on query breadth.

Environment Difficulty

Samaya AI describe FrontierFinance as the hardest open finance benchmark, with the best evaluated system achieving roughly 50% (see the announcement). In our two spot-check rollouts, gpt-5.2 with the tools above scored 0.22 and 0.44 on two low-rubric-count tasks.

Other Environment Requirements

• openai_api_key — required, for the gpt-5-mini grading judge.
• Search credentials — whichever the configured backend needs: api_key for the default backsearch backend, or tavily_api_key when the server runs with OPENREWARD_SEARCH_BACKEND=tavily. Both fall back to the server process environment (OPENREWARD_API_KEY / TAVILY_API_KEY) if not passed. An unconfigured backend surfaces as a tool error rather than failing session creation.

Safety

The agent performs read-only web research on public financial information and produces a text answer; it takes no real-world actions and handles no funds. Standard web-access considerations apply (the agent fetches live third-party content). Answers are research artifacts graded against rubrics, not investment advice, and models trained on this environment should not be treated as licensed financial advisors.

Citations

@article{zhang2026frontierfinance,
  title   = {FrontierFinance: A Benchmark for Measuring Frontier Intelligence of Finance Agents},
  author  = {Zhang, Yuhao and Koyluoglu, O. Ozan and Venkatesh, Thejas and Diehl Martinez, Richard and Bhatia, Vishank and Alidoust, Arash and Paranjape, Ashwin},
  year    = {2026},
  url     = {https://samaya.ai/blog/frontier-finance}
}