AnotherDeepResearch

Description

AnotherDeepResearch is a web-research environment. Each task is a real question that originally required live web search or a Deep Research run to answer. The agent is given web search tools, researches the question, and submits an answer, which is graded against a per-task rubric of specific factual and coverage criteria.

Tasks come in two types:

• search — quick web-research Q&A: the question is a real first-turn query whose original answer required live web search; a direct, well-supported answer is expected.
• research — Deep Research tasks: the prompt is a real research request with the clarification dialogue folded in (the original request plus the Q&A exchange that preceded the research run); a comprehensive, well-structured report is expected and is graded against a larger rubric distilled from the original research report.

A human review pass selects which candidates become eval tasks.

Capabilities

• Backdated web search and page retrieval (OpenReward's backsearch archive as of the session's start date)
• Multi-step research and evidence gathering
• Synthesising accurate, well-supported answers to open-ended questions
• Producing comprehensive, report-style research syntheses
• Grounding factual claims against retrieved sources

Compute Requirements

AnotherDeepResearch runs as an env-only server (no sandbox). It requires outbound network access for the OpenAI grader and OpenReward's backsearch web service.

Tasks

There is a single train split with 410 tasks: 363 search and 47 research (task_type field). Each task provides a research question and a hidden rubric (a list of binary criteria with points) used for grading. Tasks are built by the offline pipeline in this repo, human-reviewed in annotate.py, and stored in anotherdeepresearch.parquet.

Reward Structure

When the agent finishes with a plain-text reply, an LLM judge (gpt-5-mini) evaluates the whole reply against each rubric criterion independently, returning met / not-met. The reward is:

$\text{reward} = \frac{\sum \text{points of met positive criteria} - \sum \text{points of met penalty criteria}}{\sum \text{points of all positive criteria}}$

clamped to [0, 1]. Positive criteria reward required facts and coverage; optional negative criteria penalise common factual errors. Search tasks have rubrics of roughly 5–12 criteria; research tasks have larger rubrics (roughly 10–20 criteria) covering the key facts, figures and coverage areas of the original report.

Tools

• web_search(query, allowed_domains?, blocked_domains?) — searches OpenReward's backdated web corpus as it stood on the day the session started; returns up to 8 hits with titles, URLs and snippets, fanned out over the backend's default corpora (news, SEC filings, Wikipedia, general web, live captures, arXiv).
• web_fetch(url, prompt) — returns the archived text of a page as it existed on or before the session's start date (up to 100 KB). A fetch that 404s on a percent-encoded URL is retried with the path decoded, because the archive stores Wikipedia titles in raw Unicode.

The tools come from a BackSearchToolset subclass pinned in code, so the backend cannot be switched to a live-web provider by an environment variable. The cutoff (web_as_of) is set once per session in the constructor to the UTC date the session starts and read by the toolset on every call. A paired A/B on 2026-09-21 (gpt-5.2, same environment code, only the backend flipped, 18 tasks per arm) scored backsearch 0.51 and Tavily 0.53, within noise, with fetch failure rates of 15% and 18% respectively.

Grading uses a hidden @terminal tool: when research is complete, reply with the final answer as an ordinary message. The whole reply is graded by a gpt-5-mini judge against the per-task rubric.

Time Horizon

Single-turn task: the agent researches with the tools and then replies with one final plain-text answer. The number of search/fetch calls per task is agent-dependent; research tasks typically require more extensive tool use than search tasks.

Other Environment Requirements

This environment requires one external API key, passed via secrets:

• openai_api_key — for the gpt-5-mini rubric grader.

Local Development

pip install -r requirements.txt

# Build the dataset (see DATA_UPLOAD.md for full pipeline)
python prepare_candidates.py --source /path/to/chatgpt-export.zip
python generate_rubrics.py
python annotate.py          # review + include/exclude in the browser
python build_dataset.py

# Run the environment
python server.py            # serves on http://0.0.0.0:8080

# In another shell
export OPENAI_API_KEY=sk-...
export OPENREWARD_API_KEY=or_...           # backdated web tools
python test_agent.py                       # first task
TASK_TYPE=research python test_agent.py    # first deep-research task

Citations

@dataset{GRAnotherDeepResearch,
  author    = {General Reasoning Inc. Team},
  title     = {AnotherDeepResearch},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/anotherdeepresearch}
}