PatentQATrain

Description

PatentQATrain is an ORS training environment for patent question answering, based on the PatentQA task from LAB-Bench-2. Agents are given questions about specific details from patents across diverse technology domains and must use web search to find and verify answers from Google Patents.

Capabilities

• Researching patent details using web search
• Extracting specific information from patent claims and specifications
• Navigating patent documents to find numerical thresholds, compositions, method steps, and claim elements
• Cross-domain patent understanding spanning pharmaceutical, biotech, chemistry, electronics, software, mechanical, materials, energy, medical devices, and telecom

Compute Requirements

No sandbox or special compute requirements. Uses external web search (Tavily API) for patent retrieval.

License

MIT

Tasks

There are 909 training tasks distributed across 10 patent domains:

Every task's key_passage is verbatim text from the patent at its source_url, and every
answer was checked against that patent by an LLM judge (99.6% supported). Patents where no
passage produced a verifiable question were dropped, which is why the counts sit below 100.

Expect some drift. Both discovery and reading go through a live web-search API, and Tavily's
extraction of a Google Patents page varies between calls — the same URL has returned 12k and
132k characters minutes apart. So a task that is answerable today can become unanswerable if
extraction stops returning the section its answer came from, independently of anything in this
repo. Verification here was done against extracts captured at generation time, which is the
optimistic side of that: a row proven verifiable then is not guaranteed reachable now.

Each task presents a question about a specific patent that requires distinctive specificity — it
asks about a verifiable fact uniquely traceable to one patent. Roughly 15% of questions cite the
patent number outright; the rest describe the invention and leave the agent to identify the
patent from that description, which is the harder and more realistic research task.

Reward Structure

Sparse, binary reward:

• 1.0 for correct answers (as judged by LLM grader)
• 0.0 for incorrect or unsure answers

Grading uses semantic equivalence checking: answers that are numerically/semantically equivalent are accepted, even if phrased differently. The grader is based on LABBench2's structured evaluation prompt.

We do not use exact string matching. The LLM grader (gpt-5-mini) evaluates whether the submitted answer captures the core factual content of the expected answer.

Data

Ground-truth data consists of QA pairs derived from Google Patents documents. Each task includes:

• A question about one specific patent, sometimes citing its number and otherwise describing the invention
• A concise expected answer (under 200 characters)
• The source patent URL
• A key passage from the patent supporting the answer

Data is stored on the OpenReward platform.

Tools

Agents have access to three tools:

Grading runs through a hidden @terminal tool rather than a tool the agent can
call: replying with a plain message ends the rollout, and that message text is
graded against the expected answer.

Time Horizon

PatentQATrain is a multi-turn environment. Agents typically perform several web searches and URL fetches before submitting an answer.

Environment Difficulty

[Statistics on environment difficulty here]

Other Environment Requirements

This environment requires the following API keys passed via the secrets parameter:

• openai_api_key: For LLM-based answer grading
• tavily_api_key: For web search and URL content extraction

Safety

PatentQATrain focuses on factual information retrieval from publicly available patent records. The environment does not involve intellectual property creation, legal advice, or access to non-public information. All source patents are publicly accessible via Google Patents.

Citations

This environment is inspired by the PatentQA task from LAB-Bench-2:

@misc{labbench2,
  author    = {Laurent, Jon M. and Bou, Albert and Pieler, Michael and Igoe, Conor and Andonian, Alex and Narayanan, Siddharth and Braza, James and Vassopoulos, Alexandros Sanchez and Steenwyk, Jacob L. and Lash, Blake and White, Andrew D. and Rodriques, Samuel G.},
  title     = {LABBench2: An Improved Benchmark for AI Systems Performing Biology Research},
  year      = {2026},
  url       = {https://github.com/EdisonScientific/labbench2}
}