BrowseComp

Description

BrowseComp is an environment for evaluating web search reasoning capabilities. Based on OpenAI's simple-evals benchmark, it contains 1,266 encrypted research questions that require multi-hop reasoning and cannot be answered without current web information. The environment provides built-in web search and URL fetching tools powered by OpenReward's backdated search corpus (backsearch), with the cutoff set to the UTC date each session starts.

Capabilities

• Multi-hop web search reasoning
• Information retrieval and synthesis
• Research question answering
• Confidence calibration

Compute Requirements

Agents are given a standard environment with no sandbox or file system access.

License

MIT

Tasks

There is one split in this environment:

• test: 1,266 encrypted research questions

Questions require multi-hop reasoning across multiple web searches. Example: "What was the name of the 1995 film starring the actress who played Victoria that married the final scorer of the World Cup 1998 winner?"

Reward Structure

This is a sparse reward environment with LLM-based grading:

1. Agent receives a research question
2. Agent uses web_search and web_fetch tools to gather information
3. Agent submits answer with explanation, exact_answer, and confidence
4. An LLM grader (glm-5.2) evaluates semantic equivalence
5. Binary reward: 1.0 if correct, 0.0 if incorrect

Data

Data is sourced from OpenAI's BrowseComp benchmark. Data is stored on the OpenReward platform.

Tools

Note that the web_fetch and web_search tools require an OpenReward API key, but are optional. If you want to use a different provider for search you can exclude these tools and use external tools instead.

Optional: SerpAPI search provider (provider comparison)

Setting BROWSECOMP_SEARCH_PROVIDER=serpapi moves both web tools off backsearch: web_search makes one SerpAPI Google request per call, and web_fetch fetches pages live (trafilatura for HTML, pypdf for PDFs) with a Wayback Machine fallback when a site blocks the request. Neither tool falls back to backsearch. Output envelopes are unchanged. Search hits and fetches that would leak BrowseComp questions or answers are blocked and logged. A two-level cache (in-process, plus an optional persistent BROWSECOMP_WEB_CACHE_URL of the form gs://bucket/prefix or a directory) means a repeated query is paid for once. The SerpAPI key comes from SERPAPI_API_KEY, or from a serpapi_api_key secret when that secret holds the real value; a platform secret token cannot be used because SerpAPI only takes the key as a query parameter. See live_web.py and web_cache.py for every setting. Unset, the env behaves exactly as described above.

Time Horizon

Multi-turn. Agents can perform multiple web searches before submitting a final answer.

Environment Difficulty

This benchmark requires persistent multi-hop web navigation to find hard-to-find, entangled information.

Other Environment Requirements

• OpenAI API key required for LLM-based grading
• OpenReward API key required for web search (falls back to the server's OPENREWARD_API_KEY)

Pass via secrets={"openai_api_key": "...", "api_key": "..."}.

Safety

Agents in BrowseComp perform web searches and answer research questions. The environment does not present direct safety risks. Note: Do not share decrypted questions publicly per OpenAI's request.

Citation

@article{wei2025browsecomp,
  title={BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents},
  author={Wei, Jason and Sun, Zhiqing and Papay, Spencer and McKinney, Scott and Han, Jeffrey and Fulford, Isa and Chung, Hyung Won and Passos, Alex Tachard and Fedus, William and Glaese, Amelia},
  journal={arXiv preprint arXiv:2504.12516},
  year={2025},
  url={https://arxiv.org/abs/2504.12516}
}