ToolMind-Web-QA
ToolMind-Web-QA
Description
ToolMind-Web-QA is an ORS environment for evaluating agents on complex, multi-hop question answering that requires web search and information synthesis. Questions are synthesized from Wikipedia entity-relation knowledge graphs and require multiple search steps to resolve. Agents use web search and URL fetching tools to find information before submitting their answer.
Capabilities
- Multi-hop reasoning across multiple web sources
- Web search query formulation
- Information extraction from web pages
- Synthesizing answers from multiple documents
Compute Requirements
This is a multi-turn environment with no sandbox. Agents interact through web search and URL fetching tools only.
License
Tasks
There is one split in this environment:
- Train: 6,801 complex multi-hop questions
Each task presents a question that requires searching and synthesizing information from multiple web sources to answer.
Reward Structure
This is a multi-turn environment with binary reward:
- 1.0 — Correct answer (semantically equivalent to the reference, as judged by gpt-5-mini)
- 0.0 — Incorrect answer
The agent uses web_search and fetch_url to research, then calls submit_answer with an explanation, exact answer, and confidence score. The LLM grader evaluates semantic equivalence using both the reference answer and the solution reasoning trace, accepting minor formatting differences.
Data
Data consists of a single JSONL file (syn_wikiqa.jsonl) containing 6,801 multi-hop questions. Each instance includes the question text, ground truth answer, and a solution reasoning trace showing the multi-hop logic required.
Source: Nanbeige/ToolMind-Web-QA
Tools
| Tool | Description |
|---|---|
web_search | Search the web using Tavily API. Returns titles, URLs, and snippets. |
fetch_url | Fetch the full content of a specific URL. |
submit_answer | Submit your answer with explanation, exact answer, and confidence score. |
Note that the fetch_url and web_search tools require Tavily, but are optional. If you want to use a different provider for search you can exclude these tools and use external tools instead.
Time Horizon
ToolMind-Web-QA is a multi-turn environment. Agents typically perform several web searches and URL fetches before submitting a final answer.
Environment Difficulty
The ToolMind paper evaluates models trained on this dataset across web research benchmarks:
| Model | GAIA | BrowseComp | BrowseComp-ZH | HLE |
|---|---|---|---|---|
| DeepSeek-V3.2 | 63.5% | 67.6% | 65.0% | 40.8% |
| MiniMax-M2 | 75.7% | 44.0% | 48.5% | 31.8% |
| GLM-4.6 | 71.9% | 45.1% | 49.5% | 30.4% |
| ToolMind-Web-3B | 67.0% | 17.4% | 30.8% | 24.8% |
Questions are synthesized from Wikipedia entity-relation knowledge graphs and require multi-hop reasoning to trace connections between entities.
Other Environment Requirements
- OpenAI API key: Required for LLM-based answer grading via gpt-5-mini
- Tavily API key: Required for web search and URL content extraction
Pass via secrets={"openai_api_key": "...", "tavily_api_key": "..."}.
Safety
Agents in ToolMind-Web-QA perform web searches to answer factual questions. The environment does not involve sensitive content or actions beyond information retrieval.
Citations
@article{toolmind2025,
author = {Nanbeige Lab},
title = {Nanbeige ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset},
journal = {arXiv preprint arXiv:2511.15718},
year = {2025},
url = {https://arxiv.org/abs/2511.15718}
}