ScrapeBench

Description

ScrapeBench is an environment for evaluating an agent's ability to scrape structured data from websites. Given a natural language instruction describing what data to extract from a specific website, the agent must write code to scrape, parse, and structure the data into JSON or JSONL format. Tasks cover diverse domains including sports statistics and financial data.

Capabilities

• Web scraping using Python libraries (requests, BeautifulSoup, Selenium)
• Parsing and structuring web data into JSON/JSONL format
• Working with diverse web sources
• Developing robust data extraction pipelines

Compute Requirements

Agents are given a sandbox with 1 CPU and 2GB RAM, with network access enabled for web scraping. The sandbox includes Python with web scraping libraries (requests, BeautifulSoup, Selenium, lxml).

Tasks

There are two splits in this environment:

• Train: 29 web scraping tasks
• Test: 29 web scraping tasks

Tasks span sports statistics, financial data, and other structured web content.

Reward Structure

This is a multi-turn, sandbox-based environment. The agent writes scraping code, executes it, and submits results as submission.json or submission.jsonl. A task's ground truth file extension is the contract, and the prompt states which file and shape the task expects. Reward is continuous (0.0 to 1.0) based on deterministic algorithmic matching:

• List ground truth (a ground_truth.jsonl, submitted as a submission.jsonl of one object per line): bipartite matching with a 60% similarity threshold, reward = matched entries / total ground truth entries. Order-agnostic.
• Object ground truth: field-by-field comparison, reward = mean per-field score over the ground truth's keys. A key the submission omits scores 0.
• Per-field scoring: exact match = 1.0; strings by difflib.SequenceMatcher ratio; numbers within 10% relative error decay linearly to 0; nested objects recurse field by field; nested lists use the same bipartite matching as above.

No LLM graders are used.

Tests

The scoring logic is pure and unit-tested against the real ground truth in tasks/:

pip install -r requirements-dev.txt
pytest test_reward.py

Data

Ground truth data consists of reference JSON/JSONL files for each task stored on the OpenReward platform. Tasks reference live websites that agents must scrape.

Tools

Agents get CLI tools (bash, read, write, edit, multi_edit, grep, glob, ls, todo_write) plus 1 environment-specific tool:

Time Horizon

ScrapeBench is a multi-turn, sandbox-based environment. Agents read the scraping instructions, write code, test it, and submit structured results.

Environment Difficulty

[Put environment difficulty statistics here]

Other Environment Requirements

There are no further environment requirements; ScrapeBench works out of the box with the OpenReward endpoint without any external API keys.

Safety

Agents in ScrapeBench scrape publicly available websites in a sandboxed environment. The environment does not present direct safety risks.

Citation

@dataset{GRScrapeBench,
  author    = {General Reasoning Inc. Team},
  title     = {ScrapeBench},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/ScrapeBench}
}