SpreadsheetBench

Description

SpreadsheetBench is an environment for evaluating LLM agents on real-world spreadsheet manipulation tasks. It comprises 905 instructions (from 912 original, 7 excluded due to broken metadata) sourced from online Excel forums, each with multiple test cases (typically 3) to ensure solution generality via OJ-style evaluation.

Capabilities

• Reading and analyzing Excel spreadsheet structure
• Writing Python code to manipulate spreadsheet data
• Handling cell-level and sheet-level operations
• Producing general solutions that work across multiple test cases

Compute Requirements

• Sandbox: 0.5 CPU / 1 GB memory per session
• Network access: enabled (not blocked)
• No GPU required

License

CC-BY-SA-4.0

Tasks

Single test split with 904 tasks. Each task has 1–3 test cases (~2,700 total) with different spreadsheet values but the same instruction. Tasks are divided into:

• Cell-Level Manipulation (560 tasks): modifying specific cells or ranges
• Sheet-Level Manipulation (345 tasks): modifying entire sheets, cross-sheet operations

One task (189-9) is discarded and never served — see DISCARDED_TASK_IDS in spreadsheetbench.py. It asks for timestamps relative to TODAY(), but its answer file was frozen against the real "yesterday" at dataset-build time and nothing pins a reference date in the sandbox, so an agent reading the live clock can never match it however correct its work is.

Reward Structure

Binary reward (0.0 or 1.0). The agent's Python script is executed on each test case input file. Cell values at the specified answer_position are compared against ground-truth answer files. All test cases must pass for reward=1.0 (OJ-style hard metric).

A failure of the grading machinery itself — a missing or unloadable answer file, or an output file that exists in the sandbox but cannot be downloaded — raises InfraError instead of scoring 0.0, so a couldn't-grade is never recorded as a wrong answer. Failures that genuinely belong to the agent still score 0.0: a script that exits nonzero, writes no output file, writes an unreadable workbook, or writes the wrong cell values.

Data

• Source: KAKA22/SpreadsheetBench on HuggingFace
• Format: Excel .xlsx files with JSON metadata
• Size: ~91 MB compressed
• Input spreadsheets mounted read-only at /data/ in the sandbox

Tools

• bash — Execute bash commands in the sandbox for writing code and testing
• submit — Submit a Python script for OJ-style evaluation across all test cases
• excel_list_tabs_in_spreadsheet — List all worksheet names
• excel_read_tab — Read data from a specific worksheet
• excel_read_csv — Read CSV files
• excel_create_spreadsheet, excel_add_tab, excel_edit_spreadsheet, excel_add_content_text, excel_delete_content_cell, excel_create_chart, excel_delete_tab, excel_delete_spreadsheet — Full Excel manipulation via the ExcelToolset

Time Horizon

Multi-turn. Agents typically explore the spreadsheet, write a solution script, test it, and submit. Average interaction involves 5–15 tool calls.

Environment Difficulty

The original benchmark reports ChatGPT Agent achieving 45.5% task success rate, indicating substantial difficulty. Tasks range from simple cell extraction to complex multi-sheet operations.

Safety

Tasks involve spreadsheet data manipulation only. Input data is sourced from public Excel forum questions. No personally identifiable information or sensitive data.

Citations

@inproceedings{ma2024spreadsheetbench,
  title={SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation},
  author={Ma, Zeyao and Zhang, Bohan and Zhang, Jing and Yu, Jifan and Zhang, Xiaokang and Zhang, Xiaohan and Luo, Sijia and Wang, Xi and Tang, Jie},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}