JobBench

Description

JobBench evaluates agents on realistic white-collar professional tasks sourced from working professionals — the work those professionals actually want automated. Each task drops the agent into a Linux sandbox with a folder of heterogeneous reference materials (CSVs, spreadsheets, PDFs, Word documents, SQLite databases, images) and a TASK_INSTRUCTIONS.txt describing deliverables to produce, such as statistical analysis workbooks, legal memos, reconciliation reports, or engineering calculations. The agent saves its final deliverables to an output directory and submits them for rubric-based grading.

This environment reimplements the official JobBench evaluation harness (Job-Bench/job-bench-eval) on OpenReward, including its prompt wrapper, deliverable file extraction, and per-rubric pass/fail judging.

Capabilities

• Multi-format document analysis (xlsx, pdf, docx, csv, SQLite, images)
• Reconciling contradictory information across many sources
• Producing professional deliverables (workbooks, reports, memos) as files
• Locating public reference documents online (main split)
• Multi-step agentic execution with CLI tools in a sandbox

Compute Requirements

Agents are given a sandbox with 2 GB of RAM and 2 CPUs (generalreasoning/knowledge-worker image) with network access enabled.

License

The JobBench dataset is released under the MIT license; the official evaluation code is Apache-2.0. This environment follows the dataset's MIT license.

Tasks

128 tasks across 35 professions (biostatisticians, lawyers, civil engineers, court clerks, financial managers, reporters, supply chain managers, ...), in three test splits:

• main — 65 full tasks. Some reference documents are public materials the instructions say to obtain separately (e.g. FDA guidance PDFs); as in the official harness, these are not placed in the workspace and the agent may retrieve them online (the sandbox has network access). The dataset's ground-truth copies (files_required_to_search/) never enter the sandbox.
• easy — 63 simplified tasks with shorter prompts and no external-document requirement.
• all — the union of both (128 tasks).

Each task provides /workspace/task_folder/ (reference files + instructions) and an empty /workspace/output/. The agent works freely, saves final deliverables to the output directory, and calls submit() (no arguments) exactly once to be graded.

Reward Structure

Grading reimplements the official JobBench judge. Each task ships 6–21 weighted rubrics (mean ≈ 9 on main, ≈ 11 on easy), each with a list of fact-anchored binary criteria. Every file the agent saved to the output directory is extracted to text (spreadsheets via pandas, PDFs via pdfplumber, Word via mammoth, PowerPoint via python-pptx, SQLite via schema + row dumps, notebooks via cell/output parsing; images are attached to vision-relevant rubrics). One LLM judge call is made per rubric; a rubric passes only if all of its criteria pass, and:

$\text{reward} = \frac{\sum_{r \,\in\, \text{passed}} w_r}{\sum_{r} w_r} \in [0, 1]$

Judging uses gpt-5-mini. Parsing is fail-closed: an unparseable judge response fails the rubric, and the rubric verdict is recomputed from the per-criterion results. An empty output directory or a session that never submits scores 0.

Data

Tasks are sourced from the JobBench/job-bench dataset on Hugging Face (~624 MB), which mirrors the layout used by the official harness: dataset/<profession>/taskN/ (main) and dataset_easy/<profession>/taskN/ (easy), each with a task_folder/ of reference files, RUBRICS.json, and a task card. Task data lives on the environment host; only task_folder/ contents are uploaded into the agent sandbox — rubrics and ground-truth search materials stay host-side. For local use, run python download_data.py.

Tools

Agents get the standard CLI toolset (bash, glob, grep, ls, read, write, edit, multi_edit, todo_write) plus the environment's submit() tool. Binary formats are handled with bash + Python inside the sandbox (the agent may pip install document libraries as needed).

Time Horizon

Tasks are medium-horizon, multi-turn file-production jobs: the agent must read several reference files, perform analysis or drafting, write one or more deliverable files, and submit. The official harness allots up to 60 minutes of wall-clock per task per agent.

Other Environment Requirements

An OpenAI API key (secret key: openai_api_key) is required for the rubric judge.

Safety

Tasks are ordinary professional knowledge work over provided reference files, with no counterparties or real-world side effects. The sandbox has network access (used on the main split to retrieve public reference documents such as regulatory guidance); agents are instructed to access only the workspace and online reference materials. Deliverable text is embedded in judge prompts, so judging is fail-closed and rubric verdicts are recomputed from per-criterion results to resist prompt injection via deliverables.

Citations

@article{li2026jobbench,
  title   = {JobBench: Aligning Agent Work With Human Will},
  author  = {Li, Yuetai and Feng, Yichen and Xu, Zhangchen and Ma, Zixian and Zheng, Kaiyuan and others},
  journal = {arXiv preprint arXiv:2605.26329},
  year    = {2026},
  url     = {https://job-bench.github.io}
}