JobTasks

Description

JobTasks is an environment of synthetic professional white-collar tasks. Each task drops the agent into a Linux sandbox with a folder of real, downloaded public data files (CSV/xlsx from open-data portals, plus companion PDFs) and a TASK_INSTRUCTIONS.txt describing professional deliverables to produce (analysis workbooks, memos, reports). The agent saves deliverables to an output directory and calls submit() for rubric-based grading.

Capabilities

• Multi-format document and data analysis (CSV, xlsx, PDF)
• Multi-step quantitative work: filtering, grouping, derived metrics, reconciliation
• Producing professional deliverables (workbooks, memos) as files
• Multi-step agentic execution with CLI tools in a sandbox

Compute Requirements

Agents are given a sandbox with 2 GB of RAM and 2 CPUs (generalreasoning/knowledge-worker image) with network access enabled.

License

MIT.

Tasks

One train split with 583 tasks across the same 35 O*NET-style professions as JobBench (biostatisticians, lawyers, civil engineers, financial managers, supply chain managers, ...; 5–48 tasks per profession, most at 12–18). Each task provides /workspace/task_folder/ (reference files + instructions) and an empty /workspace/output/; the agent works freely, saves final deliverables to the output directory, and calls submit() (no arguments) exactly once to be graded. Tasks are self-contained — everything needed is in the task folder. Tasks average ~10 rubrics (≈6.5 fact-anchored) and ~2.8 MB of reference data.

Quality & Verification

Beyond the in-pipeline gates, the released set went through multi-agent verification: every task's ground-truth script re-executes deterministically against its frozen files (full-dataset sweep); a 65-task stratified sample was independently audited by verification agents that re-derived rubric-cited values from the raw data with pandas — confirmed-defective tasks (unstated grading conventions, degenerate screens, ground-truth bugs) were removed from the release, and one mechanical artifact class (sig-fig-rounded values pinned in criteria) was neutralized dataset-wide by hedging affected criteria. Blind realism judging against real JobBench tasks drove several generator revisions (stated house rules for every graded threshold, judgment-focused memos, varied deliverables). Residual imperfection to expect in a minority of tasks: one or two rubrics whose exact counts depend on defensible methodology choices; the weighted multi-rubric reward keeps this as mild noise rather than inverted signal.

Reward Structure

Each task has weighted pass/fail rubrics (weights 6–10), each with a list of fact-anchored binary criteria. Every file the agent saved to the output directory is extracted to text and one gpt-5-mini judge call is made per rubric; a rubric passes only if all its criteria pass, and

$\text{reward} = \frac{\sum_{r \,\in\, \text{passed}} w_r}{\sum_{r} w_r} \in [0, 1]$

Parsing is fail-closed and rubric verdicts are recomputed from per-criterion results. An empty output directory or a session that never submits scores 0.

Data

Reference files are real public data downloaded at generation time from open-data portals (Socrata portals such as data.cdc.gov, data.cms.gov, data.bts.gov, and the data.gov catalog), frozen into each task's task_folder/ so grading facts cannot drift. Host-side artifacts (RUBRICS.json, meta.json, task_card.md, ground_truth/) never enter the agent sandbox. See DATA_UPLOAD.md for layout and regeneration.

Tools

Agents get the standard CLI toolset (bash, glob, grep, ls, read, write, edit, multi_edit, todo_write) plus the environment's submit() tool. Binary formats are handled with bash + Python inside the sandbox (the agent may pip install document libraries as needed).

Time Horizon

Medium-horizon multi-turn file-production tasks: read several reference files, perform analysis, write one or more deliverable files, submit once.

Other Environment Requirements

An OpenAI API key (secret key: openai_api_key) is required for the rubric judge.

Safety

Tasks are ordinary professional knowledge work over provided public reference files, with no counterparties or real-world side effects. Deliverable text is embedded in judge prompts, so judging is fail-closed and rubric verdicts are recomputed from per-criterion results to resist prompt injection via deliverables.

Citations

JobTasks reimplements the task format of the JobBench benchmark with new synthetic training tasks:

@article{li2026jobbench,
  title   = {JobBench: Aligning Agent Work With Human Will},
  author  = {Li, Yuetai and Feng, Yichen and Xu, Zhangchen and Ma, Zixian and Zheng, Kaiyuan and others},
  journal = {arXiv preprint arXiv:2605.26329},
  year    = {2026},
  url     = {https://job-bench.github.io}
}

@dataset{GRJobTasks,
  author    = {General Reasoning Inc. Team},
  title     = {JobTasks},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/jobtasks}
}