SciTerminalTasks

Description

SciTerminalTasks is a training environment of scientific-computing workflows done in a terminal, built as a training-distribution counterpart to Terminal-Bench-Science: the same five domains (life, physical, earth, mathematical and engineering sciences), 19 of its 20 fields (all but formal mathematics, whose Lean proofs are better served by dedicated theorem-proving environments), the same kinds of research workflow (parameter recovery from instrument data, statistical inference under confounding, signal and image reconstruction, solvers and calibrators graded on hidden cases, optimisation under constraints, pipeline repair, model discovery) and the same verifier style (deterministic checks against hidden truth, binary reward), but with entirely new tasks. None of the benchmark's tasks or data is reused, and every generated task instruction is checked for similarity against the benchmark's instructions (maximum TF-IDF cosine 0.29, against the benchmark's own duplicate threshold of 0.80).

Each task is one seed of a family: a scientific workflow with a data generator whose hidden parameters are the ground truth. The agent receives realistic instrument or pipeline exports under /app/data, an expert-to-expert instruction stating the conventions of the field and the exact output schema, and a sandbox with a broad scientific Python stack. It must recover the hidden quantities (or write a program the verifier runs on hidden cases of the same kind) and write the results to the stated absolute paths. Every family stacks realistic confounders (drift, calibration nonlinearity, batch effects, outliers, gaps, instrument changes, censoring, unresolved components), and at least three naive analyses that ignore them are verified to miss a graded quantity by at least twice its tolerance, while a correct method passes with margin.

Capabilities

• Scientific data analysis and inverse problems across life, physical, earth, mathematical and engineering sciences
• Writing analysis code against instrument-style data files (CSV, JSON, NPZ, HDF5, NetCDF, FITS, sequence and structure formats)
• Choosing and implementing the right model for the field (forward models, estimators, solvers) and diagnosing artefacts
• Writing programs with a fixed CLI contract that must generalise to hidden cases
• Multi-step terminal work: exploring data, iterating on fits, validating outputs, respecting output schemas

Compute Requirements

Agents run in an OpenReward sandbox with 2 CPUs and 8 GB of RAM (4 GB for the lighter families; the size is set per family in its metadata). The sandbox image ghcr.io/generalreasoning/sciterm carries Python 3.12 with NumPy, SciPy, pandas, scikit-learn, statsmodels, SymPy, NetworkX, CVXPY, Astropy, healpy, xarray, netCDF4, h5py, lmfit, emcee, PyWavelets, Numba, scikit-image, OpenCV, scikit-fem, python-control, gsw, MetPy, CoolProp, Cantera, pyteomics, NiBabel, pydicom, MNE, Biopython, pysam, RDKit, pymatgen, ASE, MDAnalysis, rasterio, pyproj, Shapely, GeoPandas, ObsPy, lightgbm and Matplotlib (images/sciterm/requirements.txt lists the pins). A sciterm-ml variant adding CPU PyTorch is also built, but no current family requires it. The sandbox has no network access.

License

MIT.

Tasks

There are 1399 train tasks and 80 test tasks from 189 families:

Tiers: 706 standard and 693 hard tasks in train, 40 standard and 40 hard in test. 54 families ask for a program that the verifier runs on hidden cases; 135 ask for result files. Archetypes (families): inverse 64, program 47, inference 25, signal 22, optimisation 15, repair 9, discovery 7. Provenance (families): 160 synthetic, 17 semi-synthetic, 12 real.

Tasks are named <family>--<tier>--s<seed>. The standard tier stacks two or three disclosed confounders; the hard tier stacks more and coupled effects, typically leaves it to the agent to detect which are active, adds identification decisions (report exactly the items that apply) and, for program families, evaluates on hidden cases drawn under harder conditions. The test split holds every seed of 10 families (two per domain) that appear nowhere in train, so generalisation to unseen workflows can be tracked; the 70 Terminal-Bench-Science tasks remain an untouched external benchmark.

Twelve families (91 train tasks, slugs ending in -campaign) are campaign tasks built to come closer to the scope of a Terminal-Bench-Science task. Each couples several sub-analyses over many wells, batches, sensors, stations, patients or nights through shared calibrations and per-item applicability decisions. Their instructions run to 362-1,349 words (median 362 for the other families; the shorter campaign instructions point to a binding analysis contract in the task data), the seeds evaluated below have 26-237 graded checks each, and their generator modules are 646-1,723 lines long (median 392 for the other families). They were generated in stages: a design document first, then an implementation revised against the gates, then a stricter review that also requires a difficulty score of at least 4 out of 5.

Four campaign families came from a first pilot: characterising a chlorinated-solvent contaminated site, a multi-batch drug-substance stability study from raw HPLC exports, structural-health monitoring and model updating of a continuous bridge (these three synthetic), and quality control and homogenisation of a ten-station network of real daily climate records with injected faults. Three more are built on real data whose truth is an independent real reference, with the signals delivered unaltered: a multi-patient ambulatory ECG panel graded against the MIT-BIH expert beat and rhythm annotations, the analysis of overnight polysomnograms from a temazepam-versus-placebo crossover study graded against the technicians' sleep-stage scoring, and stage-discharge ratings and streamflow records for a panel of USGS stream gauges graded against the agency's approved daily discharge, annual peaks and withheld field measurements. Five more real-data campaigns from later batches are graded the same way against independent references: a tide-gauge network (NOAA's accepted harmonic constants and official tidal datums), a Kepler rotation-period survey (the McQuillan, Mazeh and Aigrain 2014 catalogue), a solar X-ray flare catalogue from GOES-16 irradiances (NOAA's official flare summary), ECG waveform delineation (cardiologists' boundary annotations of the QT Database) and seismic phase picking on the Westmorland swarm (the network analysts' manual picks). These eight were accepted only after a difficulty probe: gpt-5.6-sol at maximum reasoning effort attempted hard-tier seeds, an LLM audit classified every failed check as a scientific error or a task defect (ambiguous specification, unfair verifier, or a graded item too close to its decision threshold), and families were repaired until the remaining failures were scientific; when gpt-5.6-sol solved a hard tier, the family was revised to deepen it and probed again, within a budget of at most eight probe runs per batch.

Reward Structure

Reward is binary: 1.0 if every check passes, 0.0 otherwise, returned when the agent ends the rollout with a plain message (the grading tool is a hidden terminal tool; the agent never calls it). Checks are deterministic and task-specific: per-quantity relative or absolute tolerances, exact-set identification gates, metric floors (F1, RMSE, coverage) and consistency gates; per-check results are returned in the tool output metadata for analysis. No LLM graders are used.

Grading runs entirely on the environment server. The sandbox never contains the truth, the checks, the hidden cases or the generator code; for program families the hidden cases are uploaded only after the agent has finished and the agent's program is run as an unprivileged user with a time limit, and its outputs are graded server-side. Every family passed executable anti-shortcut gates before shipping: deterministic generation, a leak scan of all agent-visible bytes, an oracle solution passing every seed with at least twice the tolerance in hand, empty/schema-only/constant/prior-midpoint submissions scoring 0, random draws from the plausible prior scoring 0, and each of at least three named careless analyses (the naive method a non-expert would try) failing by at least twice the tolerance on the standard and hard seeds it is checked on. For program families a memorising program that replays the visible case must also fail. For campaign families the reference solution must also fail, on at least 80% of the seeds checked, when any one of its five to nine corrections is removed (by failing a gate or missing a tolerance by at least 25%), so every modelled effect matters to the reward; in most of them, alternative analyses that an expert could defend are also checked to still pass, and two seeds of the same tier may share at most 60% of their graded values, so a seed cannot be the same panel under new identifiers. Every family's source and instruction then passed a rubric review (adapted from the benchmark's own task-implementation rubric: scientifically grounded, well specified, genuinely difficult, sound verifier) run by an LLM reviewer; families with review findings were repaired and re-reviewed, and only seeds that pass every gate are shipped.

Data

Most families generate their data from hidden parameters, so the truth is exact by construction. 29 families are built on real public data, curated once at build time with pinned checksums (nothing is downloaded at run time and the sandbox never sees the original files). In semi-synthetic families the real record is the background and known signals, faults or effects are injected, so the truth is the injected values; in real families the truth is withheld real values or expert labels, computed by a rule stated in the instruction. Identifiers are anonymised and each seed draws a different subset of the records; for the three real-data program families we checked by fingerprinting that no hidden case shares a data segment with the records visible in the same task. The curated datasets are:

The per-task files (agent inputs, hidden cases, truth, specification) are stored on the OpenReward platform's file store for this environment; agent inputs are mounted read-only into the sandbox at session start and copied to /app/data.

Tools

Agents get the OpenReward CLI toolset (bash, glob, grep, ls, read, write, edit, multi_edit, todo_write) with /app as the working directory. There are no task-specific tools; the rollout ends when the agent replies without a tool call.

Time Horizon

The original families are short. In the first evaluation below, gpt-5.6-sol used a median of 18 tool calls per task (range 6-40), gpt-5.6-terra 13 (range 4-26) and gpt-5.2 9 (range 3-19) before ending the rollout, and the median session, including sandbox setup and grading, took under five minutes. In the re-measurement after the hard-tier rewrite the medians were 20 (range 11-38), 15.5 (11-30) and 11 (2-27) tool calls. Campaign tasks are longer: on the pilot campaigns gpt-5.6-sol at maximum reasoning effort used 37-88 tool calls (median 80.5) and 15-49 minutes per task including sandbox setup and grading, and gpt-5.6-terra at default effort 29-43 tool calls on the hard seeds; on the real-data campaigns gpt-5.6-sol used 50-172 tool calls and 13-95 minutes. All of these are still much shorter than Terminal-Bench-Science's multi-hour tasks.

Environment Difficulty

We measured difficulty on small samples rather than with a calibration sweep. Each model was driven by a plain OpenAI Responses loop over the CLI tools, through the environment server and real OpenReward sandboxes; a task counts as solved when its reward is 1.

First check (original families). 15 tasks from 15 families across four domains (8 standard, 7 hard), default reasoning effort, 80-turn cap:

With Terminal-Bench-Science's red-team prompt (the agent is told to make the verifier pass without doing the task), gpt-5.2 scored 0 / 5.

Because gpt-5.6-sol solved every sampled task, the hard tier of 34 of the 41 families generated before that measurement was afterwards rewritten (the other seven keep their original hard tier: five rewrites did not pass the gates and review, and the two hand-written exemplar families were not rewritten) with their standard tier left unchanged: which confounders are active is no longer disclosed and must be detected from diagnostics in the data, at least one identification decision is graded by an exact-set gate, the data are scaled to a multi-file analysis, effects are coupled, and program families are evaluated on hidden cases from a documented but shifted regime. Families generated afterwards used the same hard-tier targets.

Re-measurement after the hard-tier rewrite. 20 train tasks (6 standard, 14 hard) on the deployed environment, default reasoning effort, 80-turn cap. Runs that stalled in the model API are not counted (1 for gpt-5.2, 2 for gpt-5.6-terra, 3 for gpt-5.6-sol).

Pilot campaign families. One standard and one hard seed of each of the four pilot campaign families, 150-turn cap:

gpt-5.6-terra ended each hard campaign with some sub-analyses wrong (applicability sets, sequential degradation rates, temperature-frequency models, impurity identities). An earlier version of the contaminated-site family, which gpt-5.6-sol failed on both tiers because one applicability rule was ambiguous, was repaired before this measurement (the rule is now quantitative).

Real-data campaign families. Final released versions, gpt-5.6-sol at maximum reasoning effort, 150-turn cap:

Earlier versions of the tide, GOES, Kepler and phase-picking families were also solved by gpt-5.6-sol before they were deepened; the final Kepler and phase-picking versions were not probed (the probe budget was spent on the Kepler version that failed only on an output convention the instruction had not stated, since fixed, and the phase-picking probe was interrupted). The audited failures are scientific: a paced patient's beats labelled normal, ventricular beats labelled with 39% sensitivity on one patient, a missed bigeminy episode, a night staged at 50.5% epoch accuracy. Defects found in earlier versions (applicability items within one beat or one epoch of their thresholds, the same panel repeated across seeds) were repaired before release.

The environment therefore separates gpt-5.2, gpt-5.6-terra and gpt-5.6-sol. gpt-5.6-sol still solves almost every sampled task of the original families and the pilot campaigns, whereas Terminal-Bench-Science reports 22.4% for gpt-5.6-sol; it fails real-data campaigns on scientific grounds when the reference is an expert judgement on noisy signals (all probed seeds of the Holter panel, half of the sleep study's hard seeds) and solves those whose reference is a reproducible measurement (tidal constants, flare timings, waveform boundaries, stream discharge).

Other Environment Requirements

No external keys or services are needed: the sandbox has no network access and grading uses no external APIs.

Safety

Tasks are self-contained scientific analyses on generated or public data inside an isolated sandbox without network access; there is no interaction with external systems, people or real experiments. The environment rewards correct scientific reasoning and honest reporting of what the data support (including declaring that a sample shows no effect); it does not reward persuasion or speculation.

Citations

The benchmark this environment is modelled on:

@software{terminal_bench_science_2026,
  title     = {Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains},
  author    = {Dillmann, Steven and the Terminal-Bench-Science Team},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22110253},
  url       = {https://github.com/harbor-framework/terminal-bench-science}
}

This environment:

@dataset{GRSciTerminalTasks,
  author    = {General Reasoning Inc. Team},
  title     = {SciTerminalTasks},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/SciTerminalTasks}
}