GeneTasks

Description

GeneTasks is a training environment of multi-stage computational-genetics analyses. Each task gives an agent a realistic, deliberately messy dataset, a brief experimental context, and a defined target estimand tied to a downstream scientific or clinical decision. To answer correctly the agent must explore the data, choose an appropriate analysis, and navigate several decision points — inferential forks where a plausible-but-wrong choice (skipping a QC step, wrong allele orientation, ignoring a batch/ascertainment/immortal-time artifact) moves the answer by a clear margin.

Every task's data is fully simulated from a known data-generating process, so the graded answer is the value the correct analysis recovers from the realized data, and grading is deterministic — no LLM judge. Tasks span four domains: statistical genetics, clinical / carrier genetics, population genetics, and quantitative / pharmacogenomics.

Capabilities

• End-to-end quantitative analysis from raw, messy genetic data to a decision-relevant answer
• Recognising and correcting data-quality and study-design artifacts (QC filtering, allele
  harmonization, ancestry stratification, ascertainment, immortal-time bias, sequencing error)
• Multi-stage statistical reasoning where an upstream mistake propagates to the final estimate
• Iterative, tool-driven data exploration in a scientific-Python sandbox

Compute Requirements

Agents run in a sandbox with 2 CPUs and 2 GB of RAM (generalreasoning/python-ds:3.12-tools), which provides the standard scientific-Python stack (numpy, pandas, scipy, scikit-learn, statsmodels). The problems are designed not to require domain-specific bioinformatics tooling; network access is enabled so an agent may install any additional package it wants.

License

MIT.

Tasks

The suite contains 173 tasks in a single train split, in two difficulty tiers across four domains. The base tier (49 tasks, six families) is one clean end-to-end analysis with 3–4 decision points each. The hard tier (124 tasks, eight _hard families, 5–6 decision points each) demands several stacked corrections that a plausible-but-lazy pipeline gets wrong; the prompts are minimum-viable (they name the data and estimand but do not spell out which corrections are needed, so the model must discover them). Five hard families extend a base family with two extra decision points; two more (statgen_ldsc_hard, popgen_admix_hard) are standalone problem types added for diversity.

Each family is generated at many seeds; the difficulty auto-tuner drops seeds that will not separate cleanly, a near-duplicate filter drops any seed whose answer repeats an existing task, and a farmability guard ensures no constant-output policy passes more than a small fraction of a family — so survivors per family vary, and the generator supports arbitrary scale-up.

Base tier

Hard tier. The first five extend a base family with two extra stacked decision points (same estimands as the base family); the last two are standalone new problem types.

Task id format: train/<domain>/<family>_seed<N>. Base-tier tasks carry 3–4 decision points and hard-tier tasks 5–6 (GeneBench-Pro reports a median of 6 across its 129 problems).

Reward Structure

Grading is deterministic and needs no LLM. The agent submits a JSON answer of estimand fields; each field is checked against the task's known ground truth:

• numeric field: passes iff abs_error ≤ absolute_tolerance or rel_error ≤ relative_tolerance
  (tolerances are calibrated per task to admit reasonable analytic variation while excluding every
  plausible-wrong pipeline), within any stated sanity bounds;
• integer field: exact equality; categorical field: exact (case-insensitive) match within the
  stated choices.

The estimand fields correspond to the task's cascade stages, with weight increasing along the cascade.

$\text{reward} = \frac{\sum_i w_i \cdot \mathbb{1}[\text{field}_i\ \text{passed}]}{\sum_i w_i} \in [0, 1]$

This weighted-fraction reward gives partial credit for how far through the multi-stage analysis the agent got right (denser than a pure pass/fail signal, which suits RL training). The GeneBench-Pro-style binary all-fields-pass result is also reported in the tool metadata (all_required_pass) for evaluation. We do not use LLM graders for this environment.

Resistance to reward hacking. Because the reward is a numeric-tolerance check, a lazy policy that ignores the data and emits a constant or population-prior answer must not be able to farm reward. The generator enforces this: every task within a family has a numerically distinct answer (near-duplicate seeds are dropped), multi-stage estimands are checked so a task never collapses to "submit the first stage's value for every field", and per-seed parameter ranges are wide enough that the answers spread across many tolerance-widths. A generation-time guard confirms that no single fixed answer passes more than a small fraction of any family (see datagen/adversarial_probe.py).

Data

All data is synthetically generated by datagen/ — there is no external dataset to attribute. Each simulator is grounded in real genetics (real gene / panel names, gnomAD-scale allele frequencies, standard summary-statistic / genotype / time-series file schemas) and a literature-defensible estimator (Hardy-Weinberg and Bayesian residual risk; LD-aware IVW cis-Mendelian randomization; Wright-Fisher genic-selection MLE; Falconer variance-components heritability; new-user Cox proportional-hazards). The agent-visible files (*.tsv.gz + data_dictionary.tsv) are staged into the sandbox; the host-only eval_config.json (ground truth + grader) and meta.json never enter the sandbox. Data files are hosted on the OpenReward platform, not in git.

See DATA_UPLOAD.md for the layout and how to (re)generate the dataset.

Tools

Agents have the standard CLIEnvironment toolset — bash, read, write, edit, multi_edit, glob, grep, ls, todo_write — for exploring and analysing the staged files, plus one environment-specific tool:

• submit(answer, reasoning) — submit the final estimand object and a brief free-form reasoning
  string (the reasoning is collected but not graded); this grades the answer and ends the task.

Time Horizon

Each task is a single multi-turn analysis session: the agent iteratively inspects the staged files, runs its own analysis code in the sandbox, and ends the task with one submit call.

Other Environment Requirements

There are no external service dependencies or API keys required for grading (it is fully deterministic). The only requirement is the OpenReward sandbox for the agent's compute environment.

Safety

GeneTasks contains only synthetic, simulated data — no real patient, cohort, or biobank records — and the identifiers (samples, variants, cohorts) are fabricated. Tasks ask the agent to perform standard statistical-genetics analysis and report numeric estimates; they present no direct safety risk. Because the tasks mirror real clinical-genetics reasoning (e.g. carrier-screening residual risk, pharmacogenomic dosing signals), outputs from an agent trained here should not be used for actual clinical decision-making without appropriate validation.

Citations

The GeneTasks environment itself (synthetic simulators, deterministic grader, and data):

@dataset{GRGeneTasks,
  author    = {General Reasoning Inc. Team},
  title     = {GeneTasks},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/genetasks}
}