TrialCast

Description

TrialCast asks a model to forecast the outcomes of real clinical trials. Each task presents a cohort of ~100 held-out trials that were registered before an information cutoff and resolved after it, together with a read-only, point-in-time snapshot of the entire ClinicalTrials.gov registry as it stood at that cutoff — 287k–391k studies across 35 relational tables, including per-arm adverse events, primary-outcome statistical analyses with p-values and effect sizes, participant flow, eligibility criteria, and 145k–159k historical trials with their outcomes already resolved and labelled. The agent explores that corpus however it likes, fits or reasons its way to a probability for each trial in the cohort, and writes them to a CSV. Reward blends calibration against the cohort's own realised base rate with rank discrimination, so a constant or random forecast scores exactly zero and only genuine per-trial signal pays.

The task is close in spirit to published trial-outcome prediction work — HINT and the TOP benchmark, SPOT, MEXA-CTP and TrialBench — but differs in shape: rather than being handed three curated modalities per trial, the agent gets the raw registry and has to decide for itself what is predictive, and it is scored on a whole cohort at once so that calibration is measurable rather than incidental.

Capabilities

• Exploratory data analysis over a large, unfamiliar relational corpus (35 joined tables, ~1.2–1.6 GB per snapshot)
• Feature engineering from mixed structured and free-text clinical data (eligibility criteria, endpoint definitions, MeSH terms, adverse-event tables)
• Probabilistic forecasting under a proper scoring rule — calibration, not just ranking
• Building and validating a predictive model in-sandbox, or reasoning case-by-case when the data is thin
• Point-in-time discipline: reasoning from what was knowable on a date, using date-pinned literature search for context
• Domain judgement about clinical development: phase, sponsor type, endpoint choice, trial design, therapeutic area

Compute Requirements

Sandbox is 2 CPU / 8 GB RAM (machine_size="2:8") with no network access. The corpus is a read-only bucket mount, not part of the image, so the sandbox needs no local storage for it.

License

MIT.

Tasks

264 cohorts, each a self-contained forecasting task over ~100 trials — 26,400 trial-level predictions in total. Cohort membership is disjoint within each (split, family), and the train and test splits share no trials at all.

Task ids are {family}_{split}_{index}, e.g. discontinuation_test_0031. Each cohort is sampled so its realised positive rate falls in [0.20, 0.55] with at least 20 positives, which keeps AUC and Brier stable at n ≈ 100. Roughly 45% of discontinuation and efficacy cohorts are narrowed to a single phase or therapeutic area rather than being a mixed portfolio, which changes which historical comparators are informative. Every phase_advance cohort is instead confined to a single 3-year registration era, for the reason given under Safety: without that, registration recency alone scored AUC 0.76 and the family reduced to reading a date.

The three families are three different definitions of "success", all derived mechanically from the registry:

• discontinuation — of the trials that actually started, which stop early (terminated or suspended) rather than completing? Named carefully: why_stopped is frequently benign ("Funding, portfolio re-prioritization", "Parent study CIS001 was completed"), so this measures operational discontinuation risk, not scientific failure. Trials with status WITHDRAWN are excluded from the family altogether — see Safety, where a leak audit shows why.
• efficacy — did the first-listed primary outcome's first statistical analysis report p < 0.05? Restricted to primary_purpose = TREATMENT and to superiority-type analyses, with pharmacokinetic and bioequivalence endpoints excluded, since a significant p-value on an AUC endpoint does not mean the drug worked. First-listed-first-analysis rather than a minimum over analyses, which would cherry-pick across co-primary endpoints.
• phase_advance — for a programme that has not advanced as of the cutoff, will a later-phase trial for the same (MeSH drug, MeSH indication) be registered in the next 4 years? The industry's own working definition of programme success. Anchoring on the cutoff and conditioning on survival is what keeps the answer outside the corpus; historical examples use the same rule at cutoff − 4 years, so their windows close before the cutoff. See Safety for the leak that forced this design.

Two information cutoffs, one per split: train 2019-06-30, test 2022-06-30. Every task in a split shares one corpus snapshot.

Reward Structure

Terminal and fully deterministic — no LLM grader is involved. The agent's closing plain message ends the rollout, and /workspace/predictions.csv is scored as it stands.

$\text{reward} = c \cdot \left( 0.5 \cdot \mathrm{clip}_{[0,1]}(\text{BSS}) + 0.5 \cdot \mathrm{clip}_{[0,1]}\big(2(\text{AUC} - 0.5)\big) \right)$

where, over the $n$ trials of the cohort with labels $y_i$ and realised base rate $\bar{y}$:

$\text{BSS} = 1 - \frac{\mathrm{BS}}{\mathrm{BS}_{\text{ref}}}, \qquad \mathrm{BS} = \frac{1}{n}\sum_i (p_i - y_i)^2, \qquad \mathrm{BS}_{\text{ref}} = \bar{y}(1 - \bar{y})$

and $c$ is coverage, the fraction of the cohort the agent predicted. Missing rows are filled with 0.5 and still count against $c$; probabilities are clipped to $[10^{-6}, 1-10^{-6}]$, and AUC is the rank statistic with ties taking half credit.

Referencing the Brier score against each cohort's own realised base rate is what makes the reward unfarmable, structurally rather than by a bolted-on penalty: a constant forecast has $\mathrm{BS} \geq \mathrm{BS}_{\text{ref}}$ and $\text{AUC} = 0.5$, so it scores exactly 0 — even a forecast handed the true base rate. Reward is floored at 0 and reaches 1 only for a perfect forecast.

The cohort's positive count is stated in the prompt, and this is deliberate. Cohorts are stratified to a balanced positive rate, so the cohort rate differs from the historical rate in the corpus — for efficacy the corpus rate is 0.57 while cohorts sit near 0.39. Without the disclosure an agent that correctly learned the historical rate was punished for a sampling artifact: a measured gpt-5.2 rollout on efficacy_test_0000 reached AUC 0.72 but scored Brier skill −1.12, losing the whole calibration half to a base-rate mismatch rather than to any failure of judgement. Disclosing the count costs nothing in farmability — a constant forecast at exactly the disclosed rate still scores 0, which sweep.py's oracle_base_rate policy verifies — and it makes the calibration term measure what it should: whether individual probabilities are placed well.

validate_predictions returns 0 reward and exists purely so a rollout is not lost to a formatting slip; it reports coverage and parse errors and reveals nothing outcome-derived.

Data

Derived from the CTTI AACT full flat-file export of ClinicalTrials.gov (snapshot 2026-08-28: 2.4 GB, 49 pipe-delimited tables, 600,576 studies). 35 tables are kept and converted to parquet. Data files are stored on the OpenReward platform; datagen/ rebuilds the whole bundle from the public AACT export.

CTTI asks that the source be acknowledged: Aggregate Analysis of ClinicalTrials.gov (AACT) Database. Clinical Trials Transformation Initiative (CTTI). Available at: https://aact.ctti-clinicaltrials.org/. The underlying registry records are US Government public domain (U.S. National Library of Medicine).

Each snapshot is reconstructed as of its cutoff by truncation, since the AACT export is a current snapshot:

Three independent redaction rules apply, because these facts became public at different moments. A study appears only if it was first submitted on or before the cutoff. Status, why_stopped and the completion dates survive only where an ACTUAL completion date precedes the cutoff, otherwise overall_status reads ONGOING_AS_OF_CUTOFF. Results-era fields and every row of a results table survive only where results_first_posted_date precedes the cutoff — a separate test, since a trial that completed in 2018 could post its results in 2020. Update timestamps are cleared where the record was touched after the cutoff.

Agent-visible vs held back:

• Mounted read-only at /data — trials.parquet (one row per study, 66 columns), outcomes_history.parquet (resolved labels for pre-cutoff trials, from the same label functions applied to the cohort), raw/ (35 AACT tables), README.md (a data dictionary generated from the snapshot's own contents).
• Uploaded into the sandbox at /workspace/targets.csv — the cohort's 35 whitelisted feature columns: phase, sponsor name and class, conditions and MeSH terms, interventions and types, full eligibility criteria, arm and masking design, endpoint text, geography, site counts, registration year-quarter. Enrollment is deliberately not among them (see Safety).
• Never leaves the environment host — cohort labels and NCT ids (truth/), reachable only via the platform mount on the server. The sandbox mount is narrowed with SandboxBucketConfig(only_dir=...) to a single corpus directory.

Tools

• bash, glob, grep, ls, read, write, edit, multi_edit, todo_write — the SDK CLIToolset, against a sandbox with Python and the scientific stack
• web_search, web_fetch — backdated search over the GR corpus, pinned to the task's cutoff. Served by the environment host rather than the sandbox, which is why they still work with block_network=True. Both refuse the trial-registry hosts outright
• validate_predictions — format and coverage check on /workspace/predictions.csv; zero reward, reveals nothing about outcomes
• submit — terminal, hidden from the model: the agent's first plain message ends the rollout and the predictions file is graded

Time Horizon

Measured over six rollouts against the deployed environment, three per model: 13–28 tool calls (median 18) and 3.5–64.8 minutes of wall clock (median 6.7). The spread is driven by modelling choices rather than by the environment — the longest run fitted cross-validated models over the full 104,201-row history and spent most of its hour inside bash, while the shortest fitted a single logistic regression on a subset.

Most calls are bash; the stronger runs also use edit and write to build a prediction script iteratively rather than re-running one-liners, and read to consult the corpus data dictionary first. One validate_predictions call typically precedes the closing message. Note the sandbox's default bash timeout is 300 s, which heavy fits exceed — the model can pass a longer timeout, and runs that do not lose a turn to exit code 124.

Human time is not measured and no figure is claimed for it.

Environment Difficulty

Measured on the shipped build. The reference baseline is constrained to the agent's own inputs — it reads the very targets.csv files the agent receives, so feature parity holds by construction, and leak_probe.py asserts it.

Per family on the test split:

Every family shows a normal generalisation gap between the cross-validated fit on history and the score on held-out cohorts. efficacy is the hardest and legitimately so — predicting whether a primary endpoint clears p < 0.05 from protocol features alone is hard in the published literature too, and the baseline reaches only AUC 0.576 there.

Where the headroom comes from. The two baseline rows see byte-identical inputs, so the gap between them is method rather than information. The weak row fits a LogisticRegression on a single-phase subset of the history with a small feature set — the pattern every early measured rollout followed — and scores a third of what a gradient-boosted model over the full history with engineered text features achieves. The concrete, learnable choices are: train on all the labelled history rather than one slice, engineer the free-text columns, use a model that handles interactions and missing values, and recalibrate onto the disclosed base rate.

Measured models, run against the deployed environment on this build, one rollout per family:

Means: gpt-5.2 0.1125, gpt-5.5 0.2357. gpt-5.5 takes efficacy and phase_advance by wide margins; gpt-5.2 edges discontinuation on reward (0.2059 vs 0.1836) purely on ranking, since it reached the higher AUC there (0.706 vs 0.659).

The clearer pattern is calibration, not ranking: gpt-5.2 posts a negative Brier skill score on every cohort (−0.234 to −0.415), meaning its probabilities were worse than simply predicting the disclosed base rate even where its ordering carried real signal — on discontinuation it scored the best AUC of any run and still lost the entire calibration half of the reward. gpt-5.5 is positive on all three. That is exactly the distinction the reward is built to make, and it is the largest single source of headroom for a model that already ranks reasonably.

Both models sit below the reference baseline overall (0.113 and 0.236 against 0.2285), with one exception worth noting: gpt-5.5 scored 0.3866 on efficacy, well clear of the baseline's 0.0902 on that family, so the hardest family for a generic gradient-boosted model is not the hardest one for a reasoning model.

Earlier gpt-5.2 and gpt-5.5 numbers from previous builds are deliberately not restated here — including a deployed gpt-5.5 run that scored 0.61 on phase_advance by exploiting the retrievability leak described under Safety. Those describe environments that no longer exist.

The blind-policy floor is established by sweep.py over all 264 cohorts:

Every constant scores identically zero, by construction. The residual ~0.02 on the random and ordering policies is sampling noise in AUC and Brier at n ≈ 100 — the ordering rows confirm that neither the hashed trial keys nor the row order in targets.csv carries label signal. FARM_GATE = 0.10.

Other Environment Requirements

No external API keys. The backdated web_search / web_fetch tools are served by the platform's own web service.

Safety

The dataset is public clinical trial registry data; it contains no personal or patient-level information, only aggregate trial records that sponsors are legally required to post.

The environment's main risk is reward hacking by looking up outcomes rather than predicting them, and it is closed off at five layers, each verified rather than assumed:

1. Targets are removed from the corpus. All 29,700 trials used as forecasting targets are deleted from both snapshots outright, so no join can recover them. Belt-and-braces on top of the truncation rules, so that a bug in resolution-date inference cannot become a leak.
2. The corpus is truncated to the cutoff. Verified per snapshot: no study registered after the cutoff, no post-cutoff results date, no results-table row for a study whose results were not yet public, and status redacted for every unresolved trial.
3. The targets file is a whitelist. 35 columns, checked to match exactly. No NCT id, no title, no exact date, nothing results-derived, and no field whose recorded value post-dates the cutoff — see the post-hoc leak below. Registry identifiers that trials use to cross-reference each other in free text ("participants who completed NCT03928184") are scrubbed, since they would otherwise be a lookup handle.
4. The sandbox has no network (block_network=True), so ClinicalTrials.gov cannot be reached from inside it.
5. The search tools refuse registries. web_search and web_fetch are pinned to the cutoff date and, additionally, block clinicaltrials.gov and the other national registries with a list the model cannot lift — date-pinning alone is insufficient because registry pages are rewritten in place, so a page first posted in 2018 can carry present-day trial status.

datagen/verify.py runs all of layers 1–3 as pass/fail checks, golden_tests.py re-asserts them offline (30 tests), leak_probe.py covers the value- and join-reconstructible classes, and local_rollout.py verifies layers 4–5 plus the only_dir boundary against a live sandbox.

Sandbox egress was re-tested directly against the deployed environment across every route worth trying — curl over http and https, wget, urllib, a raw socket, bash /dev/tcp, openssl s_client, git clone and pip install. All fail; git clone and wget return exit 124 and pip cannot fetch a package absent from the image. DNS resolves but no TCP connection succeeds. The platform data bucket is not anonymously readable, and neither truth/ nor /orwd_data is reachable from inside the sandbox.

A post-hoc leak found by audit, and how it was closed

An audit of the whitelist against the labels found a serious leak in an earlier build, worth recording because the class of mistake is easy to repeat. AACT keeps only the current value of enrollment, so for a trial that has since resolved it is the final actual accrual, not the figure planned at registration. Exposed as enrollment_planned, it was a near-deterministic label indicator: 100% of WITHDRAWN targets had enrollment ≤ 1 (a withdrawn trial never enrols anyone), and TERMINATED targets had median actual enrollment 23 against 63 for completed ones. Measured with the reference baseline, it was worth 0.31 reward on the discontinuation family — 61% of that family's score, more than every legitimate feature combined.

Two changes closed it:

• enrollment is no longer a target column at all. Emptying it was the first fix, but a column populated throughout the corpus history and null in targets.csv is a silent trap rather than a solved problem: a measured rollout read the data dictionary, used the column anyway, validated at AUC 0.726 on held-out history and scored AUC 0.517 on its cohort, for a reward of 0.017. Removing it turns that into a loud KeyError the agent fixes in one turn. The cost is nil — the reference baseline scores the same with and without it on the two families where it survived redaction (phase_advance 0.3581 vs 0.3524).
• Trials with status WITHDRAWN are excluded from the discontinuation family entirely, as neither positive nor negative. Their whole registry record is degenerate in correlated ways — zero enrollment, 3.2× the zero-site rate of trials still ongoing at the cutoff (31.6% vs 9.8%, measured on matched drug/biologic interventional trials), no described arms — and the pre-withdrawal state cannot be reconstructed. Those values identified withdrawn trials with ~88% precision, so patching columns one at a time would not have removed the class. The family now asks the cleaner question: of the trials that started, which stop early?

After both changes, no whitelist column shows a degenerate-value tell: the strongest remaining is n_facilities == 0 on 78 discontinuation targets at 25.6% positive against a 35.9% base rate, i.e. below base. datagen/verify.py and golden_tests.py assert the whitelist and the redaction; the degenerate-value scan is documented in DEPLOY.md and should be re-run after any change to TARGET_COLUMNS.

The corpus keeps its own enrollment — for a trial that resolved before the cutoff the actual accrual genuinely was public then, so removing it would be factually wrong. The prompt and the data dictionary both state the general rule that follows: build features from the columns trials.parquet and targets.csv have in common, because the corpus also carries actual_duration_months, were_results_reported and other outcome-derived fields that no target can have.

A second leak: the label being retrievable rather than predictable

The enrollment leak above was a column whose value encoded the outcome. phase_advance had a different and larger one: the label was reconstructible from the corpus by a join, with no single column at fault.

The family originally asked whether a later-phase trial appeared within 4 years of this trial's own registration. Because targets are registered before the cutoff, that put on average 48.8% of every target's window inside the mounted snapshot. The lookup "does a later-phase trial for this (drug, indication) already exist?" fired on 31.4% of targets at 99.6% precision and recovered 81% of all positives. A gpt-5.5 rollout against the deployed environment found it without being prompted — its own summary said it "added a point-in-time later-phase signal from the registry snapshot" — and scored 0.61.

The family now anchors its window at the cutoff and conditions on survival: targets are programmes that had not advanced by the cutoff, and the label is decided strictly in the 4 years after it, which is where the corpus ends. Historical rows use the identical rule at cutoff − 4 years so their windows close before the cutoff. The same lookup now fires on 0.0% of targets in both snapshots.

Fixing that exposed a second, non-leak problem underneath it. With the window anchored at the cutoff, how recently a trial was registered became overwhelming — advance rates run 11.1% for 1999–2009 registrations against 64.6% for 2020–2022 — so registration year alone scored AUC 0.763 across cohorts spanning 22 years of registrations. Confining each cohort to a single 3-year era drops that to 0.546, in line with discontinuation (0.538) and efficacy (0.575), which never had the problem.

The tell that surfaced it is worth recording: the reference baseline reported CV AUC 0.46 on the training history but 0.82 on the targets. A model that cannot predict its own training data cannot legitimately rank targets that well, and that impossibility is what pointed at the population mismatch.

A third leak, found by a reward-hacking audit: forgeable cohort keys

targets.csv identifies trials by an opaque trial_key rather than an NCT id, so that a model cannot look up outcomes it has memorised from public registry data — the one channel a point-in-time corpus cannot close, since trial results are published and widely discussed.

That anonymisation was breakable. The key was sha256(cohort_id | nct_id) truncated to 10 hex characters, and both inputs are public: the cohort id is printed verbatim in the prompt, and NCT identifiers enumerate a small dense space. Sweeping NCT00000001–NCT06000000 and matching hashes recovered all 100 target ids of a cohort in 5.0 seconds, verified against the held-out mapping.

This was not hypothetical. In the deployed gpt-5.5 rollout on efficacy_test_0000 — its highest-scoring run — the agent wrote and executed exactly this attack, sweeping three hash algorithms across six million candidate ids. It failed only because it guessed the preimage as the bare nct_id rather than including the cohort id, and reported found [].

Keys are now an HMAC under a 32-byte secret generated at build time and stored in truth/key_salt.hex, which is never mounted into the sandbox. The same sweep now recovers 0/100, while keys stay deterministic given the salt so rebuilds remain reproducible. golden_tests.py asserts both properties.

The residual risk, stated plainly

Hiding NCT identifiers stops a model looking a trial up by id, but it does not make targets anonymous: 89.4% of them are uniquely identifiable from sponsor + drug + condition + phase (98.1% narrow to three or fewer candidates). Clinical trial results are public and widely written about, so a sufficiently large model may simply remember how a well-known trial read out. This channel cannot be closed by stripping features — sponsor track record and drug identity are exactly what makes the task predictable in the first place — so it is monitored rather than eliminated.

recall_probe.py splits a rollout's predictions by linked-publication count, a notoriety proxy that is not among the columns the agent receives, and compares accuracy on well-known against obscure trials. It reports the excess over a recall-free reference, not a raw gap: well-published trials are genuinely easier to predict, and the gradient-boosted baseline — which has no memory of anything — already shows a +0.177 gap on this data. An absolute threshold would report recall for a model that has none.

No recall signature has been observed in the measured rollouts. The direct evidence is stronger than the statistical test: the one agent that tried to de-anonymise the cohort failed and said so, no rollout asserted an outcome it had not computed, and the highest-scoring run's self-validated AUC (0.650–0.659) came in below what it actually achieved (0.776) — a model reading memorised answers would not underestimate itself.

Label honesty

The discontinuation family measures whether a trial stopped early, which is mostly not a scientific verdict. Classifying the why_stopped text of the 32,272 terminated or suspended trials that give a reason: 37.6% cite enrollment or accrual shortfall, 17.0% funding, business or portfolio decisions, 5.9% COVID, 35.5% resist simple classification, and only 12.5% explicitly cite efficacy, safety or futility. The family is named and documented as operational discontinuation risk for exactly that reason, in the prompt as well as here — it is a genuinely predictable and commercially meaningful outcome, but it is not "the drug failed".

Citations

@dataset{GRTrialCast,
  author    = {General Reasoning Inc. Team},
  title     = {TrialCast: Clinical Trial Outcome Forecasting from a Point-in-Time Registry Corpus},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/TrialCast}
}

@misc{AACT,
  author       = {{Clinical Trials Transformation Initiative}},
  title        = {Aggregate Analysis of {ClinicalTrials.gov} (AACT) Database},
  howpublished = {\url{https://aact.ctti-clinicaltrials.org/}},
  note         = {Snapshot 2026-08-28}
}

@article{HINT,
  author  = {Fu, Tianfan and Huang, Kexin and Xiao, Cao and Glass, Lucas M. and Sun, Jimeng},
  title   = {{HINT}: Hierarchical interaction network for clinical-trial-outcome predictions},
  journal = {Patterns},
  volume  = {3},
  number  = {4},
  pages   = {100445},
  year    = {2022}
}