HealthBench

Description

HealthBench is an environment for evaluating agents on healthcare conversations. It is based on the HealthBench benchmark from OpenAI, consisting of 5,000 conversations between a model and a user or healthcare professional, 42% of them multi-turn. Each conversation is graded against physician-created rubric criteria spanning diverse health contexts and behavioral dimensions. An LLM grader (gpt-4.1-2025-04-14) evaluates each rubric item independently.

Capabilities

• Answering healthcare questions across diverse medical contexts
• Multi-turn medical dialogue comprehension
• Handling safety-critical health scenarios
• Demonstrating accuracy, communication quality, and instruction following

Compute Requirements

HealthBench does not require a sandbox. It has minimal compute requirements.

License

MIT.

Tasks

There is one split: test (5,000 tasks). Each task presents a conversation between a user and an assistant, always beginning and ending with a user message; 2,915 tasks are a single user message and 2,085 (41.7%) are multi-turn, up to 19 messages deep. The agent is shown the whole conversation and writes the next assistant reply, which is graded against that task's rubric criteria (48,562 unique criteria across 57,237 rubric-item rows, median 11 per task). Each rubric item has an associated point value and tags for categorization.

Reward Structure

This is a sparse reward environment with continuous scoring. The agent replies with its response as an ordinary message; the environment grades the whole reply using an LLM grader (gpt-4.1-2025-04-14). Each rubric item is graded independently for whether its criterion is met. The overall score is the signed points of the met criteria over the total available positive points:

$\text{Reward} = \frac{\text{achieved points}}{\text{total possible points}}$

Rubric items carry negative point values as well as positive ones, and 93.6% of tasks contain at least one negative item, so a single task's reward can fall below zero — as low as -8.67 on this dataset. The published HealthBench metric clips the mean across tasks to [0, 1]; per-task scores are left unclipped, as in the reference implementation.

We do not use LLM graders from the gpt-5-mini family for this task. Grading matches the reference HealthBench configuration: gpt-4.1-2025-04-14, temperature 0.5, max_tokens 2048, and a "You are a helpful assistant." system message.

Data

Conversations are sourced from the HealthBench benchmark by OpenAI, which includes 5,000 conversations with 48,562 unique rubric criteria created by 262 physician evaluators. Data files are stored on the OpenReward platform.

Tools

Agents are given no visible tools. The environment uses a @terminal tool: the agent's final plain-text message is the answer, graded by an LLM judge (gpt-4.1-2025-04-14) against the rubric criteria.

Time Horizon

The agent takes a single action: it receives the conversation and replies with one plain-text message, which is graded. The conversation itself may be multi-turn, but the agent contributes only that final reply.

Environment Difficulty

Model performance on HealthBench from the original paper:

Frontier models have improved significantly over time, but substantial headroom remains. HealthBench Hard, a 1,000-example subset, is harder still; this environment serves the full 5,000-example set rather than that subset.

Other Environment Requirements

HealthBench requires an OpenAI API key (OPENAI_API_KEY secret) for LLM-based grading of answers.

Fidelity to the Reference Implementation

Scoring, the grader template and the graded transcript match OpenAI's published
healthbench_eval.py. Three things differ deliberately:

• The conversation is rendered as text. Task prompts here are a flat block of
  text rather than a role-tagged message list, so a multi-turn conversation is
  rendered as role: content turns — the same form the grader sees — behind a
  short instruction to reply to the final user turn. A single-message task is
  passed through verbatim with no added framing.
• The grader's own system message is not added to the graded transcript. The
  reference implementation grades the exact message list it queried, which
  includes its sampler's system message. Here the agent's system prompt belongs
  to the harness and is not visible to the environment, so it is omitted rather
  than fabricated.
• Grading failures raise instead of retrying forever. A rubric item is
  retried a bounded number of times with exponential backoff, covering both
  malformed grader JSON and transport errors; past that the tool call fails. A
  grader that cannot grade never returns a score.

Safety

Agents in HealthBench are asked to respond to healthcare conversations. The environment does not present direct safety risks, as agents only provide text answers with no access to external systems, tools, or the internet. However, the content involves medical topics and responses should be evaluated in that context.

Citations

@article{arora2025healthbench,
  title={HealthBench: Evaluating Large Language Models Towards Improved Human Health},
  author={Arora, Rahul K. and Wei, Jason and Hicks, Rebecca Soskin and Bowman, Preston and Qui{\~n}onero-Candela, Joaquin and Tsimpourlas, Foivos and Sharman, Michael and Shah, Meghan and Vallone, Andrea and Beutel, Alex and Heidecke, Johannes and Singhal, Karan},
  journal={arXiv preprint arXiv:2505.08775},
  year={2025}
}