Environments

Eigent/SETA

18

Implementation of the SETA environment https://www.camel-ai.org/blogs/seta-scaling-environments-for-terminal-agents

Eigent/toolathlon-gym

15

Large-Scale Long-Horizon Environments for Tool-Use Agents 503 multi-tool tasks backed by a local PostgreSQL database — no external APIs required This is a port of EigentAI's [Toolathlon-GYM](https://www.eigent.ai/blog/toolathlon-gym-large-scale-long-horizon-environments-for-tool-use-agents) to OpenReward.

martian/fs-review

14

Review a full set of financial statements the way an auditor does — as one interlinked whole — and flag where the numbers don't tie out.

benchflow/skillsbench

14

SkillsBench is an evaluation framework that measures how skills work, and the first dataset that measures how powerful models are at using skills on expert-curated tasks across high-GDP-value, diverse domains.

aashay96/OpenForecaster

11

Predict world events better https://github.com/OpenForecaster/scaling-forecasting-training

nebius/SWE-rebench-V2

11

SWE-rebench-V2 is an OpenReward port of the SWE-rebench V2 benchmark by Badertdinov et al. (Nebius AI). It evaluates agents on real-world software engineering tasks across multiple programming languages. Agents are given a repository checked out to a specific commit and a problem statement, and must modify the source code so that previously-failing tests pass without breaking existing tests. The dataset covers 32K+ instances across Python, JavaScript, Go, Rust, Java, Ruby, and many more languages.

kanishk/EndlessTerminals

9

EndlessTerminals is an environment of procedurally generated terminal-use tasks without human annotation for training terminal agents with reinforcement learning.

GeneralReasoning/KellyBench

7

KellyBench is a benchmark that tests an agents' ability to make machine learning models for predicting football matches and betting against market odds.

dsileo/reasoning-core

6

reasoning-core is a suite of procedural data generators for LLM pre-training and post-training. It is centered on expressive symbolic tasks, including full fledged first-order-logic, formal mathematics with TPTP, planning, and CFG syntax tasks.

GeneralReasoning/FPL

5

FPL is an environment that tests an agent's ability to play fantasy football for the English Premier League.

bys0318/LongBench-v2

5

LongBench v2 is designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks.

anjiang/SATBench

5

SATBench is a benchmark for evaluating the logical reasoning capabilities of large language models (LLMs) through logical puzzles derived from Boolean satisfiability (SAT) problems.

GeneralReasoning/Discovery30s

5

Discovery30s is a benchmark that tests the potential of vintage language models to reproduce scientific discoveries after the training data cutoff period. We construct the benchmark by taking a known discovery, e.g. Hückel's rule, and then breaking it down into a "question ladder" that asks motivating questions that eventually lead up to the discovery as the answer. For the initial test_v0_1, the discoveries tested are Hueckel's rule, Goedel's incompleteness theorem, the prediction of the Oort cloud, chromosone crossover, Haldane's unification of Mendelian genetics and evolutionary science, the theory of homeostasis, ergodic theory, von Neumann's mathematical framework for quantum mechanics, and the invention of zero-length springs. The eval consists of 196 tasks in total.

Liqiang/DSBench

4

DSBench is a benchmark designed to evaluate data science agents with realistic tasks. The benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions.

GeneralReasoning/CTF

4

CTF (Capture the flag) is an environment where agents attempt to find text strings - called flags - which are secretly hidden in purposefully vulnerable programs or websites.

NVIDIA/Nemotron-Math-Proofs-v1

4

Nemotron-Math-Proofs-v1 is a large-scale mathematical reasoning dataset containing ~580k natural language proof problems, ~550k formalizations into theorem statements in Lean 4, and ~900k model-generated reasoning trajectories culminating in Lean 4 proofs. The dataset integrates human-authored problems with systematically generated formalizations and solution traces.

GeneralReasoning/portfolio

4

Portfolio is an environment which tests agents ability to conduct portfolio optimisation tasks.

parshinsh/llmsr-bench-full

3

A comprehensive benchmark with challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization.

siegelz/corebench-easy

3

CORE-Bench is a benchmark for evaluating the ability of agents to computationally reproduce scientific papers.

siegelz/CoreBench-Hard

3

CORE-Bench evaluates the ability of agents to computationally reproduce the results of published scientific papers. In CORE-Bench Hard, the agent is only given the codebase of the paper and must install all libraries and dependencies, run the code, and read through the output and figures to answer questions about the paper. This level is most akin to fully reproducing a paper and is the most realistic and challenging level.

uiuc-kang-lab/CVEBench

3

Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities

jiayipan/SWE-Gym

3

A training environment of 2,438 real-world software engineering tasks sourced from 11 Python repositories. Each instance includes a codebase, an executable environment with pre-installed dependencies, unit tests, and a natural language problem description derived from GitHub issues. The environment supports training both agents and verifiers, with test outcomes serving as reward signals.

Naman/R2E-Gym

3

A dataset of over 8,100 software engineering problems with executable environments, generated from real-world GitHub repositories. Each entry includes a problem statement and the necessary setup to run and test code. The environments are used for training and evaluating autonomous SWE agents.

GeneralReasoning/DataScienceComps

3

DataScienceComps consists of data science competitions that agents can be trained against.