MathAnswer1000

Description

MathAnswer1000 is an assistant-end environment with 1,000 diverse training
problems. The model solves a problem in an ordinary assistant message and ends
with ANSWER: \(ANSWER\). A hidden terminal grader extracts the final answer and
checks it against a private reference answer. No submission tool is exposed.

The prompt uses this exact wrapper:

Solve the following math problem step by step. The last line of your response should be of the form "ANSWER: \(ANSWER\)" (without quotes) where ANSWER is the answer to the problem.

{problem}

Remember to put your answer on its own line at the end in the form "ANSWER: \(ANSWER\)" (without quotes) where ANSWER is the answer to the problem, and you do not need to use a \boxed command.

Capabilities

• Multi-step arithmetic and word problems
• Algebra, geometry, counting, probability, number theory, and precalculus
• Competition and olympiad-style problems
• Final-answer formatting and numerical equivalence checking

Compute Requirements

Runs in the Python environment server on CPU. No agent sandbox, GPU, browser,
or external grader is needed. Task files are bundled locally in data/ and
require no runtime download.

License

Environment code: MIT. Source data: MIT for MATH, GSM8K, and the
DeepScaleR dataset; CC BY 4.0 for AceReason-Math, according to their dataset
cards. See source attribution and license notices.

Tasks

All selected rows come from source training splits. Validation is a local
held-out subset of those training pools, not an official benchmark test split.
Each task has one finite real numerical answer, which may be an integer,
fraction, decimal, or expression such as a radical. Proofs, symbolic-variable
answers, sets, tuples, and detected diagram-dependent questions are excluded.
The MATH portion uses source difficulty levels 3–5.

Reward Structure

Sparse terminal reward: 1.0 for an equivalent final answer; 0.0 for a
wrong answer, an unsupported answer, or a missing final answer. The first
terminal call finishes the episode. Later calls
return zero without regrading.

The grader reads the final ANSWER: block, or a final line/block containing
only a mathematical answer. Correct numbers elsewhere in prose earn no credit.
It tolerates case and whitespace, Markdown emphasis or code fences, parentheses,
optional boxes, trailing punctuation, and inline/display LaTeX delimiters,
including mixed boundary markers. Unicode π, radicals, superscripts, and
arithmetic symbols are normalized. Escaped dollar signs (\$12) and ordinary
dollar signs ($12) are both accepted around numeric answers. Bold answers,
nested boxes, scientific notation (1.2e1), grouped thousands (1{,}234),
indexed roots, and parenthesized exponents are supported. For example,
ANSWER: (16\pi), $16\pi$,
\[16π\], and \[$16\pi$\] all represent the same answer. Multiline display
math is supported. Trailing explanatory text, multiple candidate answers, and
equations such as x=42 are rejected. Working steps are requested but are not
independently scored.

answer_targets.json pins reviewed grading context to
problem-text hashes: 21 multi-part source problems now explicitly identify the
single quantity to put in the final answer. Three questions asking for a
percentage accept optional %, \%, or percent after the numeric answer.
Percent signs are not silently discarded for other tasks. When the reference is
a plain number, one trailing degree mark (^\circ, ^{\circ}, °, \degree
or degrees) is accepted after the numeric answer; symbolic references keep
their meaning. Malformed decimals,
ellipses, and ungrouped chains of exponents are rejected.

Verification uses math-verify 0.9.0 with a scalar-only parser, no string fallback,
two-second parsing/comparison timeouts, and 12-place float rounding. Decimal
literals are converted to exact fractions before symbolic parsing. Equivalent
fractions and radicals are accepted. This is deterministic grading, with no LLM
judge. Reference answers come from upstream datasets, with reviewed errata in
data_corrections.json. This project does not claim an
independent mathematical proof of every source label. The cyclic-prime
quadrilateral task deepscaler-29307 has been independently checked by exact
enumeration: its answer is \sqrt{\frac{1736}{5}}. Its erroneous upstream
answer and multiple-choice options have been corrected. In deepscaler-06258,
"no two of them are 4 seats apart" admitted two readings (maximum 28 or 30); it
now reads "no two of them have exactly 4 seats between them", which matches the
upstream answer 30.

Data

data/train.jsonl and data/validation.jsonl contain the selected problems and
private reference answers. Public task specs contain only an ID. Prompts and
grading feedback do not disclose reference answers or source solutions.

prepare_data.py reproduces the files from pinned Hugging Face revisions and
verifies source-file SHA-256 hashes. Selection is deterministic, with 250
training and 25 validation tasks per source. Provenance includes the dataset,
revision, original split, and row index. data/manifest.json records counts,
checksums, and filters.

The original task IDs and splits are pinned in task_selection.json, so
improvements to answer formatting do not silently resample the dataset.

Reviewed corrections are applied after selection, preserve task IDs and split
membership, and are included in the manifest checksum. Each correction records
the original label and problem fingerprint, its replacement, and the evidence.

Normalized duplicate statements are excluded across the two shipped splits.
Known MATH/GSM8K test statements are excluded. DeepScaleR and AceReason rows
matching known MATH/GSM8K training statements are excluded too. This is text
normalization and deduplication, not a guarantee of semantic decontamination
against all public benchmarks. DeepScaleR and AceReason are themselves
aggregated, overlapping collections; see the attribution file.

Tools

There are zero model-visible tools. The grader uses OpenReward's
@terminal API.
The harness checks session.is_assistant_message_final() and submits the final
assistant message through session.call_terminal_tool(text).

Time Horizon

One assistant response per problem, followed by terminal grading.

Environment Difficulty

The mix ranges from GSM8K arithmetic to competition and olympiad-style math.
No model accuracy baseline has been measured for this particular subset.

Other Environment Requirements

No external API keys or services are required to run or grade the environment.
The optional real-model example needs the policy provider's OPENAI_API_KEY;
the default smoke test needs no model API key.

Local Development

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt
python -m pytest golden_tests.py -v
python server.py

In a second terminal with the same virtual environment:

python test_agent.py
python test_agent.py --split validation
# Optional real policy inference, using the OpenAI Responses API:
python test_agent.py --model gpt-5.2 --limit 1
# With platform credentials in the process environment, check the private deployment:
python test_agent.py --environment GeneralReasoning/MathAnswer1000

The default test sends known correct, incorrect, and malformed fixture messages
through the real HTTP/SDK terminal path, one task per source. It is an
integration test, not a model evaluation. Use --base-url to change the server
address. For example, if port 8080 is occupied:

python -c 'from server import Server, MathAnswer1000; Server([MathAnswer1000]).run(host="127.0.0.1", port=18081)'
python test_agent.py --base-url http://127.0.0.1:18081

Rebuild the dataset or container:

python prepare_data.py
docker build -t mathanswer1000:test .
docker run --rm -p 8080:8080 mathanswer1000:test

Safety

The environment accepts mathematical text and exposes no filesystem,
network, or code-execution tools to the model.

Citations

• Hendrycks et al. (2021), Measuring Mathematical Problem Solving With the MATH Dataset.
• Cobbe et al. (2021), Training Verifiers to Solve Math Word Problems.
• Luo et al. (2025), DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL.
• Chen et al. (2025), AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning.
• General Reasoning Inc. Team (2026), MathAnswer1000 environment adaptation and dataset selection.