no-start-env

By Second Nature · v0.1 · MIT ·
GitHub

An agentic diagnostic environment and eval: a simulated vehicle with electrical
no-start and charging faults, realistic shop tools, and a cheat-resistant
grader. A car will not start; the agent gets the customer complaint and seven
tools, and has to localize the fault, replace the right part, and confirm the
repair with a successful start. Nothing tells it whether the part it replaced
was faulty.

The vehicle is a resistance-network circuit model: faults perturb one physical
resistance (or the battery's internal parameters) and every voltage the agent
can measure is computed from V = I×R. Symptoms are therefore physically
consistent across probe points and engine states — the environment rewards
genuine electrical localization and punishes plausible-sounding guesswork.

This is the OpenReward (ORS) serving of the published v0.1 benchmark. Full
writeup, failure-mode taxonomy, and per-scenario results:
WRITEUP.md.

Capabilities tested

• Physical-system fault localization from two-point voltage measurements
  (choosing the right reference points, not just reading numbers)
• Hypothesis discrimination under a deliberate red herring (a tempting-looking
  battery that passes a load test)
• Intermittent-fault reasoning (a fault that only manifests on some probes)
• Compound-fault reasoning (two genuine faults; neither repair alone fixes
  the car)
• Repair discipline and cost awareness: measure before swapping parts, verify
  the fix with a successful start

Tasks

One split, test, with 5 tasks. Task specs are opaque
({"task_id": "ns-01", "tier": "easy"}) — scenario names encode ground truth
and are held server-side, so nothing in the task list can leak the answer into
an agent's context. Tiers: 1 easy, 2 medium, 2 hard.

Every task is seeded and fully deterministic: identical action sequences
produce identical observations (meter noise included) and identical rewards,
across runs and processes.

Tools

scan_dtcs, read_pid, measure_voltage(point_a, point_b, engine_state),
visual_inspect(area), replace_part(component), attempt_start, and
finish(answer) to submit the final diagnosis and end the episode. Tool
outputs are instrument readings only — no explanations, and nothing that
reveals component health or fault state. A hidden terminal tool grades the
rollout if a harness ends it on a plain assistant message (the message is
treated as the diagnosis).

Reward structure

Sparse, continuous, emitted once at episode end: reward = grader score /
100 (range 0–1). The grader reads ground-truth world state only — agent
prose is never trusted, and symptom relief from swapping the wrong part does
not count. The native 0–100 rubric:

• Root cause (60): correct component + failure mode; half credit for
  component only. On the compound task the budget splits 45 (primary) / 15
  (secondary).
• Parts discipline (25): −8 per innocent part replaced (also debits the
  total, so a parts cannon cannot ride root-cause points to a pass).
• Cost efficiency (15): time-only, linear from 1× expert time to 0 at 2×,
  negative beyond.
• Anti-cheat: finishing with no diagnostic probes before the first
  replacement caps the score at 40; a correct diagnosis without a repaired,
  start-verified vehicle takes a flat −15.

The full subscore breakdown ships in the terminal tool output's metadata
(display-formatted, matching the Inspect log format, plus numeric
score_0_100).

Prompt parity with the published benchmark

get_prompt() returns two text blocks, verbatim the strings from the
published Inspect run: block 1 is the system prompt (uncoached variant:
role, tool reference, job definition, cost one-liner — no strategy, no grader
rules), block 2 is the user message (customer complaint + task statement).
ORS blocks carry no role; harnesses that support a system role should map
block 1 there. Parity with the published run is verified at the wire level
(results/parity_diag/REPORT.md in the repo): both prompt blocks and all
tool descriptions/schemas are byte-matched to the API payloads Inspect
serialized for the published benchmark (pinned by a regression test), the
reward mapping is exact on deterministic replay, and a 20-episode
two-pipeline check found no systematic score difference. Model-level scores
remain subject to ordinary sampling variance, and harness behavior matters:
to reproduce the published conditions, run episodes basic_agent-style
(nudge on plain assistant messages — "Please proceed to the next step using
your best judgement." — 50-message cap counting every message, tool errors
rendered as is_error with the bare message; reference implementation in
scripts/validate_openreward.py).

Environment difficulty (published v0.1 results)

Uncoached prompt, 5 scenarios × 5 epochs per model, 225 episodes:

Frontier models clear the easy/medium tiers (59/60 full root-cause credit) but
not the hard tier (15/40); no model passes every episode. Best-worst
separation: 78.8 points.

Time horizon

Multi-turn, tool-calling. Expert episodes run 4–11 actions; the published runs
capped conversations at 50 messages. Episodes end when the agent calls
finish (or on the harness's own turn limit — unterminated episodes emit no
reward unless the harness invokes the terminal tool).

Compute requirements

None beyond the environment server itself — the simulation is a pure
in-process circuit computation. No sandbox, no GPU, no secrets, no external
network calls.

Data

All scenario definitions, physics constants, and grading logic live in the
GitHub repo (the benchmark is
public). Ground truth never appears in any agent-visible surface: prompts,
tool outputs, task specs, and error messages are audited for leakage (see
results/audit.md in the repo; the ORS surface is covered by
tests/test_openreward_adapter.py).

Safety

Benign, fully simulated automotive electrical diagnosis. No real-world
actuation, no web access, no user data, no dual-use content.

License & citation

MIT. Cite via the repo's
CITATION.cff.

Contact

Second Nature — snlabs.dev · Jaivir Parmar