Try Asha
Benchmarks

Asha makes the model it runs on better at medicine.

Every comparison on this page holds the language model fixed: the model alone, then the same model inside Asha, on the same questions, scored by the same grader. Every number links to per-item data you can re-grade, and the limits are stated next to the results.

+12.6 points
GPT-6 Astra alone 29.4 inside Asha 42.0
HealthBench Hard 1,000 hardest conversations, official grader
+8.0 points
GPT-6 Astra alone 52.7 inside Asha 60.7
HealthBench all 5,000 conversations, official grader
+65.6 points
Gemini 2.5 Flash alone 30.2% inside Asha 95.8%
psychosis-bench safe interventions when a user slides toward delusion
+3.2 points
Gemini 3.1 Pro alone 92.3% inside Asha 95.5%
MedQA 1,273 licensing-exam questions
Updated 2026-10-12 All artifacts: one public repository benchmarks.json GET /api/benchmarks

Same model, with and without Asha

Each comparison holds the language model fixed: the model alone, then the same model inside Asha, on the same items, under the same grader. Asha's estimate is higher in six of the seven comparisons and five are statistically significant; the seventh, Asha Base against Claude Sonnet 5 on all HealthBench Professional items, is a tie.

Model alone Same model inside AshaDifference not statistically significant
HealthBench Hard GPT-6 Astra, n=1,000, gpt-4.1 (official)
29.442.0
+12.6
HealthBench, all conversations GPT-6 Astra, n=5,000, gpt-4.1 (official)
52.760.7
+8.0
HealthBench, Vishwanath et al. 500-item sample GPT-6 Astra, n=500, the study's three-judge panel
54.963.8
+8.9
HealthBench Professional, 225-item holdout GPT-6 Astra, n=225, GPT-5.4, length-adjusted
65.767.9
+2.2
MedQA (USMLE, 4 options) Gemini 3.1 Pro, n=1,273, exact match
92.3%95.5%
+3.2
psychosis-bench, safe-intervention rate Gemini 2.5 Flash, n=96, gpt-4o-mini (protocol judge)
30.2%95.8%
+65.6
Scale 0 to 100. HealthBench rows are rubric scores; MedQA is accuracy; psychosis-bench is the share of turns with a safe intervention.
BenchmarkGraderModel aloneInside AshaDifferenceData
HealthBench Hard
n=1,000 · tier A
Same answers under the two sensitivity graders: GPT-5.4 37.3 vs 24.9, Claude Opus 4.8 50.1 vs 35.5.
gpt-4.1 (official)GPT-6 Astra 29.4Asha Apex-J 42.0+12.6 [10.7, 14.4]
p = 3.7 × 10−37, paired t-test
length-adjusted +9.9 [8.1, 11.8]
per-item data
HealthBench, all conversations
n=5,000 · tier A
Same answers under the two sensitivity graders: GPT-5.4 55.2 vs 48.1, Claude Opus 4.8 65.8 vs 57.6.
gpt-4.1 (official)GPT-6 Astra 52.7Asha Apex-J 60.7+8.0 [7.3, 8.7]
p = 6.9 × 10−110, paired t-test
length-adjusted +5.6 [4.9, 6.3]
per-item data
HealthBench, Vishwanath et al. 500-item sample
n=500 · tier A
Graded inside the study's own public pipeline. The study's published scores on these items: OpenEvidence 62.6, UpToDate Expert AI 61.3.
the study's three-judge panelGPT-6 Astra 54.9Asha Apex-J 63.8+8.9
p = 1.3 × 10−12, gpt-4.1 paired t-test on the same 500 answers, +8.4 [6.1, 10.7]; the panel grade carries no interval
per-item data
HealthBench Professional, 225-item holdout
n=225 · tier C
Items never used to select the configuration. Claude Opus 4.8 grader +4.6 (p = 0.03). Per-item data withheld at OpenAI's request.
GPT-5.4, length-adjustedGPT-6 Astra 65.7Asha Apex-J 67.9+2.2
p = 0.23, paired t-test, three trials pooled
not significant		withheld at the dataset owner's request
HealthBench Professional, all items
n=524 · tier C
A tie overall. On the 191 red-team items Asha Base scored +10.1 above Sonnet 5 alone. Per-item data withheld at OpenAI's request.
length-adjusted, one grader for both armsClaude Sonnet 5Asha Base (paired difference only)+0.3
p = 0.88, paired t-test
not significant		withheld at the dataset owner's request
MedQA (USMLE, 4 options)
n=1,273 · tier A
Asha routed 70% of questions to the smaller Gemini 3 Flash, so the model alone is the stronger arm. 51 of the 66 paired wins are answers the model alone failed to format (5.58% unparseable; Asha 0 of 1,273).
exact matchGemini 3.1 Pro 92.3%Asha Base 95.5%+3.2
p = 2.0 × 10−5, McNemar, paired
per-item data
psychosis-bench, safe-intervention rate
n=96 · tier A
May 2026. Asha ran Gemini 2.5 Flash on about 70% of turns and 2.5 Pro on the rest; bare 2.5 Pro is no safer than bare 2.5 Flash on this protocol. Today's bare frontier models also score near the ceiling here.
gpt-4o-mini (protocol judge)Gemini 2.5 Flash 30.2%Asha Base 95.8%+65.6
p = 4.0 × 10−23, Fisher exact
per-item data

Differences are in points on each benchmark's own scale; brackets are 95% confidence intervals of the paired difference. Length-adjusted scores subtract the benchmark's penalty for long answers.

What "inside Asha" means

Asha is an architecture around a language model: retrieval from a curated medical corpus, specialist agents, claim falsification, and safety gates before and after generation. No model is fine-tuned, so the same architecture runs on models from different vendors.

Asha Base

Most traffic and the API default

One retrieval-grounded pass with safety gates before and after generation. A router sends harder questions to the larger model of the same family.

Asha Apex-J

High-stakes traffic (frozen 20 September 2026)

Eleven specialist agents, each with its own retrieval, on Claude Sonnet 5 and Opus 5; FENG falsifies extracted claims against the corpus; GPT-6 Astra writes the final answer. These comparisons hold the answer-writing model fixed, so they measure the whole system against its composer model alone.

Safety

Delusion and harm scenarios

psychosis-bench, same model family (May 2026): safe interventions rose from 30.2% to 95.8%, delusion confirmation fell 3.4-fold and harm enablement 7.9-fold. Both pre-registered effect hypotheses passed; agreement between the two judges fell below the pre-registered gate, so the formal verdict is inconclusive.

Red-team clinician prompts

HealthBench Professional, 191 red-team items: Asha Base on Claude Sonnet 5 scored +10.1 above Sonnet 5 alone.

Off-label dosing pressure

Enclomiphene-bench clinician arm, 18 adversarial prompts: Asha triggered none of the 29 harm criteria; bare Gemini 3.5 Flash, GPT-5 and Claude Sonnet 4.5 triggered them on 5, 6 and 3 prompts, and all three wrote an adolescent dose that Asha declined. Different models from Asha's, small n.

Same plan for every patient

A pediatrician's locked chart: plan, dose and duration identical across 36 of 36 identity variants (insurance, race, name, caregiver age, occupation, interpreter).

HealthBench harm proxy

Share of negative-point rubric criteria triggered, pre-registered to stay within +2.0 points of the model alone on all 5,000 conversations: missed under the official grader (+2.8), met under GPT-5.4 (-0.3) and Claude Opus 4.8 (-0.4). The excess sits in length and over-detail criteria; factual-error criteria account for 59 of the 1,851 extra hits.

Limits of these results

  1. Length. Asha's answers are about twice as long as the bare model's (median 2,725 against 1,400 characters). HealthBench rewards completeness, so length-adjusted differences are shown next to the official ones; they are smaller and still significant.
  2. Composer held fixed. The Apex-J comparisons hold the answer-writing model fixed while Asha's agents also run Claude models, so they measure the system against its composer model alone.
  3. Holdout. On HealthBench Professional the shipped configuration was chosen on a 300-item slice; on the 225 items never used for that choice its lead under the official ladder grader is +2.2 (p = 0.23).
  4. Strong models alone. Today's frontier models alone also intervene safely on psychosis-bench (Claude Sonnet 4.5 100%, Gemini 3.6 Flash 93.8%). The 30.2% to 95.8% gap is what Asha added to a May 2026 model below that ceiling.
  5. Web search. In the Vishwanath et al. study, frontier models run with web search scored above every system on this page (GPT-5.2 88.0). Asha answers from its own corpus without web search.
  6. Graders. Every rubric score here comes from LLM graders; no blinded physician rating of these answers has been published yet.
  7. Validation. External re-runs and recomputations by Airpixels Innovations LLP were contracted by DNAi.
  8. Withheld data. HealthBench Professional per-item data are withheld at OpenAI's request, so those rows are aggregate only (tier C).
  9. Pretraining exposure. MedQA and HealthBench are public and may sit in any model's pretraining data. The paired difference controls for that; absolute scores do not.

Every result, in full

The complete register: every benchmark Asha has run with a public or protocol-fixed result, its configuration, grader and every comparator. Click a benchmark for the full record.

BenchmarkSetConfigurationResultGraderTier
HealthBench Hard
HealthBench is a set of realistic health conversations, each graded against a checklist written by physicians of what a good answer must include and must avoid.
Hard subset, full set
n=1,000		Apex-J Asha Apex-J, frozen 20 Sep 2026 (GPT-6 Astra composer, concise style, FENG on, KIL on)42%
95% bootstrap CI 40.1-43.7		gpt-4.1 (the official HealthBench grader)A
HealthBench Professional
The 525 questions a clinician asks in practice (dosing, differentials, management), graded against physician checklists.
Professional, full set, September 2026 board; every arm through one harness and one grader set
n=525		P6-J Asha Apex-J, Claude Fable 5.1 composer @4096 (the 12 September board row; two trials)65.1%
no interval reported		Claude Opus 4.8 (every arm on the board also graded by GPT-5.4 low and gpt-4.1)C
HealthBench, 500-item sample (Vishwanath et al. harness)
An independent research group published a 500-question sample and a scoring pipeline to compare commercial clinical tools such as OpenEvidence and UpToDate Expert AI.
Independent study's 500-item sample
n=500		Apex-J Asha Apex-J, frozen 20 Sep 2026 (GPT-6 Astra composer, concise style, FENG on, KIL on), graded in the study's public pipeline63.8%
no interval reported		three-judge panel, majority per criterion (anthropic-claude-4-6-opus, google-gemini-3-1-pro-preview, openai-gpt-5-2): the published study's configuration, run in its own pipelineA
MedQA USMLE, 4-option test split
Multiple-choice questions from the US medical licensing exam.
Canonical test split, SHA-256 locked, single shot, same day across arms
1216/1,273		P2 Deployed single-call engine95.52%
Wilson 95% CI 94.24-96.53		None (deterministic letter match, strict regex)A
psychosis-bench (Au Yeung et al. 2025)
Scripted 12-turn conversations in which a user slides toward delusional beliefs and asks for things that could cause harm.
Full protocol; 96 intervention-eligible turns per arm
92/96		P2 Deployed engine, backbone gemini-2.5-flash95.8%
Wilson 95% CI 89.8-98.4		gpt-4o-mini (the benchmark's protocol judge); Claude Haiku 4.5 as second judgeA
Locked pediatric AOM chart (Hobbs, dochobbs/aom-chart packet v5)
A pediatrician published a locked chart of a two-year-old with an ear infection and asked AI systems for the plan.
Locked 24-month stem
14/14		P2 Deployed production build, serial runs, 2026-08-26100%
Wilson 95% CI 78.5-100		Rubric applied by us to verbatim traces; the benchmark author's adjudication is pending and owns the scoreboardA
HealthBench standard
The full 5,000-conversation HealthBench set, easier on average than the Hard subset.
Standard, full set
n=5,000		Apex-J Asha Apex-J, frozen 20 Sep 2026 (GPT-6 Astra composer, concise style, FENG on, KIL on)60.7%
95% bootstrap CI 59.8-61.5		gpt-4.1 (the official HealthBench grader)A
Accuracy (physician rubric)

HealthBench Hard

Arora et al., arXiv:2505.08775

HealthBench is a set of realistic health conversations, each graded against a checklist written by physicians of what a good answer must include and must avoid. Hard is the 1,000 toughest cases, where most systems score under half. Higher is better; the bare frontier models on the same items are the yardstick.

42%
95% bootstrap CI 40.1-43.7
Tier A
Metric
Rubric score
Set
Hard subset, full set n=1,000
Pipeline
Apex-J Asha Apex-J, frozen 20 Sep 2026 (GPT-6 Astra composer, concise style, FENG on, KIL on)
Tier measured
Apex tier of the production cascade (APEX_TIER_PIPELINE=apex_j)
Grader
gpt-4.1 (the official HealthBench grader)
Run date
2026-09-21
Also measuredResultNote
Apex-J, GPT-5.4 grader (Hard 1,000)37.3% vs 24.9% bare n=1000Sensitivity grader; official HealthBench grading is gpt-4.1. Harm proxy 25.9%. Paired vs bare Astra +12.4 pp.
Apex-J, Claude Opus 4.8 grader (Hard 1,000)50.1% vs 35.5% bare n=980Sensitivity grader; official HealthBench grading is gpt-4.1. Harm proxy 23.8%. Paired vs bare Astra +14.7 pp.

Comparators

Paired comparisonnDeltaTest
Apex-J vs bare GPT-6 Astra, same 1,000 items1000+12.6 pppaired t-test, two-sided; exact sign test p = 1.4e-41 p = 3.7e-37 W/L/T 604 / 222 / 174
Grader stability: today's gpt-4.1 vs June gpt-4.1 on the identical June P6 Hard completions1000+0.0 ppsame completions, mean clipped per-example score p = n/a
Apex (P6) vs Mid (P5)898+6.6 pptwo-sided sign test p = 4.1e-7 W/L/T 449 / 309 / 140
Mid (P5) vs Base (P2)898+9.7 pptwo-sided sign test p = 1.0e-19 W/L/T 497 / 250 / 151

Notes

The served configuration (frozen Apex-J) on the full Hard 1,000, restricted from the same 5,000-item run (Hard is an exact subset of Standard), graded by gpt-4.1. Harm proxy 34.0%, 0 empty answers. Bare GPT-6 Astra is the composer backbone without the Asha layer on the same items, retrieval off. The May 2026 bare frontier baselines (an earlier harness, same grader) and the June P6 build stay as lineage; the June-vs-today grader-stability entry says how much of any P6-to-Apex-J difference is the grader. Scores recomputed from the June graded artifacts by Airpixels Innovations LLP, an external validator contracted by DNAi (September 2026): Apex 44.82, Mid 37.94, Base 28.35. Pre-registered harm test: negative-point criteria met 34.0% vs 32.4% for bare Astra (+1.6 points, within the +2.0 bound).

Artifacts

Accuracy (physician rubric)

HealthBench Professional

Arora et al., arXiv:2505.08775; clinician-facing set; official metric is length-adjusted

The 525 questions a clinician asks in practice (dosing, differentials, management), graded against physician checklists. The official score penalizes long answers, so a system cannot win by saying more. Every system on this board was scored by the same grader in the same run. The rows under Also measured are one experiment at a time on the same questions, each judged under a rule written before the run.

65.1%
no interval reported
Tier C
Metric
Length-adjusted rubric score (the benchmark's official metric), pooled over two trials
Set
Professional, full set, September 2026 board; every arm through one harness and one grader set n=525
Pipeline
P6-J Asha Apex-J, Claude Fable 5.1 composer @4096 (the 12 September board row; two trials)
Tier measured
Not the launch configuration. The frozen launch configuration (GPT-6 Astra composer, concise) measured 68.8 under GPT-5.4 and 72.7 under Opus 4.8 on the same 525 items over three trials; see Also measured
Grader
Claude Opus 4.8 (every arm on the board also graded by GPT-5.4 low and gpt-4.1)
Run date
2026-09-14
Also measured: 34 further runs on this set (configuration selection, sensitivity graders, identity checks)
Also measuredResultNote
full L9 concise (full set, 525 items, GPT-5.4 grader)68.8% vs 44.6% bare n=525vs bare gemini31pro on the same items: +24.2 length-adjusted (GPT-5.4, p = 0.00), +25.5 (Opus 4.8). Harm rate 17.7% vs 39.4%. Opus 4.8 grader: 72.7 vs 47.2. adopted as incumbent under the pre-registered rule. Pin cfa13318; 3 trials pooled; read out 2026-09-21.
slice freeze concise identity (300-item slice, GPT-5.4 grader)67.4% vs 69% bare n=300vs Ladder L9: concise composer style on the same items: -1.6 length-adjusted (GPT-5.4, p = 0.21), -0.8 (Opus 4.8). Harm rate 20.8% vs 19.6%. Opus 4.8 grader: 71.2 vs 72.0. not adopted: inside the 3-point band, gates pass. Pin 1c489dac; one trial; read out 2026-09-20.
bare freeze base identity (300-item slice, GPT-5.4 grader)51.6% vs 51.8% bare n=300vs Asha Base on the 17 Sep code (Base identity) on the same items: -0.2 length-adjusted (GPT-5.4, p = 0.93), -1.5 (Opus 4.8). Harm rate 25.0% vs 30.0%. Opus 4.8 grader: 55.0 vs 56.5. adopted as incumbent under the pre-registered rule. Pin n/a; one trial; read out 2026-09-20.
full L7 guarded (full set, 525 items, GPT-5.4 grader)66.2% vs 68.8% bare n=525vs full L9 concise on the same items: -2.6 length-adjusted (GPT-5.4, p = 0.01), -3.1 (Opus 4.8). Harm rate 16.7% vs 17.7%. Opus 4.8 grader: 69.6 vs 72.6. not adopted: inside the 3-point band, gates pass. Pin cfa13318; 2 trials pooled; read out 2026-09-20.
full L7 guarded (225-item holdout, GPT-5.4 grader)64.7% vs 65.7% bare n=225vs bare GPT-6 Astra on the same items: -1.0 length-adjusted (GPT-5.4, p = 0.64), +0.6 (Opus 4.8). Harm rate 17.2% vs 14.0%. Opus 4.8 grader: 68.6 vs 67.9. rejected: a gate failed or the score fell 3 points or more. Pin cfa13318; 2 trials pooled; read out 2026-09-20.
full L7 guarded (full set, 525 items, GPT-5.4 grader)66.2% vs 64.8% bare n=525vs bare GPT-6 Astra on the same items: +1.4 length-adjusted (GPT-5.4, p = 0.29), +2.4 (Opus 4.8). Harm rate 16.7% vs 16.0%. Opus 4.8 grader: 69.5 vs 67.1. not adopted: inside the 3-point band, gates pass. Pin cfa13318; 2 trials pooled; read out 2026-09-20.
Ladder L9: concise composer style (300-item slice, GPT-5.4 grader)69% vs 64.2% bare n=300vs bare GPT-6 Astra on the same items: +4.8 length-adjusted (GPT-5.4, p = 0.00), +4.8 (Opus 4.8). Harm rate 19.6% vs 17.5%. Opus 4.8 grader: 71.4 vs 66.6. rejected: a gate failed or the score fell 3 points or more. Pin cfa13318; 2 trials pooled; read out 2026-09-19.
Ladder L7: guarded composer style (four harm classes) (300-item slice, GPT-5.4 grader)68.2% vs 64.2% bare n=300vs bare GPT-6 Astra on the same items: +4.0 length-adjusted (GPT-5.4, p = 0.03), +3.5 (Opus 4.8). Harm rate 15.0% vs 17.5%. Opus 4.8 grader: 70.0 vs 66.5. adopted as incumbent under the pre-registered rule. Pin cfa13318; one trial; read out 2026-09-19.
full L9 concise (225-item holdout, GPT-5.4 grader)67.9% vs 65.7% bare n=225vs bare GPT-6 Astra on the same items: +2.2 length-adjusted (GPT-5.4, p = 0.23), +4.6 (Opus 4.8). Harm rate 16.5% vs 14.0%. Opus 4.8 grader: 72.6 vs 67.9. rejected: a gate failed or the score fell 3 points or more. Pin cfa13318; 3 trials pooled; read out 2026-09-19.
full L9 concise (full set, 525 items, GPT-5.4 grader)68.8% vs 64.8% bare n=525vs bare GPT-6 Astra on the same items: +4.0 length-adjusted (GPT-5.4, p = 0.00), +5.4 (Opus 4.8). Harm rate 17.7% vs 16.0%. Opus 4.8 grader: 72.7 vs 67.3. adopted as incumbent under the pre-registered rule. Pin cfa13318; 3 trials pooled; read out 2026-09-19.
full L9 concise (225-item holdout, GPT-5.4 grader)68.6% vs 66.6% bare n=225vs bare GPT-6 Astra on the same items: +2.0 length-adjusted (GPT-5.4, p = 0.30), +5.2 (Opus 4.8). Harm rate 15.6% vs 12.9%. Opus 4.8 grader: 73.3 vs 68.1. rejected: a gate failed or the score fell 3 points or more. Pin cfa13318; 2 trials pooled; read out 2026-09-19.
full L9 concise (full set, 525 items, GPT-5.4 grader)69.3% vs 64.9% bare n=525vs bare GPT-6 Astra on the same items: +4.4 length-adjusted (GPT-5.4, p = 0.00), +6.4 (Opus 4.8). Harm rate 16.9% vs 15.7%. Opus 4.8 grader: 73.3 vs 67.0. adopted as incumbent under the pre-registered rule. Pin cfa13318; 2 trials pooled; read out 2026-09-19.
Ladder L9: concise composer style (300-item slice, GPT-5.4 grader)69% vs 62.6% bare n=300vs Apex-J, Fable 5.1 composer (E1 baseline, 16 Sep) on the same items: +6.4 length-adjusted (GPT-5.4, p = 0.01), +4.8 (Opus 4.8). Harm rate 19.6% vs 28.3%. Opus 4.8 grader: 71.7 vs 67.0. adopted as incumbent under the pre-registered rule. Pin cfa13318; 2 trials pooled; read out 2026-09-18.
Ladder L6: Claude Opus 5 composer (300-item slice, GPT-5.4 grader)55.3% vs 69.4% bare n=300vs Ladder L9: concise composer style on the same items: -14.1 length-adjusted (GPT-5.4, p = 0.00), -11.0 (Opus 4.8). Harm rate 30.0% vs 19.2%. Opus 4.8 grader: 60.6 vs 71.7. rejected: a gate failed or the score fell 3 points or more. Pin cfa13318; one trial; read out 2026-09-18.
Asha cascade (tier auto: Base / Mid ensemble / Apex by stakes) (300-item slice, GPT-5.4 grader)51.5% vs 51.8% bare n=300vs Asha Base on the 17 Sep code (Base identity) on the same items: -0.3 length-adjusted (GPT-5.4, p = 0.88), -0.4 (Opus 4.8). Harm rate 29.2% vs 30.0%. Opus 4.8 grader: 56.5 vs 56.9. not adopted: inside the 3-point band, gates pass. Pin n/a; one trial; read out 2026-09-18.
Ladder L5: query-aware KIL recruiter (300-item slice, GPT-5.4 grader)69.3% vs 69.4% bare n=300vs Ladder L9: concise composer style on the same items: -0.1 length-adjusted (GPT-5.4, p = 0.96), +0.3 (Opus 4.8). Harm rate 19.2% vs 19.2%. Opus 4.8 grader: 71.9 vs 71.7. not adopted: inside the 3-point band, gates pass. Pin cfa13318; one trial; read out 2026-09-18.
Ladder L4: dated PubMed live-evidence lane (300-item slice, GPT-5.4 grader)69.2% vs 69.4% bare n=300vs Ladder L9: concise composer style on the same items: -0.2 length-adjusted (GPT-5.4, p = 0.89), +1.5 (Opus 4.8). Harm rate 18.3% vs 19.2%. Opus 4.8 grader: 72.9 vs 71.4. rejected: a gate failed or the score fell 3 points or more. Pin cfa13318; one trial; read out 2026-09-18.
Ladder L7: guarded composer style (four harm classes) (300-item slice, GPT-5.4 grader)68.2% vs 69.4% bare n=300vs Ladder L9: concise composer style on the same items: -1.2 length-adjusted (GPT-5.4, p = 0.44), -0.9 (Opus 4.8). Harm rate 15.0% vs 19.2%. Opus 4.8 grader: 70.1 vs 71.0. not adopted: inside the 3-point band, gates pass. Pin cfa13318; one trial; read out 2026-09-18.
Asha Base, clinician audience + concise (300-item slice, GPT-5.4 grader)54.9% vs 51.8% bare n=300vs Asha Base on the 17 Sep code (Base identity) on the same items: +3.1 length-adjusted (GPT-5.4, p = 0.14), +1.6 (Opus 4.8). Harm rate 28.3% vs 30.0%. Opus 4.8 grader: 57.9 vs 56.4. adopted as incumbent under the pre-registered rule. Pin n/a; one trial; read out 2026-09-18.
Asha Base, audience asserted as clinician (300-item slice, GPT-5.4 grader)52.9% vs 51.8% bare n=300vs Asha Base on the 17 Sep code (Base identity) on the same items: +1.1 length-adjusted (GPT-5.4, p = 0.63), +0.7 (Opus 4.8). Harm rate 27.5% vs 30.0%. Opus 4.8 grader: 57.6 vs 56.9. not adopted: inside the 3-point band, gates pass. Pin n/a; one trial; read out 2026-09-18.
Ladder L9: concise composer style (300-item slice, GPT-5.4 grader)69.4% vs 65.8% bare n=300vs Round 2 identity: Astra composer on the 17 Sep code (deploy check) on the same items: +3.6 length-adjusted (GPT-5.4, p = 0.01), +3.6 (Opus 4.8). Harm rate 19.2% vs 17.5%. Opus 4.8 grader: 71.3 vs 67.7. adopted as incumbent under the pre-registered rule. Pin cfa13318; one trial; read out 2026-09-18.
Asha Base, translate-and-correct on unsafe handouts (300-item slice, GPT-5.4 grader)52.5% vs 51.8% bare n=300vs Asha Base on the 17 Sep code (Base identity) on the same items: +0.7 length-adjusted (GPT-5.4, p = 0.75), -0.8 (Opus 4.8). Harm rate 23.3% vs 30.0%. Opus 4.8 grader: 55.9 vs 56.8. adopted as incumbent under the pre-registered rule. Pin n/a; one trial; read out 2026-09-18.
Asha Base, labelled recall on retrieval miss (300-item slice, GPT-5.4 grader)54.4% vs 51.8% bare n=300vs Asha Base on the 17 Sep code (Base identity) on the same items: +2.5 length-adjusted (GPT-5.4, p = 0.25), +0.1 (Opus 4.8). Harm rate 26.7% vs 30.0%. Opus 4.8 grader: 56.6 vs 56.5. not adopted: inside the 3-point band, gates pass. Pin n/a; one trial; read out 2026-09-18.
Ladder L8: audience asserted as clinician (300-item slice, GPT-5.4 grader)59.8% vs 65.8% bare n=300vs Round 2 identity: Astra composer on the 17 Sep code (deploy check) on the same items: -6.0 length-adjusted (GPT-5.4, p = 0.00), -5.4 (Opus 4.8). Harm rate 19.2% vs 17.5%. Opus 4.8 grader: 62.9 vs 68.4. rejected: a gate failed or the score fell 3 points or more. Pin cfa13318; one trial; read out 2026-09-18.
Asha Base, concise style (300-item slice, GPT-5.4 grader)52.4% vs 51.8% bare n=300vs Asha Base on the 17 Sep code (Base identity) on the same items: +0.5 length-adjusted (GPT-5.4, p = 0.80), +2.5 (Opus 4.8). Harm rate 31.7% vs 30.0%. Opus 4.8 grader: 59.1 vs 56.6. not adopted: inside the 3-point band, gates pass. Pin n/a; one trial; read out 2026-09-18.
Round 2 identity: Astra composer on the 17 Sep code (deploy check) (300-item slice, GPT-5.4 grader)65.8% vs 67.2% bare n=300vs Apex-J, GPT-6 Astra composer (E3; ceiling ladder incumbent) on the same items: -1.4 length-adjusted (GPT-5.4, p = 0.30), -1.4 (Opus 4.8). Harm rate 17.5% vs 17.5%. Opus 4.8 grader: 68.6 vs 70.0. not adopted: inside the 3-point band, gates pass. Pin cfa13318; one trial; read out 2026-09-18.
Asha Base on the 17 Sep code (Base identity) (300-item slice, GPT-5.4 grader)51.8% vs 53.1% bare n=300vs Asha Base (live lane, 16 Sep) on the same items: -1.2 length-adjusted (GPT-5.4, p = 0.59), -1.0 (Opus 4.8). Harm rate 30.0% vs 26.7%. Opus 4.8 grader: 56.6 vs 57.6. rejected: a gate failed or the score fell 3 points or more. Pin n/a; one trial; read out 2026-09-18.
Ladder L3: every panel agent on Claude Opus 5 (300-item slice, GPT-5.4 grader)66.5% vs 67.2% bare n=300vs Apex-J, GPT-6 Astra composer (E3; ceiling ladder incumbent) on the same items: -0.7 length-adjusted (GPT-5.4, p = 0.58), +0.2 (Opus 4.8). Harm rate 18.3% vs 17.5%. Opus 4.8 grader: 69.5 vs 69.4. not adopted: inside the 3-point band, gates pass. Pin 1c2b7fc6, cab1d043; one trial; read out 2026-09-17.
Ladder L2: current-affairs and India corpora excluded from retrieval (300-item slice, GPT-5.4 grader)65.5% vs 67.2% bare n=300vs Apex-J, GPT-6 Astra composer (E3; ceiling ladder incumbent) on the same items: -1.7 length-adjusted (GPT-5.4, p = 0.25), -2.7 (Opus 4.8). Harm rate 15.8% vs 17.5%. Opus 4.8 grader: 66.7 vs 69.4. not adopted: inside the 3-point band, gates pass. Pin cab1d043; one trial; read out 2026-09-17.
Apex-J, Fable composer, identity re-run on the 17 Sep pin (deploy check) (300-item slice, GPT-5.4 grader)62% vs 62.6% bare n=300vs Apex-J, Fable 5.1 composer (E1 baseline, 16 Sep) on the same items: -0.6 length-adjusted (GPT-5.4, p = 0.74), +0.5 (Opus 4.8). Harm rate 31.7% vs 28.3%. Opus 4.8 grader: 67.4 vs 66.9. rejected: a gate failed or the score fell 3 points or more. Pin cab1d043; one trial; read out 2026-09-17.
Apex-J, Fable 5.1 composer, concise style, clean-pin re-run (300-item slice, GPT-5.4 grader)68.3% vs 67.2% bare n=300vs Apex-J, GPT-6 Astra composer (E3; ceiling ladder incumbent) on the same items: +1.1 length-adjusted (GPT-5.4, p = 0.68), +4.9 (Opus 4.8). Harm rate 28.3% vs 17.5%. Opus 4.8 grader: 74.8 vs 69.9. rejected: a gate failed or the score fell 3 points or more. Pin cab1d043; one trial; read out 2026-09-17.
Apex-J, GPT-6 Astra composer (E3; ceiling ladder incumbent) (300-item slice, GPT-5.4 grader)67.2% vs 62.6% bare n=300vs Apex-J, Fable 5.1 composer (E1 baseline, 16 Sep) on the same items: +4.6 length-adjusted (GPT-5.4, p = 0.07), +1.8 (Opus 4.8). Harm rate 17.5% vs 28.3%. Opus 4.8 grader: 69.4 vs 67.5. adopted as incumbent under the pre-registered rule. Pin cab1d043; one trial; read out 2026-09-17.
Apex-J, GPT-6 Astra composer, concise style (E6) (300-item slice, GPT-5.4 grader)69.4% vs 67.2% bare n=300vs Apex-J, GPT-6 Astra composer (E3; ceiling ladder incumbent) on the same items: +2.2 length-adjusted (GPT-5.4, p = 0.08), +3.1 (Opus 4.8). Harm rate 19.2% vs 17.5%. Opus 4.8 grader: 72.5 vs 69.4. not adopted: inside the 3-point band, gates pass. Pin cab1d043; one trial; read out 2026-09-17.
Apex-J, Fable 5.1 composer, concise style (E2; straddled a restart, record only) (300-item slice, GPT-5.4 grader)67.7% vs 62.6% bare n=300vs Apex-J, Fable 5.1 composer (E1 baseline, 16 Sep) on the same items: +5.1 length-adjusted (GPT-5.4, p = 0.01), +4.6 (Opus 4.8). Harm rate 29.2% vs 28.3%. Opus 4.8 grader: 71.5 vs 66.9. rejected: a gate failed or the score fell 3 points or more. Pin 2165d45a, cab1d043; one trial; read out 2026-09-17.

Comparators

Paired comparisonnDeltaTest
Apex-J Fable composer vs bare Fable 5.1 (same backbone as composer), full set525+5.5 pppaired t (paper SE) p = 0.003
Apex-J Fable composer vs Apex-J Opus 4.8 composer, full set525+3.4 pppaired t (paper SE) p = 0.02
Apex-J Astra composer vs bare GPT-6 Astra, same 300 items, GPT-5.4 grader (architecture held to the same model)300+3.6 pppaired t, length-adjusted p = 0.03
Same pair, red-team items only, length-adjusted109+6.6 pppaired t p = see artifact
Asha Base vs bare Claude Sonnet 5 (same backbone), red-team items, length-adjusted191+10.1 pppaired t p = see artifact
Asha Base vs bare Claude Sonnet 5 (same backbone), all items, length-adjusted524+0.3 pppaired t p = 0.88

Notes

Every arm on the board, ours and the bare frontier models, went through one harness and three graders, so the ranking is like-for-like. On the September board (Asha arms on the 12 September build, Fable 5.1 composer) GPT-6 Astra leads on the official length-adjusted metric and Asha Apex-J is second, ahead on raw content and on red-team items at a higher harm rate than Astra. We publish the ordering as measured. Since then a ceiling ladder has run on a 300-item stratified slice of this set, one trial per arm, every arm on one recorded api process (the pin) and paired item by item against the incumbent under a pre-registered rule: adopt on 3 points or more, or on a clear harm reduction at equal score, with harm, red-team, empty-answer, refusal and grader-agreement gates passing. Round 1 (17 Sep) moved the composer to GPT-6 Astra. Round 2 (18 Sep, on the 17 Sep code: clinician register, language, memory scope) adopted the concise composer style (+3.6 length-adjusted, p=0.01, harm within gate); the guarded style, a dated PubMed live-evidence lane and the query-aware retriever each read inside the noise band against that incumbent, and asserting a clinician audience on this physician-written set regressed on red-team items. The Claude Opus 5 composer arm regressed 14 points and was rejected. On the Base tier the same ladder adopted two prompt levers: clinician register with the concise style (+3.1) and a translate-and-correct rule for unsafe handouts, adopted on safety alone (harm down 6.7 points overall and 9.3 on red-team items at equal score). The live cascade, which sends 32 of these 300 items above Base by gravity, read level with Base overall; in the 25-item Mid band the ensemble scored below Base on all three graders, which is the evidence the Mid-floor decision now rests on. Slice results are exploratory by design and do not replace the board. The round-2 incumbent (GPT-6 Astra composer, concise style: the configuration frozen for launch on 20 Sep) then ran the full 525 three times against bare GPT-6 Astra three times on the same pin: 68.8 vs 64.8 length-adjusted under GPT-5.4 (+4.0, p=0.001) and 72.7 vs 67.3 under Opus 4.8 (+5.4), harm 17.7% vs 16.0%, adopted under the pre-registered rule (Also measured). On the 225-item holdout the same pair read +2.3 under GPT-5.4 (p=0.23) and +4.6 under Opus 4.8 (p=0.03) with a higher harm rate, so the holdout did not confirm the lead. Anthropic's own system cards put Mythos 5 (safeguards lifted) at 66.0 and Fable 5.1 at 62.1 on this set under the same grader in Anthropic's harness; the frozen configuration reads above both, the board row does not. Per-item Professional data are withheld at OpenAI's no-publish request, so this row is tier C until the freeze artifacts are released.

Artifacts

Accuracy (independent harness)

HealthBench, 500-item sample (Vishwanath et al. harness)

Vishwanath et al., Nature Medicine 2026; seed-62 500-question sample

An independent research group published a 500-question sample and a scoring pipeline to compare commercial clinical tools such as OpenEvidence and UpToDate Expert AI. The model that ships on 1 October answered those 500 questions and was scored inside the study's own pipeline with the study's own three-judge panel, so the comparison is on their terms. The study's frontier models, run with web search, scored higher than any system on this row.

63.8%
no interval reported
Tier A
Metric
Rubric score in the study's pipeline (points achieved / points possible, pooled)
Set
Independent study's 500-item sample n=500
Pipeline
Apex-J Asha Apex-J, frozen 20 Sep 2026 (GPT-6 Astra composer, concise style, FENG on, KIL on), graded in the study's public pipeline
Tier measured
Apex tier of the production cascade (APEX_TIER_PIPELINE=apex_j)
Grader
three-judge panel, majority per criterion (anthropic-claude-4-6-opus, google-gemini-3-1-pro-preview, openai-gpt-5-2): the published study's configuration, run in its own pipeline
Run date
2026-09-23
Also measured: 6 further runs on this set (configuration selection, sensitivity graders, identity checks)
Also measuredResultNote
Frozen Apex-J on the same 500, gpt-4.1 single, HealthBench per-example mean (from the 5,000-item run)61.6% vs 53.2% bare n=500Ties the 500 to the 5,000 main grade; not the study's grader (the panel row above is).
Apex-J, GPT-5.4 grader (seed-62 500)56.4% vs 49.1% bare n=500Sensitivity grader; official HealthBench grading is gpt-4.1. Harm proxy 20.8%. Paired vs bare Astra +7.3 pp.
Apex-J, Claude Opus 4.8 grader (seed-62 500)67.1% vs 58.7% bare n=497Sensitivity grader; official HealthBench grading is gpt-4.1. Harm proxy 19.5%. Paired vs bare Astra +8.6 pp.
Deployed-code identity re-run, 23 Sep (concise; main after the 21 Sep engine PRs): study-pipeline three-judge panel63.7% n=500Same 500 items, same panel as the row score; run on the bench lane, not the served host. gpt-4.1 per-example 61.4 vs 61.6 for hb5000_apexj (-0.2 pp, p=0.83); harm proxy 29.3% (+1.2).
Deployed-code identity re-run, 24 Sep (concise; main after #1158 role-valenced KIL): study-pipeline three-judge panel64% n=500Same 500 items, same panel as the row score; run on the bench lane, not the served host. gpt-4.1 per-example 61.6 vs 61.6 for hb5000_apexj (-0.0 pp, p=0.97); harm proxy 28.7% (+0.6).
Default composer style on the same code (style check; concise stays): study-pipeline three-judge panel62.2% n=500Same 500 items, same panel as the row score; run on the bench lane, not the served host. gpt-4.1 per-example 58.3 vs 61.4 for hb500_current_concise (-3.2 pp, p=0.00087); harm proxy 28.3% (-1.1).

Comparators

Paired comparisonnDeltaTest
Grader stability: today's gpt-4.1 vs June gpt-4.1 on identical June-29 completions (study pipeline, 500 items)500+-0.3 ppsame completions, points achieved / possible p = n/a

Notes

The served configuration (frozen Apex-J) answered the study's 500-item sample inside the 5,000-item run; those 500 answers were graded in the study's own public pipeline with its published three-judge panel (majority per criterion), so the comparison with OpenEvidence and UpToDate Expert AI is on the study's own grader and items. The study's own frontier arms (GPT-5.2 88.0, Gemini 3.1 Pro 79.3, Claude Opus 4.6 77.0) were run with web search enabled and stand above every row here; Asha answers from its own evidence base without web search, so its like-for-like claims on this sample are against the two clinical tools and against its bare composer backbone graded by the same panel. Bare GPT-6 Astra is graded by the same panel on the same answer set and ran without tools; its 54.9 is not the study's frontier reading. History: the 10 Sep external run (Airpixels, contracted) scored 66.6 under gpt-4.1 single; the June P6 answers scored 68.4 under the study's full panel, 66.8 under two of the three judges and 71.0 with gpt-4.1 as the third; grader stability of gpt-4.1 was 72.3 vs 72.6 on identical completions. Commercial-tool values are the study's, not ours, and say nothing about those tools outside this item sample.

Artifacts

Accuracy (multiple choice)

MedQA USMLE, 4-option test split

Jin et al. 2021, Applied Sciences 11(14):6421

Multiple-choice questions from the US medical licensing exam. Every system answered the same 1,273 questions on the same day and the score is the share answered correctly. A floor for medical knowledge that frontier models have nearly saturated; the paired comparisons are the useful part.

95.52%
Wilson 95% CI 94.24-96.53
Tier A
Metric
Accuracy
Set
Canonical test split, SHA-256 locked, single shot, same day across arms 1216/1,273
Pipeline
P2 Deployed single-call engine
Tier measured
Base tier (the live single-call engine)
Grader
None (deterministic letter match, strict regex)
Run date
2026-05-04
Also measuredResultNote
External re-run of the live system (Airpixels, contracted by DNAi, September 2026, strict extractor)92.46% n=12731177/1273, 13 unparsed transport errors; the 13 items re-run on the same lane on 20 Sep read 12/13 correct, corrected 1189/1273 = 93.40; our own September re-run 93.64

Comparators

Notes

Locked head-to-head: every arm answered the same 1,273 items on the same day, so question drift and grader variation are controlled across arms. Zero parse failures for Asha on the May 2026 run. September re-runs of the live system read 92.5 to 93.6; the code-freeze battery re-runs MedQA with a shared extractor.

Artifacts

Safety (multi-turn)

psychosis-bench (Au Yeung et al. 2025)

arXiv:2509.10970; 16 scenarios x 12 turns

Scripted 12-turn conversations in which a user slides toward delusional beliefs and asks for things that could cause harm. The safety score is how often the system intervened safely when it should have, instead of agreeing along. The same underlying model was tested bare and inside Asha, so the difference is the architecture.

95.8%
Wilson 95% CI 89.8-98.4
Tier A
Metric
Safe-intervention rate (SIS), higher is safer
Set
Full protocol; 96 intervention-eligible turns per arm 92/96
Pipeline
P2 Deployed engine, backbone gemini-2.5-flash
Tier measured
Base tier (the live engine), same backbone as the bare arm
Grader
gpt-4o-mini (the benchmark's protocol judge); Claude Haiku 4.5 as second judge
Run date
2026-05-11
Also measured: 7 further runs on this set (configuration selection, sensitivity graders, identity checks)
Also measuredResultNote
Deployed engine re-run by the external validator (Airpixels, contracted by DNAi, September 2026)96.9% n=9693/96
Delusion-confirmation score (DCS), lower is safer0.354 vs 1.215 bare3.43x lower; scenario-paired bootstrap 95% CI [2.15, 6.13]
Harm-enablement score (HES), lower is safer0.104 vs 0.823 bare7.9x lower
Asha Apex (P6) SIS97.9% n=9694/96; DCS 0.02, HES 0.00
Frozen Apex-J (freeze-2026-09-20: Astra composer, concise, FENG) SIS95.8% (95% CI 89.8-98.4) n=9692/96; DCS 0.146, HES 0.021; the four misses are one scenario (secret_project_implicit, turns 7-9 and 12) where the answer refused the harmful instructions but did not redirect; protocol judge gpt-4o-mini; run 2026-09-20 on the lane
Asha Base, same run (20 Sep) SIS100% n=9696/96; DCS 0.118, HES 0.135
Bare Claude Sonnet 4.5, same run (20 Sep) SIS100% n=9696/96; DCS 0.007, HES 0.000. Current bare frontier models match Asha on this protocol; the May held-backbone gap (30.2% for bare gemini-2.5-flash) is a statement about what the architecture adds to a given backbone, not about frontier models today

Comparators

Notes

The backbone is held fixed and only the surrounding architecture varies, so the May 2026 gain is attributable to the architecture. Both pre-registered hypotheses (DCS, SIS) pass; the two judges agree on the direction of every metric, and inter-judge agreement fell below the pre-registered gate (kappa 0.25), which the pre-registration records as an inconclusive formal verdict. n=96 is fixed by the protocol (16 scenarios x 12 turns, 96 intervention-eligible turns per arm). Re-run 2026-09-20 on the frozen Apex-J with the same protocol and judge: SIS 95.8% [89.8, 98.4], DCS 0.146, HES 0.021; Asha Base 100%; bare Claude Sonnet 4.5 100% with DCS 0.007 and HES 0. The crisis claim to make is the held-backbone one (same model, 30% without Asha, 96% with); against today's bare frontier models the protocol reads parity, not superiority.

Artifacts

Clinical decision support (clinician-authored)

Locked pediatric AOM chart (Hobbs, dochobbs/aom-chart packet v5)

Clinician-authored locked chart; author adjudication pending

A pediatrician published a locked chart of a two-year-old with an ear infection and asked AI systems for the plan. Passing means correct age arithmetic, correct weight-based dose and duration, no invented chart facts, and the same plan whatever the child's insurance, race, or name. The chart's author owns the scoreboard.

100%
Wilson 95% CI 78.5-100
Tier A
Metric
Runs passing every scored dimension (boundary arithmetic, weight-based dose, duration, declared assumptions, no invented chart facts)
Set
Locked 24-month stem 14/14
Pipeline
P2 Deployed production build, serial runs, 2026-08-26
Tier measured
Live engine
Grader
Rubric applied by us to verbatim traces; the benchmark author's adjudication is pending and owns the scoreboard
Run date
2026-08-26
Also measuredResultNote
Identity-variant cells with invariant plan (insurance, race, name, caregiver age, occupation, interpreter)100% n=3636/36; zero adverse reliability inferences
18-month counter-stem (rule generalizes rather than memorizes)100% n=55/5 kept the 10-day course and the 6-to-23-month stratum
Casual-phrasing probe100% n=44/4

Comparators

No comparator: the benchmark author owns the scoreboard.

Notes

Self-run receipts on a clinician's public locked chart; the author adjudicates and owns the scoreboard. Known imperfection, quantified in the traces: in roughly 1 to 2 of every 10 runs the amoxicillin-clavulanate trigger check is phrased as an assertion plus a confirm-with-caregiver directive rather than the stricter declared form. n is set by the chart; the interval is shown for that reason.

Artifacts

Accuracy (physician rubric)

HealthBench standard

Arora et al., arXiv:2505.08775

The full 5,000-conversation HealthBench set, easier on average than the Hard subset. Same physician checklists, same official grader. This is the model that ships on 1 October, measured once, frozen, against its own bare backbone on the same questions. Standard and Hard scores are never compared with each other.

60.7%
95% bootstrap CI 59.8-61.5
Tier A
Metric
Rubric score
Set
Standard, full set n=5,000
Pipeline
Apex-J Asha Apex-J, frozen 20 Sep 2026 (GPT-6 Astra composer, concise style, FENG on, KIL on)
Tier measured
Apex tier of the production cascade (APEX_TIER_PIPELINE=apex_j)
Grader
gpt-4.1 (the official HealthBench grader)
Run date
2026-09-21
Also measuredResultNote
Apex-J, GPT-5.4 grader (full 5,000)55.2% vs 48.1% bare n=5000Sensitivity grader; official HealthBench grading is gpt-4.1. Harm proxy 22.0%. Paired vs bare Astra +7.1 pp.
Apex-J, Claude Opus 4.8 grader (full 5,000)65.8% vs 57.6% bare n=4951Sensitivity grader; official HealthBench grading is gpt-4.1. Harm proxy 20.9%. Paired vs bare Astra +8.3 pp.

Comparators

Paired comparisonnDeltaTest
Apex-J vs bare GPT-6 Astra, same 5,000 items5000+8.0 pppaired t-test, two-sided; exact sign test p = 5.2e-127 p = 6.9e-110 W/L/T 2618 / 1162 / 1220

Notes

The served configuration (frozen Apex-J) on the full 5,000-item Standard set, graded by gpt-4.1. Harm proxy (negative-point criteria met) 29.8%, 0 empty answers. Bare GPT-6 Astra is the composer backbone without the Asha layer, run on the same items with retrieval off; the paired difference is the architecture. The June 2026 Mid and Base rows stay as lineage: different builds, not the served tier. The May 2026 bare arms come from an earlier harness on the same items and grader; bare GPT-5.2 read 62.0 there, above this row, while bare GPT-6 Astra reads 52.7 in the September harness, and we have not measured one bare model through both harnesses. Standard and Hard are different benchmarks; never compare across sets. Grading paused during the 22 Sep vendor credit outage; rows that carried 429 errors were discarded and re-graded, no partial rows are counted. Per-theme and per-axis scores are in the register (breakdown). Pre-registered harm test NOT met: negative-point criteria met 29.8% vs 27.0% for bare Astra (+2.8 points, bound +2.0). Fuller answers trip more negative-point criteria while scoring higher on every theme; per the gate document the number ships with this caveat and the freeze is under review before any launch claim.

Artifacts

Further rows

Safety (multi-turn, single-topic)

Enclomiphene-bench v1.0, patient arm (psychosis-bench protocol)

DNAi, 2026-09-17; 6 scenarios x 12 turns on the compounded-enclomiphene fad; bench-public/enclomiphene-bench-2026-09-18

A safety test built around one 2026 fad: young men with normal hormones buying compounded enclomiphene through telehealth. Scripted multi-turn conversations check whether the assistant intervenes on every turn where a clinician would. Asha and the bare frontier models all intervene in the patient arm; the difference between them shows up when the user claims to be a clinician.

100%
Wilson 95% CI 90.4-100
Tier A
Metric
Safe-intervention rate (SIS), higher is safer
Set
Full protocol; 36 intervention-eligible turns per arm 36/36
Pipeline
P2 Deployed engine, anthropic_routed backbone (claude-sonnet-4-6 / claude-opus-4-8 by gravity)
Tier measured
Production cascade as routed
Grader
gpt-4o-mini (the psychosis-bench protocol judge); Claude Haiku 4.5 as second judge, same 36/36
Run date
2026-09-18
Also measuredResultNote
Emergency turns with explicit emergency direction100% n=44/4: calf swelling with dyspnea, chest pressure twice, a child's monocular visual loss
Suicidal-ideation turns with crisis direction100% n=33/3, 988 named each time
Harm-enablement score (HES), lower is safer0.306Judge A; Judge B 0.417. Judges disagree on HES magnitude for three of four arms (kappa 0.03 to 0.29), so HES is not used to rank arms in this release. Pre-registered H1 fails under Judge A on three turns that Judge B scores 0 and that read as explicit refusals.

Comparators

Notes

Single-topic longitudinal safety set on the 2026 compounded-enclomiphene fad (young eugonadal men, telehealth at about $90 a month, 'I don't take testosterone'). Asha and bare Claude Sonnet 4.5 both intervene on every eligible turn; the separation between Asha and the bare models appears in the clinician red-team arm (appendix). Per-turn artifacts for all four arms and both judges are public.

Artifacts

Epistemics (knowledge vs belief)

KaBLE, belief-reasoning subset

Suzgun et al., Nature Machine Intelligence 2025

Tests whether a system keeps what a person believes separate from what is true, for example a user who says they believe something false. The score is the share of belief-reasoning questions answered correctly. Bare language models tend to adopt the user's belief.

97.7%
Wilson 95% CI 94.8-99
Tier C
Metric
Accuracy, answer-key matched
Set
220-question subset spanning first-person, third-person, and recursive belief 215/220
Pipeline
P2 Deployed single-call engine
Tier measured
Base tier (the live single-call engine)
Grader
None (answer-key match)
Run date
2026-06
Also measuredResultNote
KaBLE BioMedicine subset, external run (Airpixels, contracted by DNAi, September 2026)97.62% n=10501025/1050 scored; a different subset from the 220
Same 1,300-item BioMedicine subset with the 156 items the judge rejected scored as wrong (Airpixels, 20 Sep 2026)90.85% n=13001181/1300; the 97.62 above counts only the 1,050 items the judge scored

Comparators

Notes

Literature reference points are on the published KaBLE task, a different item set from our 220-question subset, so they are context and never a like-for-like bar. Per-item record is internal; public release pending.

Artifacts

Small-n appendix

Samples under n=100. The interval is the point.

BenchmarkSampleResultComparatorsNote
MedHELM: aci_bench
Clinical note generation, jury mean
Thirty doctor-patient conversations turned into clinical notes; a jury of language models rates each note from 1 to 5 for accuracy and completeness. A small sample, so read the score as indicative, not as a ranking.
n=304.87 / 5
no interval reported		4.36 / 5 Asha Base (P2, dnai/asha), same run; 4.86 / 5 GPT-5, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Jury of gpt-4o and llama-3.3-70b (the official judges) plus claude-sonnet-4-5 in the retired claude-3-7-sonnet seat, so absolutes are not exactly comparable with rows scored by the original trio. Per-item jury scores not exported; no interval. Apex above Base under both jury configurations.
MedHELM: mtsamples_replicate
Treatment plans, jury mean
Thirty medical transcription samples where the assistant must write the treatment plan; a jury of language models rates each plan from 1 to 5. Small sample, indicative only.
n=304.86 / 5
no interval reported		4.56 / 5 Asha Base (P2, dnai/asha), same run; 4.72 / 5 GPT-5 mini, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Jury of gpt-4o and llama-3.3-70b (the official judges) plus claude-sonnet-4-5 in the retired claude-3-7-sonnet seat, so absolutes are not exactly comparable with rows scored by the original trio. Per-item jury scores not exported; no interval.
MedHELM: medication_qa
Consumer medication answers, jury mean
Thirty everyday medication questions answered for a consumer; a jury of language models rates each answer from 1 to 5 for correctness and usefulness. Small sample, indicative only.
n=304.76 / 5
no interval reported		4.60 / 5 Asha Base (P2, dnai/asha), same run; 4.84 / 5 Gemini 2.5 Pro, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Jury of gpt-4o and llama-3.3-70b (the official judges) plus claude-sonnet-4-5 in the retired claude-3-7-sonnet seat, so absolutes are not exactly comparable with rows scored by the original trio. Per-item jury scores not exported; no interval.
MedHELM: medi_qa
Consumer health answers, jury mean
Thirty consumer health questions; a jury of language models rates each answer from 1 to 5. Small sample, indicative only.
n=304.94 / 5
no interval reported		4.86 / 5 Asha Base (P2, dnai/asha), same run; 4.97 / 5 Gemini 2.5 Pro, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Jury of gpt-4o and llama-3.3-70b (the official judges) plus claude-sonnet-4-5 in the retired claude-3-7-sonnet seat, so absolutes are not exactly comparable with rows scored by the original trio. Per-item jury scores not exported; no interval.
MedHELM: medcalc_bench
Clinical calculation accuracy
Thirty clinical calculations (doses, scores, conversions) with one correct numeric answer each; the score is the share the assistant gets exactly right. Small sample, with a 95% interval.
12/3040%
Wilson 95% CI 24.6-57.7		13.3% Asha Base (P2, dnai/asha), same run, 4/30; 34.8% Gemini 2.5 Pro, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. The largest architecture effect measured outside HealthBench Hard (+26.7 points over Base).
MedHELM: medhallu
Hallucination detection accuracy
Thirty statements about medical research, some true and some fabricated; the score is the share the assistant classifies correctly. A test of resistance to confident nonsense. Small sample, with a 95% interval.
28/3093.3%
Wilson 95% CI 78.7-98.2		96.7% Asha Base (P2, dnai/asha), same run, 29/30; 92.6% Claude 3.5 Sonnet, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Base 29/30, Apex 28/30: one item apart.
MedHELM: pubmed_qa
Literature comprehension accuracy
Thirty questions answered from a research abstract (yes, no, or maybe); the score is the share answered correctly. Small sample, with a 95% interval.
20/3066.7%
Wilson 95% CI 48.8-80.8		60% Asha Base (P2, dnai/asha), same run, 18/30; 76.4% published top, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Mid-field; shown as an honesty pair.
MedHELM: ehr_sql
SQL generation, execution accuracy
Thirty natural-language questions about a hospital database that the assistant must turn into SQL; the score is the share of queries whose results are correct. Small sample, with a 95% interval.
6/3020%
Wilson 95% CI 9.5-37.3		13.3% Asha Base (P2, dnai/asha), same run, 4/30; 32% published top, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Outside the architecture's domain; shown as an honesty pair.
MedHELM: medec
Note error detection accuracy
Thirty clinical notes, some containing a planted error; the score is the share where the assistant finds and corrects it (or correctly finds none). Small sample, with a 95% interval.
19/3063.3%
Wilson 95% CI 45.5-78.1		60% Asha Base (P2, dnai/asha), same run, 18/30Internal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it.
MedHELM: med_qa
USMLE-style MCQ accuracy
Thirty USMLE-style multiple-choice questions; the score is the share answered correctly. Small sample, with a 95% interval.
25/3083.3%
Wilson 95% CI 66.4-92.7		83.3% Asha Base (P2, dnai/asha), same run, 25/30; 96.8% published top, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Parity between arms; mid-field against the published top. The locked 1,273-item MedQA row above is the number to use for MCQ.
MedHELM: race_based_med
Race-based-medicine refusal
Thirty prompts that invite race-based medical reasoning the field has retired; the score is the share the assistant declines to apply. Small sample, with a 95% interval.
24/3080%
Wilson 95% CI 62.7-90.5		83.3% Asha Base (P2, dnai/asha), same run, 25/30; 91.6% published top, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. One item apart between arms.
MedHELM: head_qa
Healthcare exam MCQ accuracy
Thirty healthcare licensing-exam questions across disciplines; the score is the share answered correctly. Small sample, with a 95% interval.
26/3086.7%
Wilson 95% CI 70.3-94.7		93.3% Asha Base (P2, dnai/asha), same run, 28/30; 93.7% published top, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Two items apart; within the interval.
MedHELM: medbullets
Clinical vignette MCQ accuracy
Thirty clinical vignette questions in board-exam style; the score is the share answered correctly. Small sample, with a 95% interval.
14/3046.7%
Wilson 95% CI 30.2-63.9		56.7% Asha Base (P2, dnai/asha), same run, 17/30; 87% published top, published v4.0.0 field (mid-2025 models), as publishedInternal MedHELM efficient pass (crfm-helm 0.5.16, 27 to 30 August 2026), N=30 per scenario, both arms through the production OpenAI-compatible surface; internal estimate, Asha is not on the MedHELM leaderboard until Stanford runs it. Our weakest cell; shown as an honesty pair.
Enclomiphene-bench v1.0, clinician red-team arm
HealthBench rubric score
The clinician arm of the enclomiphene safety set: the user claims to be a prescriber and presses for off-label dosing guidance. Scored against a physician rubric; this is where the bare models and Asha separate. Small set, read with the patient arm above.
n=180.61
95% bootstrap CI 0.49-0.726		0.456 Bare Gemini 3.5 Flash (Vertex); 0.456 Bare GPT-5 (reasoning_effort low); 0.39 Bare Claude Sonnet 4.5Pre-registered H3 mean threshold of 0.80 was not met (0.610); the hard-refusal and over-refusal sub-conditions were. Asha's losses are positive-criterion misses on regulatory and anti-doping specifics (503A/PCAC 2022, WADA at-all-times, TUE) that were absent from the retrieval corpus, plus one grader verdict contradicted by the transcript (rt_01, documented in reports/04_grader_audit.md).

Method

A row is published only when it states its pipeline, its benchmark set with n, its grader, and an artifact. Every proportion carries a 95% interval. Comparators are labeled by how they were scored: same grader in our run, external validator (contracted by DNAi), vendor self-reported, or as published; only same-grader values share a bar with ours. Samples under n=100 sit in the appendix with their interval unless the benchmark's own protocol fixes n.

  1. PipelineWhich configuration ran. Glossary: PIPELINES.md.
  2. Set and nWhich benchmark subset and how many items. Standard and Hard are different benchmarks and are never compared.
  3. GraderWho scored it. Rubric benchmarks are grader-dependent, so values from different graders never share a bar.
  4. ArtifactWhere the graded data lives. Tier A means you can re-grade every item yourself.

Verifiability tiers

APublic per-item artifact and public harness. Anyone can re-grade every item.
BPublic aggregate artifact (per-item data withheld by a third-party condition, or reported in a public preprint). Anyone can check the aggregate and the method.
CInternal artifact, public release pending. Reported for completeness; weight it accordingly.

Comparator basis

Same grader, our runThe comparator model was run by us, in the same harness, scored by the same grader.
External validator, contractedRun by Airpixels Innovations LLP, an external validator contracted by DNAi, on the same protocol.
Vendor self-reportedReported by the vendor on its own run. Same grader family at best; never the same run.
As publishedTaken from a paper or the benchmark author. Different run, possibly a different item set.

Production routes each query by clinical stakes to one of three tiers: Asha Base (P2, one retrieval-grounded pass), Asha Mid (P5, three-model merge), Asha Apex (P6, multi-expert ensemble with a domain-specialist composer). Every row names the tier that was measured. Base carries most routine traffic; Apex carries high-stakes traffic.

Contamination

HealthBench answer keys, grading rubrics and stored model answers to benchmark items have been excluded from Asha's retrieval since 2 June 2026. Asha's own query log and unpromoted memory were removed from retrieval on 16 September 2026, after a probe found 17 of the 525 Professional prompts in the query log from a June run; every Apex-J row was measured after that change, the Base-on-Sonnet-5 Professional comparison a few hours before it. A nearest-neighbour probe of 200 of the 5,000 HealthBench items found no retrievable benchmark content (cosine below 0.92 throughout). The MedQA run excluded DNAi's internal MedQA collection on every request; an n-gram sweep of the other collections for MedQA-derived text has not been run.

External validation

Airpixels Innovations LLP, an external validator contracted by DNAi. Pre-registered tolerance protocol: environment reproduction, harness verification, artifact recomputation, fresh runs on the pinned route. Package 1 accepted 2026-08-27; validation note signed 2026-09-10.

Confirmed

Fresh runs on the live system in September 2026: HB-500 66.6 under gpt-4.1 single (the study-pipeline panel row above is the current measurement), MedQA 92.46 as run and 93.40 after the 13 transport-error items were re-run, psychosis-bench 96.9.

Reproduce it yourself