Choosing an AI
assistant for CERRA

01 / Model choice

A practical balance of quality and waiting time

26B at 8-bit runs, with a tradeoff: Gemma 4 26B A4B Q8_0 passed 57/72 attempts (79.2%), with an 8.16 s median reply and partial CPU placement. This small paired test does not establish a quality advantage for either configuration. The 26B package is evaluated here but has not been added to the production selector.

The local choices remain Gemma 4 12B and Qwen3.5 9B. In their earlier September 29 paired campaign, Qwen3.5 passed 59/72 attempts (81.9%), with a 2.95 s median reply; Gemma passed 60/72 with a 3.18 s median. That small observed gap does not establish superiority or equivalence. Retired local choices remain in the historical results; a saved retired choice returns to Gemma.

Latest Qwen measurement: Qwen3.8 27B Q4_K_M with full GPU residency retained 63/72 strict successes with a 4.25 s median reply on October 7. One benchmark model ran at a time while existing GPU workloads remained active. The older 14.50 s median below describes the separate September 29 run with CPU offload.

GPT‑6.1 Sol remains hidden from production choices. A new four-model comparison including GPT‑5.5 addresses the limits of comparing earlier runs with different settings; it does not establish a general model ranking. GPT choices use OpenAI’s API and send your question and relevant source evidence to that service. Gemma and Qwen run on CERRA’s local model service and release their model allocation after each request. You can change models in the assistant.

02 / Study design

Evaluate the whole assistant workflow

The study covers Maryland crash-data questions, Baltimore High Injury Network (HIN) source queries, and changes to live map filters. Most numerical facts are calculated by CERRA after the model selects a query plan. A correct count therefore measures the combined application workflow, not the model’s arithmetic in isolation.

1. Keep the tasks fixed. Each model answers 24 primary tasks three times, giving 72 scored attempts. Four reference controls add 12 attempts per model and are excluded from the primary score.
2. Require the intended outcome. Strict success requires the right operation, source-grounded evidence, model identity, relevant caveats and exact requested UI effect. Unrequested filter changes fail.
3. Test actions in the browser. Nine tasks per model use the real Maryland UI; the other 19 use authenticated requests to the same application endpoints. There are 21 scored browser attempts, or 27 including controls.
4. Retain failures and uncertainty. No failed attempt is omitted or externally retried. Intervals resample 24 task clusters while keeping each task’s three repetitions together. HIN narratives receive AI-assisted, identity-blind rubric review, not human expert validation.

Reply timing includes planning, source-data lookup, internal validation retries and model loading where applicable. It stops before the completed UI render. One interrupted Sol request in the earlier September 30 two-model comparison has no completed application timing; it remains a primary failure and is excluded only from timing summaries, with the observed count stated. P90 describes the slower end of observed replies; none of these timings is a response-time guarantee.

03 / Results

Read quality, latency and memory together

The table contains 14 model identifiers and 26 configuration–campaign observations across 11 campaigns. Repeated models retain separate rows because settings or execution conditions can differ. Rows within the historical production group are ordered by observed strict score, not by a claim of statistically established superiority.

Task success and waiting time

Each point is one model in one campaign. The horizontal axis measures strict end-to-end task success, including the requested application action; it is not isolated answer accuracy. The vertical axis is median application request time, including measured failures. Colors identify models or local families; shapes and observation codes identify campaigns. The historical overview supplies context across different dates, reasoning settings, hardware and execution paths—not a single controlled ranking.

Horizontal and vertical bars are separate marginal 95% task-cluster intervals, not a joint confidence ellipse. The original success intervals are unchanged. Median-latency intervals for the earlier 18 observations were added post hoc, using 10,000 draws of 24 task clusters, seed 20261001. The four October 2 observations use the same task-cluster method specified before collection, seed 20261002. Both retain all three attempts per sampled task and use common draws within a campaign. The earlier Sol pair (G2) has 71 measured durations; its missing duration is not imputed and the attempt remains a scoring failure. P90 is a different descriptive statistic. Bar overlap does not replace the paired comparisons below.

The October 7 Qwen and October 9 Gemma median intervals were added post hoc on October 9, using 10,000 shared draws of 24 task clusters within each campaign, retaining all three repetitions and measured failures, seed 20261009. The original strict-success intervals and historical observations remain unchanged.

Read the reasoning setting alongside the score. The latest four GPT models all explicitly requested medium, with an 8,192-token combined reasoning/visible-output cap per call. Earlier API rows without an effort parameter are labeled “Not explicitly set”; no default is reconstructed. Local “Thinking off” reports the recorded switch, not an equivalent hosted effort level. No audited run explicitly requested Ultra. Matching requested effort and token caps still does not equalize internal compute.

Scroll the table to see reasoning settings, reply times and GPU memory →

The tasks and strict rubric are shared; dates, application versions and runtimes vary. The local-application campaigns from September 29 onward exclude the public-site network path; API campaigns include the request to OpenAI. Cross-run timing is contextual, and scores are not pooled. The intervals describe uncertainty within this small task suite. The earlier two-model Sol timing row has 71 measurements because one interrupted outcome was not retained. The September 30 three-model campaign retained all outcomes and timings. The October 2 campaign is reported separately below.

Gemma 4 12B and Qwen3.8 each completed all 15 supported filter changes with no observed UI-state violations in their paired campaigns. The October 9 Gemma 26B Q8_0 run also passed all 15 supported changes, but produced three UI-state violations in its 21 scored browser attempts. In the September 29 API extension, GPT‑6.1 Sol completed 15/15 supported filter changes, with 0/21 observed UI-state violations. These checks complement answer accuracy.

GPT‑5.5 in a four-model comparison

GPT‑5.5, GPT‑5.4, GPT‑6 Astra and GPT‑6.1 Sol were tested together on October 2 using the same frozen application, source data, prompts and 28 tasks. Each model completed three attempts per task: 72 primary attempts and 12 separate controls. Historical observations remain separate.

All four requested Medium reasoning, an 8,192-token combined reasoning/output budget and a 60-second model-call timeout through the US Responses API. No temperature was set. The provider returned gpt-5.5-2026-04-23 for GPT‑5.5 and gpt-5.4-2026-03-05 for GPT‑5.4; Astra and Sol returned their aliases. These settings match at the API boundary, while provider-internal compute and hardware remain unknown.

The four models ran adjacently for each task and repetition. A balanced schedule gives each model equal exposure to every position across the primary tasks and, separately, the controls. With three repetitions, each model visits three of four positions per task. Identity checks were registered and tested before collection; scoring requirements and source oracles were retained.

Six exploratory, unadjusted comparisons using 10,000 paired task-cluster resamples. These intervals do not establish a universal ranking or model equivalence. The chart’s marginal success and median-latency intervals were specified before this campaign; they are separate from the paired-difference intervals.

GPT-5.4: 63/72 strict successes, 3.91 s median and 11.35 s P90 (72 recorded primary timings); 12/12 controls. GPT-5.5: 68/72 strict successes, 3.35 s median and 9.02 s P90 (72 recorded primary timings); 12/12 controls. GPT-6 Astra: 63/72 strict successes, 2.54 s median and 5.06 s P90 (72 recorded primary timings); 12/12 controls. GPT-6.1 Sol: 65/72 strict successes, 3.24 s median and 6.81 s P90 (72 recorded primary timings); 12/12 controls.

GPT-5.5 has the highest observed strict score in this campaign: 68/72 (94.4%), compared with Sol 65/72 (90.3%) and GPT-5.4 and Astra each 63/72 (87.5%). Its advantage over GPT-5.4 is 6.94 percentage points, with a paired 95% interval from -2.78 to +19.44 points. All six exploratory pairwise intervals include or touch zero. These data establish neither a stable ranking nor equivalence. Astra has the shortest observed median request time (2.54 s); GPT-5.5 takes 3.35 s.

All four models passed 12/12 separate controls and 15/15 supported filter changes, with zero unrequested state violations in 21 primary browser attempts each. All 336 application outcomes and timings were retained, including two GPT-5.4 HTTP 422 validation rejections on MD07.

On MD01, all 12 original model plans request the correct filtered-data scope. The unchanged application then switches them to viewport scope and returns zero instead of 266 reports. All four fail the end-to-end task because of this shared reconciliation defect.

On MD04, GPT-5.5 returns the correct Baltimore County total of 151,679 plus all eight annual rows in all three attempts. Sol supplies the required rows twice; Astra gives the correct aggregate but no yearly rows; GPT-5.4 refuses all three. The frozen rubric requires annual rows although the question does not explicitly request them, so this difference partly concerns an ambiguous output contract.

On MD07, GPT-5.5, Astra and Sol pass all three selected-crash queries. GPT-5.4 fails all three: two validation rejections and one unsupported-operation response. Conversely, GPT-5.4 and GPT-5.5 pass all three MD09 original-versus-estimated location comparisons, while Astra and Sol refuse them. These are task-specific differences, not evidence that one model is worse across every capability.

One GPT-5.5 HIN14 answer correctly refuses a causal road-diet effect estimate but additionally asserts that project designs and exposure changes are absent. The supplied evidence does not establish those extra absences, so the sealed identity-blind review fails claim support for that answer. This is a semantic judgment by one independent AI reviewer, not human domain-expert validation.

All 336 scheduled attempts enter their designated primary or control denominator. The campaign retained 336 application outcomes; 0 interrupted outcomes are recorded as failures. The 48 narrative records received independent, identity-blind AI-assisted review before aggregate results were released. This is not human domain-expert validation.

The shared MD01 scope-reconciliation defect and MD04 annual-row ambiguity remain part of the frozen historical test. The proposed English transportation benchmark is a separate, untested review draft.

Earlier three-model comparison: GPT‑6 Astra

GPT‑6 Astra, GPT‑5.4 and GPT‑6.1 Sol were evaluated together on September 30 using the same frozen application, source data, 28 tasks and prompts. Each task was repeated three times per model. This contemporaneous comparison and the earlier two-model campaign remain separate historical observations.

All three used the US Responses API, explicit medium reasoning, an 8,192-token combined reasoning/output budget and a 60-second model-call timeout. Each task’s three models ran adjacently, with counterbalanced order: each model occupied each position once per task across repetitions. Matching exposed settings does not equalize internal compute or provider load.

Three exploratory pairwise comparisons, each using 10,000 task-cluster resamples. These intervals are not adjusted for multiple comparisons and do not establish a universal ranking or model equivalence.

GPT-6 Astra: 63/72 strict successes, 2.49 s median and 4.92 s P90 (72 recorded primary timings); 12/12 separate controls. GPT-5.4: 64/72 strict successes, 3.82 s median and 11.16 s P90 (72 recorded primary timings); 12/12 separate controls. GPT-6.1 Sol: 64/72 strict successes, 3.19 s median and 5.91 s P90 (72 recorded primary timings); 12/12 separate controls.

All three configurations passed 12/12 controls and had zero UI-state violations in 21 scored browser attempts. Astra and Sol completed 15/15 supported filter actions; GPT-5.4 completed 14/15. Its one strict failure redundantly included unchanged controls, so the inherited functional-equivalence sensitivity credits it (65/72 versus 64/72 strict).

All nine raw plans for MD01 selected the correct filtered scope. The shared application changed that scope to the viewport and returned zero instead of 266 reports. On MD04, Astra returned the correct aggregate but omitted the rubric-required annual rows in all three attempts; Sol supplied them once and GPT-5.4 twice. This question does not explicitly request yearly rows, so the rubric ambiguity affects interpretation.

On the supported selected-crash query MD07, Astra and Sol passed all three attempts; GPT-5.4’s plans led to three HTTP 422 application-validation rejections. On the supported location-comparison query MD09, GPT-5.4 passed all three, while Astra and Sol each refused all three. These are distinct workflow failures, not evidence of a broad capability ranking.

Scorer registration correction. The inherited HIN scorer did not recognize Astra’s new model identifier. Its allowlist was amended after collection, with the original grades retained. The correction changes only model registration; task requirements, source oracles and the sealed narrative review are unchanged. Independent replay verified exactly 30 affected Astra records (24 primary and six controls), with all other checks unchanged and 1,260 historical grades preserved. Astra’s incomplete-registry result was 39/72 primary and 6/12 controls; registration gives 63/72 and 12/12. The former counts reflect the scorer defect, not model performance. Final scores use this disclosed amendment; no inference was repeated.

The frozen MD01 scope-reconciliation defect and MD04 output-granularity ambiguity remain in this comparison. Their correction belongs to the separately reviewed next benchmark version, so a change in the application or scoring cannot be mistaken for a model improvement.

Why GPT‑5.4 remains competitive here

In the September 30 three-model campaign, GPT‑5.4 and Sol tie at 64/72; Astra passes 63/72. The earlier 66/72 GPT‑5.4 result used a different date and unspecified reasoning effort with smaller output caps. It cannot establish that GPT‑5.4 generally outperforms the newer models. Model age alone does not predict performance on this particular application contract.

CERRA constrains the model to choosing supported query plans and actions, while the application calculates most numerical results. Models can therefore differ in how well they follow these specific interfaces. The observed tradeoffs run in both directions: GPT‑5.4 passes all three MD09 location comparisons where Astra and Sol refuse, but fails all three MD07 selected-crash attempts where both newer models pass. Shared application defects and ambiguous task requirements also affect scores. With only 24 primary task clusters, this is evidence about configured assistants in CERRA, not a causal explanation based on training, model generation or general transportation expertise.

Earlier two-model GPT comparison

GPT‑5.4 and GPT‑6.1 Sol were rerun with the same frozen application, 28 tasks, source data, prompts and graders. Both used the US Responses API, explicit medium reasoning, an 8,192-token combined reasoning/output limit and a 60-second model-call timeout. Adjacent model pairs used balanced first-model order in each repetition. Matching requested settings does not imply equal internal compute, and this experiment does not change GPT‑5.4’s production settings.

Sol’s median/P90 recorded request times were 2.83/5.97 s (71 measurements), versus 3.51/8.58 s for GPT‑5.4 (72). Both completed 15/15 supported filter changes, had zero unrequested state violations in 21 scored browser attempts, and passed 12/12 separate controls. The earlier 66/72 GPT‑5.4 and 64/72 Sol scores remain historical observations with different settings and dates.

One infrastructure interruption is retained. The runner stopped during Sol’s second MD07 attempt after its provider call returned, but before the application outcome was recorded. It was not retried or reconstructed from provider output. It counts as a primary non-success, with no invented request time. Excluding both members of that pair gives Sol 63/71 versus GPT‑5.4 64/71 (difference −1.41 points; paired interval −11.27 to +5.80). The missing outcome permits 63–64 Sol successes out of 72, so it can account for one of the two observed pass-count differences. These are disclosed post-interruption sensitivity checks.

Raw plans now make one shared failure attributable: both models requested the correct filtered-data scope in all six MD01 attempts, but the application changed it to the visible map and returned zero instead of 266 reports. On MD04, GPT‑5.4 refused all three attempts; Sol returned the correct total but omitted yearly rows twice, and passed once. The frozen rubric requires those rows although the question does not explicitly request them. On MD09, Sol refused the supported location comparison in all three attempts while GPT‑5.4 passed all three. Both passed all MD11 year-filter changes. These localized differences do not support the claim that Sol is worse across almost every capability.

Historical GPT‑6.1 Sol API extension

The earlier run retained all 84 attempts, including 12 reference controls, and used the same 28 frozen tasks, data snapshot and strict rubric. It passed all three repetitions on 21/24 primary tasks; 64/72 primary attempts were strictly successful within 30 seconds. Its 12 HIN narratives received the same identity-blind, AI-assisted review procedure.

The tested configuration uses gpt-6.1-sol through the US Responses API, with medium reasoning and an 8,192-token combined reasoning/output budget per call. Earlier API configurations used the application’s smaller output budgets without explicitly setting reasoning effort. The new budget accommodates reasoning tokens; prompts, validators, source queries, internal repair rules and the 60-second model-call timeout are unchanged. This is a configured-assistant observation, not an isolated model-generation comparison.

04 / Local comparisons

Configuration and model placement both matter

Historical Qwen comparison · September 29

To examine the newer Qwen configuration under a shared setup, we reran Qwen3 14B alongside Qwen3.8 27B using the same frozen application, data, questions and scoring. Each model completed 72 primary attempts and 12 controls: 168 attempts in total.

This supports higher strict success for the newer, larger configuration within this suite and runtime. It also took longer: median replies were 14.50 s versus 3.77 s, and P90 replies were 29.81 s versus 7.20 s.

Both used Q4_K_M weights, temperature 0.1, an 8,192-token context and disabled thinking. Retained runtime logs show both models used partial CPU offload on the shared RTX 5090. Model generation, parameter count, architecture, cold loading and shared load are not separated by this design. The paired interval applies to these two Qwen configurations; it is not a Qwen3.8–Gemma comparison.

A lighter model: Qwen3.5 9B versus Gemma 4 12B

A separate completed run kept the same 28 tasks, source snapshots and strict scoring, with 168 attempts and 24 identity-blind narrative reviews. Qwen3.5 passed 59/72 primary attempts, with a 2.95 s median reply; the contemporaneous Gemma run passed 60/72, with a 3.18 s median. The Qwen-minus-Gemma paired 95% task-cluster interval was −20.8 to +19.4 percentage points, so the small observed difference establishes neither superiority nor equivalence.

Sampled model residency was 5.34 GiB for Qwen3.5 and 7.80 GiB for Gemma, with no CPU offload observed in retained samples. Qwen3.5 completed 12/15 supported filter changes, versus Gemma’s 15/15; both had zero unrequested changes in 21 scored browser attempts. Qwen omitted the requested GPS-filter reset and one HIN narrative added an unsupported claim. Gemma remains the production default; Qwen3.5 is the lighter production alternative.

Both used Q4_K_M, an 8,192-token context, temperature 0.1, disabled thinking and unloading after each call. Package sampling defaults differ, and loading is included in reply times. This campaign retained the shared GPU workload and preceded the later removal of idle Qwen3 8B residency.

05 / Failure patterns

A higher score still leaves important gaps

Map actions and source queries are different capabilities

In the October 9 Gemma pair, both configurations passed 15/15 supported filter changes and all 24 HIN source/reasoning attempts. On one ambiguous-location task, however, the 26B Q8_0 model changed the year filter on all three repetitions instead of clarifying; the 12B Q4_K_M model clarified and preserved the interface. The 26B model’s 3/21 UI-state violations remain failures under both strict and secondary scoring.

Qwen3.8 passed 15/15 supported filter changes in both Qwen campaigns. Its Qwen3 baseline passed 11/15 on September 29 and 12/15 on October 7. The baseline applied the requested year before clarifying the ambiguous Baltimore location in three attempts in each campaign, violating the frozen clarify-first policy. In the earlier production evaluations, GPT-5.4 left the map unchanged on all six ambiguous or unsupported UI requests; GPT-5.4 mini completed 14/15 supported changes, versus 12/15 for GPT-5.4. Neither GPT model had an observed UI-state violation in its scored browser attempts.

Some failures persist across models

All eight configurations in the production evaluations failed the current-crash-selection data-query task. Both models in the September 29 Qwen paired run also failed that task and the year-breakdown task. These are end-to-end failures. The shared application forces viewport scope when it detects “visible,” including “outside the visible map”; raw planner outputs were not retained, so the model’s separate contribution cannot be isolated. For the year-breakdown task, both returned the correct aggregate and filters but missed the frozen requirement for yearly rows, which the prompt did not state explicitly. Qwen3.8 additionally failed all three selected-crash weather queries with HTTP 422 validation rejections—a regression relative to its paired baseline.

In the September 29 extension, GPT‑6.1 Sol failed all three repetitions of the current-selection and year-breakdown tasks, with the same scope limitation and yearly-row rubric ambiguity. It returned “unsupported” on two of three available original-versus-estimated location comparisons; the third repetition passed. All 84 application requests returned HTTP 200, illustrating why a successful request alone is insufficient evidence of task success.

Strict failures need interpretation

Operation labels, exact response fields and relevant caveats are part of the frozen contract. A strict failure can therefore occur even when a displayed number is correct. The inherited functional-equivalence check raises Qwen3 from 49/72 to 55/72 on September 29 and from 51/72 to 57/72 on October 7; Qwen3.8 stays at 63/72 in both. These secondary results do not replace the strict score.

The earlier Gemma 4 review found two narratives that confused the segment-boundary caveat with the intersection-distance caveat. Three Qwen3 control answers omitted the distinction between people and crash counts. Such errors matter even when the application returns the right data.

See the complete task-family breakdown and UI outcomes below ↓

06 / Interpretation

A decision aid, within a bounded test

This is a small, implementation-aware convenience sample. It does not establish a universal model ranking or validate a scientific HIN method. Data retrieval, planning, validation and narration all contribute to the outcome; failures cannot always be attributed to the model alone.

Several apparent differences remain uncertain. In the production tests, GPT-5.4’s 11.1-point advantage over GPT-5.4 mini had a paired 95% interval of −4.2 to +26.4 points. Gemma 4’s 9.7-point advantage over Qwen3 14B had an interval of −8.3 to +29.2 points. Neither establishes a reliable general advantage.

Local configurations vary in size, quantization, sampling and execution conditions. Historical API runs use provider aliases rather than independently pinned snapshots; the matched GPT‑5.4 and GPT‑5.5 runs verify dated provider identities, while Astra and Sol remain aliases. Network time and service load also vary. The September 29 paired Qwen interval excludes zero, while the October 7 interval reaches zero. Neither comparison isolates a release-generation effect, and neither supports a universal model ranking.

For current interactive use, the evidence supports a practical policy: start with Gemma 4 for a balance of observed quality and response time, offer the smaller Qwen3.5 alternative, and make the model choice explicit. Source evidence and the assistant’s caveats remain essential whichever model is selected.

Next: review the questions before expanding the claims

The v0.2 transportation benchmark review draft proposes 20 English crash-analytics tasks, four separate controls and 16 broader transportation and asset-management candidates. Baltimore HIN remains in this historical case study and is excluded from the proposed general core. The draft has no new scores: domain review, independent answers, data and implementation checks must precede a new frozen evaluation.

Future records will separate the model’s original plan, application changes, data/tool results and final interface state. Scoring will make scope, units, required rows and permitted UI changes explicit, with semantic-equivalence rules decided before testing. Reviewers can annotate the question CSV or the blank worksheet, or submit Feedback using a question ID.