Which model should power the support bot?
4 models, the same 80-conversation test suite, 3 runs each. Quality, hallucinations, consistency from run to run, speed and cost per conversation, measured on one real use case instead of a leaderboard.
Pick GPT-6.1 Terra for this bot: the same quality as the flagship at half the list price.
- GPT-6.1 Terra (93.3%) and GPT-6.1 Sol (93.8%) are tied within run-to-run noise (91–95% vs 92–95% across three runs), but Sol costs twice as much at list price: $2.60 vs $1.29 per 1,000 conversations. At 100,000 conversations a month that is about $130 for no measurable gain.
- Both fail the same four conversations in every run. Three start in retrieval or clarification, the gaps found in the original evaluation (one of them traced to a tokenizer bug in review); the fourth (old-01) is an incomplete answer. A bigger model does not fix them.
- GPT-6 Luna is 10× cheaper than Terra at list price ($0.13 per 1,000) and passes 89.6%, but it is the only GPT model that distorted policy: it repeated the same subtle error about postage charges in all three runs, and its results are less consistent (90% vs 96%). Fine for low-risk FAQ deflection, not for billing or policy answers.
- Nemotron 3 Ultra (free) scored 73.8% as integrated: without enforced structured output it dropped the JSON format in every multi-turn conversation. A one-line format reminder lifted its mean to 81.7%, but it stays the least consistent (82%) and slowest at p95 (6.3 s). Free costs engineering time and quality.
- Every model routed 100% of must-escalate conversations to a human, with zero policy violations.
- The GPT judge did not favor GPT models. The second judge (Nemotron) rated every model higher, by 1.3 to 6.3 points for the GPT models and 10 points for itself, and it missed both substantive policy distortions the GPT judge caught.
Side by side
Means over 3 runs of all 80 conversations; the pass-rate range shows the worst and best run. Consistency is the share of conversations with the same pass/fail result in every run. Costs are for the bot only: real token usage priced at each model's list price without prompt caching, because how much input hit OpenAI's cache varied between runs and would skew the comparison. The billed column shows what was actually paid. Hallucination counts answers only. GPT-6.1 Terra's run 1 is reused from the support chatbot evaluation suite, so its latency includes one session measured at a different time.
| Model | Pass rate | Range | Hallucination | Right action | Handoff recall | Valid output | Consistency | Latency p50 / p95 | $ per 1,000 conv. (list) | Billed per 1,000 | $ per passing conv. |
|---|---|---|---|---|---|---|---|---|---|---|---|
GPT-6.1 Solgpt-6.1-sol |
93.8% | 92%–95% | 0.0% | 97% | 100% | 100.0% | 96% | 2.0s / 2.9s | $2.60 | $1.80 | $0.00277 |
GPT-6.1 Terragpt-6.1-terra |
93.3% | 91%–95% | 0.0% | 97% | 100% | 100.0% | 96% | 1.7s / 2.6s | $1.29 | $1.09 | $0.00138 |
GPT-6 Lunagpt-6-luna |
89.6% | 88%–91% | 3.5% | 98% | 100% | 100.0% | 90% | 1.5s / 2.0s | $0.13 | $0.09 | $0.00014 |
Nemotron 3 Ultra (free)nvidia/nemotron-3-ultra-550b-a55b:free |
73.8% | 71%–78% | 4.3% | 88% | 100% | 90.0% | 82% | 2.8s / 6.3s | $0.00 | $0.00 | $0.00000 |
Integration finding: the free model and JSON output
All of Nemotron 3 Ultra (free)'s invalid outputs were in multi-turn conversations. Its OpenRouter endpoint has no structured-output mode, so the JSON format was only requested in the system prompt, and when earlier turns in the history were plain text the model answered in plain text too, often with the right content. The GPT models never hit this because the API enforces the schema. A follow-up re-ran only those conversations with a one-line reminder at the end of the last message ("Respond with the JSON object only…").
| Run | Passed as integrated | Multi-turn passed with reminder | Passed with reminder |
|---|---|---|---|
| Run 1 | 62/80 (77.5%) | 7/8 (was 0) | 69/80 (86.2%) |
| Run 2 | 58/80 (72.5%) | 6/8 (was 0) | 64/80 (80.0%) |
| Run 3 | 57/80 (71.2%) | 6/8 (was 0) | 63/80 (78.8%) |
With the reminder, multi-turn passed 79% and the model's mean pass rate rises to 81.7%; 3 of 24 replies still broke format. The side-by-side table keeps the as-integrated numbers so every model is measured on identical inputs. Lesson: a model without enforced structured output needs format handling in the integration (reminders, validation and retry), and that is engineering cost a price list doesn't show.
Quality against cost
Each point is a model's mean pass rate; the vertical line spans its worst and best run. Up and to the left is better. Cost uses a log scale because the models differ by orders of magnitude; the free model sits at $0 on the left.
Pass rate by category
Mean over runs. Small categories swing a lot: one conversation in "Legacy / outdated info" is 25 points. Nemotron's 70% on "Needs a human" sits beside 100% handoff recall because those replies routed correctly but were failed by the judge for adding a follow-up timeline ("shortly") the articles don't state.
| Category | GPT-6.1 Sol | GPT-6.1 Terra | GPT-6 Luna | Nemotron 3 Ultra (free) |
|---|---|---|---|---|
| Factual questions | 93% | 93% | 83% | 82% |
| Multi-turn follow-ups | 100% | 100% | 96% | 0% |
| Not in knowledge base | 100% | 100% | 97% | 93% |
| Needs a human | 100% | 100% | 97% | 70% |
| Off-topic | 100% | 100% | 94% | 100% |
| Too vague to answer | 80% | 80% | 93% | 80% |
| Legacy / outdated info | 75% | 75% | 75% | 75% |
| Adversarial | 89% | 85% | 89% | 74% |
Bold marks the best model in each category (ties all bold).
Where the models differ
Every conversation that some model failed in at least one run. Each cell is runs passed out of runs made: green passed every time, red failed every time, grey is inconsistent between runs.
| Conversation | Category | GPT-6.1 Sol | GPT-6.1 Terra | GPT-6 Luna | Nemotron 3 Ultra (free) |
|---|---|---|---|---|---|
fact-10 | Factual questions | 0/3 | 0/3 | 0/3 | 0/3 |
old-01 | Legacy / outdated info | 0/3 | 0/3 | 0/3 | 0/3 |
adv-06 | Adversarial | 0/3 | 0/3 | 0/3 | 0/3 |
clar-04 | Too vague to answer | 0/3 | 0/3 | 2/3 | 0/3 |
fact-18 | Factual questions | 2/3 | 1/3 | 2/3 | 1/3 |
fact-01 | Factual questions | 3/3 | 2/3 | 0/3 | 2/3 |
fact-11 | Factual questions | 3/3 | 3/3 | 1/3 | 1/3 |
multi-01 | Multi-turn follow-ups | 3/3 | 3/3 | 2/3 | 0/3 |
fact-07 | Factual questions | 3/3 | 3/3 | 0/3 | 3/3 |
fact-25 | Factual questions | 3/3 | 3/3 | 1/3 | 2/3 |
fact-27 | Factual questions | 3/3 | 3/3 | 3/3 | 0/3 |
multi-02 | Multi-turn follow-ups | 3/3 | 3/3 | 3/3 | 0/3 |
multi-03 | Multi-turn follow-ups | 3/3 | 3/3 | 3/3 | 0/3 |
multi-04 | Multi-turn follow-ups | 3/3 | 3/3 | 3/3 | 0/3 |
multi-05 | Multi-turn follow-ups | 3/3 | 3/3 | 3/3 | 0/3 |
multi-06 | Multi-turn follow-ups | 3/3 | 3/3 | 3/3 | 0/3 |
multi-07 | Multi-turn follow-ups | 3/3 | 3/3 | 3/3 | 0/3 |
multi-08 | Multi-turn follow-ups | 3/3 | 3/3 | 3/3 | 0/3 |
hand-06 | Needs a human | 3/3 | 3/3 | 3/3 | 0/3 |
unk-03 | Not in knowledge base | 3/3 | 3/3 | 2/3 | 2/3 |
fact-12 | Factual questions | 3/3 | 3/3 | 3/3 | 1/3 |
hand-02 | Needs a human | 3/3 | 3/3 | 3/3 | 1/3 |
hand-05 | Needs a human | 3/3 | 3/3 | 3/3 | 1/3 |
hand-10 | Needs a human | 3/3 | 3/3 | 3/3 | 1/3 |
adv-05 | Adversarial | 3/3 | 3/3 | 3/3 | 1/3 |
fact-08 | Factual questions | 2/3 | 3/3 | 3/3 | 3/3 |
fact-19 | Factual questions | 2/3 | 3/3 | 3/3 | 3/3 |
adv-04 | Adversarial | 3/3 | 2/3 | 3/3 | 3/3 |
hand-08 | Needs a human | 3/3 | 3/3 | 2/3 | 3/3 |
off-03 | Off-topic | 3/3 | 3/3 | 2/3 | 3/3 |
fact-16 | Factual questions | 3/3 | 3/3 | 3/3 | 2/3 |
unk-10 | Not in knowledge base | 3/3 | 3/3 | 3/3 | 2/3 |
adv-08 | Adversarial | 3/3 | 3/3 | 3/3 | 2/3 |
adv-09 | Adversarial | 3/3 | 3/3 | 3/3 | 2/3 |
Is the judge fair to every model?
The main judge, gpt-6.1-sol, is also a contestant, so run 1 of every model was graded again by a judge from a different company, nvidia/nemotron-3-ultra-550b-a55b:free, which is a contestant too. If either judge favored its own model, its pass rate for that model would sit above the other judge's.
| Model (run 1) | Pass rate, judge 1 (GPT-6.1 Sol) | Pass rate, judge 2 (Nemotron 3 Ultra (free)) | Judges agree on the verdict | Hallucinations flagged (1 / 2) |
|---|---|---|---|---|
| GPT-6.1 Sol | 92.5% | 93.8% | 98% of 80 | 0 / 0 |
| GPT-6.1 Terra | 95.0% | 97.5% | 98% of 80 | 0 / 0 |
| GPT-6 Luna | 87.5% | 93.8% | 94% of 80 | 3 / 0 |
| Nemotron 3 Ultra (free) | 77.5% | 87.5% | 89% of 72 | 4 / 0 |
Hallucination flags here count every graded reply, not only answers. Of the 7 replies the main judge flagged, 2 are substantive policy distortions (both GPT-6 Luna); the rest are minor phrasing, mostly a "will follow up shortly" timeline the articles don't state. The second judge flagged none, including the 2 substantive ones, so it is the more lenient grader, and it gave its largest increase (+10 points) to its own model. Both judges rank the models the same way.
Method
- Fixed setup. The support chatbot evaluation suite's 15-article knowledge base, BM25 retrieval (top 3), prompt v2, output schema and all 80 golden conversations with the original suite's corrected test definitions. Only the model changes.
- Models. OpenAI models through the Responses API with native structured output and low reasoning effort. Nemotron 3 Ultra through OpenRouter's free tier, which offers no structured-output mode, so the JSON schema is in its prompt and its reply is parsed and validated; a reply that doesn't parse counts as invalid output and fails.
- Grading. The original suite's deterministic checks plus the LLM judge (gpt-6.1-sol, medium effort). A conversation passes only if every check passes and the judge rates it correct, grounded and policy-safe.
- Repeats. Each model ran the suite 3 times as separate requests. The judge grades identical replies identically (its responses are cached), so variation measures the bot, not the judge.
- Limitations. 80 conversations in one domain; results describe this use case, not general ability. Latency for the free model depends on OpenRouter's free-tier queueing and isn't comparable to a paid endpoint. Both judges are also contestants; the cross-check above is the guard against that.