BENCHMARKS · NOV 2026
A private 4B model
ties Claude Opus 5.
We trained a 4-billion-parameter US open model on frontier answers for one workflow, contract-clause classification, and tested it on 200 held-out documents against the same prompt run on frontier APIs. It scored 81.0%, tying Claude Opus 5 and beating every GPT model we ran, and serves for $0.013 per 1,000 documents on one NVIDIA L4.
01 / ACCURACY
Same 200 documents, same prompt, every model.
LEDGAR is a public dataset of contract clauses labelled by type: governing law, termination, confidentiality, notices and 96 more. We froze 200 test documents before training and never trained on them.
| Setup | Runs on | Accuracy | Invalid answers |
|---|---|---|---|
| Inferoid private 4B, trained | One L4, your cloud | 81.0% | — |
| Claude Opus 5 | API | 81.0% | 0.0% |
| GPT-5.6 Luna | API | 78.5% | 0.0% |
| GPT-5.6 Sol | API | 78.0% | 0.0% |
| GPT-5.5 | API | 78.0% | 0.0% |
| GPT-5.4 | API | 78.0% | 1.0% |
| GPT-5.6 Terra | API | 74.0% | 1.0% |
| Same 4B, untrained | One L4, your cloud | 65.5% | — |
Frontier models called through Amazon Bedrock (Converse API, max 1,024 output tokens) on 2026-11-02, zero request errors. Invalid means the answer could not be parsed to an allowed label; it counts as wrong. A dash means the rate was not recorded. 200 documents gives roughly ±5.5 points of sampling error, so a small gap is a tie.
02 / KNOWING WHEN IT IS UNSURE
Send the unsure 14.5% to a person, and the rest are 86% right.
Every answer comes with a calibrated confidence, so “90% sure” means about 90% right. A team can send only the least-sure answers to a person; their corrections become training data for the next version.
Each point is a confidence threshold: everything below it goes to a person, and the line shows accuracy on what remains. Same 200 held-out documents.
03 / OTHER US OPEN MODELS
Training lifts every model we tried by 15 to 33 points.
An earlier run trained three US open models the same way, each on 2,000 frontier-labelled examples in about 13 minutes on one NVIDIA L40S.
Grey: untrained. Red: after training. 2026-10-31, same 200 held-out documents. Gemma 4 E4B later reached 81.0% in the product run above.
04 / HARDWARE YOU ALREADY OWN
Untrained models on GPU and CPU.
Before training, we measured accuracy and cost per 1,000 documents for small open models on an NVIDIA L4 and on AMD and Intel server CPUs. EC2 on-demand prices stand in for hardware a company already owns.
| Model, untrained | NVIDIA L4 | AMD EPYC CPU | Intel Xeon CPU |
|---|---|---|---|
| Google Gemma 4 E4B | 65.5% · $0.013 | 44% · $0.42* | 38% |
| Mistral Ministral 3 8B | 59.5% · $0.024 | 55% · $0.27 | 55.5% · $0.41 |
| Microsoft Phi-4-mini | 46.5% · $0.014 | 46% · $0.10 | 46% · $0.095 |
| IBM Granite 4.0 H-Tiny | 45% · $0.023 | 48% · $0.29 | 48% · $0.48 |
| IBM Granite 4.2 8B | 45% · $0.041 | 41% · $0.29 | 42.5% · $0.42 |
| Hugging Face SmolLM3 3B | — | 34% · $0.13 | 33.5% · $0.19 |
| Ai2 OLMo 3 7B | — | 20.5% | 22% |
L4: AWS g6.xlarge, vLLM. CPUs: AWS c8a.8xlarge (AMD EPYC, llama.cpp Q4) and c7i.8xlarge (Intel Xeon). *Gemma on CPU likely hit a chat-template issue. Phi-4-mini CPU figures use vLLM on CPU. A dash means not run on that hardware. This was a separate run from section 03, so untrained scores differ by about a point.
05 / METHOD, AND WHAT THIS DOES NOT SHOW
One task, measured carefully.
- Training. A LoRA fine-tune on examples labelled by a frontier model on Amazon Bedrock, inside our AWS account. Training, validation and test sets are separate, and the test set was frozen first.
- Same harness for everyone. Identical system prompt, label list and answer parsing; no model got extra examples or tools.
- One workflow. This is classification with a fixed label set. It does not show the same result on open-ended writing, reasoning or agent tasks.
- Cost. Serving cost uses AWS on-demand list prices and measured throughput, excluding idle time. Frontier cost uses Claude Opus API list prices with measured token counts.
- Versions move. Every figure is a model version on a date. We rerun rather than cite a stale number.