Fine-tuned clinical models versus general-purpose LLMs — where the performance gap materialises

Published comparisons between general-purpose language models and clinically fine-tuned models frequently show a narrow gap, and sometimes show the general model ahead. Those results are usually honest and usually irrelevant, because the aggregate figure averages over sub-tasks with very different difficulty profiles and very different consequences for a claim.

A general model with strong medical knowledge will identify diagnostic categories, recognise procedure names, and summarise a clinical narrative at a level that is genuinely competitive. Where it diverges is on the mechanical sub-tasks that determine whether output is billable: reading fragmented dictation, resolving assertion status under coordination and hedging, and traversing an ontology to full specificity while respecting instructional notes it cannot see.

The consequence is that aggregate parity and production divergence coexist without contradiction. Two systems can post the same F1 on a mixed evaluation set and produce materially different denial rates, because the errors are not distributed evenly across the metric — they concentrate in the encounter types with the highest specificity requirements and the highest dollar value.

Real clinical documentation is not the clean prose that public benchmark corpora contain. It is dictated, transcribed with recognition errors, missing sentence boundaries, dense with locally-defined abbreviations, and structured by section headers that vary by author and template. A general model handles this by inferring intent from context, which usually works and occasionally produces a confident, fluent, wrong reading.

The specific failure modes are predictable. Transcription substitutions that produce a real but different clinical term. Abbreviations whose expansion depends on specialty — the same three letters meaning different things in cardiology and orthopaedics. Run-on dictation that merges two clinical statements, causing a qualifier from the first to attach to the entity in the second. Section headers dictated as ordinary sentences, so section-based reasoning loses its anchor.

Models fine-tuned on production documentation absorb these patterns as distribution rather than as noise. That is not a knowledge advantage; it is a robustness advantage, and it is only visible when evaluation uses documents with the artefact profile of real dictation instead of cleaned reference text.

The second concentration is assertion status. General models identify simple negation reliably and degrade on coordinated lists, on cues separated from the entity by intervening clauses, and on the distinction between hedged and asserted findings where the correct coding behaviour depends on encounter setting. Because a false-positive code from a ruled-out finding is the least defensible error class in an audit, a small percentage difference here has outsized consequence.

The third is ontology depth. Full specificity requires enumerating legal children at each level of the hierarchy and applying Excludes1 and other instructional notes against the codes already assigned on the claim. A model asked to emit a code from parametric memory will produce a plausible code, sometimes one that no longer exists in the current annual release, and will not reliably decline when documentation fails to support a branch. Retrieval and traversal close this gap regardless of base model, which is why architecture and model choice are separable decisions.

Notably, the second and third gaps are addressable without changing base models. Explicit assertion classification and live ontology traversal are pipeline components. The first gap — dictation robustness — is the one most genuinely tied to what the model was trained on.

Model selection is not only an accuracy decision. Coding is a high-volume batch workload where per-document cost multiplies across hundreds of thousands of encounters, and where a small task-specific model that runs on controlled infrastructure has a different operational profile than an API call to a frontier model.

Data residency and retention constrain the choice further. Where PHI handling requires that no document text leave a defined boundary, self-hosted models are not a cost optimisation but a compliance precondition — a point that interacts directly with the de-identification sequencing question.

Version stability is the third operational factor. A hosted general model can change underneath a validated pipeline without notice, which in a regulated coding workflow means revalidation on someone else's schedule. A pinned, self-hosted model changes when you decide it changes.

Evaluate on your own documents. Sample from your actual encounter mix, including the specialties and document types that carry the most volume and the most revenue, and keep the artefacts — the transcription errors, the copy-forward blocks, the idiosyncratic abbreviations — rather than cleaning them out.

Report the sub-tasks separately: category identification, full-specificity exact match, assertion classification precision and recall, and decline rate on unsupported branches. A single number will hide the differences you are trying to measure.

Then treat the base model as one component rather than as the system. In most production deployments the achievable accuracy is determined more by whether assertion status is modelled explicitly and whether the ontology is traversed live than by which family the underlying model came from.