Clinical NLP on real documentation — dictation artefacts, abbreviations, and fragmented syntax

Vendor demonstrations typically run on well-formed structured notes: complete sentences, labelled sections, consistent terminology, no transcription noise. Those documents exist in production, but they are not representative of the mix. A realistic corpus from a mid-sized health system contains dictated operative notes with dropped verbs, speech-to-text substitutions that are phonetically plausible and clinically absurd, templated text with unfilled placeholders, copy-forward blocks that contradict the current encounter, and addenda that silently supersede earlier statements.

The performance delta between the demonstration corpus and the production corpus is routinely fifteen to twenty-five points of F1. That delta is not a model defect in any interesting sense — it is a measurement artefact of evaluating on the wrong distribution. But it becomes an operational failure the moment a system is deployed on the assumption that the demonstration figure transfers.

The practical consequence is that document mix is the first thing to establish in any evaluation, before any accuracy number is discussed. Two systems evaluated on different mixes are not comparable, and a single aggregate figure over an unstated mix carries almost no information.

Dictation produces a specific and recognisable class of error. Function words disappear because the speaker elided them. Sentence boundaries are absent because the speaker did not dictate punctuation, which collapses several clinical statements into one run-on span and destroys the sentence-level structure that scope resolution depends on. Numbers and units are transcribed inconsistently, so dosages and measurements arrive in several surface forms.

Speech-to-text substitution is the most dangerous category, because the output is well-formed text. A substituted drug name or a substituted anatomic term produces a confident, fluent, wrong extraction with no surface signal that anything went wrong. Detection requires clinical plausibility checking against the rest of the document rather than any property of the token itself.

Systems that were trained on clean text tend to fail on these inputs in a particular way: they degrade quietly. Confidence scores stay high because the text is fluent. The failure is only visible against a reference standard, which is exactly what production lacks.

Clinical abbreviations are massively ambiguous and the ambiguity is not resolvable from local context alone. The same three-letter token resolves to different concepts depending on the specialty of the author, the section of the note it appears in, and occasionally the institution's local conventions. A global lexicon mapping abbreviation to expansion will be wrong a substantial fraction of the time, and the errors will cluster in specific specialties rather than distributing evenly.

Resolution must therefore be conditioned on document type, section, and authoring context. In practice this means the pipeline needs section segmentation before abbreviation expansion, and it means the expansion step needs access to metadata about the note's provenance. Both are architectural requirements that are easy to omit and expensive to retrofit.

Local convention is the part vendors most often underestimate. Every health system accumulates institution-specific shorthand that appears in no public terminology resource. A deployment plan that does not include a corpus-driven pass to discover and register local abbreviations will carry a persistent, silent error floor.

Where a statement appears in a note changes what it means for coding. A condition named in Past Medical History is history, not an active problem for this encounter. A condition in Family History belongs to someone else entirely. A finding in the Assessment carries diagnostic weight that the same finding in a copied Review of Systems does not.

Section headers in production documentation are inconsistent, sometimes absent, sometimes duplicated, and frequently rendered as formatting rather than text. Segmentation therefore has to be a modelled step with its own error rate, not a regex over expected header strings.

When segmentation fails, the downstream effect is a class of error that looks like a coding mistake but is actually a structural one: history coded as active, family history coded as patient history. These are among the most common findings in coding audits of automated systems.

Normalisation belongs upstream of extraction, and its failures must be observable. Silent normalisation errors surface much later as missing or incorrect codes with no audit trail back to the transformation that caused them. Every normalisation step should emit both its output and a record of what it changed, retained alongside the extraction result.

Confidence should be composed across stages rather than reported only at the end. A code derived from an extraction that depended on an uncertain abbreviation expansion inside a section whose boundary was uncertain is a low-confidence code, even if the final classification step was confident. Systems that report only terminal confidence systematically under-route exactly the cases that most need review.

Finally, build the evaluation corpus from your own documents, sampled to match your production mix, including the ugly documents. A corpus that excludes difficult notes will produce an accuracy figure that is accurate about nothing.