ICD-10/CPT accuracy benchmarks — production F1 by document type (2024 cohort)

Controlled evaluation sets are curated, and curation is itself a filter that removes exactly the documents on which systems fail. Notes that are incomplete, internally contradictory, truncated by a dictation error, or missing the addendum that contains the pathology result tend not to survive into a benchmark corpus, because they are difficult to assign reliable ground truth to. Their absence inflates every figure computed on the remainder.

Production cohorts contain all of it: unsigned drafts pulled at the wrong moment, notes amended after the initial code assignment, addenda that arrive days later and change the principal diagnosis, copy-forward blocks carrying resolved conditions from prior encounters, and templated text that describes a normal exam the clinician did not perform. The spread between a curated figure and a production figure on the same system is the number actually worth knowing, and it is routinely five to fifteen points.

The cohort described here is production output across deployed environments over a full year, adjudicated by coding staff rather than scored against historical billed claims. That distinction matters: billed claims encode the organisation's own existing error patterns, and scoring against them rewards a system for reproducing them.

Accuracy is reported as F1 at encounter level over the complete code set, with sequencing and required modifiers inside the unit of measurement. A code set that is complete but misordered, or missing a modifier that the payer requires, counts as incorrect. This is a stricter unit than per-code accuracy and produces lower numbers; it is also the unit that corresponds to a clean claim.

Precision and recall are retained separately rather than being collapsed, because the two error modes carry different costs. Reference standard is coder adjudication with a second reviewer on disagreements, and inter-rater agreement is reported per stratum — where coders themselves disagree materially, the ceiling on measurable model accuracy is correspondingly lower and the stratum figure must be read against it.

Every stratum carries a sample size. Strata below a few hundred encounters produce confidence intervals wide enough to accommodate almost any claim, and figures reported without sample sizes should be treated as unmeasured rather than as evidence.

Radiology, pathology, and laboratory reports sit at the top of the distribution. The reason is structural rather than clinical: these documents are produced against a template, the findings section is stereotyped, the diagnostic conclusion is stated explicitly in an impression, and the code set drawn on is comparatively shallow. Assertion status is usually unambiguous because negation in a radiology impression follows a small number of recurring forms.

Screening and preventive encounters behave similarly. The documentation is short, the procedure is one of a small set, and the diagnosis is frequently a Z code determined by the encounter reason rather than by clinical inference.

These document types are the appropriate first automation targets, and they are also where direct-to-bill rates are highest — not because the model is better on them, but because the confidence distribution is tighter and fewer encounters fall into the review band.

Outpatient office visits occupy the middle of the range. Diagnosis extraction is generally strong; the loss is concentrated in evaluation and management level assignment, which under the current guidelines turns on medical decision making complexity or total time, and both are frequently underdocumented. A system can identify every condition addressed and still assign the wrong level, and level assignment is the part with direct revenue consequence.

Emergency department encounters sit slightly lower. The documentation is written under time pressure, the differential is often broader than the final assessment, and the note frequently contains conditions considered and excluded alongside conditions treated. This makes assertion status the dominant error source rather than code selection.

Copy-forward content is the other systematic contributor in this band. Problem lists carried forward from prior visits produce codes for conditions that were not addressed at this encounter, and distinguishing addressed from merely present requires reasoning about the plan section rather than the assessment.

Inpatient discharge summaries sit at the bottom of the distribution and by a wide margin. They are long, they span an entire admission, they must be reconciled against the rest of the chart, and their coded output depends on principal diagnosis selection — a determination about which condition, after study, occasioned the admission. That is a judgement over the whole record, not an extraction from a document, and it drives DRG assignment and therefore the majority of the reimbursement.

Complication and comorbidity capture compounds it. Whether a documented condition qualifies as a CC or MCC, and whether it was present on admission, changes the DRG. Present-on-admission determination frequently requires reading the admission note against the discharge summary, and the two may disagree.

Operative reports fail differently. Extraction of the individual procedures is usually reliable; the loss is in sequencing, bundling, and modifier assignment, as covered in the multi-procedure article. The claim-level figure is therefore substantially below the per-code figure for this document type specifically.

The ordering is stable across deployments and is the single most useful planning input. Structured reports first, with a low review rate and a high direct-to-bill target. Office visits second, with review policy concentrated on E&M level rather than on diagnosis selection. Emergency encounters third, with assertion status as the monitored failure mode. Inpatient and dense operative work last, with full review by policy until the rule layer and DRG logic are validated locally.

Set confidence thresholds per document type rather than globally. A single global threshold either over-reviews the structured documents, wasting the automation, or under-reviews the narrative ones, which is the more expensive error. Per-type thresholds calibrated against local adjudication data are the mechanism that makes a mixed deployment safe.

Finally, treat these figures as a shape rather than as targets. The ordering transfers between organisations; the absolute values do not, because they depend on local documentation quality, template design, specialty mix, and payer composition. The only figure that predicts your outcome is one measured on your own encounters.