<- Back to digest

JOA - 2026-11-12 - Journal Article

New Large Language Models Outperform Older Models and Hallucinate Less in Operative Note Data Extraction.

Mulford KL, Yang L, Williams DW, Khosravi A, Le M, Ulrich MN, Kremers HM, Wyles CC

retrospective cohortLOE IIIn = 1,000 THA operative notes from 47 surgeonsN/A

Topics

arthroplasty
PMID: 43020159DOI: 10.1016/j.arth.2026.11.047View on PubMed ->

Key Takeaway

A 2026 LLM extracted THA operative variables with up to 98% accuracy and reduced bearing surface hallucination rates from 49% to 11% compared to a 2024 model.

Summary Depth

Choose how much analysis to show on this article page.

Summary

This study compared accuracy and hallucination rates of a 2024 versus 2026 locally hosted LLM in extracting surgical approach, fixation technique, and bearing surface from 1,000 primary THA operative notes. Prompts were iteratively developed on 25% of the dataset and tested on the remaining 75%. The newer model achieved 98%, 98%, and 93% accuracy versus 96%, 96%, and 89% for the older model, while hallucination rates dropped dramatically—from 49%, 20%, and 49% to 5%, 3%, and 11% respectively—using simpler prompts.

Key Limitation

Residual hallucination rates of up to 11% for bearing surface extraction mean human oversight remains mandatory before any LLM output is used for registry submission or quality reporting.

Original Abstract

BACKGROUND

Recent improvements in large language models (LLMs) may provide a scalable solution for registry data curation. However, trust in model outputs remains limited by hallucinations, wherein models invent information. Given the rapid pace of LLM development, the purpose of this study was to compare the accuracy and hallucination rates of a "newer" and an "older" and previously evaluated LLM in extracting total hip arthroplasty (THA) operative variables.

METHODS

Surgical approach, fixation technique, and bearing surface were manually abstracted from 1,000 primary THA operative notes from 47 different surgeons. Using 25% of the dataset, we iteratively developed prompts to query two locally hosted LLMs (one released in 2024, the older model, and the other released in 2026, the newer model) to extract those three data points. We tested the extraction pipeline for the best prompts on the remaining 75% of the dataset. We calculated performance metrics and manually assessed 100 sample output reasonings from each model for hallucinations.

RESULTS

Overall Performance: The newer model exhibited higher accuracy than the older model across all three data extraction tasks: surgical approach (98 versus 96%), fixation technique (98 versus 96%), and bearing surface (93 versus 89%). For each task, the highest-performing prompt technique for the newer model was comparatively simpler than the highest-performing prompt technique for the older model. Hallucination Rate Analyses: Across all evaluated extraction domains, the newer model demonstrated significantly lower hallucination rates compared with the older model. Hallucination rates for surgical approach, fixation technique, and bearing surface extraction were 5, 3, and 11% for the newer model versus 49, 20, and 49% for the older model, respectively.

CONCLUSION

The newer LLM extracted THA operative variables more accurately and hallucinated substantially less with simpler prompts than the 2-year-old model. These findings suggest LLMs are improving their ability to follow directions and reason without inventing information. However, lower rates of hallucinations persist, limiting trustworthiness.