JOA - 2026-11-12 - Journal Article
New Large Language Models Outperform Older Models and Hallucinate Less in Operative Note Data Extraction.
Mulford KL, Yang L, Williams DW, Khosravi A, Le M, Ulrich MN, Kremers HM, Wyles CC
Topics
Key Takeaway
A 2026 LLM extracted THA operative variables with up to 98% accuracy and reduced bearing surface hallucination rates from 49% to 11% compared to a 2024 model.
Summary Depth
Choose how much analysis to show on this article page.
Summary
This study compared accuracy and hallucination rates of a 2024 versus 2026 locally hosted LLM in extracting surgical approach, fixation technique, and bearing surface from 1,000 primary THA operative notes. Prompts were iteratively developed on 25% of the dataset and tested on the remaining 75%. The newer model achieved 98%, 98%, and 93% accuracy versus 96%, 96%, and 89% for the older model, while hallucination rates dropped dramatically—from 49%, 20%, and 49% to 5%, 3%, and 11% respectively—using simpler prompts.
Key Limitation
Residual hallucination rates of up to 11% for bearing surface extraction mean human oversight remains mandatory before any LLM output is used for registry submission or quality reporting.
Original Abstract
BACKGROUND
Recent improvements in large language models (LLMs) may provide a scalable solution for registry data curation. However, trust in model outputs remains limited by hallucinations, wherein models invent information. Given the rapid pace of LLM development, the purpose of this study was to compare the accuracy and hallucination rates of a "newer" and an "older" and previously evaluated LLM in extracting total hip arthroplasty (THA) operative variables.
METHODS
Surgical approach, fixation technique, and bearing surface were manually abstracted from 1,000 primary THA operative notes from 47 different surgeons. Using 25% of the dataset, we iteratively developed prompts to query two locally hosted LLMs (one released in 2024, the older model, and the other released in 2026, the newer model) to extract those three data points. We tested the extraction pipeline for the best prompts on the remaining 75% of the dataset. We calculated performance metrics and manually assessed 100 sample output reasonings from each model for hallucinations.
RESULTS
Overall Performance: The newer model exhibited higher accuracy than the older model across all three data extraction tasks: surgical approach (98 versus 96%), fixation technique (98 versus 96%), and bearing surface (93 versus 89%). For each task, the highest-performing prompt technique for the newer model was comparatively simpler than the highest-performing prompt technique for the older model. Hallucination Rate Analyses: Across all evaluated extraction domains, the newer model demonstrated significantly lower hallucination rates compared with the older model. Hallucination rates for surgical approach, fixation technique, and bearing surface extraction were 5, 3, and 11% for the newer model versus 49, 20, and 49% for the older model, respectively.
CONCLUSION
The newer LLM extracted THA operative variables more accurately and hallucinated substantially less with simpler prompts than the 2-year-old model. These findings suggest LLMs are improving their ability to follow directions and reason without inventing information. However, lower rates of hallucinations persist, limiting trustworthiness.