JBJS - 2026-08-05 - Journal Article; Comparative Study
Comparison of Large Language Models with Rules-Based Natural Language Processing Algorithms for Extracting Data from Operative Notes.
Yang L, Mulford KL, Girod-Hoffman MM, Khela M, Khosravi A, Crossman DM, Kanabar A, Saniei S, Ulrich MN, Taunton MJ, Wyles CC
Topics
Key Takeaway
LLM-based extraction pipelines outperformed rules-based NLP algorithms for bearing surface identification (89% vs 74% accuracy) across 719 THA operative notes, with LLMs correctly inferring bearing surface in 80% of ambiguous notes.
Summary Depth
Choose how much analysis to show on this article page.
Summary
This study evaluated whether LLM-based extraction pipelines could outperform existing rules-based NLP algorithms in extracting three registry-relevant data points—surgical approach, bearing surface, and fixation technique—from primary THA operative notes. Two human annotators labeled 1,000 notes; LLMs were tested against rules-based NLP on a held-out set of 719 notes. LLMs achieved superior accuracy for all three fields: surgical approach 96% vs 94%, bearing surface 89% vs 74%, and fixation technique 96% vs 95%, with 80% accuracy on ambiguous bearing surface notes.
Key Limitation
Single-institution dataset means the LLM prompts were iteratively customized to one center's operative note style, and accuracy on external institution notes is untested.
Original Abstract
BACKGROUND
We aimed to develop automated data extraction pipelines with large language models (LLMs) to extract registry data from total hip arthroplasty (THA) operative notes and compare the performance with that of existing natural language processing (NLP) algorithms.
METHODS
We randomly sampled 1,000 primary THA cases from our institutional registry. Two human annotators manually reviewed each operative note for 3 data points: surgical approach, bearing surface, and fixation technique. All labeled THA notes were split into the development set (n = 239) and the testing set (n = 719). We developed a custom data extraction pipeline for each data point by combining an iteratively customized prompt with an LLM. The performance was compared with that of existing rules-based NLP algorithms.
RESULTS
The accuracy of LLMs was superior to that of NLP algorithms for all data points: surgical approach (96% compared with 94%), bearing surface (89% compared with 74%), and fixation technique (96% compared with 95%). Furthermore, the LLM accurately inferred the bearing surface for 80% of the notes that were ambiguous about the bearing surface.
CONCLUSIONS
We developed LLM pipelines for extracting 3 registry-relevant data points from THA operative notes, demonstrating superior performance to existing NLP algorithms.
CLINICAL RELEVANCE
LLMs have the potential to impact clinical care, including the evaluation of electronic medical record free-text data. As registries serve as a cornerstone of orthopaedic evidence, this work demonstrates promise for LLMs to simplify, improve, and democratize the construction of registry databases from operative notes.