JOA - 2026-10-28 - Journal Article
Evaluating Large Language Models for Patient-Facing Total Knee Arthroplasty Information in Knee Osteoarthritis: Development and Application of the Composite Clinical Reliability Score.
Muthu S, Kolarpatti Ponnusamy DV, Rathinam MM, Viswanathan VK, Rajappan Chandra SK, Sharun K
Topics
Key Takeaway
Among three LLMs evaluated on 90 patient-facing TKA questions, Program 2 achieved the highest composite clinical reliability score (median CCRS 84.0) and best readability (CLI 12.3), though Program 3 demonstrated superior evidence citation validity (ECVI 7.7 vs 7.1 vs 5.6).
Summary Depth
Choose how much analysis to show on this article page.
Summary
This study evaluated three commercial LLMs on 90 patient-facing TKA questions sourced from Google's People Also Ask, using a novel six-domain Composite Clinical Reliability Score (CCRS) assessed by two fellowship-trained orthopaedic surgeons with ICC 0.87. Program 2 achieved the highest median CCRS (84.0, IQR 82.0–85.0) versus Program 3 (76.5) and Program 1 (73.0) (P<0.0001), with superior factual accuracy, hallucination control, and safety framing. Program 3 led in evidence citation validity (ECVI 7.7) but was least readable (CLI 13.7 vs 12.3 for Program 2), and all three LLMs produced content at a reading level exceeding recommended patient health literacy targets.
Key Limitation
LLM non-determinism and continuous model updating render the specific performance rankings non-reproducible and likely obsolete within months of publication.
Original Abstract
BACKGROUND
Total knee arthroplasty (TKA) is the definitive surgical treatment for advanced knee osteoarthritis. As patients increasingly consult large language models (LLMs) for medical guidance, reliability, safety, and readability are concerns. This study compared patient-facing TKA information from three LLMs using the Composite Clinical Reliability Score (CCRS).
METHODS
We used 90 patient-facing TKA questions from Google's People Also Ask to test three LLMs. We employed two fellowship-trained orthopaedic surgeons to independently score the 270 responses at two timepoints across six CCRS domains: factual accuracy, United States Medical Licensing Examination (USMLE)-style clinical correctness, Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guideline adherence, hallucination severity index (HSI, inverse), clinical safety score (CSS), and evidence citation validity index (ECVI). Readability was assessed with the Coleman-Liau Index (CLI), corroborated by the Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook (SMOG). Inter-rater reliability used the intraclass correlation coefficient (ICC), and inter-model differences the Friedman test.
RESULTS
Program 2 achieved the highest CCRS (median 84.0, interquartile range [IQR] 82.0 to 85.0), outperforming Program 3 (76.5, 65.0 to 83.8) and Program 1 (73.0, 69.0 to 79.0) (P < 0.0001), with superior accuracy, guideline adherence, hallucination control, and safety framing. Program 3 showed stronger ECVI (7.7 ± 1.5) than Program 2 (7.1 ± 0.8) and Program 1 (5.6 ± 0.6). Readability differed significantly (P < 0.0001): Program 2 was most accessible (CLI 12.3 ± 1.6), Program 1 intermediate (12.8 ± 1.7), and Program 3 least readable (13.7 ± 2.5). Agreement was excellent (ICC 0.87, 95% confidence interval [CI] 0.82 to 0.91).
CONCLUSION
Within this framework and evaluation period, Program 2 ranked highest on composite reliability and readability, Program 3 offered better evidence traceability, but less accessible text, and Program 1 was intermediate. Because LLM outputs are non-deterministic and continuously updated, these findings are exploratory rather than a durable ranking, and LLMs should serve only as adjuncts to professional consultation.