<- Back to digest

Archives of Orthopaedic and Trauma Surgery - 2026-10-15 - Journal Article

Expert evaluation of ChatGPT-generated responses to patient questions in conventional and fast-track total knee arthroplasty.

Maciag B, Jegierski D, Frankiewicz A, Tarkowski P, Żarnovsky K, Mietlicki F, Stolarczyk A

surveyLOE Vn = 70 ChatGPT responses, 210 expert assessments from 3 ratersN/A

Topics

arthroplasty
PMID: 42878881DOI: 10.1007/s00402-026-06534-7View on PubMed ->

Key Takeaway

Three arthroplasty surgeons rated ChatGPT responses to 70 TKA patient questions at near-ceiling scores (accuracy 4.99/5, completeness 4.98/5, safety 4.98/5, readability 5.00/5), but ceiling clustering and non-equivalent question-set construction prevent any equivalence claim between conventional and fast-track responses.

Summary Depth

Choose how much analysis to show on this article page.

Summary

Three arthroplasty surgeons independently scored ChatGPT-generated responses to 50 conventional and 20 fast-track TKA patient questions across accuracy, completeness, safety, and readability on 5-point scales. All 20 fast-track responses received perfect 5/5 scores from all three raters in all four domains; conventional responses scored 4.98–5.00 across domains. Ceiling-level clustering, non-equivalent question-set selection methodology, and absence of source-data verification for fast-track responses preclude any conclusion of equivalence between question sets.

Key Limitation

Scores clustered at the ceiling for all domains, eliminating discriminative power and making it statistically impossible to detect meaningful differences between conventional and fast-track response quality.

Original Abstract

BACKGROUND

Fast-track total knee arthroplasty (TKA) asks patients to understand early mobilization, early discharge, opioid-sparing analgesia, and selective use of drains and catheters. We evaluated expert-rated quality of ChatGPT responses to patient questions about these pathway-specific concepts alongside conventional TKA questions.

MATERIALS AND METHODS

In this cross-sectional evaluation, three arthroplasty surgeons independently scored responses to 50 conventional TKA and 20 investigator-developed fast-track questions in four domains on 5-point scales. The conventional questions were selected by a multidisciplinary panel; the fast-track questions did not undergo the same selection process. Response-level means were compared descriptively and with exploratory Mann-Whitney U tests.

RESULTS

Seventy responses received 210 expert assessments. Recalculated mean ± SD scores across individual ratings were 4.99 ± 0.15 for accuracy, 4.98 ± 0.14 for completeness, 4.98 ± 0.17 for safety, and 5.00 ± 0.07 for readability. Each of the 20 fast-track responses received a score of 5 from each of the three raters in all four domains (Supplementary Table S1). Nevertheless, question-set construction differed and scores clustered at the ceiling, so the exploratory comparison does not establish equivalence.

CONCLUSIONS

Experts gave high ratings to the evaluated responses, including those on fast-track concepts. The observed data do not establish equivalence between question sets: the fast-track ratings require source-data verification, the sets were selected differently, and scores were concentrated at the maximum. Clinician review remains appropriate for protocol-dependent and safety-related advice.

TRIAL REGISTRATION

Not applicable.