<- Back to digest

Journal of Pediatric Orthopaedics - 2026-10-19 - Journal Article

Readability and Content Integrity of Discharge Documents for Common Pediatric Orthopaedic Procedures Before and After Artificial Intelligence Simplification.

A Prempeh AG, Saeed R, Hill C, Kopp L, Tenfelde A

retrospective cohortLOE IIIn = 15 documents from 9 US statesN/A

Topics

pediatrics
PMID: 42895514DOI: 10.1097/BPO.0000000000003508View on PubMed ->

Key Takeaway

ChatGPT reduced discharge document reading level by 3.94 grade levels (FKGL 8.91→4.97) but introduced 12 safety-critical errors across 14 of 15 documents, with greater word-count reduction correlating with higher error burden (Spearman rho 0.63).

Summary Depth

Choose how much analysis to show on this article page.

Summary

This study evaluated whether AI simplification of publicly available discharge documents for CRPP supracondylar fractures, forearm fracture ORIF/CRPP, and SCFE pinning could achieve target readability without compromising content integrity. ChatGPT reduced mean FKGL from 8.91 to 4.97 (p<0.001, Cohen d=1.69), with 73% of documents reaching the FKGL ≤6 threshold versus 0% at baseline. However, structured content review identified 35 flags across 14 of 15 documents—including 12 safety-critical errors—and 3 documents required clinician revision before clinical use.

Key Limitation

The study used only 15 documents and a single AI model (ChatGPT), limiting generalizability across institutions, document types, and competing AI platforms.

Original Abstract

BACKGROUND

Caregivers of pediatric orthopaedic surgical patients routinely receive discharge documents written at reading levels exceeding established health literacy recommendations. This disparity creates a significant barrier to comprehension and adherence to postoperative instructions. While artificial intelligence (AI) language models offer a scalable solution for simplification of medical text, a systematic evaluation of the ability to preserve critical content in surgical discharge materials remains lacking.

METHODS

Publicly available caregiver-facing discharge documents for 3 common pediatric orthopaedic conditions, supracondylar humerus fractures (CRPP), pediatric forearm fractures (ORIF/CRPP), and slipped capital femoral epiphysis (SCFE pinning), were collected. All documents underwent AI simplification using a standardized prompt with ChatGPT (version 5.2; OpenAI). Readability was assessed using 6 validated indices for both original and simplified documents. A structured content integrity review was conducted on all simplified documents by 2 independent reviewers to assess for safety-critical errors, completeness gaps, and factual accuracy, with inter-rater reliability quantified by Cohen kappa.

RESULTS

Fifteen documents were obtained from institutions across 9 US states. After AI simplification, mean Flesch-Kincaid Grade Level (FKGL) decreased from 8.91 (SD: 1.38) to 4.97 (SD: 2.14), a mean reduction of 3.94 grade levels (95% CI: 2.65-5.23; P<0.001; Cohen d=1.69). All 6 readability indices improved across all documents. Eleven of 15 documents (73%) achieved the target readability threshold of FKGL 6 or below, compared with zero at baseline. Content integrity review identified 35 flags across 14 of 15 documents: 12 safety-critical errors, 16 completeness gaps, and 7 accuracy concerns. Cohen kappa was 0.87 for agreement on whether a change was clinically meaningful, 0.85 for severity classification, and 0.89 for flag category. Three documents had content omissions sufficient to require clinician revision before clinical use. Greater word-count reduction was associated with a higher burden of safety-critical flags (Spearman rho 0.63; P=0.012).

CONCLUSIONS

While ChatGPT substantially reduced the reading complexity of pediatric orthopaedic discharge documents, it also introduced safety-critical alterations. AI-assisted simplification should be implemented only with structured clinician review. Larger word-count reductions were associated with more safety-critical alterations, so the magnitude of reduction may help prioritize which simplified documents receive the closest clinician review.

LEVEL OF EVIDENCE

Level III.