Spine Journal - 2026-10-18 - Journal Article
Development of a Natural Language Processing Algorithm to Aid Creation of a Surgical Spine Database.
Wong HJ, Song M, Miller SL, Damle EB, Muniz R, Mehta S, McFarlane K, Berg A, Li YD, Koltsov JCB, Xiong GX
Topics
Key Takeaway
A rule-based NLP algorithm extracted operated spinal levels from operative reports with 93% agreement and kappa 0.98, processing 1,039 reports per minute versus 1 report per minute for the fastest human reviewer.
Summary Depth
Choose how much analysis to show on this article page.
Summary
This study asked whether a rule-based NLP algorithm could replace manual chart review for surgical spine database creation, validating performance against human reviewers of varying clinical experience using 100 randomly sampled operative reports from a 16-year institutional dataset. The NLP met the a priori acceptability threshold (agreement 93%, kappa 0.98, F1 0.98) for spinal level classification and outperformed human reviewers in training, but failed to meet threshold for diagnosis, operative procedure, and additional surgical detail extraction. Processing speed was 1,039 reports per minute versus a human median of 1 report per minute, representing a >1,000-fold efficiency gain for level classification tasks.
Key Limitation
Validation on only 100 operative reports from a single tertiary institution limits generalizability, as operative report dictation style varies substantially across surgeons, institutions, and training programs.
Original Abstract
BACKGROUND
Manual effort required for feature extraction from operative reports can be time-intensive, inaccurate, and costly.
PURPOSE
Develop a natural language processing (NLP) algorithm for surgical spine database creation and validate performance against human reviewers with varying clinical experience.
DESIGN/SETTING
Retrospective study at a tertiary academic institution from 2008-2024.
PATIENT SAMPLE
Operative spinal cases on adult patients from 22,951 spinal operative reports.
OUTCOME MEASURES
Primary outcome was accuracy in operated spinal level classification. Secondary outcomes include clinical diagnoses, operative procedures, and additional surgical detail classification.
METHODS
A rule-based NLP algorithm was developed and validated on a random sample of 100 operative reports. Performance of the NLP algorithm and human reviewers was assessed against a reference standard. An a priori acceptability threshold was set at 0.90 for agreement, kappa, and F1. Time required for chart review/processing were also assessed for the NLP algorithm and human reviewers.
RESULTS
The NLP algorithm matched the reference standard on operated spinal levels (Percent Agreement=93%; Kappa=0.98; F1=0.98), which outperformed human reviewers in clinical training. Performance on procedural and diagnostic feature extraction did not meet the accepted threshold but largely outperformed human validators in clinical training. NLP algorithm computation time averaged speed of 1,039 reports per minute, compared to the fastest median time of 1 minute per operative report for human chart reviewers.
CONCLUSIONS
The NLP algorithm can accurately and reliably extract operated spinal levels from high volumes of operative reports. However, classification of diagnosis, operative procedures, and additional surgical procedures were not sufficient for practical implementation.