AUTHORS
Sri Vaishnavi Krishnan, Mei Wang, MSc2, Sarah Addison, MD, BS3, Ryan Dingle4, Cynthia Ruoqi Dong, BA5, Kristen Marie Sanfilippo, MD, MPHS6, Theodore Thomas, MD7, Martin Schoen, MD, PMH8, Su-Hsin Chang, PhD9
AFFILIATIONS
1 University of Wisconsin-Madison; Madison, Madison, Wisconsin, United States, Washington University School of Medicine, St. Louis, Missouri, United States
2 St. Louis Veterans Affairs Medical Center, St. Louis, Missouri, United States, Washington University School of Medicine, St. Louis, Missouri, United States
3 St. Louis Veterans Affairs Medical Center, St. Louis, Missouri, United States, Washington University School of Medicine, St. Louis, Missouri, United States
5 St. Louis Veterans Affairs Medical Center, St. Louis, Missouri, United States, Washington University School of Medicine, St. Louis, Missouri, United States
6 St. Louis Veterans Affairs Medical Center, St. Louis, Missouri, United States, Washington University School of Medicine, St. Louis, Missouri, United States
7 St. Louis Veterans Affairs Medical Center, St. Louis, Missouri, United States, Washington University School of Medicine, St. Louis, Missouri, United States
8 Saint Louis University School of Medicine, St. Louis, Missouri, United States, St. Louis Veterans Affairs Medical Center, St. Louis, Missouri, United States
ABSTRACT
Introduction: Cytogenetic markers are essential for risk-stratification and clinical management in multiple myeloma (MM). However, these data are embedded within unstructured electronic health records (EHRs). Dual-human data abstraction is the standard approach to collect cytogenetic data but can be labor-intensive and time-consuming. This limits the feasibility of large-scale, population-based outcomes research. Past studies have found that using artificial intelligence (AI) to conduct data abstraction increased efficiency (Jensen et al., PLOS One, 2025). We adapted this approach and evaluated if a hybrid AI-human abstraction workflow, with AI as the second independent abstractor to abstract MM cytogenetic abnormalities in unstructured EHRs, could replace the standard dual-human data abstraction.
Methods: We performed analyses within the Veterans Health Administration (VHA) database. As the largest integrated healthcare system in the U.S., VHA captures extensive, longitudinal data on cytogenetic tests across the nation. Individuals with ≥1 International Classification of Diseases (ICD) code for monoclonal gammopathy of indeterminate significance (MGUS) or MM who underwent conventional karyotyping and Fluorescence in Situ Hybridization (FISH) between 1/2013 and 6/2025 were identified using CPT codes. We randomly selected 100 patients and retrieved their clinical records, laboratory reports, and pathology reports <6 months after the testing date (846 records). We targeted 9 cytogenetic abnormalities due to their clinical relevance in MM including amplifications of 1q, monosomy 13/deletion of 13q, monosomy 14, trisomies of odd-numbered chromosomes, deletion of 1p, monosomy 17/deletion 17p, IgH translocations t(4;14), t(14;16), and t(11;14).
We performed a hybrid AI-human workflow for cytogenetic data abstraction and compared it to the standard dual-human workflow. In the dual-human workflow, 2 human abstractors independently reviewed each EHR document and noted the presence of each target abnormality as yes, no, or unknown. In the hybrid AI-human workflow, 1 human abstractor paired with an AI model using a large language model (Ministral-3) with a zero-shot prompt independently extracted the same abnormalities. The discordant findings were adjudicated by another human abstractor. Concordance was assessed using document-level complete-match ratio, abnormality-specific precision, recall, F1-score, and accuracy for each abnormality. Macro-averages of precision, recall, F1-score, and accuracy across all target abnormalities were calculated to assess overall performance. Efficiency was measured by the amount of time taken to perform data abstraction, the proportion of EHRs requiring adjudication, and total workflow completion time.
Results: Among theincluded 846 records for the 100 randomly selected patients, 37.1% of the records were identified to have ≥1 of the 9 target abnormalities present. The AI-human workflow attained macro-averaged accuracy/precision/recall/F1-score of 99.8/97.9/99.9/98.7% across all target MM cytogenetic abnormalities. For abnormality-specific performance, the human-AI workflow yielded an F1-score of 100% for monosomy 13/deletion of 13q, trisomies of odd-numbered chromosomes, and IgH translocations t(4;14) and t(14;16) and the lowest F1-score of 87.3% for monosomy 14 abnormality. Average human review time was 102.5 seconds per record, while it was 8.7 seconds for AI. However, the AI-human workflow required adjudication in 35.9% of EHRs compared with 18.1% for dual-human workflow. The completion time, comprising data abstraction and result adjudication, was 34.8 hours for the AI-human workflow and 52.5 hours for the dual-human workflow, a 40.5% reduction.
Conclusions: A hybrid AI-human data abstraction in which AI served as the second abstractor increased the efficiency of cytogenetic data extraction by ~40%, while maintaining non-inferior accuracy to the standard dual-human workflow. These findings support the implementation of AI-human workflow as a scalable alternative to dual-human review for cytogenetic data extraction from unstructured EHRs. The proposed workflow has the potential to be generalized to accelerate clinical data extraction in other malignancies. External validation is warranted in other target subjects of interest, EHR systems, or patient populations.
DISCLOSURES
Ryan Dingle: No relevant conflicts of interest to declare.