AUTHORS
Robert Norberg, Omkar Bhabad2, Myles Kim, PhD3, Larry Grullon-Polanco, N/A4, Adam Duca, BS5, Emily C. Merritt6, Daniel De Avila7, Gabriel De Avila8, Omar Alexis Castaneda Puglianini, MD9, Hien Liu, MD10, Melissa Alsina, MD11, Taiga Nishihori, MD12, Kenneth H. Shain, MD, PhD13, Ariosto Siqueira Silva, PhD14, Rachid Baz, MD15, Brandon Blue, MD16, Ariel Grajales-Cruz, MD17, Doris K. Hansen, MD18, John L Cleveland, PhD19, Frederick L. Locke, MD20, Nicholas Figura, MD21, Ghulam Rasool, PhD22, Ciara Louise L Freeman, MD, PhD23
AFFILIATIONS
1 Moffitt Cancer Center, Tampa, Florida, United States
2 Moffitt Cancer Center, Tampa, Florida, United States
3 Moffitt Cancer Center, Tampa, Florida, United States
4 Moffitt Cancer Center, Tampa, Florida, United States
5 Moffitt Cancer Center, Tampa, Florida, United States
6 Moffitt Cancer Center, Tampa, Florida, United States
7 Moffitt Cancer Center, Tampa, Florida, United States
8 Moffitt Cancer Center, Tampa, Florida, United States
9 Moffitt Cancer Center, Tampa, Florida, United States
10 Moffitt Cancer Center, Tampa, Florida, United States
11 Moffitt Cancer Center, Tampa, Florida, United States
12 Moffitt Cancer Center, Tampa, Florida, United States
13 Moffitt Cancer Center, Tampa, Florida, United States
14 Moffitt Cancer Center, Tampa, Florida, United States
15 Moffitt Cancer Center, Tampa, Florida, United States
16 Moffitt Cancer Center, Tampa, Florida, United States
17 Moffitt Cancer Center, Tampa, Florida, United States
18 Moffitt Cancer Center, Tampa, Florida, United States
19 Moffitt Cancer Center, Tampa, Florida, United States
20 Moffitt Cancer Center, Tampa, Florida, United States
21 Moffitt Cancer Center, Tampa, Florida, United States
22 Moffitt Cancer Center, Tampa, Florida, United States
23 Moffitt Cancer Center, Tampa, Florida, United States
ABSTRACT
Background
Trial pre-screening and real-world cohort assembly in relapsed/refractory multiple myeloma (RRMM) require accurate identification of key clinical features from longitudinal free-text documentation. These include drug-specific exposures and refractoriness, presence of extramedullary disease (EMD), and plasma cell leukemia (PCL). Manual abstraction is time-intensive and susceptible to incomplete or inconsistent classification. In a cohort of patients treated with BCMA-directed CAR T-cell therapy, we evaluated a large language model (LLM; Claude) pipeline that reconstructs these phenotypes from longitudinal notes using prompts operationalizing IMWG definitions, benchmarked against a human-curated database, with discordant cases adjudicated by physician review and validated in a blinded held-out cohort.
Methods
The single-center cohort included 734 patients with RRMM and 5,225 clinical notes. Feature-specific prompts applied predefined IMWG-based criteria to classify exposure and refractory status for bortezomib, carfilzomib, lenalidomide, pomalidomide, and anti-CD38 monoclonal antibodies. Each drug was assigned a status of not exposed, exposed, or refractory. Refractory status required documented progression during or within 60 days of a regimen containing the drug; whereas discontinuation for toxicity or planned sequencing without progression did not. The pipeline also classified active EMD prior to lymphodepletion using clinical notes and radiology reports and identified PCL from clinical documentation. Per-drug status was determined by reconciling notes within a prespecified pre-apheresis window to align with clinical trial criteria and human-curated database. A physician re-reviewed discordant classifications. Prompts, models, and note-level inputs were version-controlled, run at temperature 0 with full-prompt hashing. Agreement, sensitivity, specificity, and Cohen's kappa with 95% confidence intervals were calculated for each feature. Weighted kappa was used for ordinal drug status. Among patients represented in Flatiron real-world data, structured medication records were used as an independent comparison for pre-CAR-T drug exposure.
Results
Across 1,834 agent classifications, agreement between the LLM and the curated database for ordinal exposure and refractory status was 86.1% (84.2%-88.8% by drug; 95% CI, 84.4%-87.6%; weighted κ=0.73). Physician adjudication was performed for 95 patients with discordant results (475 classifications). After discordant cases were re-reviewed by a physician to establish an adjudicated reference, agreement between the LLM and this adjudicated reference was 96.6%, compared with 88.6% against the original database.
For active EMD prior to lymphodepletion, agreement with the original curated database was 80.3% among 142 evaluable patients (κ=0.45). Physician review of 53 discordant cases assessed whether discrepancies represented previously unrecorded soft-tissue disease or false-positive LLM classifications. Against the physician-adjudicated reference, agreement was 88.7% and κ was 0.71. For prior PCL, agreement was 97.2% among 325 evaluable patients (prevalence, 3.7%; κ=0.63); after physician adjudication, LLM agreement rose to 99.4%. Among 351 patients also represented in Flatiron real-world data, the LLM agreed with structured records for 1,223 of 1,277 evaluable pre-CAR-T drug-exposure classifications (95.8%). In the independently curated, blinded held-out cohort, agreement was 79.1% for ordinal drug exposure and refractory status (weighted κ=0.74, compared with 0.73 in the development cohort), 81.2% for active EMD, and 98.6% for PCL. The held-out cohort was more class-balanced, which may account for the lower raw agreement despite a similar weighted κ.
Conclusions
This criterion-guided LLM pipeline identified trial-relevant myeloma phenotypes from longitudinal clinical documentation, including agent-specific exposure and refractoriness, active EMD, and prior PCL, with improved agreement after expert adjudication. Discordance review showed that some apparent LLM errors reflected incomplete or inconsistent original abstraction, while others highlighted cases requiring targeted physician review. The approach also improved phenotype completeness compared with existing curated fields. These findings support further evaluation of LLM-assisted trial pre-screening workflows to identify potentially eligible patients and prioritize cases for human confirmation before trial enrollment.
DISCLOSURES
Robert Norberg: No relevant conflicts of interest to declare.