AUTHORS
Rosario Ruiz Velazquez, Andre De Menezes Silva Corraes, MD2, Christoph Schaefers, MD3, Matthew Jankowski, MS4, Melinda Tan, MD5, Kenneth JC Lim, MBBS6, Monica Shaw7, Kimberly Riester8, Jody Mclean9, Ricardo Daniel Parrondo, MD10, Saurabh Chhabra, MD, MBBS11, Patrick Johnston, MD, PhD12, Jonas Paludo, MD13, Yi Lin Jr, MD, PhD14
AFFILIATIONS
1 Mayo Clinic, Rochester, Minnesota, United States
2 Mayo Clinic, Rochester, Minnesota, United States
3 Mayo Clinic, Rochester, Minnesota, United States
4 Mayo Clinic, Rochester, Minnesota, United States
5 Mayo Clinic, Rochester, Minnesota, United States
6 Mayo Clinic, Rochester, Minnesota, United States
7 Mayo Clinic, Rochester, Minnesota, United States
8 Mayo Clinic, Rochester, Minnesota, United States
9 Mayo Clinic, Rochester, Minnesota, United States
10 Mayo Clinic, Rochester, Minnesota, United States
11 Mayo Clinic, Rochester, Minnesota, United States
12 Mayo Clinic, Rochester, Minnesota, United States
13 Mayo Clinic, Rochester, Minnesota, United States
14 Mayo Clinic, Rochester, Minnesota, United States
ABSTRACT
Introduction
Chimeric antigen receptor (CAR-T) T-cell therapy has transformed the treatment of various hematological cancers. Increased use in real world practice has led to the recognition of uncommon, delayed toxicities, most commonly present in the first 6 months after infusion: immune-effector cell (IEC) associated cranial nerve palsy (IEC-CNP), IEC-Parkinsonism (IEC-PKS), and IEC-enterocolitits (IEC-EC), more commonly seen after BCMA-targeting CAR-T, such as cilta-cel. Accurate reporting to regulatory agency and data registries such as the CIBMTR (Center for International Bone Marrow Transplant and Cell Therapy Registry) is important to understand real world incidence and inform evolving strategies to evaluate and manage these toxicities. The objective of this study was to evaluate the application of a large language model (LLM) to identify delayed toxicities from clinical documentation compared with manual chart review
Methods
479 patients (pts) who received cilta-cel across Mayo Clinic 3 sites between 2022 – 2026 with clinical data available from the IEC compliance database were used to develop the LLM: 40 pts with IEC-CNP (28), IEC-PKS (11) and or IEC-EC (7) and 20 randomly selected pts without these events. A Gemini large language model (LLM) was used to analyze clinical documentation obtained from Mayo Clinic electronic medical records (EMR), including clinical notes, pathology reports, and procedure notes, using prompts describing the clinical characteristics of each toxicity. The model independently evaluated IEC-CNP, IEC-PKS, & IEC-EC. LLM results were compared with manual EMR review and the database Performance were assessed using confusion matrix statistics, including accuracy, precision, sensitivity, and the F1 score.
Results
For IEC-CNP, initial LLM (v1) had 90% sensitivity, 80% specificity, with 1 false positive (FP) case. This was found to be due to clinical note documentation of symptoms occurring before CAR-T infusion. Prompt was updated to specify symptoms occurrence after infusion regardless of the timing of the clinical note. Updated LLM (v2) yielded 100% sensitivity, specificity & F1 score.
For IEC-PKS, LLMv1 had 92.31% sensitivity, 90.48% specificity, with 4 FP cases. Prompts were updated to remove tremor only symptom. In LLMv2, those 2 FP cases became true negatives. EMR review of the other 2 FP cases determined that LLM identification was correct, but missed in the database due to limited clinical understanding of clinical presentation at the time of the events. 1 false negative (FN) case occurred in LLMv2 without provision of justification from LLM.
For IEC-EC, LLMv1 had 92.31% sensitivity, 90.48% specificity, with 3 FP. FP were due to infections identified as etiology for diarrhea. Repeat run of LLMv1 on the same dataset yielded 100% sensitivity, 92.86% specificity, 2 FP & 2 FN cases. One FP case in the first run became true negative without edits to the prompt, demonstrating output variations of LLM.
In general, the FN cases with both IEC-PKS & IEC-EC needed additional details to guide quality program review of the cases.
The initial analysis demonstrated that the LLM correctly identified most cases across the three toxicities. However, discrepancies were observed due to limitations in prompt design and the complexity of clinical documentation. Review of these cases guided iterative prompt refinement by incorporating more explicit toxicity criteria, improving the distinction between pre-CAR-T and post CAR-T clinical events, excluding nonspecific symptoms and enhancing contextual interpretation of clinical notes. Following prompt optimization, LLM performance improved, although some discrepancies remained.
Discussion
We demonstrate that LLM has the potential to support clinical data abstraction, synthesis and reduce manual review burden for the IEC compliance team. Iterative prompt refinement improved the model's performance to identify CAR-T associated delayed toxicities. The remaining discrepancies were primarily attributable to limitations in clinical documentation, the evolving clinical understanding of these uncommon toxicities, and the variability of the LLM. Future work will focus on prospectively validating this LLM in patients undergoing CAR-T therapy to support their integration into clinical and regulatory workflows.
DISCLOSURES
Rosario Ruiz Velazquez: No relevant conflicts of interest to declare.