AUTHORS
Joyce Tiong, MD, Omid Jafari, Senior Software Engineering and Programming Associate2, Emma Kitchel3, Elizabeth Chiang4, Aliya Lackan5, Ryan Bailey, MD6, Mary North Jones7, Dang Nhu Hai Pham, PhD8, Ang Li, MD, MS9
AFFILIATIONS
1 Baylor College of Medicine, Houston, Texas, United States
2 Baylor College of Medicine, Houston, Texas, United States
3 Baylor College of Medicine, Houston, Texas, United States
4 Baylor College of Medicine, Houston, Texas, United States
5 Baylor College of Medicine, Houston, Texas, United States
6 Baylor College of Medicine, Houston, Texas, United States
7 Baylor College of Medicine, Houston, Texas, United States
8 Baylor College of Medicine, Houston, Texas, United States
9 Baylor College of Medicine, Houston, Texas, United States
ABSTRACT
Background: Clinically Relevant Bleeding (CRB) includes Major Bleeding (MB) and Clinically Relevant Non-Major Bleeding (CRNMB) as defined by ISTH. While it is an important outcome to capture in studies of cancer-associated thrombosis, it is difficult and labor-intensive to ascertain accurately at scale. Most retrospective studies rely on ICD-10 codes, which inadequately distinguish incident from historical bleeding due to carrying forward from prior encounters. Conventional natural language processing (NLP) models pose a similar problem since the evidence needed to adjudicate a true incident event is scattered across unstructured and structured data. We prompt-engineered and evaluated a Large Language Model (LLM) for patient-level detection, severity classification, and structured characterization of CRB events longitudinally.
Methods: We manually annotated a cohort of 400 cancer patients (50% with CRB) from our local health system. For each patient, two trained and blinded medical students and residents independently reviewed clinical notes, hemoglobin values, and blood transfusion counts. Each discrete CRB event was annotated with its date, severity, and site. A senior medical resident reconciled discrepancies, and a hematologist adjudicated remaining disagreements. Prompts were iteratively developed on 200 training patients for Anthropic Claude Opus 5 model (with max thinking effort) in a HIPAA-compliant, zero-retention environment. After each iteration, discrepancies between model predictions and manual annotations were systematically reviewed to refine prompt instructions, and the final prompt was locked before test-set inference. Model inputs were full discharge summaries, radiology impressions, preprocessed clinical notes, and hemoglobin and transfusion data from -3 months to +12 months after cancer therapy. The prompt instructed the LLM to first screen permissively for candidate overt bleeding, then apply specificity-oriented adjudication rules using ISTH guidelines to determine clinical relevance and severity, assign date, site, and episodes. Structured output also captured suspected cause and ISTH criteria met. In the 200 held-out patients, predicted and gold-standard events were matched 1:1 by site using a 30-day window. Patient-level status was defined as CRB versus none (binary) and MB/CRNMB/none (3-class), with severity assigned from the first CRB event, consistent with time-to-first-event definition. Wilson score 95% confidence intervals (CIs) were calculated for patient-level metrics.
Results: The test-set gold-standard contained 164 CRB events (65 MB; 99 CRNMB) in 106 of 200 patients (73 with 1 event, 19 with 2 events, 14 with 3+). The LLM detected patient-level CRB with 97% precision (95% CI 92-99%) and 96% recall (95% CI 91-99%). Event-level precision and recall were both 92%. For MB, patient-level precision and recall were 94% (95% CI 81-98%) and 91% (95% CI 78-97%), respectively, while event-level outcomes were both 91%. For CRNMB, patient-level precision and recall were both 93% (95% CI 85-97%), compared with 86% precision and 87% recall at the event level. Performance for second CRB events remained strong, with 91% precision and 88% recall overall, 80% for both metrics in MB, and 82% precision and 78% recall for CRNMB. Overall exact-site agreement was 99%, with 92% agreement for genitourinary bleeds and 100% for all other sites. Using AWS Bedrock, batch inference processed 200 patients in ~10 minutes at an average cost of $0.13 per patient.
Conclusions: Benchmarked against labor-intensive, multi-reviewer manual annotation of full records, a prompt-engineered LLM achieved high accuracy at the identification of 1-year CRB status, separated MB from CRNMB, and reconstructed recurrent episodes with strong site and date fidelity in cancer patients. Remaining errors primarily stemmed from recurrent tumor bleeds and confounding anemia from chemotherapy or cancer-related marrow suppression, reflecting the unique challenges to capturing longitudinal CRB in cancer patients. Because the model returns structured, episode-level output rather than a single flag, it may enable bleeding endpoint ascertainment at a scale that manual review cannot reach, with significantly less time and labor intensity and at a reasonable cost. External validation in independent health systems remains necessary for future work.
DISCLOSURES
Joyce Tiong, MD: No relevant conflicts of interest to declare.