2026 ASH®
Annual Meeting & Exposition December 12-15, 2026 ▪ New Orleans, Louisiana

Paper Details

Abstracts, authors, affiliations, and disclosures.

SESSION

653. Multiple Myeloma: Clinical and Epidemiological: Poster III

Program: Poster Abstract Session
Monday, December 14, 2026
06:00 p.m. - 08:00 p.m.
MCCNO - Hall F
Access, Representation, Population Health, Technology and Procedures, Imaging, Treatment Considerations, Combination Therapy, Supportive Care, Therapy Sequence, Biological Therapies, Transplantation, Bispecific Antibody Therapy, (CAR)-T Cell Therapies, Immunotherapy, Monoclonal Antibody Therapy, Non-Biological Therapies, Pharmacology, Radiation Therapy, Women’s Health, Adverse Events, Artificial Intelligence (AI), Machine Learning, Biological Processes, Clinical Practice (Health Services and Quality), Clinical Guidelines, Patient Safety, Practice Models, Quality Improvement, Quality of Care, Value, Diseases, Bone Marrow Failure Syndromes, Hemoglobinopathies, Infectious Diseases, Viral, Lymphoid Malignancies, Plasma Cell Disorders, Myeloid Malignancies, Research, Translational Research, Clinical Research, Clinical Trials, Epidemiology, Patient-Reported Outcomes, Real-World Evidence, Registries, Survivorship, Study Population, Human, Adult, Elderly
PAPER

5760 Comparative evaluation of chatgpt, claude, and gemini for patient education in multiple myeloma and its precursor conditions.

Sree Manasa Puttamreddy, MD, Yumna Khalid2, Saad Abdul Kareem3
1 University of Alabama at Birmingham, Montgomery, Alabama, United States
2 University of Alabama at Birmingham, Montgomery, Alabama, United States
3 University of Maryland Capital Region Health, Largo, United States

Introduction: Large language models (LLMs) are increasingly used by patients as readily accessible sources of health information. Approximately 17% of adults report monthly use of AI chatbots for medical questions. Unlike traditional health websites, LLMs provide conversational, personalized responses that may influence patient understanding and decision-making. However, their reliability in hematologic malignancies remains poorly characterized. To our knowledge, this is the first study to comprehensively compare commercially available LLMs for accuracy, hallucinations, safety, quality, and readability using common patient questions spanning multiple myeloma (MM), smoldering multiple myeloma (SMM), and monoclonal gammopathy of undetermined significance (MGUS). Responses were compared with guideline-based reference standards from the International Myeloma Working Group (IMWG), National Comprehensive Cancer Network (NCCN), and International Myeloma Foundation (IMF).

Methods: We performed a cross-sectional evaluation of three publicly available LLMs (free versions of ChatGPT, Claude Sonnet 4.6, and Gemini 3.5 Flash) using responses generated on June 24, 2026. Nineteen patient-facing questions regarding MM, SMM, and MGUS were identified using Google Trends and covered disease knowledge, diagnosis, symptoms, and treatment. Free versions were evaluated because they represent the most accessible LLMs for patients seeking health information. Responses were generated using newly created accounts, with each question entered into a new chat without follow-up prompts. Each response was evaluated against predefined guideline-based answer keys, yielding 74 scorable accuracy items per LLM. Safety and hallucination assessments were performed only for predefined applicable responses. Two independent reviewers assessed accuracy, readability, major and subtle hallucinations, safety, and overall information quality using the DISCERN instrument. 

Results: Inter-rater agreement for accuracy was 96.8% (215/222); Cohen's κ 0.35, consistent with the prevalence paradox due to overwhelmingly accurate responses. Following consensus review, ChatGPT achieved an accuracy of 98.6% (73/74; 95% CI, 92.7-99.8%), Gemini 97.3% (72/74; 95% CI, 90.7-99.3%), and Claude 95.9% (71/74; 95% CI, 88.7-98.6%). No statistically significant differences in accuracy were detected among models (Cochran's Q(2)=1.20, P=0.549).

Major hallucinations were rare, occurring once each with ChatGPT and Gemini (1.3%) and not observed with Claude (0.0%) (Cochran's Q(2)=2.00, P=0.368). Subtle hallucinations occurred more frequently with Claude (3/41, 7.3%) than with ChatGPT or Gemini (0%) (Cochran's Q(2)=6.00, P=0.050).

Safety rates were 72.2% (13/18) for ChatGPT, 88.9% (16/18) for Gemini, and 94.4% (17/18) for Claude. No statistically significant differences were detected  (Cochran's Q(2)=4.33, P=0.115). Overall DISCERN scores were 44 (Fair) for ChatGPT, 49 (Fair) for Claude, and 51 (Good) for Gemini, indicating the highest overall information quality for Gemini. 

Readability differed significantly among models. ChatGPT generated the most readable responses (mean Flesch-Kincaid grade level 9.6 ± 1.6), followed by Claude (11.0 ± 1.7) and Gemini (12.4 ± 1.8) (ANOVA, P < 0.001). However, responses from all three models exceeded the NIH- and AMA-recommended sixth-grade reading level for patient education materials.

Conclusion: While most research has focused on clinician applications of artificial intelligence, our findings evaluate LLM performance from the patient perspective. ChatGPT, Claude, and Gemini demonstrated high accuracy with infrequent hallucinations when answering common patient questions about MM, supporting their potential as adjuncts for patient education. However, safety concerns were identified in a subset of responses, and all models generated content above the NIH- and AMA-recommended sixth-grade reading level. As patient use of LLMs expands, AI-specific quality standards, patient AI literacy initiatives, and clinician awareness of AI-generated misinformation will be essential for the safe integration of these tools into cancer care. Limitations include evaluation at a single time point without assessment of patient comprehension or clinical outcomes, and use of DISCERN, which may not fully capture AI-specific quality characteristics.

Sree Manasa Puttamreddy, MD: No relevant conflicts of interest to declare.