AUTHORS
Lang Min, Kuanyu Xia2, Ting MD, PhD Niu, MD, PhD3
AFFILIATIONS
1 Department of Hematology, Institute of Hematology, and Center for High Altitude Medicine, West China Hospital, Sichuan University, Chengdu, Sichuan, China
2 Department of Dermatology, Xiangya Hospital, Central South University, Changsha, Hunan, China
3 Department of Hematology, Institute of Hematology, and Center for High Altitude Medicine, West China Hospital, Sichuan University, Chengdu, Sichuan, China, National Facility for Translational Medicine (Sichuan), West China Hospital, Sichuan University, Chengdu, Sichuan, China, State Key Laboratory of Biotherapy, Collaborative Innovation Center of Biotherapy, West China Hospital, Sichuan University, Chengdu, Sichuan, China
ABSTRACT
Introduction: The treatment of lymphoma has become increasingly complex with the expansion of targeted agents, antibody-based therapies, and cellular immunotherapies, requiring clinicians to integrate disease subtype, treatment sequencing, contraindications, toxicity monitoring, and patient-specific risk factors within evolving guideline frameworks. Large language models (LLMs) can rapidly retrieve and synthesize medical information and are increasingly considered for clinical decision support. However, most prior evaluations have focused on earlier-generation models or general medical questions, and the guideline adherence and safety of contemporary reasoning-enhanced LLMs in specialty-specific lymphoma treatment—particularly across low-risk informational tasks and high-risk clinical decisions—remain uncertain. We therefore compared five contemporary LLMs with board-certified hematologic oncologists using a risk-stratified, guideline-based framework to evaluate response completeness and safety.
Methods: A set of 45 open-ended clinical scenarios was developed from the National Comprehensive Cancer Network Clinical Practice Guidelines in Oncology for Lymphomas and stratified into low- and high-risk categories. Five LLMs—ChatGPT-5.4, Claude Sonnet 4.6, Gemini 3.0 Pro, Grok 4.1 Thinking, and DeepSeek-R1—were compared with five board-certified hematologic oncologists. The median expert score for each scenario served as the human benchmark. Responses were evaluated against predefined guideline-based reference answers for normalized completeness and critical errors, defined as recommendations that contradicted guideline requirements or posed potential patient harm. Two independent evaluators blinded to response source assessed all responses. Overall and risk-stratified differences were evaluated using Kruskal–Wallis tests followed by Bonferroni-adjusted Dunn tests.
Results: Normalized completeness scores differed significantly among the five LLMs and the hematologic oncology expert benchmark (χ² = 22.637, df = 5, P < 0.001; ε² = 0.073). Gemini 3.0 Pro, DeepSeek-R1, and Grok 4.1 Thinking achieved significantly higher completeness scores than the expert benchmark, whereas no statistically significant differences were detected among the five LLMs. Risk-stratified analysis showed that this advantage was context dependent. In low-risk scenarios, the LLMs significantly outperformed the expert benchmark (χ² = 18.245, P = 0.003), demonstrating stronger coverage of guideline-based information in retrieval, routine management, and explanatory tasks. In contrast, performance converged in high-risk scenarios, with no significant between-group differences (χ² = 5.482, P = 0.360), indicating that the overall completeness advantage was driven primarily by lower-risk questions. Safety analysis showed the opposite pattern. Critical error frequency differed significantly among respondents (χ² = 25.104, P < 0.001). The expert benchmark yielded only 1 critical error across all 45 scenarios, whereas each LLM produced 4–7 critical errors. The safety gap was particularly important in high-risk scenarios: the expert benchmark yielded no critical errors, while every evaluated LLM produced at least 1. Thus, models that provided more complete guideline-based responses were not correspondingly safer, revealing a clear divergence between information coverage and the ability to avoid potentially harmful recommendations.
Conclusions: Contemporary LLMs demonstrated high completeness in guideline-based lymphoma treatment recommendations, and several models outperformed the hematologic oncology expert benchmark, particularly in low-risk scenarios. However, their higher frequency of critical errors, including errors in high-risk scenarios, limits their suitability for autonomous clinical decision-making. LLMs may support information synthesis and preliminary decision support, but their treatment recommendations should not be used without review by qualified hematologic oncology specialists.
DISCLOSURES
Lang Min: No relevant conflicts of interest to declare.