AUTHORS
Jonathan Berry, MD, Evan Garrad2, Alec Seidman Sorsby, MD3, Samantha Armstrong, MD4, Richard C. Godby, MD5, Ariela Marshall, MD6, Ronak Mistry, DO7, Vivek Patel, MD8, Karine Tawagi9
AFFILIATIONS
1 Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States
2 University of Illinois College of Medicine at Chicago, Chicago, Illinois, United States
3 Boston Medical Center, Boston, Massachusetts, United States
4 Indiana University School of Medicine, Indianapolis, Indiana, United States
5 Mayo Clinic, Rochester, Minnesota, United States
6 University of Minnesota Medical School, Minneapolis, Minnesota, United States
7 Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, United States
8 Vanderbilt University Medical Center, Nashville, Tennessee, United States
9 University of Illinois College of Medicine at Chicago, Chicago, Illinois, United States
ABSTRACT
INTRODUCTION: Generative artificial intelligence (AI) tools, including large language models (LLMs), are rapidly being integrated into clinical practice and medical education. Our previous work demonstrated that H/O trainees report limited formal education on AI capabilities, limitations, and safe implementation. From these findings, we developed and evaluated the impact of a multicenter pilot educational workshop on fellows' understanding of generative AI, perceptions of its clinical applications, and intended future use.
METHODS: Five U.S. Hematology/Oncology fellowship programs were invited to participate in this study between March and June 2026. We conducted a standardized 90-minute interactive educational workshop about AI led by an AI expert that was delivered either in person or virtually over Zoom. Topics during the workshop included introducing generative AI concepts, practical clinical and administrative applications, and limitations including bias, impacts on learning, and responsible use. Participants completed anonymous pre- and immediate post-workshop surveys assessing AI knowledge, familiarity with bias, and perceptions of AI's impact on patient care, workflow, and diagnostic reasoning. Participants completing both surveys were matched using the last four digits of their phone number. Changes in paired ordinal responses were categorized as improved, unchanged, or declined and compared using the Wilcoxon signed-rank test. Post-workshop surveys also evaluated participant satisfaction and anticipated future AI use.
RESULTS: A total of 78 fellows completed the pre-workshop survey and 48 completed the post-workshop survey, corresponding to an overall response rate of 48/78 (62%). Of the 78 fellows who completed the pre-workshop survey, 34 (44%) also completed the post-workshop survey and were included in the paired analysis. Prior to the workshop, 37 (47%) participants reported an intermediate or advanced understanding of LLMs and generative AI, however 33 (42%) were not at all familiar with how bias may manifest in generative AI outputs. Following the workshop, familiarity with AI bias improved in 24 participants (71%), remained unchanged in 10 (29%), and declined in none (P < 0.001). Perceptions of AI's impact on patient care improved in 17 participants (50%), remained unchanged in 15 (44%), and declined in 2 (6%) (P < 0.001). Perceptions of AI's impact on workflow improved in 10 participants (29%), remained unchanged in 20 (59%), and declined in 4 (12%) (P = 0.058). Perceptions of AI's impact on diagnostic and management reasoning improved in 12 participants (35%), remained unchanged in 18 (53%), and declined in 4 (12%) (P = 0.077). Across all post-workshop respondents, 32 (94%) reported a moderate or significant improvement in their understanding of LLMs and generative AI. Additionally, 36 of 48 participants (75%) reported planning a specific change or new use of generative AI, with common planned applications including personal education, administrative tasks, and patient education materials. All participants rated the workshop as either excellent 33 (85%) or good 6 (15%).
CONCLUSIONS: This multicenter pilot educational workshop significantly improved H/O fellows' familiarity with AI bias and perceptions of AI's impact on patient care. Participants also reported high satisfaction and a strong intention to incorporate generative AI into future educational and clinical practice. These findings suggest that structured AI education is feasible, well received, and may facilitate the safe and responsible integration of generative AI into H/O training. This study was limited by small sample size and respondent bias. A large multi-center cluster randomized trial is underway to validate these findings.
DISCLOSURES
Jonathan Berry, MD: No relevant conflicts of interest to declare.