AFFILIATIONS
1 German Cancer Consortium (DKTK), Heidelberg, Germany, German Cancer Research Center (DKFZ), Heidelberg, Germany, Heidelberg Institute for Stem Cell Technology and Experimental Medicine (HI-STEM gGmbH), Heidelberg, Germany, Heidelberg University Hospital, Heidelberg, Germany
2 German Cancer Consortium (DKTK), Heidelberg, Germany, German Cancer Research Center (DKFZ), Heidelberg, Germany, Heidelberg Institute for Stem Cell Technology and Experimental Medicine (HI-STEM gGmbH), Heidelberg, Germany, Heidelberg University Hospital, Heidelberg, Germany
3 Comprehensive Cancer Center Ulm, Ulm University, Ulm, Germany
4 Helmholtz AI, Helmholtz Munich, Munich, Germany, Institute of AI for Health, Helmholtz Munich, Munich, Germany, LMU University Hospital, Munich, Germany
5 Helmholtz AI, Helmholtz Munich, Munich, Germany, Institute of AI for Health, Helmholtz Munich, Munich, Germany, LMU University Hospital, Munich, Germany
6 Helmholtz AI, Helmholtz Munich, Munich, Germany, Institute of AI for Health, Helmholtz Munich, Munich, Germany, LMU University Hospital, Munich, Germany
7 Heidelberg University Hospital, Heidelberg, Germany
8 Heidelberg University Hospital, Heidelberg, Germany
9 LMU University Hospital, Munich, Germany
10 LMU University Hospital, Munich, Germany
11 Comprehensive Cancer Center Ulm, Ulm University, Ulm, Germany
12 LMU University Hospital, Munich, Germany
13 German Cancer Consortium (DKTK), Heidelberg, Germany, German Cancer Research Center (DKFZ), Heidelberg, Germany, Heidelberg Institute for Stem Cell Technology and Experimental Medicine (HI-STEM gGmbH), Heidelberg, Germany, Heidelberg University Hospital, Heidelberg, Germany
14 German Cancer Consortium (DKTK), Heidelberg, Germany, German Cancer Research Center (DKFZ), Heidelberg, Germany, Heidelberg Institute for Stem Cell Technology and Experimental Medicine (HI-STEM gGmbH), Heidelberg, Germany, Heidelberg University Hospital, Heidelberg, Germany
15 Heidelberg University Hospital, Heidelberg, Germany
16 Heidelberg University Hospital, Heidelberg, Germany
17 Heidelberg University Hospital, Heidelberg, Germany
18 LMU University Hospital, Munich, Germany
19 German Cancer Consortium (DKTK), Heidelberg, Germany, Heidelberg Institute for Stem Cell Technology and Experimental Medicine (HI-STEM gGmbH), Heidelberg, Germany
20 Heidelberg University Hospital, Heidelberg, Germany
21 LMU University Hospital, Munich, Germany
22 Comprehensive Cancer Center Ulm, Ulm University, Ulm, Germany
23 Institute of AI for Health, Helmholtz Munich, Munich, Germany, LMU University Hospital, Munich, Germany
24 German Cancer Consortium (DKTK), Heidelberg, Germany, German Cancer Research Center (DKFZ), Heidelberg, Germany, Heidelberg Institute for Stem Cell Technology and Experimental Medicine (HI-STEM gGmbH), Heidelberg, Germany, Heidelberg University Hospital, Heidelberg, Germany
ABSTRACT
Background
Translating clinical AI from controlled evaluation into reproducible cross-institutional decision support remains a central challenge. Hematology is a stringent test case: decisions integrate longitudinal treatment histories, guidelines, molecular findings, patient constraints and evolving evidence. HemaGuide is a locally deployable, case-grounded AI agent that converts unstructured clinical documents into structured case representations, autonomously routes them to guideline, advanced or molecular decision modes, and grounds recommendations in disease-specific pathways, targeted literature and a decision memory of >2,000 real-world tumor board cases (Zoller, Kalz et al., Nat Med 2026). We evaluated HemaGuide across progressively more realistic settings: blinded benchmarking, ablation, molecular interpretation, simulated practice, cross-institutional validation and prospective silent operation, and are now translating it into multicenter prospective use.
Methods
Completed stages comprised six foundation models benchmarked plain versus embedded in HemaGuide on 45 high-complexity cases with blinded expert scoring on six Likert dimensions; an 11-level ablation isolating each component; oncogenicity assessment of 70 missense variants against ClinGen/CGC/VICC references; a simulated practice study of residents unaided versus agent-assisted against senior attendings; cross-institutional validation on 555 de-identified cases from a second academic center; and prospective, non-interventional silent operation, in which recommendations were generated from de-identified documents, time-stamped and locked before the independently documented reference decision, and withheld from treating teams. Concordance was rated by two hematologists with a third adjudicator. The protocol is now being extended prospectively across four academic hematology centers with local workflows and guideline resources, and to selected gynecologic and dermatologic malignancies to test generalizability beyond hematology. Secondary endpoints are clinical utility, guideline adherence, patient-context integration, additional actionable options, processing time, clinically relevant omissions, harmful recommendations and hallucinations. The system runs on a dedicated workstation inside the hospital network; no clinical data leave the institution.
Results
Agentic grounding, not model scale, determined clinical adequacy: an ungrounded model reached 0% concordance versus 86.7% for the full agent (kappa = 0.85), with gains strictly routing-dependent (guideline-solvable cases resolved by flowcharts alone, 100%; advanced cases requiring case memory, literature and enrichment, 80%; molecular cases the variant toolkit, 100%). Blinded expert ratings improved from 3.21+/-0.39 to 4.22+/-0.06 (P=0.002) with markedly reduced case-to-case variability across backbones. Molecular mode reproduced expert classification in 56/70 variants (sensitivity 0.88, specificity 0.84, r=0.84); no oncogenic variant was downgraded to benign, and interpretation ran in real time on local hardware. Agent-assisted residents rose from ~60% unaided concordance to near-senior performance (P=0.015-0.035). Cross-institutional validation at a second academic center gave 81.8% concordance (454/555) across 47 entities (kappa=0.93) without re-optimization. Under prospective silent operation, 92 consecutively listed cases yielded 64 evaluable cases (28 excluded a priori: 21 imaging demonstration only, 6 incomplete diagnostic information, 1 prior trial enrollment) across 25 entities with 82.8% concordance (53/64; kappa=0.80): lymphomas 81.6% (31/38), plasma cell dyscrasias 82.4% (14/17), leukemias 88.9% (8/9). Hallucinations occurred in 2 of 664 evaluated cases (0.3%); discordance was dominated by alternative regimen selection (36.4% prospectively) with guideline-consistent alternatives rather than unsafe errors.
Conclusions
Locally hosted open-weight models can deliver auditable, clinically relevant decision support entirely within institutionally controlled hospital infrastructure, with concordance above 80% in all currently tested external and prospective real-world cohorts. The ongoing multicenter silent evaluation extends this to centers that neither developed the system nor contributed to its case memory, and beyond hematology; its structured discordance analysis defines endpoints and guardrails for a large-scale controlled interventional study of AI-integrated tumor board decisions.