To construct a high-accuracy, interpretable prognostic prediction model for breast cancer by integrating whole slide images (WSIs) and corresponding pathology report text from patients in the TCGA-BRCA public dataset, thereby providing a reliable auxiliary decision-making tool for clinical practice.

This study utilized The Cancer Genome Atlas Breast Cancer dataset (TCGA-BRCA). A total of 702 samples with complete WSIs, matched pathology reports, and survival data were obtained through stringent filtering. To handle the high resolution of WSIs, morphological methods were employed to segment tissue regions and partition them into image patches, constructing multi-instance feature bags. Pathology reports were parsed and structured using a large language model (LLM) to extract key semantic features. For model construction, a Transformer-based multimodal survival analysis framework was proposed: the image branch aggregated global and local features via improved multi-instance learning with an attention mechanism; the text branch encoded pathology reports into semantic embeddings. Deep fusion of textual and imaging features was achieved through a cross-attention mechanism. Finally, the output from the fully connected layer is a patient-level survival risk score, and the model is trained end-to-end using negative log-likelihood loss based on discrete-time survival analysis.

This study first developed a unimodal survival prediction model based solely on digital pathology whole-slide images (WSIs), employing a multiple instance learning framework to generate patients' recurrence risk scores and survival probabilities. Through 5-fold cross-validation, the model achieved a C-index of 0.687, significantly outperforming previously reported results in the literature. Further integration of clinical and pathological text information further improved the performance of the multimodal model: the C-index increased to 0.698, and the standard deviation decreased from ±0.046 to ±0.022—a reduction of approximately 50%—indicating a significant enhancement in model stability and reliability.

By developing and validating a novel multimodal fusion model, this study provides a more accurate and reliable solution for survival prediction in breast cancer patients.