Multimodal Federated Learning for Robust Lung Cancer Prognosis

Lung cancer is a leading cause of cancer-related deaths, yet accurate prognosis remains a challenge due to data limitations, privacy concerns, and institutional data silos. This presentation explores how federated learning—a decentralized machine learning approach—combined with multi-modal data integration (EHR and imaging) can enhance predictive models while preserving patient privacy. We discuss the benefits of this approach, key challenges such as data heterogeneity and computational costs, and proposed solutions to improve robustness and generalization. Discover how AI-driven innovations can lead to more personalized and effective cancer prognosis to improve patient outcomes.

1. Multimodal Federated Learning for Robust Lung Cancer
Prognosis

AI & Data Science Group,
Department of Bioinformatics,
Fraunhofer Institute for Algorithms and Scientific Computing (SCAI)

February 19
th
, 2025
Yusuf Brima

2. Outline
●Introduction
●Background and Context
●Deep Learning in Diagnosis, Prognosis, and Treatment Monitoring
●Federated Learning Overview
●Unimodal Learning
●Multimodal Learning
●Combining Federated Learning with Multimodal Data Integration
●Prominent Challenges and Proposed Solutions
●Key Takeaways
2

3. Introduction


3
Lung Cancer Facts

●Lung cancer accounts for approximately 1.8
million deaths annually
2
●Cancer is a leading cause of death
worldwide, accounting for nearly 10 million
deaths in 2020
2
●Approximately 5.7 percent of men and women
will be diagnosed with lung and bronchus
cancer at some point during their lifetime
1
Source
[2] World Health Organization (WHO), "Lung cancer," https://www.who.int/news-room/fact-sheets/detail/cancer

4. Introduction


4

●Significance: Lung cancer is one of the leading
causes of cancer-related deaths worldwide
1
.
●Opportunity: The use of Deep Learning (DL)
for early detection/diagnosis/prognosis.
●Challenge: Limited (annotated) data
availability, trust, and privacy concerns hinder
its applications.
●Objective: Explore how federated learning
combined with multimodal data (EHR and
imaging) can enhance prognosis accuracy
while preserving privacy.
Source
[1] National Cancer Institute (NCI), "Cancer Stat Facts: Lung and Bronchus Cancer," https://seer.cancer.gov/statfacts/html/lungb.html

5. Contextualizing Deep Learning in Diagnosis, Prognosis, and
Treatment Monitoring


●Diagnosis:
○Definition: The identification of a health condition or disease via examination.
○Goal: Detect the presence of a disease.
○Example: Identifying a lung tumor from imaging (e.g., chest X-ray or CT scans).
●Prognosis:
○Definition: Predicting the future course or outcome of a disease.
○Goal: Forecast the disease progression and long-term outlook.
○Example: Predicting 5-year survival rates for lung cancer patients.
●Treatment Monitoring:
○Definition: Assessing the effectiveness of a treatment over time.
○Goal: Track patient response and adjust treatment as needed.
○Example: Monitoring tumor shrinkage during chemotherapy.
5

6. Motivation for Lung Cancer Prognosis




●Supports personalized treatment strategies.
●Recent works, such as
3,4
, have shown that multi-modal representation
learning can significantly improve survival prediction in lung cancer
by combining structured gnomic data with imaging (CT and/or PET)
data.
●Current Limitations:
○Data silos across institutions impede large-scale analysis for robust and
generalizable predictive models.
6
[3] Farooq, A., Mishra, D. and Chaudhury, S., 2024. Survival Prediction in Lung Cancer through Multi-Modal Representation Learning. arXiv preprint arXiv:2409.20179.
[4] Z. Ning et al., "Multi-Constraint Latent Representation Learning for Prognosis Analysis Using Multi-Modal Data," in IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 7, pp.
3737-3750, July 2023, doi: 10.1109/TNNLS.2021.3112194.

7. Federated Learning Overview


●Definition: A decentralized ML
approach where data remains
local, and only model updates
are shared.
●Benefits in Healthcare:
○Preserves patient privacy.
○Leads to more robust ML systems
○Enables inter-institutional
collaboration.
○Key Mechanism: Federated
averaging (e.g., FedAvg).
7

8. Unimodal Representation Learning




●EHR-Only Model: Predict outcomes
based on structured clinical data.
○Strength: Often low-dimensional, thus less
computationally intensive and models if
traditional are interpretable and explainable.
○Limitation: Misses spatial information from
imaging.
●Imaging-Only Model: Leverage CT
scans for feature extraction.
○Strength: Visual representation of tumors.
○Limitation: Lack of contextual patient
information, computational intensive and
often less interpretable/explainable.
8

9. Multimodal Representation Learning




●Definition: Combining data
from multiple modalities to
enhance performance.
●Methods:
○Feature-level fusion (e.g.,
concatenation).
○Decision-level fusion (e.g.,
ensemble models).
●Advantages:
○Richer representation (e.g., more
holistic view of a patient state).
○Improved generalization across
diverse cases.
9

10. Combining Federated Learning with Multimodal Data Integration






●Framework:
○Local multimodal training on structured and imaging data at
participating institutions.
○Federated aggregation of model weights.
●Key benefits:
○Preserves data privacy.
○Leverages complementary strengths of different modalities to enable
better robustness and generalization.
10

11. Key Challenges and Proposed Solutions

●Challenges:
○Limited and imbalanced datasets.
○Data heterogeneity across institutions.
○Communication overhead for federated learning and
computational cost for multimodal learning.
●Proposed Solutions:
○Data augmentation, class weighting, regularization and other
training recipes.
○Efficient model updates using sparsification (e.g., quantization,
pruning, distillation, etc).
11

12. Key Takeaways




●Federated learning enables privacy-preserving model development.
●Multimodal integration enhances the robustness and generalization of
lung cancer prognosis models.
●Addressing challenges such as data scarcity and heterogeneity is crucial
for success.
●Addressing the highlighted challenges and combining these two
techniques could enable more personalized prognosis of cancer to help
improve patient outcomes.
12

13. Thank you!
13