J. Kim, CVPR 2024, MLILAB, KAIST AI.

J.Kim, CVPR 2024, MLILAB, KAIST AI

1. Data-Efficient Unsupervised Interpola6on
Without Any Intermediate Frame for 4D Medical Images
JungEun Kim1*,HangyulYoon1*,GeondoPark1, KyungsuKim2, EunhoYang1,3
1 Korea Advanced Institute of Science and Technology (KAIST)
2 Massachusetts General Hospital and Harvard Medical School 3 AITRICS

2. Introduction
•4D (3D+!) medical images are crucial in clinical practice
•Main task -Interpolating 4D medical images
•Similar to 3D Video Frame Interpolation (VFI) methods
•However, it is not appropriate to extend and applythe 2D natural VFI methods to 3D medical domain
Each frame consists of a 3D volume

3. Introduction
•Why directly extending the 2D natural VFI methods to 3D medical are not appropriate?
1) Lack of high-quanNty intermediate frames
•Limited due to privacy concerns
2) Lack ofhigh-quality intermediate frames
•CT exposes paNents to radiaNon → potenNally increasing secondary cancer risk
•MRI has lengthy scan Nmes (up to 1h) →causing logisNcal and comfort issues
•Ground truth intermediate frames are oOen degraded by several factors.
(e.g., paNent movement, unstable breathing, and difficulty maintaining a stable posiNon during long scans)

4. Motivation
Can we train a VFI model
without depending on any ground truth intermediate frames?

5. Previous Works on 4D Medical VFI
•Relies merely on post-hoc multiplication of the flow calculation model
!! !"
"!→"×$
Start frameEnd frame
Full registration
Linear estimation%!!→$
Estimated frame
for time !∈(0,1)
: Spatial Transform
<Limita'ons>
•PronetospaNaldistorNon→doesnotaccountforthestructuralsmoothness
•PotenNaloverfiUngtothelinearassumpNon

6. Methodology
•Training stage (two-stage process)
•Given two real input images (%!&%")
→Generate virtual intermediate samples, using interpolaNon & extrapolaNon
!"!→#!$%& !"!→#"$%&
2. Create virtual intermediate samples
#=#' #=#(#=0
"!
1. Ini4al frames ("' omi8ed)
#=0

7. Methodology
•Training stage (two-stage process)
•Make candidate images with re-interpolation
•Reconstructthe real images ('%!
#$#&'%"
#$#) based on these virtual candidates
"!
1. Initial frames ("' omitted)
#=0
!"!→#!$%& !"!→#"$%&
2. Create virtual intermediate samples
#=#' #=#(#=0
3. Interpolate to acquire image candidates
#=#' #=#(#=0
!"#!→!)*+,!"#"→!)*+,
!=|!!|
!!+!" !=!"
!!+!"
4. Generate final image using candidate images
!"!
)-)
#=0

8. !"#!→!)*+,!"#!→!)*+,
: Multi-scale Features
Methodology
•Training stage (two-stage process)
•Make candidate imageswith re-interpolation
•Reconstructthe real images ('%!
#$#&'%"
#$#) based on these virtual candidates →Use a cycle-consistency constraint
"!
1. Ini4al frames ("' omi8ed)
#=0
!"!→#!$%& !"!→#"$%&
2. Create virtual intermediate samples
#=#' #=#(#=0
3. Interpolate to acquire image candidates
#=#' #=#(#=0
!"#!→!)*+,!"#"→!)*+,
!=|!!|
!!+!" !=!"
!!+!"
4. Generate final image using candidate images
!"!
)-)
#=0
and featuresand features

9. Methodology
Flow
Calculation
(!
("
)!→"
)"→!
: Spatial Transform
)!→$
)"→$
*(!→$%&'(
*("→$%&'(
×/
×(1−/)
•Inference stage
•Use two authenNc frames %!&%", create two virtual candidate frames for Nme (('%!→&#'()&'%"→&#'())

10. Methodology
Flow
Calculation
(!
("
)!→"
)"→!
: Spatial Transform
*($Reconstruction
)!→$
)"→$
*(!→$%&'(
*("→$%&'(
×/
×(1−/)
Feature Extractor
Multi-scale Warped Feature
4!→#
Feature Extractor
4$→#
•Inference stage
•Use two authentic frames %!&%", create two virtual candidate frames for time (('%!→&#'()&'%"→&#'())
•Generate final prediction ̂%&, using multi-scale warped features & reconstruction module

11. Methodology
Flow
Calculation
(!
("
)!→"
)"→!
: Spatial Transform
*($Reconstruction
)!→$
)"→$
*(!→$%&'(
*("→$%&'(
×/
×(1−/)
Feature Extractor
Multi-scale Warped Feature
4!→#
Feature Extractor
4$→#
•Inference stage
•Use two authentic frames %!&%", create two virtual candidate frames for time (('%!→&#'()&'%"→&#'())
•Generate final prediction ̂%&, using multi-scale warped features & reconstruction module
•(Optional) Instance-Specific Optimization: Can be fine-tuned for each test sample

12. •Cardiac (ACDC dataset)
•Lung (TCIA 4D-Lung dataset)
Results –Quantitative Evaluation

13. •Cardiac (ACDC dataset)
•Lung (TCIA 4D-Lung dataset)
Results –Quan?ta?ve Evalua?on

14. •Our method successfully retains fine-grained details and maintains the structural integrity
Results –Qualita?ve Evalua?on
Cardiac
Lung
Unsupervised baselinesSupervised baselines

15. •Qualitative Video Results -Cardiac
Results –Qualita?ve Evalua?on

16. •Qualitative Video Results -Lung
Results –Qualita?ve Evalua?on

17. Thank You!
Any Questions?