Similarity Maps Using SSIM Index

Using Similarity Maps in Visual Comparisons. In this presentation we used the SSIM metric to build maps for visual comparisons.

1. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Ten Minute Speech :: Overview of Activities Developed in Disciplines and Guided Studies :: Laboratory Seminars and Meetings
Federal University of Rio de Janeiro (UFRJ) :: Campus Cidade Universitária :: Rio de Janeiro :: Fundão Island :: COPPE/PESC/LCG :: March/2015
Using Similarity Maps in
Visual Comparisons
Michel Alves dos Santos
Ten Minute Speech
(March, 2015)
Advisors: D.Sc. Ricardo Marroquim & Ph.D. Claúdio Esperança

2. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
IntroductionIntroduction
How to measure the visual
similarity between images?
[Lima'15][Nasc'08]
...Using IQA Metrics...
Introduction
-1-1
+1+1
+1+1
-1-1
NCC
SSIM

3. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
MetricsMetrics
What metric should I use?
MSE?PSNR?
<<No!!! Avoid ambiguous metrics!>>
Metrics
Impulsive Noise, MSE = 225
"A Universal Image Quality Index", Wang & Bovik, 2002, IEEE Signal Processing Letters.
Gaussian Bluring, MSE = 225 Contrast Stretching, MSE = 225
MAE?

4. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Today MetricsToday Metrics
Today we have a hundred of
metrics and several surveys!
We will compare the performance of two...
Each of these metrics have specific use...
Metrics
Metrics: NCC and SSIM...

5. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Range:
NCC vs SSIMNCC vs SSIM
Normalized Cross-Correlation:
Structural Similarity Index: [Wang'04]
In our comparison we will assemble similarity maps with these metrics...
NCCxSSIM
Range:

6. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
NCC – Normalized Cross-CorrelationNCC – Normalized Cross-Correlation
covariance
stdev
mean mean
stdev
NCC

7. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
NCC – Normalized Cross-CorrelationNCC – Normalized Cross-Correlation
Homogeneous Patches
...Numerical Instability...
Suggestion: add a small ε
in the denominator...
NCC

8. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
SSIM – Structural Similarity IndexSSIM – Structural Similarity Index
bits per pixel
dynamic range of pixels
suggestion
suggestion
SSIM

9. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
SSIM – Structural Similarity IndexSSIM – Structural Similarity Index
Homogeneous Patches
...Constants Avoid Numerical Instability...
SSIM

10. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Montage of MapMontage of Map
3x3 sub in 5x5 window 7 pixels in 5x5 window 9 pixels in 5x5 window
Centered Window
Directional Window
3x3 sub in 5x5 window
Tsuji­Tomita Nagao7 Nagao9 NagMod
(compute a mean neighbors value around a central pixel)
(target pixel value)(types of centralized neighborhood)
Montage

11. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
ComparisonsComparisons
Cartoon Lighting Color Harmonization
Image-Based Reconstruction
Comparisons

12. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
NCC Map[Lima'15][Nasc'08]
*NCC Map with 3x3 window (centered neighborhood)
Comparisons – Gingerbread - NCCComparisons – Gingerbread - NCC
-1-1 +1+1
*Map obtained using YIQ luminance.
Comparisons

13. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
SSIM Map[Lima'15][Nasc'08]
*SSIM Map with 3x3 window (centered neighborhood)
Comparisons – Gingerbread - SSIMComparisons – Gingerbread - SSIM
-1-1 +1+1
*Map obtained using YIQ luminance.
Comparisons

14. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
NCC MapSSIM MapOverlay
-1-1 +1+1-1-1 +1+1
*NCC and SSIM Maps with 3x3 window (centered neighborhood)
Comparisons – Gingerbread - MapsComparisons – Gingerbread - Maps
-1-1 +1+1
Comparisons

15. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Purple Flower - NCCComparisons – Purple Flower - NCC
NCC Map[Lima'15][Nasc'08]
-1-1 +1+1
Observe the behavior of this index in
homogeneous areas...
*NCC Map with 3x3 window (centered neighborhood) *Map obtained using YIQ luminance.
Comparisons

16. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Purple Flower - SSIMComparisons – Purple Flower - SSIM
SSIM Map[Lima'15][Nasc'08]
-1-1 +1+1
Observe the behavior of this index in
homogeneous areas...
*SSIM Map with 3x3 window (centered neighborhood) *Map obtained using YIQ luminance.
Comparisons

17. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Purple Flower - MapsComparisons – Purple Flower - Maps
Overlay[Lima'15][Nasc'08]
N
C
C

M
a
p
S
S
I
M

M
a
p
-1-1
+1+1
-1-1
+1+1
Comparisons

18. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Color HarmonizationComparisons – Color Harmonization
-1-1
+1+1
-1-1
+1+1
I
n
p
u
t
H
a
r
m
o
n
i
z
e
d
*Maps obtained using average luminance.[Color Harmonization Project'14]
Comparisons
N
C
C

M
a
p
S
S
I
M

M
a
p

19. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Color HarmonizationComparisons – Color Harmonization
-1-1
+1+1
-1-1
+1+1
I
n
p
u
t
H
a
r
m
o
n
i
z
e
d
*Maps obtained using YIQ luminance.[Color Harmonization Project'14]
Comparisons
N
C
C

M
a
p
S
S
I
M

M
a
p

20. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Image-Based ReconstructionComparisons – Image-Based Reconstruction
-1-1
+1+1
-1-1
+1+1
N
C
C

M
a
p
S
S
I
M

M
a
p
[Cout'15]
P
h
o
t
o
R
e
n
d
e
r
*Maps obtained using YIQ luminance.
*Dataset: Nana.
Comparisons

21. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Image-Based ReconstructionComparisons – Image-Based Reconstruction
-1-1
+1+1
-1-1
+1+1
N
C
C

M
a
p
S
S
I
M

M
a
p
[Cout'15]
P
h
o
t
o
R
e
n
d
e
r
*Back vision of the previous model.
*Maps obtained using YIQ luminance.
Comparisons

22. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Image-Based ReconstructionComparisons – Image-Based Reconstruction
-1-1
+1+1
-1-1
+1+1
N
C
C

M
a
p
S
S
I
M

M
a
p
[Cout'15]
P
h
o
t
o
R
e
n
d
e
r
*Maps obtained using YIQ luminance.
*Dataset: Buddha.
Comparisons

23. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
Comparisons – Image-Based ReconstructionComparisons – Image-Based Reconstruction
-1-1
+1+1
-1-1
+1+1
N
C
C

M
a
p
S
S
I
M

M
a
p
[Cout'15]
P
h
o
t
o
R
e
n
d
e
r
*Maps obtained using YIQ luminance.
*Dataset: Gertrud.
Comparisons

24. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Introduction ConclusionComparisonsMetrics MontageNCCxSSIM NCC SSIM
ConclusionsConclusions
NCC:
SSIM:
fast and simple...
has numerical instability control...
is not so simple...
Advice: Use appropriate metric ever!
excellent perception of structural change...
not designed for images only...
good perception of local variation...
needs revaluation in pathological cases...
presents numerical instability...
Conclusion

25. Michel Alves – Laboratory of Computer Graphics (LCG) – Graduate Program in Systems Engineering and Computing – UFRJ/COPPE/PESC
Ten Minute Speech :: Overview of Activities Developed in Disciplines and Guided Studies :: Laboratory Seminars and Meetings
Federal University of Rio de Janeiro (UFRJ) :: Campus Cidade Universitária :: Rio de Janeiro :: Fundão Island :: COPPE/PESC/LCG :: March/2015
Thanks for your attention!
References:
[Nasc'08]: Representation and Ilumination of 2D Objects, 2008,
Renata Nascimento et al.
[Lima'15]: Local lighting cartoons method using tree regions,
2015, Ailton Lima.
[Cout'15]: Per Vertex BRDF Acquisition, 2015, Daniel Coutinho.
[CHPr'14]: Color Harmonization Project, 2014, Michel Alves.
[Wang'04]: Image quality assessment: From error visibility to
structural similarity, 2004, Zhou Wang et al.
Michel Alves dos Santos