PR12-151 The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Paper review: "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric"
Presented at Tensorflow-KR paper review forum (#PR12) by Taesu Kim
Paper link: https://arxiv.org/abs/1801.03924
Video link: https://youtu.be/VDeJFb5jt5M (in Korean)

1. The Unreasonable Effectiveness of
Deep Features as a Perceptual Metric
R. Zhang et al, UC Berkeley, OpenAI, Adobe
Presented by Taesu Kim
Mar.24,2019

2. Perceptual Similarity
›Which patch (left or right) is closer to the middle patch?
Peak Signal to Noise RatioStructural Similarity

3. Contributions
›A large-scale, highly varied, perceptual similarity dataset,
containing 484k human judgments
–Both parameterized distortion and real algorithm distortion
›Show all typesofdeepfeatures outperforms traditional metrics
›Show network architecture alone doesn’t account for the
performance
›Train new metric (LPIPS) on perceptual judgments
›Improve performance by calibrating feature response from a pre-
trained network.

4. Psychophysical Similarity Measurements
›Two Alternative Forced Choice (2AFC)
–Goal: Collect large-scale set of human perceptual judgements on distortions
–Procedure: Sample apatch.Distortittwice. Ask human which is smaller.
–2judgmentpertrainpatch, 5 judgment per valpatch
›Just Noticeable Differences (JND)
–Goal: validate 2AFC with less “cognitively penetrable” test
–Procedure: ask human if 2 patches are identical or not
–Provide a short training period of 10 pairs, containing 4 ”same” pairs, 1
obviously different pair, 5 different with distortions
–3judgments perpatch

5. Database
Traditional distortion CNN based distortion
Database comparison

6. Database

7. Deep Networks as a Perceptual Metric
›Lin: keep pre-trained network weights and learn linear weights w
›Tune: initialize from a pre-trained classification model and allow all the weights for network to be fine-tuned
›Scratch: Initialize the network from random Gaussian weights and train it on our judgment
Learned Perceptual Image Patch Similarity (LPIPS) metric

8. How well do low-level metrics and classification networks perform?
Does the network have to be trained on classification?
Can we train a metric on traditional and CNN-based distortions?
Does training on traditional and CNN-based distortions transfer to
real-world scenarios?

10. Do metrics correlate across different perceptual task?

11. Where do deep metrics and low-level metrics disagree?
Primary difference is that deep embeddings appear to be more sensitive to blur

12. Conclusions
›Introduce a new dataset of human perceptual similarity judgments
›Evaluate deep features across different architectures and task
›Deep features outperform all previous metrics
›The result is not restricted to VGG, but holds across different deep architectures and
level of supervision
›Perceptual similarity is an emergent property shared across deep visual
representations
›The networkstrainedtosolvechallengingvisualpredictionandmodelingtasksend
uplearningarepresentation of the world that correlates well with perceptual
judgments
›Thestrongerafeaturesetisatclassificationanddetection,thestrongeritisasa
modelofperceptualsimilarityjudgments
›Features that are good at semantic tasks are also good at self-supervised and
unsupervised tasks, and also provide good models of human perceptual behavior

13. Follow us:
Contact us:
[email protected]
For more information:
http://www.neosapience.com