Densely Connected Convolutional Networks

A class presentation in soft computing course, June 2018

1. DENSELY
Prof. Mohammad-R. Akbarzadeh-T
Ferdowsi University of Mashhad
A Presentation by:
•Hosein Mohebbi
•M.-SajadAbavisani

2. CVPR 2017 BEST PAPER AWARD
2
Gao Huang
Cornell University
h-index: 12
Zhuang Liu
Tsinghua University
h-index: 5
Laurens van der Maaten
Facebook AI Research
h-index: 29
Kilian Weinberger
Associate Professor
Cornell University
h-index: 41

3. 3

4. Normalized initialization and intermediate normalization layers
The main culprit : Vanishing/exploding gradients
Not caused by overfitting
4

5. 5
RESNET

6. STOCHASTIC DEPTH
Deep network during testing, but shallower network during training.
6
??????
??????=??????�??????????????????
??????�
??????(??????
??????−1+??????�??????
??????−1) ??????
??????∈0,1
They all share a key characteristic:
They create short paths from early layers to later layers

7. DENSE
7
??????(??????+1)
2
direct connections

8. DENSE
8
The growth rate regulates how much new information each
layer contributes to the global state.

9. Traditional Convolutional feed-forward networks :
9
??????
??????=??????
??????(??????
??????−1)
??????
??????=??????
??????(??????
??????−1) + ??????
??????−1
ResNets:
DenseNets:
??????
??????=??????
??????([??????
0,??????
1,…,??????
??????−1])
Where [??????
0,??????
1,…,??????
??????−1] refers to the concatenation of the feature-maps produced
in layers 0….. ??????-1.

10. 10

11. DENSENET
11

12. DENSENET
WITH BOTTLENECK LAYER
12

13. DENSENET
13
Compression in transition layer
Pooling reduces
Feature map sizes
Feature map sizes match
Within each block

14. DENSE

15. 15
Direct access: Deep Supervision with single classifier
Reduces overfitting on tasks with smaller training set sizes

16. 16

17. 17
ResNetconnectivity:
DenseNetconnectivity:
#parameters:
k: Growth rate

18. 18
Standard Connectivity :

19. 19
Feature visualization of convolutional net trained on ImageNetfrom [Zeiler& Fergus 2013]
Remember feature visualization

20. 20
“Collective Knowledge”

21. Feature reuse
Information flow from the first to the last layers of the block
Compression in transition layer
Concentrate on high level feature for final classification
21

23. CIFAR-10
23

24. DENSENET IMAGENET
24

25. IMAGENET
25

26. 26
KaimingHe, et al. "Deep residual learning for image recognition" CVPR
2016
Chen-Yu Lee, et al. "Deeply-supervised nets" AISTATS 2015
Gao Huang, et al. "Deep networks with stochastic depth" ECCV 2016
CS231n: Convolutional Neural Networks for Visual Recognition
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der
Maaten. Densely connected convolutional networks. Conference on
Computer Vision and Pattern Recognition, 2017
Geoff Pleiss, et al. "Memory-Efficient Implementation of DenseNets”,
arXivpreprint arXiv:1707.06990 (2017)