DeepLab V3+: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

A presentation introducting DeepLab V3+, the state-of-the-art architecture for semantic segmentation. It also includes detailed descriptions of how 2D multi-channel convolutions function, as well as giving a detailed explanation of depth-wise separable convolutions.

1. DeepLabV3+ Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

2. Background DeepLabV3+ is the latest version of the DeepLab models. DeepLab V1 : Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. ICLR 2015. DeepLab V2 : DeepLab : Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. TPAMI 2017. DeepLab V3 : Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv 2017. DeepLab V3+ : Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv 2018.

3. Semantic Segmentation Classifying all pixels in an image into classes. Classification at the pixel level. Does not have to separate different instances of the same class. Has i mportant applications in Medical Imaging.

4. Current Results on Pascal VOC 2012

5. Motivation and Key Concepts Use Atrous Convolution and Separable Convolutions to reduce computation. Combine Atrous Spatial Pyramid Pooling Modules and Encoder-Decoder Structures. ASPPs capture contextual information at multiple scales by pooling features at different resolutions. Encoder-Decoders can obtain sharp object boundaries.

6. Architecture Overview

7. Advanced Convolutions

8. Convolution (Cross-Correlation) for 1 Channel Convolution with Zero-Padding Display with Convolution Kernel Blue maps: inputs, Cyan maps: outputs, Kernel: not displayed

9. Other Convolutions (Cross-Correlations) Strided Convolution with Padding Atrous (Dilated) Convolution with r=2 Blue maps: inputs, Cyan maps: outputs, Kernel: not displayed

10. Atrous Convolution à trous is French for “with holes” Atrous Convolution is also known as Dilated Convolution. Atrous Convolution with r=1 is the same as ordinary Convolution The image on the left shows 1D atrous convolution

11. Receptive Field of Atrous Convolutions Left: r=1, Middle: r=2, Right: r=4 Atrous Convolution has a larger receptive field than normal convolution with the same number of parameters.

12. Depth-wise Separable Convolution A special case of Grouped Convolution. Separate the convolution operation along the depth (channel) dimension. It can refer to both (depth -> point) and (point -> depth). It only has meaning in multi-channel convolutions (cross-correlations).

13. Review: Multi-Channel 2D Convolution

14. Exact Shapes and Terminology Filter : A collection of Kernels of shape concatenated channel-wise. Input Tensor Shape: Filters are 3D, Kernels are 2D, All filters are concatenated to a single 4D array in 2D CNNs. ( , , , , , , )  

15. Step 1: Convolution on Input Tensor Channels

16. Step 2: Summation along Input Channel Dimension

17. Step 3: Add Bias Term Each kernel of a filter iterates only 1 channel of the input tensor. The number of filters is . Each filter generates one output channel. Each 2D kernel is different from all other kernels in the 3D filter.   Key Points

18. Normal Convolution Top: Input Tensor Middle: Filter Bottom: Output Tensor

19. Depth-wise Separable Convolution Replace Step 2. Instead of summation, use point-wise convolution (1x1 convolution). There is now only one filter. The number of 1x1 filters is . Bias is usually included only at the end of both convolution operations. Usually refers to depth-wise convolution -> point-wise convolution. Xception uses point-wise convolution -> depth-wise convolution.  

20. Depth-wise Separable Convolution

21. Characteristics Depth-wise Separable Convolution can be used as a drop-in replacement for ordinary convolution in DCNNs. The number of parameters is reduced significantly (sparse representation). The number of flops is reduced by several orders of magnitude (computationally efficient). There is no significant drop in performance (performance may even improve). Wall-clock time reduction is less dramatic due to GPU memory access patterns.

22. Example: Flop Comparison (Padding O, Bias X) Ordinary Convolution For a 256x256x3 image with 128 filters with kernel size of 3x3, the number of flops would be There is an 8-fold reduction in the number of flops.   Depth-wise Separable Convolution Left : Depth Conv, Right : Point Conv For a 256x256x3 image with 128 filters and a 3x3 kernel size, the number of flops would be  

23. Example: Parameter Comparison (Excluding Bias Term) Ordinary Convolution For a 256x256x3 image with 128 filters and 3x3 kernel size, the number of weights would be   Depth-wise Separable Convolution For a 256x256x3 image with 128 filters and 3x3 kernel size, the number of flops would be There is also an 8-fold reduction in parameter numbers.  

24. Atrous Depth-wise Separable Convolution

25. Architecture Overview

26. Encoder-Decoder Structures The Encoder reduces the spatial sizes of feature maps, while extracting higher-level semantic information. The Decoder gradually recovers the spatial information. UNETs are a classical example of encoder-decoder structures. In DeepLabV3+, DeepLabV3 is used as the encoder.

27. Architecture Overview

28. Decoder Layer Structure Apply 4-fold bilinear up-sampling on the ASPP outputs. Apply 1x1 Convolution with reduced filter number on a intermediate feature layer. Concatenate ASPP outputs with intermediate features. Apply two 3x3 Convolutions. Apply 4-fold bilinear up-sampling. Purpose & Implementation The ASPP is poor at capturing fine details. The decoder is used to improve the resolution of the image. The intermediate layer has 1x1 convolutions to reduce channel number.

29. ASPP: Atrous Spatial Pyramid Pooling

30. The ASPP Layer Encodes multi-scale contextual information through multiple rates. Concatenate all extracted features and an up-sampled global average pooling layer channel-wise. Use Atrous Depth-wise separable convolutions for multiple channels. Bad at capturing sharp object boundaries.

31. Modified Aligned Xception Network Xception : Extreme Inception Network. Backbone network for DeepLabV3+ Uses residual blocks and separable convolutions.

32. Explanation of Xception Takes the “ Inception Hypothesis ”, which states that cross-channel correlations and spatial correlations are sufficiently decoupled that it is preferable not to map them jointly, to the extreme. The extensive use of separable convolutions and atrous convolutions allows the model to fit in GPU memory despite the huge number of layers. Originally applied point-wise convolution before depth-wise convolution. Invented by François Chollet .

33. Architecture Review

34. The End