“An Introduction to Semantic Segmentation,” a Presentation from Au-Zone Technologies

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/08/an-introduction-to-semantic-segmentation-a-presentation-from-au-zone-technologies/

Sébastien Taylor, Vice President of Research and Development for Au-Zone Technologies, presents the “Introduction to Semantic Segmentation” tutorial at the May 2024 Embedded Vision Summit.

Vision applications often rely on object detectors, which determine the nature and location of objects in a scene. But many vision applications require a different type of visual understanding: semantic segmentation. Semantic segmentation classifies each pixel of an image, associating each pixel with an object class (e.g., pavement, pedestrian). This is required for separating foreground objects from background, for example, or for identifying drivable surfaces for autonomous vehicles. A related type of functionality is object segmentation, which associates each pixel with a specific object (e.g., pedestrian #4), and panoptic segmentation, which combines the functionality of semantic and instance segmentation.

In this talk, Taylor introduces deep learning-based semantic, instance and panoptic segmentation. He explores the network topologies commonly used and how they are trained. He discusses metrics for evaluating segmentation algorithm output and considerations when selecting segmentation algorithms. Finally, he identifies resources useful for developers getting started with segmentation.

1. Introduction to Semantic
Segmentation
Sébastien Taylor
V.P. of Research & Development
Au-Zone Technologies

2. •Introduction to segmentation
•Practical examples and applications
•Various types of segmentation
•Accuracy metrics
•Computational requirements
•Resources
Outline
2© 2024 Au-Zone Technologies

3. •Image Segmentation is a process that subdivides
an image into its constituent parts or objects.
•Key task in computer vision and image
processing
•It can be formulated as a pixel classification
problem with three different approaches
(semantic, instanceand panoptic)
Introduction to Segmentation
3© 2024 Au-Zone Technologies

4. Image Segmentation vs. Object Detection
4© 2024 Au-Zone Technologies

5. •Autonomous vehicles
•Smart agriculture
•Drones and aerial imaging
•Medical image diagnosis
•Image editing
•Dataset augmentation
Practical Examples
5© 2024 Au-Zone Technologies

6. •Semantic segmentation:produces a contextual description of the “stuff” in the image. Classes are isolated but not
objects within the same class. We don’t have access to a single object.
•Instance segmentation: produces a better description that can list objects as individual instances of “things” but
lower generalization on the environment and background “stuff”.
•Panoptic segmentation: Combines semantic and instance segmentation. We have access to the environmental
context but also to the individual objects. So, we see both “stuff” and “things”.
Instance, Semantic and Panoptic Segmentation
6
Image Semantic Segmentation Instance Segmentation Panoptic Segmentation
© 2024 Au-Zone Technologies

7. Image Segmentation Using Deep Learning
7
•Deconstruction: Feature extraction (backbone, encoder)
•Reconstruction: Upsampler(decoder)
Deconstruction Reconstruction
© 2024 Au-Zone Technologies

8. Deep Learning Segmentation Architecture
8© 2023 Au-Zone Technologies
Encoder Decoder

9. •1-hot encoding, just like classification
•Score applied to each pixel
•Class with highest score sets the pixel
Semantic Segmentation Output
9 © 2024 Au-Zone Technologies

10. Instance Segmentation –Naïve
10
Detections
•Additional model output for
computing bounding boxes
•Same as SSD, YOLO, etc…
•Boxes are post-processed to re-
colourmasks in order to
distinguish instances.
•Overlapping instances will be
poorly segmented because of box
limitations.
© 2024 Au-Zone Technologies

11. •Additional model output computes
per-instance mask predictions.
•Learns to separate objects in each
mask which are then fused with
semantic mask.
•Handles overlapping instances.
Instance Segmentation –Proto Masks
11© 2024 Au-Zone Technologies

12. •Extension of detection models. Inherently instance based.
•Instead of predicting boxes for objects, the model predicts masks.
Instance Segmentation –Box Masks
12© 2024 Au-Zone Technologies

13. Fusing semantic and instance segmentation to detect “things”and “stuff”
Panoptic Segmentation
13© 2024 Au-Zone Technologies

14. •Label masks
•Object polygons
•Very high annotation effort
•“Segment Anything Model” has been
a game changer for annotation effort
Dataset Types
14© 2024 Au-Zone Technologies

15. •Similar IoUconcept as detection
•Panoptic Quality “PQ” is a new
metric and applied, in part, to all
segmentation challenges
•PQ metrics for “things” and “stuffs”
categories
•COCO metrics “Panoptic Evaluation”
Accuracy Metrics
15© 2024 Au-Zone Technologies

16. •Same backbone as detection
•Segmentation head incurs ~20% overhead
•Post-processing demands
•Instance and panoptic incur additional overhead
Computational Requirements
16© 2024 Au-Zone Technologies

17. •Semantic segmentation is a technique that enables us to isolate different
objects in an image along their contours.
•Improves on detection models for objects with more complex shapes.
•It can be considered an image classification task at a pixel level.
Conclusions
17© 2024 Au-Zone Technologies

18. •Semantic segmentation classifies all pixels in an image by their class.
•Instance segmentation refines the semantic masks to separate each
object instance.
•Panoptic segmentation fuses semantic and instance segmentation into a
single unified model with knowledge of “things” and “stuff”.
Conclusions
18© 2024 Au-Zone Technologies

19. •Datasets
•https://cocodataset.org/
•https://www.cityscapes-dataset.com/
•https://ai.facebook.com/datasets/segment-anything/
•Models
•https://towardsdatascience.com/u-net-explained-understanding-its-image-segmentation-
architecture-56e4842e313a
•https://learnopencv.com/yolov5-instance-segmentation/
•https://segment-anything.com/
Resources
19© 2024 Au-Zone Technologies

20. 20
Thank you! Questions?