[20240603_LabSeminar_Huy]TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking.pptx

TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking

1. Quang-Huy Tran Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: [email protected] 2024-06-03 TransMOT : Spatial-Temporal Graph Transformer for Multiple Object Tracking Peng Chu et al. ICCV-23: Proceedings of the IEEE/CVF Winter Conference on applications of computer vision

2. OUTLINE MOTIVATION INTRODUCTION METHODOLOGY EXPERIMENT & RESULT CONCLUSION

3. MOTIVATION Multiple object tracking : Input: Image sequence from video. MOT tracker: Feature representation, Affinity Estimation, Optimization for Association . Output: Tr aj ector ies of all targets across the whole video. Overview

4. MOTIVATION Traditional methods use strict threshold to track only most confident detection candidates Overview Our method leverages low confidence detection candidates for robust tracking in occluded scenes.

5. INTRODUCTION propose a novel spatial-temporal graph Transformer for MOT ( TransMOT ) objects are arranged as a temporal series of sparse weighted graphs that are constructed using their spatial relationships within each frame. handles a large and varied number of tracked targets and detection candidates during tracking Using the graph representation, TransMOT encodes the features and spatial-temporal relationships for all tracked targets through its Spatial-Temporal Graph Transformer Encode TransMOT can learn the spatial-temporal clues for association and directly generate the assignment matrix for MOT. By relying on the discriminative spatial-temporal clues and capability of modeling many candidates, TransMOT can associate candidates from a large number of loosely filtered detection predictions.

6. METHODOLOGY Main Architecture - TransMOT

7. METHODOLOGY Problem Setting At the t- th frame, TransMOT maintains a set of tracklets. tracklet maintains a set of states: past locations and appearance features on the previous T image frames. Given a new image frame : determines whether any tracked objects are occluded, computes new locations for the existing tracklets ,and generates new tracklets for new objects that enter the scene. At each frame, the detection and feature extraction sub-networks generate Mt candidate object detection proposals .  

8. METHODOLOGY Spatial-Temporal Graph Transformer Encoder Spatial Graph Transformer Encoder Input: the states of the tracklets for the past T frames. Adjacency matrix is defined by Intersection over Union( IoU ).

9. METHODOLOGY Spatial-Temporal Graph Transformer Encoder Node features are embedded through a source embedding layer (a linear layer) independently for each node. Then passed into the spatial graph transformer encoder layer together. self-attention weights for the i-th head: Graph multi-head attention weighted feature tensor. Temporal Graph Transformer Encoder transposes the first two dimensions of SGTE and apply standard transformer model in temporal dimension.

10. METHODOLOGY Spatial Graph Transformer Decoder produces extended assignment matrix from candidate graph and STGTE. candidate graph same as graph of tracklets .   Apply graph multi-head attention into candidate graph Add virtual node from STGTE handle the candidates that either is false positives or need to initiate a new tracklet in the current frame t to form an extended tracklet embedding . Feed-forward layer and a normalization layer to generate output.

11. METHODOLOGY Training and Tracking Framework Relax the constraints in finding assignment adjacency matrix for efficient computation: a tracklet is always associated with a detection candidate or a virtual source. a row of the assignment matrix can be treated as a probability distribution over a total of categories.   Cross-entropy loss:

12. EXPERIMENT AND RESULT EXPERIMENT SETTINGs Measurement: standard ID score metrics: measure the long-term ID consistency and compare whole trajectories with ground truth for ID precision (IDP), ID recall (IDR), and their IDF1 scores. multiple object tracking accuracy (MOTA): bounding box false positives(FP), false negatives (FN) and the identity switches (IDS). The percentage of mostly tracked targets (MT) and the percentage of mostly lost targets (ML). Four standard MOT challenge datasets for pedestrian tracking: MOT15, MOT16, MOT17, and MOT20.

13. EXPERIMENT AND RESULT RESULT – Overall Perfor mance

14. EXPERIMENT AND RESULT RESULT – Visualization

15. CONCLUSION proposed a novel spatial-temporal graph Transformer for Multi-Object Tracking ( TransMOT ). By formulating the tracklets and candidate detections as a series of weighted graphs, the spatial and temporal relationships of the tracklets and candidates are explicitly modeled and leveraged. achieves higher tracking accuracy, but also is more computationally efficient than the transitional Transformer-based methods.