[20240729_LabSeminar_Huy]Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction.pptx

Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction

1. Quang-Huy Tran Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: [email protected] 2024-07- 2 9 Spatio -Temporal Self-Supervised Learning for Traffic Flow Prediction Jiahao Ji et al. AAAI -2023 : The Thirty-Seventh AAAI Conference on Artificial Intelligence

2. OUTLINE MOTIVATION METHODOLOGY EXPERIMENT & RESULT CONCLUSION

3. MOTIVATION Robust traffic flow prediction across different spatial regions at different time periods is crucial for advancing intelligent transportation systems : not only enable effective traffic controls, but also mitigate tragedies caused by peak traffic flow . Overview and Limitation Challenges: Lack of modeling spatial heterogeneity exhibited with skewed traffic distributions across different regions. ignore spatial heterogeneity and easily biased. High computational and memory cost. Infeasible to handle large-scale traffic data. Existing modeled temporal dynamics with a shared parameter space for all time periods. Hard to preserve the temporal heterogeneity in the latent embedding space.

4. INTRODUCTION Propose an adaptive heterogeneity-aware data augmentation scheme over the graph-structured spatial temporal graph against the noise perturbation . Two self-supervised learning tasks are incorporated to supply the main task Enforcing the model discrimination ability with the awareness of both spatial and temporal traffic heterogeneity. Contribution P resent a novel Spatio -Temporal Self Supervised Learning framework (ST-SSL) for predicting traffic flow : First to propose a novel self-supervised learning . May be applied to other applications: air quality prediction .

5. METHODOLOGY Problem Definition Spatial region, partition a city into disjoint grids to denote set . A traffic flow graph (TFG) : set of spatial regions (nodes) with size . : set of edge connecting two adjacent regions. : adjacency matrix. : traffic inflow and outflow data over previous T time steps. traffic volume of all regions V at time slot t- th : .   Problem: given historical traffic flow graph G till current time step aim to learn a predictive function to estimate accurately traffic volume of all regions at the future time step t + 1.

6. METHODOLOGY Main Architecture a) Overall architecture of ST-SSL. b) Spatial heterogeneity modeling. c) Temporal heterogeneity modeling.

7. METHODOLOGY Spatio -Temporal Encoder Jointly preserve the ST contextual information over the traffic flow graph jointly model the sequential patterns of traffic data across different time steps and the geographical correlations among spatial regions . Use as backbone for spatial temporal relational representation . where represents region embedding matrix at the time step t, D is embedding dimension. is length of the output embedding sequence.   Temporal traffic pattern (TC) is 1-D causal convolution with gated mechanism. Spatial convolution (SC) encoder is message passing mechanism. ST encoder is built with a “sandwich” block structure: TC-SC-TC. Obtain a sequence of embedding matrix with the temporal dimension of T’. After propagation and aggregation, we generate the final embedding matrix .  

8. METHODOLOGY Adaptive Graph Augmentation on TFG Devise two phases of graph augmentation Traffic-level data augmentation and graph topology-level structure augmentation . Adaptively learn heterogeneity-aware region dependencies in terms of their traffic regularities. where is the aggregated representation based on aggregation weight , is index of time step range (t-T, t).   Region-wise Heterogeneity Measurement: For region , its embedding sequence within T time steps from has   Propose to estimate the heterogeneity degree between two region, reflect traffic distribution difference over time:

9. METHODOLOGY Adaptive Graph Augmentation on TFG Heterogeneity-guided Data Augmentation : Traffic-level Au gmentation: L earned time-aware traffic pattern dependencies of each region adaptively. Mask less relevant traffic volume at - th time step of region against noise perturbation based on Bernoulli distribution . The augmented data: Graph Topology-level Augmentation : Perform the topology-level augmentation over the region TFG . not only debias region connections with low inter-correlated traffic patterns, but also capture the long-range region dependencies with the global urban context. For adjacent regions and , their connection edge are masked based on Bernoulli distribution in low heterogeneity degree . For 2 non-adjacent regions, the low heterogeneity degree will result in adding an edge between and based on the masking probability from Bernoulli. Augmented TFG: .  

10. METHODOLOGY SSL for Spatial Heterogeneity Modeling Self-supervised objective over all regions: where is temperature parameter to control the smoothing degree of softmax output. is predicted assignment score of .   Aim to preserve the spatial heterogeneity with auxiliary self-supervised signals. Design a soft clustering-based self-supervised learning (SSL) task over regions. G enerate K cluster embeddings as latent factors for region clustering. represents estimated relevance score between embedding region and embedding of k- th cluster. Cluster assignment of region is generated with . The self-supervised augmented task is optimized:  

11. METHODOLOGY SSL for Spatial Heterogeneity Modeling For any assignment, map embedding matrix into cluster matrix . Thus, search for optimal solution by maximizing the similarity between the embeddings and the clusters   where tr(·) is trace operator that sums elements on main diagonal of a square matrix, is the entropy function and is a parameter that controls smoothness of assignment.   Distribution Regularization for Region Clustering: Generate the cluster assignment matrix as self-supervised signals for generative data augmentation. However, no guarantee that each region’s cluster assignment sums up to 1. To avoid the trivial solution, employ the principle of maximum entropy. Hence, define a feasible solution set:  

12. METHODOLOGY SSL for Temporal Heterogeneity Modeling Design to inject the temporal heterogeneity into time-aware region embeddings : Enforcing the divergence among time step-specific traffic pattern representations. Fuse the encoded time-aware region embeddings from both the original and augmented TFGs: where is element-wise product. are learnable parameters.   Generate city-level representation at time step t through aggregating embeddings of all regions:   To enhance representation discrimination ability among different time steps: Treat the region-level and city-level embeddings from the same time step as the positive pairs, embeddings from different time steps as negative pairs.   where t and t′ denote two different time steps. is learnable transformation matrix.   where is sigmoid function.  

13. METHODOLOGY Model Training For prediction in learning: F eed the embedding of each region into an MLP.   Incorporating the self-supervised spatial and temporal heterogeneity modeling: where , are ground truth of inflow and outflow. is parameter to balance the influence of each type of traffic flow   Loss function: minimize overall loss

14. EXPERIMENT AND RESULT EXPERIMENT SETTINGs Dataset: NYCBike1, NYCBike2, NYCTaxi , BJTaxi . Baselines: Statistical-based methods: ARIMA, SVR. Deep Learning: ST- ResNet [1]. STGNN: STGCN[2], GMAN[3], AGCRN[4], STSGCN[5], and STFGNN[6]. [1] Zhang, J., Zheng, Y., & Qi, D. (2017, February). Deep spatio -temporal residual networks for citywide crowd flows prediction. In Proceedings of the AAAI conference on artificial intelligence (Vol. 31, No. 1). [2] Yu, B., Yin, H., & Zhu, Z. (2017). Spatio -temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875. [3] Zheng, C., Fan, X., Wang, C., & Qi, J. (2020, April). Gman : A graph multi-attention network for traffic prediction. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 01, pp. 1234-1241). [4] Bai, L., Yao, L., Li, C., Wang, X., & Wang, C. (2020). Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural information processing systems, 33, 17804-17815. [5] Song, C., Lin, Y., Guo, S., & Wan, H. (2020, April). Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 01, pp. 914-921). [6] Li, M., & Zhu, Z. (2021, May). Spatial-temporal fusion graph neural networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 5, pp. 4189-4196). Measurement : Mean absolute error (MAE) and Mean Average Percentage Error (MAPE).

15. EXPERIMENT AND RESULT RESULT – Overall Perfor mance

16. EXPERIMENT AND RESULT RESULT – Visualization

17. CONCLUSION Proposed a novel spatio -temporal self-supervised learning(ST-SSL) framework: I ntegrated temporal and spatial convolutions to encode spatial-temporal traffic patterns . Devised a spatial self-supervised learning paradigm that consists of an adaptive graph augmentation and a clustering-based generative task. a temporal self-supervised learning paradigm that relies on a time-aware contrastive task, to supplement the main traffic flow prediction task with spatial and temporal heterogeneity aware self-supervised signals. Summarization