240617_Thuy_Labseminar[Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning​].pptx

Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning​

1. Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning Van Thuy Hoang Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: [email protected] 2024-06-17 NeurIPS 2023

2. Preliminaries: Graph Convolutional Networks (GCNs) Key Idea: Each node aggregates information from its neighborhood to get contextualized node embedding. Limitation: Most GNNs focus on homogeneous graph. Neural Transformation Aggregate neighbor’s information

3. Graph contrastive learning (GCL) methods GCL applies augmentations to generate different views of the original graph, and learn node or graph representations by contrasting positive and negative data pairs. GCL requires domain-specific designs for graph, the design of GCL generally obeys the same paradigm as VCL with three key components: Data augmentations positive pairs for feature alignment negative pairs for feature uniformity

4. Preliminaries: Graph Contrastive Learning Contrastive learning aims to maximize the agreement of latent representations under stochastic data augmentation. Three main components: Data augmentation pipeline Encoder and representation extractor Contrastive objective

5. Preliminaries: Contrastive Learning Objectives Usually implemented with an n-way softmax function: Commonly referred to as the InfoNCE loss. The critic function can be simply implemented as Distinguish a pair of representations from two augmentations of the same sample (positives) apart from (n – 1) pairs of representations from different samples (negatives).

6. Findings: Graph Contrastive Learning (GCL) 1) Positive samples are not a must for GCL 2) Negative sample s are not necessary for graph classification, neither for node classification when adopting specific normalization modules 3) Data augmentations have much less influence on GCL Simple domain-agnostic augmentations (e.g., Gaussian noise) can also attain fairly good performance.

7. How GCL Works without Positive Samples? By maximizing the agreement between positive samples, the neural networks can effectively learn semantic information relevant to downstream tasks It is widely recognized that without this alignment effect, learned representations may lose meaning and incur poor performance For example, optimizing only the uniformity loss significantly degrades performance: Positive Samples Are NOT a Must in GCL

8. How GCL Works without Positive Samples? In node classification: the accuracy gap between the contrastive loss (Contrast) and the loss without positives (NO Pos) is relatively narrow across In graph classification task: similar phenomena Positive Samples Are NOT a Must in GCL N_neg is the set comprising negative samples of u

9. How GCL Works without Positive Samples? Suggestions: The removal of positive samples in GCL has minimal impact on the performance of downstream benchmarks , in stark contrast to the results in VCL (Visual Contrastive Learning) The difference between VCL and GCL: Positive Samples Are NOT a Must in GCL

10. The Implicit Regularization of Graph Convolution in GCL All the GCL methods analyzed above adopt messagepassing graph neural networks (GNNs) To demonstrate that these GNNs inherently possess an implicit regularization effect that facilitates the aggregation of positive samples. . This finding helps elucidate why GCL can achieve satisfactory performance without explicitly incorporating an alignment objective. At the l-th layer, node representations are aggregated : The reasons behind the observation

11. The Implicit Regularization of Graph Convolution in GCL Importantly, the utilization of GraphConv within encoders is the key for most existing GCL methods to generalize well in the absence of positive samples. GraphConv implicitly achieves feature alignment among neighborhood samples. This alignment process is particularly effective in homophilic graphs, where neighbors predominantly belong to the same class.

12. How GCL Works without Negative Samples? In the absence of negative samples: There are works showing contrastive learning can get rid of negative samples by specific designs on architectures or objective functions Findings: Negative samples are dispensable without any specific designs for the graph classification task From the perspective of feature collapse, we emphasize the significant role played by the projection head in the graph classification task.

13. Graph Classification Task: No need for negative samples Both Negative Samples and Specific Designs Are Not Needed Findings: In contrast to the VCL scenario: GCL can perform well by utilizing the vanilla positive alignment loss alone, without the need for negative samples or any modifications to the network architecture The role of the projection head: Although the model learns a collapsed solution for optimizing the alignment loss, the downstream results remain unaffected. Estimate the average similarity of the representations H = f(X) output by the encoder and Z = g(H)

14. Node Classification: Normalization in the Encoder Is Enough The absence of negative samples in GCL leads to a significant performance drop Different from graph classification, the representations learned by the encoder also collapse in the node classification task. In graph-level: One plausible conjecture is that learning a collapsed solution is relatively easier for the global graph representation, which can be achieved solely by the projection head

15. Node Classification: Normalization in the Encoder Is Enough Question: How does GCL manage to work without negative samples for node classification? Findings: just incorporating a normalization component called ContraNorm (CN) into the encoder of GCL is enough. ContraNorm is originally designed for alleviating the over-smoothing problem in GNNs and Transformers with the formulation: computes the row-wise normalized similarity matrix between node features.

16. Simple Augmentations Do Not Destroy GCL Performance How we do multi views without augmentation: Encoder perturbation: The perturbation term which samples from Gaussian distribution with zero mean and variance SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation, WWW 22

17. Simple Augmentations Do Not Destroy GCL Performance A substantial degradation (83.51% → ≈30%) in VCL’s performance when removing all augmentations or applying only random Gaussian noise. However, data augmentations have a much smaller influence on GCL methods. A simple augmentation: random Gaussian noise, where a random noise sample drawn from a Gaussian is directly added to node features. Formally, given a graph G = (A, X), the random noise augmentation is defined as:

18. Discussion and Conclusion GCL can work in the absence of positive samples. without negative pairs for the graph classification task. achieve comparable performance with domain-agnostic data augmentations like random Gaussian noise.