240923_Thuy_Labseminar_ONE TRANSFORMER CAN UNDERSTAND BOTH 2D & 3D MOLECULAR DATA.pptx

ONE TRANSFORMER CAN UNDERSTAND BOTH 2D & 3D MOLECULAR DATA

1. ONE TRANSFORMER CAN UNDERSTAND BOTH 2D & 3D MOLECULAR DATA Van Thuy Hoang Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: [email protected] 2024-09-02 ICLR 23

2. BACKGROUND: Graph Convolutional Networks (GCNs) Key Idea: Each node aggregates information from its neighborhood to get contextualized node embedding. Limitation: Most GNNs focus on homogeneous graph. Neural Transformation Aggregate neighbor’s information

3. Molecular property prediction with GNNs Learning molecular structures though GNNs Inputs: Molecules Outputs: a score for specific task prediction Graph Neural Networks Molecules Pooling Function Task Prediction

4. Problem: Multi-view problem In image processing: In molecules: 2D views 3D views Hierarchical views Single-view self-supervised learning (SV-SSL). Multi-view self-supervised learning (MV-SSL). Fusing representations Clustering module (CM): how consistency between different views (pseudo labels) On the Effects of Self-Supervision and Contrastive Alignment in Deep Multi-View Clustering ; CVPR2023

5. Problem Most previous works focus on designing neural network models for either 2D or 3D structures, making the model learned in one form fail to be applied in tasks of the other form A general-purpose neural network model in chemistry should at least be able to handle molecular tasks across data modalities.

6. Backgrounds: BACKBONE TRANSFORMER Transformer layer: Positional encoding: plays a crucial role in extending standard Transformer to more complicated data structures beyond language.

7. TRANSFORMER-M Encoding pair-wise relations in E Encoding pair-wise relations in R. Encoding atom-wise structural information in R

8. TRANSFORMER-M: Encoding pair-wise relations in E encode the shortest path distance (SPD) between two atoms to reflect their spatial relation.

9. TRANSFORMER-M: Encoding pair-wise relations in R. encode the Euclidean distance to reflect the spatial relation between any pair of atoms in the 3D space. K is the number of Gaussian Basis kernels.

10. TRANSFORMER-M: Combining E and R All pair-wise encodings defined above capture the interatomic information, which is in a similar spirit to the relative positional encoding for sequential tasks

11. TRANSFORMER-M: Combining E and R Encoding atom-wise structural information in E: Encoding atom-wise structural information in R :

12. EXPERIMENTS Prediction head for position output. (3D Position Denoising): Adopt the 3D Position Denoising task as a self-supervised learning objective. During training, if a data instance is in the 3D mode, we add Gaussian noise to the positions of each atom. The model is required to predict the noise from the noisy input.

13. EXPERIMENTS: Results on PCQM4Mv2 Results on PCQM4Mv2 validation set in OGB Large-Scale Challenge

14. EXPERIMENTS: Results on PDBBind core set Results on PCQM4Mv2 validation set in OGB Large-Scale Challenge

15. EXPERIMENTS: Results on QM9 Results on PCQM4Mv2 validation set in OGB Large-Scale Challenge

16. CONCLUSION Transformer-M offers a promising way to handle molecular tasks in 2D and 3D formats. use two separate channels to encode 2D and 3D structural information and integrate them into the backbone Transformer. Through simple training tasks on 2D and 3D molecular data, the model automatically learns to leverage chemical knowledge from different data formats and correctly capture the representations.