250310_Thuy_Labseminar[MORE: Molecule Pretraining with Multi-Level Pretext Task].pptx

MORE: Molecule Pretraining with Multi-Level Pretext Task

1. MORE: Molecule Pretraining with Multi-Level Pretext Task Van Thuy Hoang Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: [email protected] 2025-03-10 Yeongyeong Son , et. al.; AAAI 25

2. BACKGROUND: Graph Convolutional Networks (GCNs) Key Idea: Each node aggregates information from its neighborhood to get contextualized node embedding. Limitation: Most GNNs focus on homogeneous graph. Neural Transformation Aggregate neighbor’s information

3. Graph data as molecules Molecules can be naturally represented as graphs with their atoms as nodes and chemical bonds as edges.Graph data such as molecules and polymers are found to have attractive properties in drug and material discovery Molecules as graphs

4. Level view points Four-level viewpoints in a molecular graphs This paper integrates four graph viewpoints: node-, subgraph-, graph-, and 3D-level Note that subgraphand graph-level pretext task is different from conventional approaches, learning predefned local or global information.

5. Methodology: MORE A molecular graph with some nodes masked as input and learns a meaningful representation, and the decoders, which learn multiple attributes of the molecule. In the decoders, (a) reconstructs node-level molecular structures, (b) predicts subgraph-level molecular structures, (c) predicts graphlevel molecular attributes, and (d) learns 3D-level molecular structures.

6. Node-level Pretext Task where He is the re-masked node representation, and Onode ∈ R N×119 denotes the predicted atomic numbers for all nodes,

7. Subgraph-level Pretext Task The graph representation is derived from H via the READOUT function and used to predict 155 MACCS keys. Binary Cross Entropy loss is applied for learning. Given the subgraph-level decoder. MACCS: a type of molecular fngerprint representing unique chemical patterns

8. Graph-level Pretext Task The graph representation obtained through the READOUT function is used to predict the 194 molecular descriptors. We use the MSE loss function.

9. 3D-level Pretext Task Calculate the relative distances between every node by computing the Euclidean distances from the 3D coordinates of each node to generate the true distance matrix

10. Optimizing Multi-level Pretext Task Loss MORE is updated based on following loss L:

11. Experiments Prediction performance of pretrained models on seven downstream tasks

12. Scalability with Dataset Size in Pretraining MORE exhibits the highest and most consistent performance improvement in linear probing and outperforms signifcantly in full fne-tuning. These results suggest the potential of MORE as a foundation model.

13. Conclusions MORE integrates four levels of a molecular graph: node-, subgraph- graph-, and 3D-level. MORE learns highlevel semantic properties by predicting molecular descriptors as well as considering both local- and global-level information. Compared to the baseline models with the same model structure, MORE demonstrates the ability to learn comprehensive representations, with the best performance. In the quantitative forgetting analysis of pretrained models, MORE shows consistently minimal and stable parameter changes with the smallest performance gap. The consistent performance gains with larger pretraining datasets indicate the potential for development as a foundation model.