250405_HW_LabSeminar[CoLAKE: Contextualized Language and Knowledge Embedding].pptx

CoLAKE: Contextualized Language and Knowledge Embedding

1. K im Hyun W oo Network Science Lab The Catholic University of Korea E-mail : [email protected] CoLAKE : Contextualized Language and Knowledge Embedding

2. LLM with KG Deep contextualized language models pre-trained on large-scale unlabeled corpora have achived significant improvement on a wide range of NLP tasks However, they are shown to have difficulty capturing factual knowledge Sentence is not a “FACT”, it is a “Indicator of FACT”. PLM learn relations between the sentences. And by the learning process, the model seems like it know real world knowledge. But Knowledge graph(KG) is literally FACT. It shows relations between entities, direct knowledge. By put KG in PLM, we can make model can capture factual knowledge. Introduction

3. KEPLER: aims to benefit both sides jointly learn language model and knowledge embedding. However, KEPLER does not directly learn embeddings for each entity but learns to generate entity embedding with PLMs from descriptions. K-BERT: it is similar with this model, but K-BERT injecting triplet during fine-tuning Previous work LLM with KG

4. LLM with KG Model Architecture

5. LLM with KG Model Architecture By calculating attention socre only if tokens are connected, they can preserve structural information . (It works like GAT) On training stage, the model predicts masked words and entities, model learn contextualized representation for entities

6. LLM with KG Experiment They do three type of tasks. Kno wledge-driven tasks Entity typing Relation Extraction Knowledge-Probing Language understanding tasks Word-Kno wledge Graph Completion

7. Knowledge-Driven tasks LLM with KG

8. Knowledge Probing LLM with KG

9. Language understanding tasks LLM with KG

10. Word-Knowledge Graph Completion LLM with KG

11. They propose the model to jointly learn contextualized representation for language and knowledge. they show CoLAKE is essentially a powerful GNN that is structure aware. And demonstrate performance in various task. LLM with KG Conclusion

12. QA