250310_JH_Seminar[Translating Embeddings for Modeling Multi-relational Data].pptx

Translating Embeddings for Modeling Multi-relational Data

1. Cho junhee Network Science Lab The Catholic University of Korea E-mail : [email protected] Translating Embeddings for Modeling Multi-relational Data

2. Abstract Objective: Embed entities and relationships into a low-dimensional vector space. Key Features: Simple, easy to train, and requires fewer parameters. Scales efficiently to large datasets. Methodology: Relationships are modeled as translations between entity embeddings. Current Status

3. Introduction Directed graph where nodes are entities and edges represent relationships in the form (head, label, tail). Objection of this paper is to develop an efficient tool to automatically complete KBs without extra knowledge Multi-Relational Data & Its Importance

4. Introduction Modeling Multi-Relational Data = Extract local/global connectivity patterns to predict relationships. Since multiple types of relationships and entities are involved simultaneously, a more generalized approach is required. To handle multiple data, they tried to increasing the expressivity and universality of the model imitations of Complex Models Difficult to interpret due to high expressivity. High computational cost and risk of overfitting/underfitting. Research Insight: Simpler linear models can perform almost as well as complex ones, balancing accuracy & scalability. Modeling Multi-Relational Data: Challenges & Approaches

5. Introduction Hierarchical Relationships in KBs Common in knowledge bases → Naturally represented by translations Tree Structure Representation: Siblings → Close in embedding space Parent-Child → Translation along the y-axis Zero translation → Represents equivalence (e.g., sibling relationships) 1-to-1 Relationships (e.g., Country-Capital) Found in word embeddings from free text Suggests that translation naturally models structured relationships Motivation Behind TransE

6. Introduction Algorithm

7. Translation-based model Goal Learn entity (h,t) and relation (ℓ) embeddings in vector space Model relations as translations in embedding space If (h,ℓ,t) is true: Otherwise:   TransE Model Overview

8. Translation-based model Loss Function & Negative Sampling Objective: Lower energy for positive triplets Higher energy for corrupted triplets

9. Translation-based model Corrupted Triplets (Negative Sampling) Randomly replace head or tail (not both) Encourages model to learn triplet validity Loss Function & Negative Sampling

10. Translation-based model Stochastic Gradient Descent (SGD) : Optimization over h,ℓ,t embeddings Assitional Constraints : L2-normalization of entity embeddings No constraints for relations (ℓ) Why? To p revents trivial solutions (infinite norm growth) Optimization & Training Process

11. Translation-based model Stochastic Gradient Descent (SGD) : Optimization over h,ℓ,t embeddings Assitional Constraints : L2-normalization of entity embeddings No constraints for relations (ℓ) Why? To p revents trivial solutions (infinite norm growth) Optimization & Training Process

12. Experiment wordnet, Freebase Data sets

13. Experiment Link prediction result

14. Conclusion Advantages: Highly efficient & fast (SGD-based training) Scalable to large-scale datasets Simple yet powerful model Limitations: Struggles with 1:N, N:1, N:N relations Cannot model symmetric relationships effectively Does not consider relation-specific transformations (→ Improved models: TransH, TransR) Motivation Behind TransE