2024-03, EACL, A RelEntLess Benchmark for Modelling Graded Relations between Named Entities

Relations such as "is influenced by", "is known for" or "is a competitor of" are inherently graded: we can rank entity pairs based on how well they satisfy these relations, but it is hard to draw a line between those pairs that satisfy them and those that do not. Such graded relations play a central role in many applications, yet they are typically not covered by existing Knowledge Graphs. In this paper, we consider the possibility of using Large Language Models (LLMs) to fill this gap. To this end, we introduce a new benchmark, in which entity pairs have to be ranked according to how much they satisfy a given graded relation. The task is formulated as a few-shot ranking problem, where models only have access to a description of the relation and five prototypical instances. We use the proposed benchmark to evaluate state-of-the-art relation embedding strategies as well as several recent LLMs, covering both publicly available LLMs and closed models such as GPT-4. Overall, we find a strong correlation between model size and performance, with smaller Language Models struggling to outperform a naive baseline. The results of the largest Flan-T5 and OPT models are remarkably strong, although a clear gap with human performance remains.

1. A RelEntLess Benchmark for Modelling
Graded Relations between Named
Entities
EACL 2024 Main Conference
Asahi Ushio, Jose Camacho-Collados, Steven Schockaert
Cardiff NLP

2. Relational Knowledge
Capability to understand relationship between two
words.
➢Word Embedding Mikolov (2013)
➢LMs (eg. GPT-3)
➢RelBERT




King
Queen
Woman
Man
Word Embedding
Asahi Ushio, et al. “Distilling Relation Embeddings from Pretrained Language Models” (EMNLP 2021)

3. Word Analogy
Word analogy as a probing task of
relational knowledge.
➢Solvable without training.
➢Different Levels
○Primary school to college
➢Various Relation Types
○Named entity, common noun

4. Research Question
Word analogies are discriminative.
➢(“Tokyo”, “Japan”) is capital-of, but (“U.K.”, “Japan”) is not.

Relations in real-world are often graded.
➢(“Amazon”, “Google”) is more prototypical example of competitor than
(“Netflix”, “Disney”).

Can LMs understand such graded relation?

5. Graded Relation Ranking
New challenging tasks.
➢5 relation types.
➢Pairs of named entities.
➢Rank the pairs based on prototypicality.
Relation Types Examples (Ordered by Prototypicality)
competitor of [Dell, HP] > [Neoclassicism, Romanticism] > [Steve Jobs, Atlanta]
friend of [Australia, New Zealand] > [The Beatles, Queen] > [KGB, CIA]
influenced by [Plato, Socrates] > [Hip Hop, Jazz] > [Sauron, Shiba Inu]
known for [Apple, iPhone] > [Apple, Apple Watch] > [Pixar, Novosibirsk]
similar to [Coca-Cola, Peps] > [Christmas, Easter] > [Italy, Superman]

6. Results
competitorfriendinfluencedknownsimilaraverage
Human 75.9 78.0 70.5 82.0 80.2 80.2
FastText 25.0 10.0 7.0 24.0 20.0 17.0
RelBERT 64.0 20.0 20.0 44.0 53.0 40.0
FlanT5 74.0 56.0 44.0 70.0 66.0 62.0
Flan-UL2 79.0 51.0 47.0 67.0 57.0 60.0
GPT3 72.0 39.0 64.0 73.0 47.0 59.0
GPT4 62.5 55.8 35.9 60.8 69.3 56.9

7. Analysis
➢It scales with the model size (bigger models are often better).
➢Choice of template matters.
➢Few-shot improves most models (except Flan-UL2).
➢Typical error involves
○Biased by entity domain: “Rihanna” / “Stevie Wonder” for
“influenced” (music domain)
○New relationship: “OpenAI” / “Microsoft”
○Surface similarity: “New York”/”York”

8. ?????? Thank you!??????