To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Reflecting on new architectures for knowledge based systems in light of generative ai

1. To Graph or Not to Graph
Knowledge Graph Architectures
and LLMs
Prof. Paul Groth | @pgroth | pgroth.com | indelab.org
Text2KG Workshop - ESWC - 2024

2. Effy Xue Li Stefan Grafberger
Corey Harper
Daniel Daza
Prof. Paul Groth
James Nevin
Melika Ayoughi
Dr. Frank NackDr. Jacobijn Sandberg Dr. Sebastian Schelter
Stian Soiland-Reyes Thiviyan
Thanapalasingam
Shubha GuhaDr. Victoria Degeler
Pengyu Zhang
Zeyu ZhangFina Polat Erkan Karabulut Danru Xu
Bradley AllenDr. Jan-Christoph Kalo
Dr. Hazar Harmouch
22
Teresa Liberatore Yichun Wang
Thanks

3. Building knowledge graphs

4. What does the knowledge graph development cycle look like?

6. Gytė Tamašauskaitė and Paul Groth. 2023. Defining a
Knowledge Graph Development Process Through a
Systematic Review. ACM Trans. Softw. Eng. Methodol.
32, 1, Article 27 (January 2023)
https://doi.org/10.1145/3522586

8. Timely question
•Dagstuhl seminar 22372, 11-14.09.2022
•Organised by
•Paul Groth (University of Amsterdam, NL)
•Elena Simperl (King's College London, UK)
•Marieke van Erp (KNAW Humanities Cluster - Amsterdam,
NL)
•Denny Vrandecic (Wikimedia - San Francisco, US)
•More information at: https://www.dagstuhl.de/
seminars/seminar-calendar/seminar-details/22372
•Other places too: aaai-make.info
8

9. Knowledge engineering: before
•Gathering highly curated knowledge
from experts and encoding it into
computational representations in
knowledge bases.
•Mostly manual process, focusing on
how knowledge was structured and
organised rather than the domain data.
•Results used in expert systems,
requiring considerable up-front
investment.
9

10. Knowledge engineering: today
Automatic process with human-in-the-loop
Large knowledge bases, drawn from heterogeneous data, using a mix of data
management, machine learning, knowledge representation, crowdsourcing
Provided access to data and (off-the-shelf) AI capabilities, costs are a fraction from what
they were decades ago.
This has led to mainstream adoption in search, intelligent assistants, digital twins, supply
chain management, legal compliance etc.
10

11. KE Requirements over time
11
See Allen et. al 2023
https://arxiv.org/abs/2306.15124

12. LLMs have changed our thinking
Knowledge Engineering Using Large Language Models
Bradley P. Allen
1
￿
University of Amsterdam, Amsterdam, The Netherlands
Lise Stork￿
Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
Paul Groth￿
University of Amsterdam, Amsterdam, The Netherlands
Abstract
Knowledge engineering is a discipline that focuses
on the creation and maintenance of processes that
generate and apply knowledge. Traditionally, know-
ledge engineering approaches have focused on know-
ledge expressed in formal languages. The emergence
of large language models and their capabilities to
e!ectively work with natural language, in its broad-
est sense, raises questions about the foundations
and practice of knowledge engineering. Here, we
outline the potential role of LLMs in knowledge
engineering, identifying two central directions: 1)
creating hybrid neuro-symbolic knowledge systems;
and 2) enabling knowledge engineering in natural
language. Additionally, we formulate key open re-
search questions to tackle these directions.
2012 ACM Subject ClassificationComputing methodologiesæNatural language processing; Computing
methodologiesæMachine learning; Computing methodologiesæPhilosophical/theoretical foundations
of artificial intelligence; Software and its engineeringæSoftware development methods
Keywords and phrasesknowledge engineering, large language models
Digital Object Identifier10.4230/TGDK.1.1.3
CategoryVision
Related VersionPrevious Version:https://doi.org/10.48550/arXiv.2310.00637
FundingLise Stork: EU’s Horizon Europe research and innovation programme, the MUHAI project
(grant agreement no. 951846).
Paul Groth: EU’s Horizon Europe research and innovation programme, the ENEXA project (grant
Agreement no. 101070305).
AcknowledgementsThis work has benefited from Dagstuhl Seminar 22372 “Knowledge Graphs and
Their Role in the Knowledge Engineering of the 21st Century.” We also thank Frank van Harmelen for
conversations on this topic.
Received2023-06-30Accepted2023-08-31Published2023-12-19
EditorsAidan Hogan, Ian Horrocks, Andreas Hotho, and Lalana Kagal
Special IssueTrends in Graph Data and Knowledge
1Introduction
Knowledge engineering (KE) is a discipline concerned with the development and maintenance of
automated processes that generate and apply knowledge [4,93]. Knowledge engineering rose to
prominence in the nineteen-seventies, when Edward Feigenbaum and others became convinced that
automating knowledge production through the application of research into artificial intelligence
required a domain-specific focus [32]. From the mid-nineteen-seventies to the nineteen-eighties,
knowledge engineering was mainly defined as the development of expert systems for automated
decision-making. By the early nineteen-nineties, however, it became clear that the expert systems
approach, given its dependence on manual knowledge acquisition and rule-based representation
1
Corresponding author
©Bradley P. Allen, Lise Stork, and Paul Groth;
licensed under Creative Commons License CC-BY 4.0
Tra n s a c t i o n s o n G ra p h D a t a a n d Kn o w led g e, Vol. 1, Issue 1, Article No. 3, pp. 3:1–3:19
Transactions on Graph Data and Knowledge TGDK
Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Germany
Bradley P. Allen, Lise Stork, and Paul Groth. Knowledge Engineering Using Large Language Models. In
Special Issue on Trends in Graph Data and Knowledge. Transactions on Graph Data and Knowledge
(TGDK), Volume 1, Issue 1, pp. 3:1-3:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2023)
https://doi.org/10.4230/TGDK.1.1.3

13. https://king-s-knowledge-graph-lab.github.io/knowledge-prompting-hackathon//

14. The Multimodal Nature of Knowledge

15. Embracing the diversity of knowledge forms
•domain knowledge is often best represented in a variety of modalities, i.e., images, taxonomies, or free text,
•each modality with its own data structure and characteristics which should be preserved,
•no easy way of integrating, interfacing with or reasoning over multimodal knowledge in a federated way exists;
•provenance of data is paramount in understanding knowledge within the context in which it was produced;
•fuzzy, incomplete, or complex knowledge is not easily systematized;
•Data standards
•using data standards for describing and reasoning over data can aid in countering unwanted biases via transparency;
•making data comply with data standards can lead to oversimplification or reinterpretation;
•the production of structured domain knowledge, for instance from images or free text, requires domain expertise, and is
therefore labor intensive and costly;
•knowledge evolves, and knowledge-based systems are required to deal with updates in their knowledge bases.

16. LLMs for KB and LLMs as KB

17. LLMs for KBs

18. LLMs for Information Extraction

19. 27.09.2319
Relation Extraction & Instruction Tuning
Do Instruction-tuned Large Language Models Help with Relation Extraction?
Xue Li, Fina Polat and Paul Groth. LM-AKBC Workshop at ISWC 2023
Results on REBEL dataset
Results on Post-Hoc Human Eval
Can we preserve relation extraction performance
while perserving in-context capabliities?
Method: Instruction Tune Dolly LLM with
LORA using a relation extraction dataset
(REBEL)

20. Language Models as Encoders
https://github.com/dfdazac/blp
Inductive Entity Representations from Text via Link Prediction
Daza, Daniel, Cochez, Michael, and Groth, Paul
In Proceedings of The Web Conference 2021.
DOI:10.1145/3442381.3450141

21. BioBLP: Domain Specific Attribute Encoders
Daza, Daniel, Dimitrios Alivanistos, Payal Mitra, Thom Pijnenburg, Michael
Cochez, and Paul Groth. BioBLP: a modular framework for learning on
multimodal biomedical knowledge graphs. J Biomed Semant 14, 20 (2023).
https://doi.org/10.1186/s13326-023-00301-y

22. Daza, Daniel, Dimitrios Alivanistos, Payal Mitra, Thom Pijnenburg, Michael
Cochez, and Paul Groth. BioBLP: a modular framework for learning on
multimodal biomedical knowledge graphs. J Biomed Semant 14, 20 (2023).
https://doi.org/10.1186/s13326-023-00301-y

23. LLMs for Data Wrangling
Zeyu Zhang, Paul Groth, Iacer Calixto, Sebastian Schelter (2024). Directions Towards
Efficient and Automated Data Wrangling with Large Language Models. Databases and
Machine Learning workshop at ICDE.
Code and experimental results available at https://github.com/Jantory/cpwrangle

24. Data Wrangling with Large Language Models (LLMs)
1
•Huge potential of LLMs for long-standing data wrangling tasks such as entity matching,
missing value imputation and error detection [1, 2]
•Automation and scalability challenges (e.g. for data wrangling services in the cloud)
•Manual few-prompt selection from [1] not automatable and scalable
•Disadvantages of automatable alternatives such as fully fine-tuning a model per customer
•High storage costs (for copies of model parameters)
•High computational costs (for model training)


→ We need parameter- and compute-efficient ways to employ LLMs for data wrangling
[1] Narayan et al.: Can Foundation Models Wrangle Your Data?, VLDB’22
[2] Fernandez et al.: How large language models will disrupt data management, VLDB’23

25. Parameter Efficient Fine-Tuning
Transfer Learning techniques for LLMs
•Manual prompt engineering -- no training (+), hard to automate (-)
•Full finetuning (FT) -- high performance (+), requires substantial computational resources (-)
•Parameter Efficient Tuning (PEFT) -- fewer parameters trained (+), on par performance (+)
Prefix-tuning
[1]
LoRA adapter
[2]
3
[1] Li et al., “Prefix-tuning: Optimizing continuous prompts for generation,” ACL’21.
[2] Huetal.,“LoRA:Low-RankAdaptationofLargeLanguageModels,” ICLR’22.

26. Results on Prediction Quality
How does prediction quality vary among different PEFT methods and base models?
4
LLMMethod# of Parameter UpdatesMean Predictive ScoreGPT3 (175B)Zero-Shot -66.71 -AutoML -76.88T5-small (60.5M)Prompt48K81.94P-tune212K80.11Prefix309K67.66LoRA296K90.96Finetune60,500K89.95T5-base (223M)Prompt67K81.22P-tune312K85.09Prefix914K84.49LoRA892K92.03Finetune223,000K90.36T5-large (783M)Prompt74K82.04P-tune369K76.62Prefix2,435K88.65LoRA2,362K92.24Finetune770,000KTrain Failed
Evaluated four PEFT methods (Prompt, P-tune,
Prefix, LoRA) on three variants of Google’s T5
model on benchmark data from Narayan et al.
Findings:
•PEFT methods outperform GPT3 baseline and
AutoML in many settings
•LoRA provides highest performance
•Applying PEFT methods to larger models
provides higher performance

27. Results on Computational Efficiency
How does computational efficiency vary among different PEFT methods and base
models?
5
Training time per epoch on AMGO dataset
Mean inference throughput over all datasets
Training Times for FT on AMGO Dataset: 38s, 109s, and 312s,
respectively.
Findings:
•Only minor differences in training and inference times
between PEFT methods, parameter size has highest impact

•PEFT methods designed for parameter efficiency (two
orders of magnitude less parameters than full finetuning),
but not for compute efficiency!
Even the fastest method Prefix-Tuning is only twice as fast as
full fine-tuning on t5-base

28. LMs as KBs

29. 27.09.2329
Prompt-contexts for obtaining knowledge from LLMs
Knowledge-centric Prompt Composition for Knowledge Base Construction from Pre-trained
Language Models. Xue Li, Anthony Hughes, Majlinda Llugiqi, Fina Polat, Paul Groth and
Fajar J. Ekaputra. LM-KBC Workshop at ISWC 2023
https://github.com/effyli/lm-kbc/
Task: retrieve the object of a triple
given the subject and object
LM-AKBC challenge 2023
2nd Place!

30. ▫SemEval competition
SHROOM
▿https://helsinki-nlp.github.io/
shroom/
▫Goal: detect whether a given
LLM output contains
hallucination or not
▫4
th Best Performing system
LLMs as Curators:
Dealing with hallucinations
SHROOM-INDElab at SemEval-2024 Task 6: Zero-and Few-Shot LLM-Based Classification for Hallucination Detection
BP Allen, F Polat, P Groth
18th International Workshop on Semantic Evaluation (SemEval-2024)

31. LLMs as Curators: Class membership relation evaluation 

by an LLM
domain
knowledge in
natural language
corpus C
    = arg max
?????? L (?????? | (e, instance-of, o) )
knowledge
graph G
pre-training
sampling
(e, instance-of, c)
decision
For more details: Wednesday 14:40 LLM-KE Track
Evaluating Class Membership Relations in Knowledge Graphs using Large Language Models
by Bradley Allen and Paul Groth

32. Architectures

33. RAG
Source: The Future of Work With AI - Microsoft March 2023 Event

https://www.youtube.com/watch?v=Bf-dbS9CcRU&ab_channel=Microsoft

34. Faculty of Science
SPARQL Queries over Text
Groth, P., Scerri, A., Daniel, R., & Allen, B. (2019). End-to-end learning for answering structured queries directly over text.
Proceedings of the Workshop on Deep Learning for Knowledge Graphs (DL4KG2019) (Vol. 2377, pp. 57–70). CEUR.
https://ceur-ws.org/Vol-2377/#paper7

35. Querying text like a DB
Mohammed Saeed, Nicola De Cao, and Paolo Papotti. 2023. Querying Large Language Models with SQL. https://doi.org/10.48550/arXiv.2304.00472

36. Rethinking DB architecture
Matthias Urban and Carsten Binnig: "CAESURA:
Language Models as Multi-Modal Query Planners",
CIDR’2024
https://github.com/DataManagementLab/caesura

37. Adaptive Source Architecture
Input
Documents
Input Schema
Generative LLM
Prompt
Templates
Assembled
Prompt (s) Knowledge
Graph
Elements
Training

Data
Queries

38. Conclusion
•LLMs through the notion of encoders allow us to take advantage of
more of what’s in our KGs
•LLMs as robust information extractors can be live components in
systems. Break away from pipeline view of information extraction.
•LLMs can be effectively treated information sources and curators.
•This allows new flexible architectures that take advantages of the
different formats of knowledge
Paul Groth | @pgroth | pgroth.com | indelab.org