Knowledge engineering: from people to machines and back

Keynote at the 21st European Semantic Web Conference

1. Knowledge
engineering: from
people to machines
and back
Elena Simperl, @esimperl

2. Knowledge
engineering
•Building and maintaining
knowledge-based systems
•Sits between
•Software engineering: how to
build and maintain software
systems;
•AI: how to represent, learn,
and reason over knowledge
computationally

3. KNOWLEDGE ENGINEERING: BEFORE
•Gathering curated knowledge
from experts and encoding it into
knowledge bases.
•Mostly manual process,
supported by methodologies,
methods, tools.
•Not scalable, considerable up-
front investment.
3
Source: NeOn project, 2010

4. KNOWLEDGE ENGINEERING: TODAY
•Large knowledge bases, drawn from
heterogeneous data, automatic process w/
human-in-the-loop.
•Mainstream adoption in search, intelligent
assistants, digital twins, supply chain
management, legal compliance etc.
•W/ access to data and (off-the-shelf) AI
capabilities, costs are a fraction from what they
were decades ago.
•New knowledge engineering experiences:
prompting, conversational tools, co-auditing
automatically generated content
4
Source: Fathallah et al., 2024

5. My KE
journey
5/28/2024 5

6. Standard
knowledge
engineering
process during
my PhD
6
Source: me, 2007

7. Case studies,
guidance,
templates to
reuse ontologies
effectively
7

8. Predicting costs,
understanding
the most
resource-
intensive part of
the process
8

9. Two ways forward
Automation Involve more people
9

10. Two challenges
Participation
Data quality
10

11. Participation:
aligning tasks
to people’s
motivations,
designing
incentives
11

12. People implicitly
contribute to the
knowledge base
while going about
their regular work.
Benefits are
highlighted
Challenges: not suitable
for every use case,
shallow semantics
12

13. People play
games. Games
data is turned
into
computational
knowledge
Challenges: design,
attracting and
retaining players,
costs
13

14. People are part of
online communities
that build
knowledge bases
Challenges: uneven
participation, data quality
14

15. What if
automation
were an
option?
15
Human computation Large language models

16. Knowledge
integrity in
Wikidata
Wikidata is a widely used secondary source of knowledge
Data is trusted when it is backed by its primary source
How references are encoded in Wikidata

17. Are
references
easy to
navigate,
authoritative,
relevant?
Yes, but some variations
across languages
5/28/2024 17
English, Swedish, Japanese, Spanish, Dutch, Portuguese

18. Does the
source
support
the claim?
Yes, but challenging task to
automate because of
heterogeneity and
multimodality of evidence
18

19. (Chandler Fashion Center, directions, Highway 101 & Highway 202)
«■ Not logged in Talk Contributions Create account Login
voyage
PageDiscussion
Read Edit View history | Search Wikivoyage
This is an old revision of this page, as edited by SelfieCity (talk | contribs) at 20:58,13 October 2019 (page banner). (diff)
•— Older revision | Latest revision (diff) | Newer revision —» (diff)
Main page
Travel destinations
Star articles What's
Nearby?
Tourist office
Random page
North America> United States of America >
Chandler
Southwest (United States of America) >Arizona >Greater Phoenix > Chandler (Arizona)
Travellers' pub
Recent changes
Community portal
Maintenance panel
Policies
Help
Interlingual lounge
Donate
Chandler is a city in Arizona, and a medium-sized suburb of Phoenix with over 240,000 residents. It is a delightful place to visit.
Get in
Related changes
Upload file
By plane
Phoenix Sky Harbor International Airport (PHX
IATA
) [phone] [1]i? is the main air gateway to Arizona. It is in East Phoenix 3 miles from downtown. All major U.S. carriers serve Phoenix
Special ।
Perm am
Page inft
Cite this
F rirt s r
* Q Chandler Fashion Center^ (Chandler Mali), 3111 W Chandler Blvd (Highway 101 & Highway 202), W [phone]. M-Sa 10AM-9PM, Su 11AM-6PM; restaurant and dept store hours vary An upscale shopping mall with various
department stores, (updated Sep 2018)
• H me Shoppes at Casa Palomas’, 7131 West Ray Road (Highway 10 & Ray). Upscale outdoor shopping and dining center, (updated Sep 201 B>
•BPhoenix Premium Outletss’, 4976 Premium Outlet Way (Highway 10 & Highway 202), [phone]. A shopping destination for locals and visitors looking for upscale shopping in a casual, family-friendly atmosphere.
(updated Sep 2018)
In other projects Get around
Wikimedia Commons
Wikipedia
Car and bus are the easiest ways to get around Chandler Rideshare services Uber and Lytt also operate in the city. Google's Waymo autonomous car service is also available in parts of the city.

26. Datasets
WDV:
verbalisation
WTR: textual
references
WTR
416 triples, 3 classes (supports,
refutes, not-enough-information),
free-text rationale to manage
diversity of reference layouts

27. Complex task, 87.5% accuracy when
explicit textual evidence present
27

28. What’s
next
28

30. 5/28/202430

31. Dagstuhl
seminar
09/22

32. Hackathon
08/23

33. 5/28/2024 33

34. How do
knowledge
engineers use
LLMs
Ethnographic observations of the
hackathon
Documentary analysis of documents
collected during the hackathon
Semi-structured interviews with hackathon
participants

35. Findings
Challenges around lack of datasets and
evaluation protocols
Guidance and best practices needed on
knowledge prompting and auditing LLM
outputs
Limited awareness of responsible AI
concerns like biases, fairness
35

36. LLMs and
requirements
engineering
AI assistants that
automate tasks,
generate
documentation,
remove technical
jargon
84.1% of knowledge engineers lack access to
specialised tools
66.1% lack access to guidelines
Manual process, using text editors and spreadsheets
[1] MonfardiniGK, SalamonJS, BarcellosMP. Use of Competency Questions
in Ontology Engineering: A Survey. Int. Conf. on Conceptual Modeling2023

37. Example: OntoChat
https://huggingface.co/spaces/b289zhan/OntoChat
37

38. User research: mock-ups

39. Using LLMs and
knowledge
graphs to
reverse
engineer
competency
questions
39

40. Explanations
5/28/2024 40

41. Cards for KGs and KG embeddings

42. Final thoughts
•Knowledge engineering is transforming. We need
•Methodologies w/ stronger focus on hybrid workflows, new roles (prompting), new
skills (co-auditing, responsible AI)
•Viable AI copilots, multimodal support, scalable to real knowledge graphs
•Datasets, benchmarks, and evaluation protocols to improve the technical
performance of LLMs in knowledge engineering tasks (e.g. knowledge prompts,
multi-turn dialog, safety, user stories, CQs, documentation etc.)
•Campaigns to share and reuse data where possible to reduce data-related costs
(prompting collectives)
•Guidance, best practices to use AI effectively
•Mechanisms to demonstrate AI-co-generated knowledge graphs can be trusted
42

43. Where is
the web in
all this?
43

44. This talk presented work by: Gabriel Amaral, Jacopo
de Berardinis, Fiorela Ciroku, Jonathan Hare, Lucie
Kaffee, Jongmo Kim, Elisavet Koutsiana, Albert
Meroño-Peñuela, Michelle Nwachukwu, Alessandro
Piscopo, Valentina Presutti, Odinaldo Rodrigues,
Johanna Walker, Bohui Zhang, Yihang Zhao
Thank you also to participants of the Dagstuhl
seminar, hackathon, and ESWC special track.
Elena Simperl
@esimperl
44