Webinar week day 3, 3 October 2024: domenichini-encore-presentation.pdf

Webinar week day 3, 3 October 2024: domenichini-encore-presentation

1. LLMs for Knowledge Modeling:
NLP Approach to Constructing
Learner Knowledge Profile
for Personalized Education


DIANA DOMENICHINI
NLP Approach to Develop
Learner Knowledge Profile
for Personalized Education
Diana Domenichini
National PhD in Artificial Intelligence

2. Problem Definition
Facts:
●Rise of interdisciplinary courses, that attracts students from varied
backgrounds.
●Lifelong learning era: workers with diverse past learning experiences follow
courses to update their knowledge and skills
●Abundance of online educational resources, that encourages self learning
Consequences:
●Challenges in managing the abundance of educational resources.
●Difficulty in designing adequate and challenging classes due to the diverse
backgrounds and knowledge levels of learners.
Possible solution:
●Personalization of the learning experience based on: background knowledge

3. Profiling Tool characterizes the the knowledge
profile of the learners or of the courses
Learner Profile is a list of knowledge that the
student has faced during his or her past academic
studies
Educational Material Profile is a list of knowledge
that that encompassess the content of the course
Personalization Tool works on finding the
difference and similarities between the two
profiles
Personalized Educational Materials are the
educational material adapted to the student prior
knowledge
Solution Design
Profiling Tool
Educational
Material
Learners
Educational
Material Profile
Learners
Profile
Personalization Tool
Personalized
Educational Material

4. Profiling Tool characterizes the the knowledge
profile of the learners or of the courses
Learner Profile is a list of knowledge that the
student has faced during his or her past academic
studies
Educational Material Profile is a list of knowledge
that that encompassess the content of the course
Personalization Tool works on finding the
difference and similarities between the two
profiles
Personalized Educational Materials are the
educational material adapted to the student prior
knowledge
Solution Design
Profiling Tool
Educational
Material
Learners
Educational
Material Profile
Learners
Profile
Personalization Tool
Personalized
Educational Material

5. Method
Evaluation
Compare the two
extraction method and
evaluate them with a
test set
Dataset Selection
Select the appropriate
data that represents the
knowledge of the
learner
Profiling
Extract knowledge concepts
from data using two method:
-Gazetteer-based Named
Entity Recognition
-Prompt engineering with
ChatGPT

6. Learners: three students graduated from the University of Pisa in:
●Bachelor and Master in Physics,
●Bachelor and Master in Management Engineering,
●Bachelor in Digital Humanities and Master in Data Science.



Case study

7. Method
Dataset Selection
Select the appropriate
data that represents the
knowledge of the
learner

8. ● A course can be summarized by a set of
educational concepts representing the
knowledge addressed in the course.
●The learner’s academic knowledge can
be synthesized by educational concepts
derived from all completed courses.
Dataset Selection
Learner Knowledge Profile
=
List of educational concepts
found on the courses the
learner has attended

9. Dataset Selection
Learner’s
academic
learning
experiences
Courses the
learner has
completed and
passed.
outline learning objectives,
course overviews, and
program details, set by the
course teacher
brief descriptions of topics
covered, including date,
time, and lesson type, and
are updated regularly.
Syllabus
Lessons
Records

10. Dataset Selection
Learner’s
academic
learning
experiences
Courses the
learner has
completed and
passed.
outline learning objectives,
course overviews, and
program details, set by the
course teacher
brief descriptions of topics
covered, including date,
time, and lesson type, and
are updated regularly.
Syllabus
Lessons
Records
High level of detail ensuring deeper insights into the
concepts addressed in a university course.

11. Lesson Record
Example of a lesson record (the first
3 lessons of the Introductory Physics
course):
●date
●time
●duration
●type of the lesson
●content of the lesson
●name of the instructor

12. Method
Profiling
Extract knowledge concepts
from data using two method:
-Gazetteer-based Named
Entity Recognition
-Prompt engineering with
ChatGPT

13. Gazzetta: ESCO (European Skills, Competences, and Occupations)

From the ESCO database, we selected the list of Italian and English
skills and chose those of type "knowledge".

We obtained a gazette of knowledge composed of 22,348
alternative labels, grouped into 3,059 ESCO preferred labels

https://esco.ec.europa.eu/en
Gazetteer-based Named Entity Recognition

14. Prompt Engineering: the practice of designing and optimizing prompts to
effectively utilize and enhance the capabilities of large language models (LLMs)
for various tasks and applications.

-Zero-shot prompting: the prompt directly instructs the model to perform a task
without any additional examples to steer it.
- Few-shot prompting can be used as a technique to enable in-context learning
where we provide demonstrations in the prompt to steer the model to better
performance. The demonstrations serve as conditioning for subsequent input text
where we would like the model to generate a response.
https://www.promptingguide.ai/
Prompt engineering with ChatGPT

15. TASK INSTRUCTION:
Identify all the educational concepts in the given description of a university course lecture.

OUTPUT FORMAT SPECIFICATIONS:
Answer by writing only the concepts separated by “;”. If no concept exists, then just answer NA.

DEMONSTRATION EXAMPLES:
University course: Chemistry
Description: Electronic configurations. Electronic configurations from H to Ne (Z = 1-10). Pauli exclusion principle,
Hund rule. Internal electronic shells and shells of valence.
Answer: electronic configurations; pauli exclusion principle; hund rule
…

INPUT TEXT:
University course: Introductory Physics Course
Description: Problems with moments of inertia and rigid body. Hollow cylinder, double bar, physical pendulum
Answer:



Prompt

16. Demonstration Example for Physics Curriculum
1)University course: Chemistry
Description: Electronic configurations. Electronic configurations from H
to Ne (Z = 1-10). Pauli exclusion principle, Hund rule. Internal electronic
shells and shells of valence.
Answer: electronic configurations; pauli exclusion principle; hund rule

2)University course: Mathematical Analysis 1 and 2
Description: Series with any sign terms. Absolute convergence and
simple convergence. Leibniz criterion. Variant of
the Leibniz criterion for indeterminate series.
Answer: absolute convergence; simple convergence; leibniz criterion

3)University course: Introductory Physics Course
Description: The vector shift. Operations with the displacement vector:
sum of vectors, multiplication by a scalar, linear combination of vectors.
Form, direction, and direction of a vector. Unit vector. Reference
systems. Orthonormal reference systems.
Answer: displacement vector; unit vector; reference systems;
orthonormal reference systems


4)University course: Computer Science
Description: Introduction to operating systems
Answer: operating systems

5)University course: Laboratory of Physics 1
Description: Recovery session in the laboratory
for interested students
Answer: NA

6)University course: Laboratory of Physics 1
Description: The Laboratory Report. Introduction
to scientific communication between scientists
and Laboratory report model: suggestions and
mistakes to avoid.
Answer: laboratory report, Scientific
communication

17. Method
Evaluation
Compare the two extraction
method and evaluate them
with a test set
we considered subset of the overall case study
comprising 9 course from physics curricula.
Manually annotation of knowledge made by an
expert

18. Statistics on the number of unique concepts extracted per lesson
Comparison
Method Av. St. Dev Min Max Total
ESCO 0.43 0.89 0 7 65
ChatGPT 5.1 3.7 0 27 4403
Method Av. St. Dev Min Max Total
ESCO 6.3 4.0 0 15 65
ChatGPT 186 111 30 528 4403
Statistics on the number of unique concepts extracted per course

19. TestSet: subset of the overall case study comprising 9 course from physics curricula.
manually annotation of knowledge made by an expert

Evaluation
Method Accuracy Recall Precision
ESCO 0.37 0.05 0.65
ChatGPT 0.54 0.81 0.61

20. Visualization for Personalization

21. Physicists
Engineer
Data Scientist
Knowledge Network
●Nodes: concepts.
●Weight of Links: no. of lessons
concepts appear together

22. Future Developments
●Explore personalization method based on the knowledge profile
●Explore the capability of ChatGPT in extracting entities at different level of
detail from the given unstructured text.
●Testing ESCO results with a test set designed for a lower level of detail.
●Knowledge Profile: meaningful representation to inform teacher and instructional
designer about the student previous knowledge
●Knowledge profile from ESCO extraction is shorter compared to ChatGPT generated
profile.
●ESCO's level of detail can be useful for certain applications (e.g. during
university-to- professional transitions)

Findings

23. Thank you!
[email protected]
https://www.researchgate.net/profile/Diana-Domenichini-2