Learning Design meets Learning Analytics and Artificial Intelligence: �Potentials and challenges

Technology-enhanced learning ecosystems are becoming quite complex, especially when non-conventional approaches, such as collaborative or inquiry learning are employed.

On the other hand, the recent advances in the learning analytics field have been very promising, for purposes of understanding and optimizing learning and the environments in which it occurs.

However, the alignment between design for learning and learning analytics has been recently shown to be a pending, albeit essential, issue that would allow for effective and trustful pedagogical interventions and orchestration.

This talk discusses the interdependence and mutual benefits of bringing together learning design and learning analytics, as well as the eventual support of (generative) artificial intelligence to the learning design and orchestration lifecycle.

It also argues that teachers’ agency should be safeguarded through human-centered design approaches. Illustrating examples and relevant pending challenges complement the overall discussion.

1. Learning Design meets Learning
Analyticsand Artificial
Intelligence:
Potentials and challenges
Prof. Yannis Dimitriadis
GSIC/EMIC group
University of Valladolid, Spain
Open Universiteit of the
Netherlands
October 11, 2024

2. Acknowledgements to international
and GSIC/EMIC colleagues
2

3. What is this talk about?
Not about Artificial Intelligence in Education
Not about Design for Learning
Not about Analytics
But essentially on the relations among all of them
and why they may be complementary and mutually
supportive
And eventually how they may jointly contribute to
better teaching and learning
3

4. The complexity of TEL ecosystems
4

5. How to design for complex/innovative TEL?
nUse scripts/pedagogical patterns/recurrent
routines to increase chances of success
nDesign for Learning (Learning Design)
nOrchestrate learning designs/scripts
nReflect and redesign
5

6. Pedagogical Patterns (scripts)
Learning and artefact flow
6

7. Recurrent routines
Atomic patterns, teacher moves
7

8. Design for Learning
or Learning Design
8

9. Design for Learning
“In media res framework”
nWhat can be designed for learning?
nThe learning (performed by students) and
support (made by teachers) tasks
nThe “physical” environment
nSpaces, tools, infrastructures, artifacts-resources (to be
consumed and/or produced)
nThe social architecture
nGroupings, interactions with external agents
nDesign is indirect (tasks vs. activities)
nLearners may change-interpret tasks in learntime
9

10. Design for Learning
Focus on student activity (ACAD)
10

11. A learning design in action
Orchestration graphs
11

12. The Learning Design lifecycle
nAn iterative, bidirectional, spiral process …
–Conceptualizing
–Authoring
–Deploying
–Orchestrating
–Reflecting
–Redesigning
nBy (communities of) teachers, Teaching and
Learning Support Units …
12

13. Is design for learning sufficient?
nCan we achieve TEL through Learning Design
and Orchestration that is
–Innovative and effective
–Efficient and scalable
–Ethically appropriate?
nWe still need
–Supporting tools
–Pedagogical frameworks
–Teacher professional development programs
–Etc …
13

14. The METIS ILDETools, repositories, frameworks …

15. But still LD adoption is low
15
nFirst Order Barriers
–Lack of institutional support
–Lack of adequate teacher training
–Time/workload factors
–Conceptual complexity of method and tools
–Adoption by peers
nSecond order barriers
–Use of ICTs in teaching practice
–Teachers’ motivation

16. Eventually we also need evidence
nWe definitely need
–Theory of Teaching and Learning
–Design for Learning and Orchestration
nBut what about evidence and data?
–Will the use of analytics solve the problem?
nThe emerging power of Data Science
–Learning (+ Teaching + Community) Analytics
nBut most (Learning) Analytics proposals did
not sufficiently consider Design and Theory
16

17. Theory, Design, Orchestration, Data
17

18. However in most LA projects …
nAnalytics are usually created and delivered by
other stakeholders to the teacher in a
dashboard, without
–Making explicit connection to LD process and product
–Involving “the teachers or the students in the loop”
–Considering teacher-student agency and trust (versus
external, institutional control through LA)
nPost-hoc analyses are performed to
understand processes (student retention …)
–Aiming to extract patterns, identify the most influencing
variables in predictive models, analyze strategies, etc.
18

19. LD guides LA-based sense-making
19

20. LA-based monitoring complement LD-
based scripting
20
nProposals
–Monitoring-aware design
process
–Script-aware monitoring
process
nPositive implications in
ethical aspects (agency,
transparency, trust)
María Jesús Rodríguez-Triana,Alejandra Martínez-Monés,Juan I. Asensio-Pérez,Yannis Dimitriadis, Scripting and
monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL
situations, BJET, vl46 (2), 330-3045 (Soller et al., 2005; Rodríguez-Triana et al., 2015)

21. LA guides creation of learning designs
and supporting tools
21

22. LD informs design of LA environments
Embedded/extracted continuous/cumulative analytics
22

23. LD allows better LA predictive models
n“There is a need to create models for academic
success prediction for individual courses,
incorporating instructional conditions into the
analysis model”
nFeatures drawn from the LD product can be derived
that create better prediction models
nBetter in-situ prediction models allow better
pedagogical interventions and orchestration
236















Figure 1. Learning design of the Translation MOOC (adapted from [7])
BLOCK 6
Introduction to
module
Discussion
Forum
Peer Review for
IA3 in Week 5
Quiz 7 Peer Review for
GA2 in Week 5
Peer Review for
OA3 in Week 5
Optional
Activity
BLOCK 5
Introduction
[BLK5_0]
Lecture Content
[BLK5_1]
Discussion Forum
[BLK5_2]
Detailed
Description
[BLK5_31]

Introduction &
Submission
[BLK5_30]

Peer Review for
GA1 in Week 4
Peer Review for
OA2 in Week 4
Optional
Activity
Discussion
Forum
[BLK5_32]
Group Assignment 2 [GA2]
BLOCK 4
Introduction
[BLK4_0]
Lecture Content
[BLK4_1]
Discussion Forum
[BLK4_2]
Peer Review for
IA2 in Week 3
[BLK4_4]
Review Video
[BLK4_5]
BLOCK 3
Introduction
[BLK3_0]
Lecture Content
[BLK3_1]
Discussion Forum
[BLK3_2]
Detailed
Description
[BLK3_31]
Optional Activity
[BLK3_4]
Discussion
Forum
[BLK3_32]
Group Assignment 1 [GA1]
Review Video
[BLK3_5]
Review Video
[OA2]
[OA3]
[OA4]
[IA3]
Introduction &
Submission
[BLK4_3]

Introduction &
Submission
[BLK3_30]

24. LD and orchestration information
increase efficacy, agency and trust
nCustomization of LD and orchestration increases
efficacy and teacher agency and trust
nLA solutions may be based on concrete LD
information provided by the teacher
–course checkpoints, script and activity constraints,
orchestration problems
24

25. Teacher Inquiry into Student Learning
25
Teaching and design analytics help teachers to inquire on their
practice and enhance it, having learning analytics as means to
understand the students’ learning processes

26. Analytics Layers for Learning Design
26

27. But what about the design process?
nWhat are the relations so far?
–Design for learning and orchestration are relevant
–Learning Analytics may support better learning
design, better Teacher Inquiry into Student Learning
–Learning Design may allow for better design of
Learning Analytics
–Learning Theory is necessary for deriving learning-
relevant analytics
nWhat about design process of TEL/LA/AI for …
–Learning, orchestration, reflection, redesign, systems,
pedagogical interventions, programs, etc. ?
27

28. Pattern mining using LA
(Detection of learning strategies)
28
J. B. J. Huang, A. Y. Q. Huang, O. H. T. Lu and S. J. H. Yang, "Exploring Learning Strategies by Sequence Clustering
and Analysing their Correlation with Student's Engagement and Learning Outcome," 2021 International Conference
on Advanced Learning Technologies (ICALT), 2021, pp. 360-362, doi: 10.1109/ICALT52272.2021.00115.
How are the proxies for strategies DEFINED AND COMPUTED?
Who can interpret this data and how?
Is there any student bias regarding these strategies?

29. Smart Learning Environments
(Personalized recommendations-resources)
29
S. Serrano-Iglesias, E. Gómez-Sánchez, M. L. Bote-Lorenzo, G. Vega-Gorgojo, A. Ruiz-Calleja and J. I. Asensio-Pérez
(2021) "From Informal to Formal: Connecting Learning Experiences in Smart Learning Environments," 2021 International
Conference on Advanced Learning Technologies (ICALT), pp. 363-364, doi: 10.1109/ICALT52272.2021.00116.
How is the student model built?
Do teachers/students get involved in the reaction scripts?
What about privacy in informal learning settings?

30. Generative AI
(Outline of existing essay)
30
Hodges, C.B., Kirschner, P.A. (2024) InnovationofInstructionalDesignand Assessmentin theAge ofGenerative Artificial
Intelligence. TechTrends68, 195–199. https://doi.org/10.1007/s11528-023-00926-x.
Why should a STUDENT TRUST this outline?
Is there a need for CRITICAL ASSESSMENT of the product?
How has the outline been GENERATED?

31. Two dilemmas on Agency (I)
Dilemma 1: Learning Analytics (LA) may be
helpful when embedded in Technology-Enhanced
Learning (TEL) contexts. They are typically
designed by researchers and developers, that best
know about efficiency and effectiveness. But
existing LA solutions mostly ignore teachers as
orchestrators (designers and enactors).
What about teachers’ agency?
31

32. Two dilemmas on Agency (II)
Dilemma 2: AI agents that are using LA or GenAI
may support student productivity and eventually
maximizestudents’ learning but how can they be
transparent, trustful, responsible or ethical?
What about students’ agency?
32

33. Human-Centered Learning Analytics
33
HCD should involve:
Inclusion via stakeholder participation in the design process
+
Empathic experiences (particularly when making design
decisions).
Giacomin, J. (2014). What is human centred design? The Design Journal,
17(4), 606–623. https://doi.org/10.2752/175630614X140561854801.

34. Human-Centeredness in MMLA-AIED
34
Cukurova, M. (2022). “Multimodal Learning Analytics in Real-world Practice: A Bridge Too Far?”, Webinar at Spanish
Network of Learning Analytics (SNOLA), May 2022. https://snola.es/2022/05/03/webinar-multimodal-learning-
analytics-in-real-world-practice-a-bridge-too-far-mutlu-cukurova/

35. The Future of HCLA?
S. BuckinghamShum,R. Martínez-Maldonado,Y. Dimitriadis,P. Santos(2024),
Human-CentredLearningAnalytics: 2019–24, Firstpublished: 26 February2024
https://doi.org/10.1111/bjet.13442

36. (Gen)AI for LD and LA (I)
Automatic content creation
nCreate quick and personalized learning activities for learners
nTailor learning experiences based on different learning needs
nContribute to design, co-creation, and collation of activities
faster and easier
nConsider expertise, quality assurance, the rapidly changing
policy environment, and the focus on (automatic) content
rather than (automatic) pedagogy
nReflect on reuse of commercially sensitive content
nDeep and authentic learning is more than just the
assimilation of written texts and artifacts
36
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., … Rienties, B. (2024).
Thepromiseand challengesofgenerative AI in education. Behaviour& InformationTechnology, 1–27.
https://doi.org/10.1080/0144929X.2024.2394886

37. (Gen)AI for LD and LA (II)
Analytics Layers and Learning Design
nReduce complexity of Learning Design lifecycle
nInvolve stakeholders in human-centered design approaches
nExploit evidence based on analytics layers of learning, design
and (teacher) communities
37
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., … Rienties, B. (2024).
Thepromiseand challengesofgenerative AI in education. Behaviour& InformationTechnology, 1–27.
https://doi.org/10.1080/0144929X.2024.2394886

38. (Gen)AI for LD and LA (III)
Analytics Layers and Learning Design
38
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., … Rienties, B. (2024).
Thepromiseand challengesofgenerative AI in education. Behaviour& InformationTechnology, 1–27.
https://doi.org/10.1080/0144929X.2024.2394886

39. Some final thoughts
nDesign for Learning and Orchestration is still
necessary
n(Learning) Analytics are compulsory companions
of Design for Learning
nHuman-Centered approaches to design of tools
(LA or AI-based) and effective TEL are even
more relevant in the GenAI era
nTheory (and practice) of learning and teaching
should accompany design
nA delicate balance is sought for agency,
effectiveness, scaling, innovation, efficiency …
39