Webinar Innovative assessments for SOcial Emotional Skills

Presentations by Adriano Linzarini and Daniel Catarino da Silva of the OECD Rethinking Assessment of Social and Emotional Skills project from the OECD webinar "Innovations in measuring social and emotional skills and what AI will bring next" on 5 July 2024

1. INNOVATIONS IN THE DIRECT ASSESSMENT
OF SOCIAL AND EMOTIONAL SKILLS
Behavioural-based assessments
Daniel Catarino da Silva & Adriano Linzarini
Webinar –05 July 2024
Centre for Educational Research and Innovation

2. 2
Whyisinnovation socritical?
We need robust, valid, and comparable assessments to monitor socio-
emotional development and improve teaching practices
Self-reported and teacher-reported questionnaires offer valuable, yet
limited, information
Many skills are better assessed in action
Behavioural assessments flesh out more authentic behaviours >>
assessment of choices and actions >> more objective and comparable
measurement of SES

3. Almost 60 different tasks and digital games mapped >> assessing 20
different SES
Unevenassessment coverage of skills
All instruments reviewedagainst 3 different major criteria:
3
Ecological
validity
Validity &
Reliability
Feasibility
Whatbehaviouralassessmenttoolsare out there?

4. 4
Interactive table comparinginnovative SES assessmenttools

5. 5
Examplesof tasksand digital games
Rafner et al. (2023)
CREA (creativity)
VESIP (social problem-solving)
Russo-Ponsaran et al. (2018)

6. 6
Strengths Challenges
Tasks
•Comprehensive coverage
•Efficient extraction of data
•Robust reliability and validity
•Simple stimuli and context
•Difficulty in representing multifaceted
constructs
•Can overlook richness of SES
manifestation
Digital
games
•Storytelling, audiovisual,
immersiveness and interactivity
•Simultaneous assessment of
multiple SES
•Stealth assessment
•Significant time to administer
•Not sufficiently tested
•Complexity and cognitive load generate
noise in the data
•Costly and complex to develop
Whatare the differentstrenghtsand challenges of theseinnovative tools?

7. Assessment tools offer rich and diverse designs, with very distinct
levels of psychometric quality
They were developed for different purposes –research, clinical and
educational –which conditions their use in other contexts
The majority does not offer large-scale validation, lacking testing with
sufficiently representative samples
Further research and validation is required before classroom-level and
system-level implementation
7
Whatcan weconclude?

8. INNOVATIONS IN THE DIRECT ASSESSMENT
OF SOCIAL AND EMOTIONAL SKILLS
Advanced technological approaches
Adriano Linzarini& Daniel Catarino da Silva
Webinar –05 July 2024
Centre for Educational Research and Innovation

9. New technologicalapproachesfor assessment
Biophysiologicalmeasures
Virtual and AugmentedReality
Heartactivity
Cortisol
Eye tracking ArtificialIntelligence applications
Digital footprintsand behavioural
measuresfromvideogames
Authors - left to right up/down: John Campbell, Gordon Dylan Johnson, Julia M Cameron, http://www.norden.org/

10. New technologicalapproachesfor assessment
Biophysiologicalmeasures
Virtual and AugmentedReality
Heartactivity
Cortisol
Eye tracking ArtificialIntelligence applications
Digital footprintsand behavioural
measuresfromvideogames
Authors - left to right up/down: John Campbell, Gordon Dylan Johnson, Julia M Cameron, http://www.norden.org/

11. New technologicalapproachesfor assessment
Biophysiologicalmeasures
Virtual and AugmentedReality
Heartactivity
Cortisol
Eye tracking ArtificialIntelligence applications
Digital footprintsand behavioural
measuresfromvideogames
Authors - left to right up/down: John Campbell, Gordon Dylan Johnson, Julia M Cameron, http://www.norden.org/

12. New technologicalapproachesfor assessment
Biophysiologicalmeasures
Virtual and AugmentedReality
Heartactivity
Cortisol
Eye tracking ArtificialIntelligence applications
Digital footprintsand behavioural
measuresfromvideogames
Authors - left to right up/down: John Campbell, Gordon Dylan Johnson, Julia M Cameron, http://www.norden.org/

13. Strengthsand challenges
Biophysiologicalmeasures
Virtual and AugmentedReality
ArtificialIntelligence applications
Digital footprintsand behavioural
measuresfromvideogames
• Adaptable into existing tasks &
games
• Less subject to conscious
alteration
• Fine-grained information about
internal states
• Biodata and behaviours/skills
correlations are complex
• Strongly affected by stress, sleep,
food
• Requires specialized staff and
equipment
• Usually used to assess skills
related to stress and attention

14. Strengthsand challenges
Biophysiologicalmeasures
Virtual and AugmentedReality
ArtificialIntelligence applications
Digital footprintsand behavioural
measuresfromvideogames
• Offer greater levels of
immersiveness
• Strong engagement and
embodiement
• Mainly targeting learning
purposes (not assessment)
• Equipment costs
• No clear proof of gains compared
to 2D screen yet

15. Strengthsand challenges
Biophysiologicalmeasures
Virtual and AugmentedReality
ArtificialIntelligence applications
Digital footprintsand behavioural
measuresfromvideogames
• Increase in quality and decrease in
costs and time for assessment
design
• Analyse large amounts of data
from different sources
• Identification of profiles of social
and emotional skills based on
behaviours/choices beyond test
contexts
• Clear validation studies are not
yet available
• Biases related to culture,
ethnicity, gender, socioeconomic
status etc.
• Ethical concerns and limited
regulatory constraints

16. In short…
New technologies provide powerful tools for better data analysis
and broader data sources
The cost-benefit trade-off of some of these technologies compared
to traditional assessments remains unproven
Interpreting data and relating it to actual skills is often complex
There are ethical concerns, particularly in educational contexts
Open-access validation research is essential for classroom and
system-level implementation

17. Noémie Le Donné – Senior Project Manager
[email protected]
Adriano Linzarini – Lead Analyst
[email protected]
Daniel Catarino da Silva – Junior Analyst
[email protected]
For more information, find us on
https://www.oecd.org/en/about/projects/rethinking-assessment-of-
social-and-emotional-skills.html
or contact [email protected]
Team and publication