A happier technology-enhanced PhD?​ Value-sensitive design of ​ single-case learning analytics for doctoral education (​SNOLA webinar, 29.04.2024)

Worldwide, about half of the students who start a doctorate never complete it, and the prevalence of mental health problems (e.g. anxiety, depression) among them is higher than in the general population. These problems, which affect hundreds of thousands of students worldwide, are linked to systemic and socio-economic issues over which students themselves have little control... but also to motivational and socio-emotional skills over which students may have more control. A key aspect of these problems with the doctorate, and the lack of reliable and scalable solutions to mitigate them, is the inherent uniqueness of the doctorate. This webinar will look at recent research in doctoral education on these motivational factors, as well as recent work where we are designing and implementing Learning Analytics (LA) systems that aim to harness these motivational factors to support doctoral students. The resulting human-AI collaboration approach uses analytical methods specifically designed to address the contextual challenges of doctoral education, its uniqueness and heterogeneity among others. Importantly, these systems are designed from a human-centered perspective using value-sensitive design (VSD) methods - an approach that is particularly relevant given the ethical nature of the problem. Thus, the talk can also serve as an example of how these techniques from the field of human-computer interaction and AI ethics can be applied to increase the acceptance and trustworthiness of LA and AI-based learning technologies.

1. A happier technology-enhanced PhD?
Value-sensitive design of
single-case learning analytics for
doctoral education
Luis P. Prieto
GSIC-EMIC research group
Universidad de Valladolid, Spain
SNOLA Webinar, 29.04.2024

2. The Team
•Luis P. Prieto, UVa
•Paula Odriozola-González, UCantabria
•María Jesús Rodríguez-Triana, UVa
•Yannis Dimitriadis, UVa
•Gerti Pishtari, DUK (Austria)
•Tobias Ley, DUK (Austria)
•...

3. Widespread problems of doctoral education
HIGH DROPOUT RATES
•About 50% of doctoral students never
finish their degrees
•Higher rates for online/distance
programs
EMOTIONAL HEALTH ISSUES
•Depression, anxiety, stress...
•Prevalences 10-25%… up to 40+% in
some studies
•Higher than in the general population
and other university students
Terrell, S. R., Snyder, M. M., Dringus, L. P., & Maddrey, E. (2012). A grounded
theory of connectivity and persistence in a limited residency doctoral
program. Qualitative Report, 17, 62.
Wollast, R., Boudrenghien, G., Van der Linden, N., Galand, B., Roland, N.,
Devos, C., De Clercq, M., Klein, O., Azzi, A., & Frenay, M. (2018). Who Are
the Doctoral Students Who Drop Out? Factors Associated with the Rate
of Doctoral Degree Completion in Universities. International Journal of
Higher Education, 7(4), 143. https://doi.org/10.5430/ijhe.v7n4p143
Evans, T. M., Bira, L., Gastelum, J. B., Weiss, L. T., & Vanderford, N. L. (2018).
Evidence for a mental health crisis in graduate education. Nature
Biotechnology, 36(3), 282–284.
Satinsky, E. N., Kimura, T., Kiang, M. V., Abebe, R., Cunningham, S., Lee, H., Lin, X.,
Liu, C. H., Rudan, I., Sen, S., Tomlinson, M., Yaver, M., & Tsai, A. C. (2021).
Systematic review and meta-analysis of depression, anxiety, and suicidal
ideation among Ph.D. students. Scientific Reports, 11(1), 14370.
https://doi.org/10.1038/s41598-021-93687-7

4. How big is the problem?
More details and references
athttps://ahappyphd.org/posts
/how-many-phd-students/

5. Related factors and the circle of control
•Many factors have been related to these two overlapping problems
oGender
oSocioeconomic (e.g.,wages, funding)
oDiscipline or Area of knowledge
oOrganizational (e.g., department/lab atmosphere)
oRelationship with supervisor
oPersonal situation (e.g., children, family obligations)
oCandidate skills
oMotivational factors
o...
What is under the control
of the individual student
(or amenable to our
interventions as
researchers)?
Sverdlik, A., C. Hall, N., McAlpine, L., & Hubbard, K. (2018). The PhD Experience: A Review of the Factors Influencing Doctoral Students’ Completion,
Achievement, and Well-Being. International Journal of Doctoral Studies, 13, 361–388. https://doi.org/10.28945/4113
Wollast, R.,Boudrenghien, G., Van der Linden, N., Galand, B., Roland, N.,Devos, C., De Clercq, M., Klein, O., Azzi, A., &Frenay, M. (2018). Who Arethe Doctoral
Students Who Drop Out? Factors Associated with the Rateof Doctoral Degree Completion in Universities. International Journal ofHigher Education, 7(4),
143.https://doi.org/10.5430/ijhe.v7n4p143

6. Motivational aspects and underlying problems
Uniqueness!
•of topic
•of process
•of student
•of context
•...
Devos, C., Boudrenghien, G., Van der Linden, N., Azzi, A., Frenay, M.,
Galand, B., & Klein, O. (2017). Doctoral students’ experiences leading
to completion or attrition: A matter of sense, progress and distress.
European Journal of Psychology of Education, 32(1), 61–77.

7. Large-scale,
heterogeneous,
wicked problems
No existing scalable, reliable,
personalizable solution
(technological or otherwise)

8. Goal: Can technology
help support doctoral
students?
Starting hypothesis:
Learning Analytics (LA) and
Human-AI collaboration help!

9. Example LA-HAIC pattern:
Single-case Learning Analytics (SCLA)
Prieto, L. P., Pishtari, G., Dimitriadis, Y., Rodríguez-Triana, M. J., Ley, T., & Odriozola-González, P. (2023). Single-case learning analytics: Feasibility of a human-
centered analytics approach to support doctoral education. JUCS - Journal of Universal Computer Science, 29(9), 1033–1068.
https://doi.org/10.3897/jucs.94067

10. How to design
ethicalLA/HAIC?
The value alignment
problem
•Any technology emphasizes certain values, to the
detriment of others
oE.g., gathering lots of data can help the app be
more engaging, at the cost of students'
privacy
•In the case of an AI, how can we ensure the AI acts
aligned with human values?
oExample: the "paper clip producing AI"
optimized for producing lots of paper clips
•Unsolved problem with a long history in AI
research, philosophy/ethics, …
oHuman values are messy!
•Some hints of solution (or at least, inquiry methods)
from the Human-Computer Interaction research
community...
Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., Zeng, F., Ng, K. Y., Dai, J., Pan, X., O’Gara, A., Lei, Y., Xu, H., Tse, B., Fu, J., …
Gao, W. (2023). AI Alignment: A Comprehensive Survey. https://doi.org/10.48550/ARXIV.2310.19852

11. Value-
sensitive
design (VSD)
•"Value Sensitive Design is a theoretically grounded
approach to the design of technology that accounts
for human values in a principled and comprehensive
manner throughout the design process."
•Recognition of technology as NOT being value neutral
--> it fosters/espouses certain values, hindering others
oWhich values? Privacy? Performance? Informed
consent? Calmness?
oWhose values? Designers? Shareholders? (direct)
Users?
•VSD tries to systematically inquire and design with
these questions in mind
•VSD has specific methods of data gathering/analysis
(see later for an example)
Friedman, B., Kahn, P., & Borning, A. (2002). Value sensitive design: Theory and methods. University of Washington technical report, 2(8).
Friedman, B., Hendry, D. G., Borning, A., & others. (2017). A survey of value sensitive design methods. Foundations and Trends® in Human–Computer Interaction,
11(2), 63–125.

12. What are
values?
•In VSD, values are "what is important to people in their
lives, with a focus on ethics and morality."
•Example values studied in VSD/HCI: privacy, ownership
and property, physical welfare, freedom from bias,
universal usability, autonomy, informed consent, and
trust.
•Values also studied a lot in cross-cultural psychology
(key characteristic of individuals and cultures)
•What distinguishes people (and cultures) is their
hierarchy (what is often preferred over what)

13. Hands-on exercise:
What are our values?
•Please answer this short
questionnaire. It should not take
more than 5 min
ohttps://tinyurl.com/SNOLA-VAL
•… let's look at the results!
Sandy, C. J., Gosling, S. D., Schwartz, S. H., & Koelkebeck, T. (2017). The Development and Validation of Brief and Ultrabrief Measures of Values. Journal of
Personality Assessment, 99(5), 545–555. https://doi.org/10.1080/00223891.2016.1231115

14. Schwartz's ten basic human values
•POWER: Social status and prestige, control or dominance over people and resources. (He likes to be in charge and tell others what to do. He
wants people to do what he says.)
•ACHIEVEMENT: Personal success through demonstrating competence according to social standards. (Being very successful is important to
him. He likes to stand out and to impress other people.)
•HEDONISM: Pleasure and sensuous gratification for oneself. (He really wants to enjoy life. Having a good time is very important to him.)
•STIMULATION: Excitement, novelty, and challenge in life. (He looks for adventures and likes to take risks. He wants to have an exciting life.)
•SELF-DIRECTION: Independent thought and action-choosing, creating, exploring. (He thinks it’s important to be interested in things. He is
curious and tries to understand everything.)
•UNIVERSALISM: Understanding, appreciation, tolerance and protection for the welfare of all people and for nature. (He thinks it is important
that every person in the world should be treated equally. He wants justice for everybody, even for people he doesn’t know.)
•BENEVOLENCE: Preservation and enhancement of the welfare of people with whom one is in frequent personal contact. (He always wants to
help the people who are close to him. It’s very important to him to care for the people he knows and likes.)
•TRADITION: Respect, commitment and acceptance of the customs and ideas that traditional culture or religion provide the self. (He thinks it is
important to do things the way he learned from his family. He wants to follow their customs and traditions.)
•CONFORMITY: Restraint of actions, inclinations, and impulses likely to upset or harm others and violate social expectations or norms. (He
believes that people should do what they’re told. He thinks people should follow rules at all times, even when no one is watching.)
•SECURITY: Safety, harmony and stability of society, of relationships, and of self. (The safety of his country is very important to him. He wants
his country to be safe from its enemies.)
Sagiv, L., & Schwartz, S. H. (2022). Personal Values Across Cultures. Annual Review of Psychology, 73(1), 517–546. https://doi.org/10.1146/annurev-psych-020821-
125100

15. What are values?
•In VSD, values are "what is important to people in their lives, with a focus on ethics
and morality."
•Example values studied in VSD/HCI: privacy, ownership and property, physical
welfare, freedom from bias, universal usability, autonomy, informed consent, and
trust.
•Values also studied a lot in cross-cultural psychology (key characteristic of
individuals and cultures)
•What distinguishes people (and cultures) is their hierarchy (what is often preferred
over what)
•Values are contextual! Somevalues we apply across contexts (e.g., Schwartz's basic
values), others we may apply in learning situations,others when we use technology...
Sagiv, L., & Schwartz, S. H. (2022). Personal Values Across Cultures. Annual Review of Psychology, 73(1), 517–546. https://doi.org/10.1146/annurev-psych-020821-
125100

16. Example VSD+LA/HAIC study
Study context
•Problem: dropout and emotional health in doctoral education
•Assumption: Perception of one's own progress is a critical factor in these
problems (something doctoral students consider very important)
•Goal: understand values of the doctoral students interested/affected by these
problems
•Research question:What values should a technology to support doctoral
progress embody?
oWhat do doctoral students value at the basic (i.e., across-situations), doctorate(i.e.,
learning experience) and technology use levels?
oHow do doctoral students experience progress, at the group-level and individually?
Prieto, L. P., Rodríguez-Triana, M. J., Dimitriadis, Y., Pishtari, G., & Odriozola-González, P. (2023). Designing Technology for Doctoral Persistence and Well-Being:
Findings from a Two-Country Value-Sensitive Inquiry into Student Progress. In O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.),
Responsive and Sustainable Educational Futures (Vol. 14200, pp. 356–370). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42682-7_24

17. One example VSD+LA study
Study methods
Prieto, L. P., Rodríguez-Triana, M. J., Dimitriadis, Y., Pishtari, G., & Odriozola-González, P. (2023). Designing Technology for Doctoral Persistence and Well-Being:
Findings from a Two-Country Value-Sensitive Inquiry into Student Progress. In O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.),
Responsive and Sustainable Educational Futures (Vol. 14200, pp. 356–370). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42682-7_24

18. One example VSD+LA study
Selected results RQ1 (student values)
Prieto, L. P., Rodríguez-Triana, M. J., Dimitriadis, Y., Pishtari, G., & Odriozola-González, P. (2023). Designing Technology for Doctoral Persistence and Well-Being:
Findings from a Two-Country Value-Sensitive Inquiry into Student Progress. In O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.),
Responsive and Sustainable Educational Futures (Vol. 14200, pp. 356–370). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42682-7_24
(lower values = higher priority)

19. One example VSD+LA study
Selected results RQ2 (experience of progress)
Prieto, L. P., Rodríguez-Triana, M. J., Dimitriadis, Y., Pishtari, G., & Odriozola-González, P. (2023). Designing Technology for Doctoral Persistence and Well-Being:
Findings from a Two-Country Value-Sensitive Inquiry into Student Progress. In O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.),
Responsive and Sustainable Educational Futures (Vol. 14200, pp. 356–370). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42682-7_24

20. Implications & limitations of the example study
•Some design implications of our VSD inquiry into doctoral student values:
–Self-Direction as the key basic value of "doctoral culture"
–Time tensions and task conflicts (e.g., with other jobs or non-thesis tasks) as a critical
struggle of our participants
–The quest for focus (vs. procrastination or interruptions) as another hurdle for progress
–Progress is uniquely experienced: allow for a wide variety of contextual factors and
direction of effect (+/-)
•Limitations
–Non-representative (but purposeful) sample: cannot generalize beyond these
participants/institutions/countries
–Inductive coding done by just one researcher
Prieto, L. P., Rodríguez-Triana, M. J., Dimitriadis, Y., Pishtari, G., & Odriozola-González, P. (2023). Designing Technology for Doctoral Persistence and Well-Being:
Findings from a Two-Country Value-Sensitive Inquiry into Student Progress. In O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.),
Responsive and Sustainable Educational Futures (Vol. 14200, pp. 356–370). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42682-7_24

21. Ongoing work: Piloting SCLA prototypes
•Socio-technical intervention: coaching/personalization interviews + diary/self-
tracking + reflection/motivational dashboard

22. Wider open questions
•How to includemore stakeholders (and their values) in the picture?
oE.g., supervisors in a doctoral process
•As researchers we often focus on ONE value in our work (e.g., privacy)
oWhat isthe possible catalog of values we should investigate/choose
from when designing LA and AI for education?
oAnd who gets to decide those? Researchers? Students? Institutions?
•How to navigate value tensions between conflicting values?
oE.g., privacy set at the societal level by GDPR vs. privacy as valued by
the individual student, vs.beneficence of more accurate models that
use more data

23. Thank you!
Questions?
More info at my blog, ahappyphd.org and our website, doctoraledtech.uva.es
This work has been supported by grant PID2020-112584RB-C32 and grant
RYC2021-032273-I, financed by MCIN/ AEI/ 10.13039/501100011033 and the
European Union's "NextGenerationEU/PRTR”. It has also been supported by
the Regional Government of Castile and Leon, under project grant VA176P23.