Educational Technology and Assessment in Health Professions Education

Plenary presentation at the NTTC-HP Interuniversity Workshop 19 June 2024, Manila Prince Hotel.

1. CC: BY NC ND

EDTECH ao

ASSESSMENT

Iris Thiele Isip Tan MD, MSc
Professor 12, UP College of Medicine
Director, UP Manila Interactive Learning Center
Chief, UP Medical Informatics Unit

@endocrine_witch 0

2. @%
Technology ii Open Book
enhanced — Examination
assessment
vOSCE

Ottawa consensus
(2022)

3. STAGE1 STAGE 2 STAGE 3

Assessing readiness to Applying technology Evaluating,
adopt technology to within the ‘assessment disseminating and
enhance assessment life cyle’ accelerating TEA

Fuller, R., Goddard, V. C. T., Nadarajah, V. D. Treasure-Jones, T., Yeates, P., Scott, K, .. Pyorala, E. (2022). Technology
enhanced assessment: Ottawa consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

4. Tech-enh ed Assessmen

Quality of existing assessment
practice

Purpose of introducing TEA

Engagement with digital
equity challenges

Institutional capacity to adopt
technology

Fuller, R., Goddard, V. C. T., Nadarajah, V. D. Treasure-Jones, T., Yeates, P., Scott, K, .. Pyorala, E. (2022). Technology
enhanced assessment: Ottawa consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

5. Tech-enh ed Assessmen

Quality of existing assessment
practice

Purpose of introducing TEA

Engagement with digital
equity challenges

Institutional capacity to adopt
technology

Fuller, R., Goddard, V. C. T., Nadarajah, V. D. Treasure-Jones, T., Yeates, P., Scott, K, .. Pyorala, E. (2022). Technology
enhanced assessment: Ottawa consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

6. Digital Equity

Two dimensions to digital
accessibility (UNESCO):

+ Technical: infrastructure for
connectivity and capacity of
devices

+ Socioeconomic: affordability
and students’ digital
capabilities

Fuller, R., Goddard, V. C. T., Nadarajah, V. D., Treasure-Jones, T., Yeates, P., Scott, K.,.. Pyorala, E. (2022). Technology
enhanced assessment: Ottawa consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

7. 4 P’s Approach

Programs: frame the context and
educational need

People (students, IT, faculty): buy-
in, participation and feedback are
crucial

Processes: review impact on data
governance, security, transfer and
reporting

Products: Align with scope of
work, accessibility, and affordability

Fuller, R., Goddard, V. C. T., Nadarajah, V. D., Treasure-Jones, T., Yeates, P., Scott, K.,.. Pyorala, E. (2022). Technology
enhanced assessment: Ottawa consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

8. 4 P’s Approach

Programs: frame the context and
educational need

People (students, IT, faculty): buy-
in, participation and feedback are
crucial

Processes: review impact on data
governance, security, transfer and
reporting

Products: Align with scope of
work, accessibility, and affordability

Fuller, R., Goddard, V. C. T., Nadarajah, V. D., Treasure-Jones, T., Yeates, P., Scott, K.,.. Pyorala, E. (2022). Technology
enhanced assessment: Ottawa consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

9. Assessme

© ©

Life Cycle

@ ©

Principle 1. Principle 2. Principle 3. Principle 4. Principle 5.
Advance Engage Enhance Optimize Track learner
assessment learners with design & assessment progress and
authenticity. assessment. scheduling of delivery 8 faculty
assessment. recording of activity.
learner

Fuller, R, Goddard, V. C.T., Nadarajah, V. D., Treasure-Jones, T., Yeates, P,

Scott, K.... Pyorala, E. (2022), Technology enhanced assessment: Ottawa achievement.
Consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

10. Assessme

© ©

Life Cycle

@ ©

Principle 1. Principle 2. Principle 3. Principle 4. Principle 5.
Advance Engage Enhance Optimize Track learner
assessment learners with design & assessment progress and
authenticity. assessment. scheduling of delivery 8 faculty
assessment. recording of activity.
learner

Fuller, R, Goddard, V. C.T., Nadarajah, V. D., Treasure-Jones, T., Yeates, P,

Scott, K.... Pyorala, E. (2022), Technology enhanced assessment: Ottawa achievement.
Consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

12. Assessme

© ©

Life Cycle

@ ©

Principle 1. Principle 2. Principle 3. Principle 4. Principle 5.
Advance Engage Enhance Optimize Track learner
assessment learners with design & assessment progress and
authenticity. assessment. scheduling of delivery 8 faculty
assessment. recording of activity.
learner

Fuller, R, Goddard, V. C.T., Nadarajah, V. D., Treasure-Jones, T., Yeates, P,

Scott, K.... Pyorala, E. (2022), Technology enhanced assessment: Ottawa achievement.
Consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

13. Assessme

© ©

Life Cycle

@ ©

Principle 1. Principle 2. Principle 3. Principle 4. Principle 5.
Advance Engage Enhance Optimize Track learner
assessment learners with design & assessment progress and
authenticity. assessment. scheduling of delivery 8 faculty
assessment. recording of activity.
learner

Fuller, R, Goddard, V. C.T., Nadarajah, V. D., Treasure-Jones, T., Yeates, P,

Scott, K.... Pyorala, E. (2022), Technology enhanced assessment: Ottawa achievement.
Consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

14. Technology futurists point to ‘human only
traits’ (e.g. compassion, creativity) as

increasingly valuable as we learn, and provide
healthcare, using more automation.

-Leonhard, 2021

Singko ka! | oh

\ Ib

How we ‘assess’ these traits, practice-based
ethics, and decision making (and abdication
from automated systems) in our co-existence
with technology will prove major challenges for
assessment in HPE.

Fuller, R., Goddard, V. C. T,, Nadarajah, V. D., Treasure-Jones, T., Yeates, P.,
Scott, K... Pyorala, E. (2022). Technology enhanced assessment: Ottawa
consensus statement and recommendations. Medical Teacher, 44(8), 836-850.

15. @%
Technology ii Open Book
enhanced — Examination
assessment
vOSCE

Ottawa consensus
(2022)

16. irtual OSCE (vOSCE)

Systematic review: 17 studies
Synthesized using
meta-ethnography

1190 candidates and assessors from
faculties of medicine, dentistry,
nursing, pharmacy, and osteopathy

Chan SCC, Choa G, Kelly J, Maru D, Rashid MA. Implementation of virtual OSCE in
health professions education: A systematic review. Med Educ. 2023 Sep;57(9):833-843.

17. Suitable assessment of verbal
communication skills

Inadequate assessment of physical
examination skills

Diminished rapport

Standardization of examination
process

Chan SCC, Choa G, Kelly J, Maru D, Rashid MA. Implementation of virtual OSCE in
health professions education: A systematic review. Med Educ. 2023 Sep;57(9):833-843.

18. vOSCE: Process

Improved accessibility

Unpredictable logistical
challenges

Impact on resources

Challenges with assessment
security

Chan SCC, Choa G, Kelly J, Maru D, Rashid MA. Implementation of virtual OSCE in
health professions education: A systematic review. Med Educ. 2023 Sep;57(9):833-843.

19. vos

Confidence

Reduced intimidation

Anxiety associated with
technology

Chan SCC, Choa G, Kelly J, Maru D, Rashid MA. Implementation of virtual OSCE in
health professions education: A systematic review. Med Educ. 2023 Sep;57(9):833-843.

20. vOSCE: Future Role

Appropriate temporary
alternative for face-to-face
OSCEs

Increased relevance to
telehealth

Assimilation of virtual OSCEs

Chan SCC, Choa G, Kelly J, Maru D, Rashid MA. Implementation of virtual OSCE in
health professions education: A systematic review. Med Educ. 2023 Sep;57(9):833-843.

21. Telemedicine Cart

Telemedicine cart for OSCE (nurse
practitioner students) to conduct a
virtual health care evaluation

Cart equipment: camera,
stethoscope, otoscope, etc

Patient (student)
Tele-presenter (faculty)

Provider (student)
Evaluator

Chrostowski, S. K, & Tietze, M. (2022). Using a Telemedicine Cart for an Objective
Structured Clinical Examination (OSCE) in Nurse Practitioner Education. Clinical
‘Simulation in Nursing, 70, 21-27. 2

22. Telemedicine Cart

Telemedicine cart for OSCE (nurse
practitioner students) to conduct a
virtual health care evaluation

Cart equipment: camera,
stethoscope, otoscope, etc

Patient (student)
Tele-presenter (faculty)

Provider (student)
Evaluator

Chrostowski, S. K, & Tietze, M. (2022). Using a Telemedicine Cart for an Objective
Structured Clinical Examination (OSCE) in Nurse Practitioner Education. Clinical
‘Simulation in Nursing, 70, 21-27. 2

23. vOSCE: Dentistry

Harvard School of Dental Medicine

OSCE results (n=241) from
2016-2022: 170 in-person vs 71
virtual OSCE

OSCE scores were not significantly
different

kim, J. Y., AlDosari, M. A, & Park, S. C. (2022). Application of OSCE using a Virtual
Assessment Platform. Journal of Education and Training Studies, 11(1), 27-27.

24. 1 felt prepared for the virtual OSCE 46% 49% B%3%

The virtual OSCE was effective 34% 60%

1 felt had enough time

during the virtual OSCE 6% 9%

The virtual environment was

easy to navigate 68% 30%

My laptop/tablet was adequate

for the virtual OSCE un 17% 8%
Iwas able to access the models oa Le
and materials needed J %
| would like to see the virtual OSCE a 3
offered in the future, post-COVID-19 he 51% 6% 6%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

mStrongly Agree mSomewhat Agree mNeutral #Somewhat Disagree Strongly Disagree

Students

Kim, J. Y., AlDosari, M. A, & Park, S. C. (2022). Application of OSCE using a Virtual
Assessment Platform. Journal of Education and Training Studies, 11(1), 27-27.

vOSCE: Dentistry

25. proctored in the PCLC lab GS Ban 5%
| would have preferred an
in-person OSCE

tops in a inual OS EM + 23%
participate in a virtual OSCE
| could focus on students
more na vituol OSCE SLA 7%

with an in-person OSCE i 2
a
experience to an in-person OSCE MM Gus aie ES
| would recommend the
virtual OSCE format

| would like to see the virtual OSCE A
offered in the future, post-COVID-19 21% 14%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

15% 15% 15% 8%

mStrongly Agree mSomewhat Agree mNeutral = Somewhat Disagree — Strongly Disagree

vo Dentistry

Evaluators

Kim, J. Y., AlDosari, M. A, & Park, S. C. (2022). Application of OSCE using a Virtual
Assessment Platform. Journal of Education and Training Studies, 11(1), 27-27.

26. Accessibility [EPA

Stress on student
participants

Stress on faculty
participants

Scheduling

Digital instead of
physical models

Convenience
Timekeeping

Organization

Interacting of
with students 122 14% 29%

0% 10% 20% 40% 50% 60% 70% 80% 90% 100%

Much prefer virtual = Somewhat prefer virtual = Neutral = Somewhat prefer in-person Much prefer in-person

Evaluators vOSCE: Dentistry
Kim, J. AlDosar, M.A, & Park, S.C. (2022) Application of OSCE using a Virtual

Assessment Platform. Journal of Education and Training Studies, 11(1), 27-27.

27. vos

rmacy

Zoom: formative then summative

Scenario: counsel a patient on new
medication dispensed ina
community pharmacy setting

Communication-based
assessment: introduction, patient
history, patient education, teach
back, and a communication barrier

Wollen, J. T., Gee, . S., Nguyen, K. A, & Surati, D. D. (2023). Development of a communication-based virtual patient
counseling objective structured clinical examination (OSCE) for first year student pharmacists. PEC innovation, 3, 100215.

28. Prescription Label

Wollen, J. T., Gee, J. 5, Nguyen, K. À, & Surati, D. D. (2023). Development of a
communication-based virtual patient counseling objective structured clinical
examination (OSCE) for first year student pharmacists. PEC innovation, 3, 100215.

Pharmacy

29. vos

Pharmacy

Zoom format reduces time with
negligible impact on student
performance vs in-person
alternative: 121 students in 157
min

Standardized patients were
able to interact with the
students virtually

Normalize a telehealth
encounter to students

Wollen, J. T., Gee, . S., Nguyen, K. A, & Surati, D. D. (2023). Development of a communication-based virtual patient
counseling objective structured clinical examination (OSCE) for first year student pharmacists. PEC innovation, 3, 100215.

30. vOSCE: Rheumatolo:

5 fellowship training programs

vROSCE (virtual rheumatology
OSCE):
+ Simulated patient encounter(n=1)
+ Giving advice via telephone to a
colleague (n=2)
+ Oral examination-style stations (n=3)
+ Removed interactive rehab station
with ambulatory aids & procedural
station

Wolfe, R. M. Hant, F. N, Ishizawar, R. C., Criscione-Schreiber, L. G., Jonas, B. L., O'Rourke, K.S., &
Bolster, M. B. (2023). Virtual Learning and Assessment in Rheumatology Fellowship Training:
Objective Structured Clinical Examination Revisited. Arthritis care & research, 75(12), 2435-2441.

31. vOSCE: Rheumatolo:

23 fellows completed the vROSCE

Fellows received immediate
formative feedback using
standardized rubrics

Fellows interacted with more
faculty from other (geographically
diverse) institutions with unique
expertise and teaching skills

Wolfe, R. M. Hant, F. N, Ishizawar, R. C., Criscione-Schreiber, L. G., Jonas, B. L., O'Rourke, K. S., &
Bolster, M. B. (2023). Virtual Learning and Assessment in Rheumatology Fellowship Training:
Objective Structured Clinical Examination Revisited. Arthritis care & research, 75(12), 2435-2441.

32. @%
Technology ii Open Book
enhanced — Examination
assessment
vOSCE

Ottawa consensus
(2022)

33. Trinary??

Open Book

34. Tools

Software/
hardware to do or
make task easier,
more efficient, or

higher quality

Information

Books, notes,
websites etc

Dawson, P., Nicola-Richmond, K, & Partridge, H. (2023), Beyond open book versus closed book:a
taxonomy of restrictions in online examinations. Assessment Evaluation in Higher Education, 1-13,

35. Information ROM Tools
None, specified list, — No tools,
offline only, full openness specific tools allowed,
minus prohibited list, People specific tools disallowed,
full openness all tools allowed

Dawson, P., Nicola-Richmond, K, & Partridge, H. (2023), Beyond open book versus closed book:a
taxonomy of restrictions in online examinations. Assessment Evaluation in Higher Education, 1-13,

36. Learning Feasibility Authenticity Values
Outcomes
Consequential
validity

Dawson, P., Nicola-Richmond, K., & Partridge, H. (2023). Beyond open book versus closed book: a
taxonomy of restrictions in online examinations. Assessment & Evaluation in Higher Education, 1-13.

37. Learning Feasibility Authenticity Values
Outcomes
Consequential
validity

Dawson, P., Nicola-Richmond, K., & Partridge, H. (2023). Beyond open book versus closed book: a
taxonomy of restrictions in online examinations. Assessment & Evaluation in Higher Education, 1-13.

38. ROOBE

Remote online open book
examinations (ROOBE)

Assess students’ higher order
thinking skills

Concerns about validity and
reliability if exams are not
proctored

Hui Meng Er, Pei Se Wong, & Vishna Devi Nadarajah. (2023). Remote online open book examinations: through the
lenses of faculty and students in health professions programmes. BMC Medical Education (Online), 23(1).

39. Faculty: ROOBE

Validity: assessment blueprinting,
faculty competencies, quality
assurance

Reliability: student performance,
standard setting

Educational impact: promotion of
higher-order thinking skills

Hui Meng Er, Pei Se Wong, & Vishna Devi Nadarajah. (2023). Remote online open book examinations: through the
lenses of faculty and students in health professions programmes. BMC Medical Education (Online), 230).

40. Faculty: ROOBE

Acceptability: recognition by
stakeholders, change
management, impact of COVID-19
pandemic, academic integrity

Cost: IT software, internet
connection, technical support,
faculty time

Hui Meng Er, Pei Se Wong, & Vishna Devi Nadarajah. (2023). Remote online open book examinations: through the
lenses of faculty and students in health professions programmes. BMC Medical Education (Online), 230).

41. 100% 93%
90%
80%
70%
60%
50%
40% 35%
30%

Number of students

20% 12%
10%
0%
0% —
Desktop Laptop Smart Tablet Smart Phone Other device(s)

Fig. 1 Types and frequencies of devices used by students during ROOBE
ROOBE: Stud s

Hui Meng Er, Pei Se Wong, & Vishna Devi Nadarajah. (2023). Remote online open book
‘examinations: through the lenses of faculty and students in health professions programmes.
BMC Medical Education (Online), 23(1).

42. (A) Were questions challenging? (8) Questions required me to apply
knowledge in real life situations

nor calnging ata song agree
suite changing Disagree
very challenging mare
Strong agree
(C) Questions tested intended (D) Quesions tested what | learned
learning outcomes in T&L activities
‘Sonal dire irony agree
sbiagree Disagree
agree agree
Strong ares Song are

Fig. 2 Students perceptions of the questions in ROOBE

Hi eng E, Pei Se Wong & Vin Devi Nadarajah (2023, Remote online open book .
txaminations: through he lenses of faulty and students in heath professions programmes ROOBE: Students

BMC Medical Education (Online), 23(1)

43. 100%

Number of students

90%
80%
70%
60%
50%
40%
30%
20%

10%

0%

The time was not sufficient

13%

You could answer the
questions without
referring to them

38%
i >
‘The answers were not Other reason(s)
available in the learning
materials/resources

Fig. 3 Students’ reasons for not referring to any learning resources during ROOBE

Hui Meng Er, Pei Se Wong, & Vishna Devi Nadarajah. (2023). Remote online open book
‘examinations: through the lenses of faculty and students in health professions programmes.

BMC Medical Education (Online), 23(1)

ROOBE: Students