OSCE for UAHEB Makerere University(2).pptx

Makerere

1. OBJECTIVE STRUCTURED CLINICAL EXAMINATIONS/OBJECTIVE STRUCTURED PRACTICAL EXAMINATIONS AND HOSPITAL CLINICAL EXAMINATIONS BY LM LLMML

2. Session Objectives By the end of this session participants should be able to Organize OSCE/OSPE at institution level O rganize the hospital clinical examination Explain the role of the examiner in the assessment session Objectively mark practical examination Discuss the Dos and don'ts during the conduct and marking of practical examinations

3. Introduction OSCE/OSPE is a method of clinical/practical examination where by the candidates rotate through a series of stations at which they are asked to carry out a (usually clinical/practical) task.

4. Introduction OSCE/OSPE started in 1972 in Dundee , Scotland by R. Harden and F.Glesson Used in undergraduate as well as postgraduate. Formative and summative that is used in many disciplines Mainly used to measure clinical competence . Should be planned or structured (predetermined clinical competences).

5. Introduction Candidates are observed by one examiner in most of the stations. Scored as they carry out the task or interpret clinical materials , answer question. Can be used with large number of students.

6. Mr Y’s story about integrity Mr Y was a Tutor in School T . He loved money and women so much. Any female student who offered herself got exam papers. When offered money he would give out exam papers to even male students. One day his mother died in a health unit in the village. He discovered that his mother died due to incompetence and the incharge was his former student who used to give him momey for exams. Mr Y realized his mistakes and vowed to change after killing his mother.

7. ACTIVITIES 1. a . How do you organize and conduct OSCE/OSPE at the institution? b. What challenges do you meet in organizing and conducting OSCE/OSPE at the institution? 2. a. How do you organize and conduct hospital clinical examinations? b . What challenges do you meet in organizing and conducting hospital clinical examinations? 3. a. What should you keep in mind when setting OSCE/OSPE questions? b. How do you objectively mark practical examinations?

8. Organizing OSCE/OSPE at the institution What is to be assessed ? The OSCE/OSPE tasks should cover the core competences expected of the students. Once a station list is compiled, the stations can either be pulled from an item bank or prepared

9. Organizing OSCE/OSPE at the institution What is to be assessed ? OSCE stations should be of the integrated type that assess soft skills such as; Communication Ethics Attitudes Professionalism Clinical task The skills tested should be in line with the cadre.

10. EXAMPLE Please critique this question for laboratory technician STATION ONE Using the provided glass slides, identify the microorganisms in the provided specimen using appropriate tools and report your findings to the examiner Please critique this question for clinical officers, Nurses, Midwives among others STATION TWO Discuss the adverse effects the provided medicine on diabetic patients.

11. Organizing OSCE/OSPE at the institution What is to be assessed? Clear instructions should be written for examiners, examinees and patients. Scenarios should use the patients’ language and not overloaded with unnecessary detail.

12. Sample of a station A ) SCENARIO:  . A 30 -year-old, presented to your clinic with headache, generalized loss of interest in life, poor social performance, chest discomfort.

13. Sample of a station A) Instruction for the candidate You have 8 minutes to assess the emotional/mental state of this patient, by taking the proper and relevant history An observer/examiner, using a checklist, will assess your performance while you interact with the patient

14. Sample of a station B) Briefing to the examiner: The student is required to assess the emotional/mental state of Mr. Dan, by taking the proper and relevant history. Observe your students using the following checklist, without any probing or interference from you.

15. Sample of a station C) Briefing to the patient : Do not give more information than asked for. Try to answer the questions as clearly as possible. Try to be consistent with all the students as they interview you.

16. Sample of a station Checklist Student greeted the patient Introduced her/himself Asked the patient ’ s: Name Mood state Duration of mood disturbance Checklist Interest & enjoyment of life Daytime swings of mood Recent change in sleep pattern Change in appetite Suicidal thoughts or attempts

17. QUESTION How do you ensure that all stations require the stated time?

18. Organizing OSCE/OSPE at the institution Duration of station Its good idea to pre-test questions in order to determine how long it takes to test chosen competences effectively. Pre-testing is possible during formative assessment with a limited number of stations. Avoid questions that need a lot of time

19. QUESTION What do you consider when determining the number of stations?

20. Organizing OSCE/OSPE at the institution Number of stations Having more stations is better than less Its better to add many rest stations than having multiple circuits when students are many Multiple stations e.g each station can have more than one (parallel stations), so that a number of students are at the same station at the same time.

21. Organizing OSCE/OSPE at the institution Orientation and debriefing of the examinees, patients and examiners Orientation of examiners, examinees and patients is essential. Brief examiners and simulated patients early enough. Examiners , examinees and patients should be asked to arrive at the examination site at least one hour prior to the examination. S omeone must be available to guide examiners and patients to their stations and ensure that they know where to find any essential materials.

22. Organizing OSCE/OSPE at the institution It is also valuable to have a debriefing meeting of the organizers shortly after the examination to review all aspects of content and organization . More value will be added if feedback has been obtained from examiners, simulated patients and a representative of the examinees. This feedback provides useful information about the performance of each station and can assist decisions made to modify and improve stations before being banked for future use- Don’t repeat same stations from time to time.

23. Organizing OSCE/OSPE at the institution Considering number of students You have been appointed head of examiners at examination center X. There are 200 students to do practical examination. How do you control students to avoid cheating How do you arrange stations to save time? What if students are very few, how do you save resources? What if examiners are very few?

24. Organizing OSCE/OSPE at the institution Organization of the examination Arrange everything the day before and start in time Have as many stations as possible Keep students in one place Those who have finished should not be in touch with others No phones Monitor staff movements, no phones for staff Keep checking private places-rest rooms Staff should strictly speak english

25. Organizing OSCE/OSPE at the institution Organization of the examination For writing stations, let all students sit at once and do them first. That is, if students are very many or examiners very few. If students are very few use less examiners and have more circuits in order to save resources

26. Organizing OSCE/OSPE at the institution Resource requirements A document with details of required resources should be provided in advance and should be in place early enough. A meeting should be held with staff in charge at the institution to ensure everything required is in place. Resources required at each station should be clearly indicated e.g simulated patient, patient, equipment e.t.c Every station must have a number

27. Organizing OSCE/OSPE at the institution Resource These include: Candidate instructions. Examiner’s instructions, including grading. Simulated patient instructions including history to be given if relevant Equipment list, including disposables and sterile packs A scoring form/ checklists Required furniture

28. Organizing OSCE/OSPE at the institution Resource requirements-human Station developers Exam coordinator Examiners Support staff Simulated patients enough for each station

29. Organizing OSCE/OSPE at the institution Resource requirements Make sure equipment required are relevant- Do not set a question that needs microscopes for student clinical officers Do not set questions which require rare resources e.g grasshoppers.

30. Organizing hospital clinical examination QUESTION: What challenges do you meet when organizing and conducting hospital clinical examinations?

31. Organizing hospital clinical examination Who allocates cases Awarding of marks- big gaps between examiners Students’ readiness- most of them have no experience

32. The role of the examiner in the assessment session Contribution to the overall good conduct of the examination Awareness of the examination process Attendance at training for their role as an examiner Be attentive to candidates as soon as they enter the station Careful observation of the candidates' performance

33. The role of the examiner in the assessment session Score according to the marking guide provided Only prompt the student as required and indicated on their marking schedule or examiner instructions Maintenance of a passive behavior Prevention of students encroaching on other students' 'time'

34. The role of the examiner in the assessment session No mobile phones Upholding the confidentiality of the OSCE mark sheet content Restriction of unnecessary noise to a minimum Refrain from commenting about a candidate's performance to the simulated patient

35. The objective marking of practical examination The assessment is structured using prepared mark sheets or checklists with instructions. At each station, candidates’ performances are rated on checklists Scoring in a checklist is usually of the yes/no, done/not done type . Some examiners tick everything and its discovered when a student fails in other stations and writing stations- some teachers have been disqualified.

36. QUESTION What is the possibility that a student can get 10/10 in taking vital observations?

37. The objective marking of practical examination Like the checklist, a rating scale has a list of statements detailing the task to be performed. The difference is in the scoring style as examiners are interested in differentiating performance based on quality and level of mastery. The candidate will be rated across a scale such as, satisfactory, borderline, weak, not done. A mark is assigned for each level in the scale, like 2, 1.5, 1, 0.5 and 0

38. The objective marking of practical examination Consider situations when answers are two but only one appears on check list. E.g piriton -Chloropheramine Give marks and indicate in comments. Unless the question specified that a generic name is required Examiners should be knowledgeable in the content area being examined

39. Dos and don'ts during the conduct and marking of practical examinations Avoid setting theoretical questions that test only knowledge Consider tools required when setting exams. Allocate marks appropriately. Avoid questions requiring long procedures Be objective as an examiner Adhere to the rules set Do not cheat for a student Be a team player Do your best

40. Summary When used correctly, the OSCE can be highly successful as an instrument to assess competence

41. Training Examiners The programme should cover; Principles of OSCE/OSPE Role of Examiners Marking (group members act as candidates, SP and assessors) Standard setting procedures

42. Challenges of OSCE Although OSCE was developed to decrease Bias in the clinical competence, it is not without pitfalls of other type of Examinations. Because of limited time at each station, a narrow range of knowledge and skills is examined (unlike in long case exams) Objectivity of OSCE rest on standardization of the task and the scoring checklist for the stations. However differences between raters observing the same station has been recorded.

43. Challenges of OSCE iii) Unreliability may arise if students are tested on different patients for same examination. Some patients may be uncooperative, others may help some students. iv) It may be difficult to have a number of similar real patients. Simulated standardized patients can be used but training them for use in test situations incurs costs and manpower.

44. Challenges of OSCE v) Students, patients and examiners may get fatigued thus affecting scoring ability. vi) OSCE is resource intensive; more time, more money and more human resource are required compared to other traditional assessment methods.

45. CONCLUSION There is no assessment method that satisfies all the criteria of a good test. To ensure validity, objectivity and reliability, great care needs to be taken in its planning and administration. Learning objectives that need to be assessed over a number of stations, training academic staff and using appropriate checklist, OSCE/OSPE can be a reliable, valid, objective and practicable test method for any HTI for assessing several skills and competences in one Examination.