Evaluating Assessment in medical education.pptx

evalaution of medical education

1. Evaluating Assessment Afsheen Zafar MBBS, FCPS (General Surgery), MHPE Professor of Surgery, IIMC, Rawalpindi.

2. Evaluation ‘‘the act of judgement of the worth of . . .’’. (Collin’s English Dictionary)

3. “the systematic process of determining the extent to which instructional objectives are achieved” (Gronlund, 1976)

4. ‘‘an examination conducted to assist in improving a programme and other programmes having the same general purpose’’. (Cronbach et al, 1980)

5. PROGRAM “A planned series of future events or performances.” Individual class session Course Clerkship rotation Whole educational program

6. Assessment vs evaluation assessment Measurement of student performance evaluation Process of obtaining information for judgment or decision making in future. Broader term May use assessment results.

7. The assessment cycle Fowell SL, Southgate LJ, Bligh JG. Evaluating assessment: the missing link?. Medical education. 1999 Apr 1;33(4):276-81.

8. Why evaluate assessment? Influences quality of assessment Determines effectiveness of assessment Can be accreditation need

9. STEPS IN EVALUATION (HOW?)

10. STEPS IN EVALUATION OF A PROGRAM Planning & preparation Data collection & analysis Interpretation/ reflection Reporting

11. Planning and preparation Starts before the program Define stakeholders Define purpose & goals Define evaluator’s role Choosing the evaluation questions Choosing appropriate methodology

12. Common methods of evaluation quantitative Pre-post test designs Post test only designs Interrupted time series Comparison group designs Cost analysis qualitative Case studies Action research approach Naturalistic and ethnographic approaches

13. Data collection quantitative Achievement testing Attitude scales Rating scales Questionnaires Observation schedules qualitative Interviews Observations Diaries/ self reports Documentary analysis

14. DATA ANALYSIS & INTERPRETATION Analysis Explanation

15. Interpreting findings

16. REPORTING Students Other stakeholders Decision makers

17. Evaluation Cycle

18. Levels of Assessment Evaluation Long term view

19. First Level Looks at individual Questions and their quality Inbuilt item analysis Difficulty index Discrimination index Clarity of question statements Matching with LOs

20. Second Level Validity assessment Content validity (usually built in if blue printing has already been done) Criterion relationships Construct evidence Consequence of using (either for knowledge or for skills) Modern concept is of a unified single concept rather than the traditional view of separate types of validity assessment

21. Tests of Reliability Measure of internal consistency with Cronbach alpha Content specificity

22. Third Level Usually to identify issues around validity of the assessment items Done by external assessors/examiners Can be useful to examine or review the pass/fail standards as well before implementation

23. Fourth Level Identifying larger areas where students are performing badly and identifying the cause May result in reviewing and revising assessment methods of that particular area. May initiate another assessment cycle

24. Assignment Think of an evaluation question that you might like to ask for evaluating your assessments. Identify your stakeholders What methodology will you use to evaluate this question?