Critique_ Scenario Based MCQs FOR MEDICAL PHYSIOLOGY STUDENTS.pptx

MCQS ON SCENARIO BASED ON VARIOUS MEDICAL CONDITIONS TO TEST PHYSIOLOGICAL PRINCIPLES

1. COMPETENT IMG: HOW YOU ASSESS IS WHAT MATTERS Dr. Sandeep Kadu Dr. Payal Bansal Dr. Supriya Dhakne-Palwe MUHS Nashik Dr. Praveen B Iyer Seth G S Medical College, Mumbai Dr. Yanamandra Uday Armed Forces Medical College, Pune Dr. Gokul Jorvekar Rural Medical College, Loni

2. Scan this QR Code for live interaction

3. Introduction to Written Assessment

4. Assessment of Students Introduction to Written Assessment Dr.Payal Bansal MUHS Chair Professor, Medical Education

5. Session Objectives Review purpose and characteristics of an effective assessment Explore the role of written methods to assess competence (knowledge and performance) Analyse written methods in the context of UTILITY of an in the context of assessing professional

6. Purpose of assessment What are the purposes of assessment?

7. Purpose of assessment Judge mastery of essential skills and knowledge Measure improvement in student performance over time Make pass/fail decision Rank students

8. Purpose of assessment Give feedback & diagnose their difficulties Evaluating teaching methods Evaluating effectiveness of course Motivating students

9. Effective assessment Groups of 4: What are the characteristics of an effective assessment? Write list on flip chart

10. Effective Assessment The assessment system must be based on the purpose of the course Purpose of the course must be expressed in the curriculum. Assessment , therefore, must be based on the curriculum.

11. Important Formats Formative Assessment Summative Assessment

12. Types of Assessment Norm Referenced Criterion Referenced

13. Effective assessment What are the characteristics of a “ good ” assessment? Write list and share with your partner – 2 minutes

14. Effective assessment? Representative sampling of performance Variety of assessment methods Uniform test and procedures for all Explicit, specific criteria in judging and grading successful performance Feasible Timely feedback

15. Miller’s Pyramid Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. Knows Shows how Knows how Does Written, Oral or Computer based assessment Performance or hands on assessment

16. Selection Methods MCQ Matching Extended Matching True False Multiple True False Knows Knows how Shows how Does

17. Supply Methods MEQ Essay (long & short) Practical Short Answer Written simulations Knows Knows how Shows how Does

18. Written Assessment tools Strengths and Limitations

19. Methods - Knowledge Long Answer Short Answer Essay/ Modified Essay Question ISSUES - Unstructured (Subjectivity) , Inadequate Sampling STRENGTH: All cognitive levels can be covered

20. Methods - Knowledge Multiple Choice Questions Needs Systematic Coverage of all topics Training is necessary : Writing Quality Control STRENGTH – Structured, Adequate Sampling, Most Cognitive levels can be covered

21. Cognition as Performance Assessment of Cognitive Skills Ability to analyse, interpret , synthesize Critical Thinking Clinical decision making Planning & Organising the Management

22. Utility of written assessment tools Reliability - MCQ Validity - MCQ Educational Impact MCQ Feasibility - MCQ Acceptability - MCQ

23. ASSESSMENT METHODS All assessment methods have limitations and no one method can assess all types of knowledge & skill Multiple methods are recommended

24. Summary Define carefully, the level of performance expected. A variety of tools measurements of performance that are reliable and valid NOT A SINGLE METHOD IS BEST OR BAD A good assessment uses multiple encounters, observed multiple times a mix of methods that depend on the context as defined by the curriculum

25. Assessment Exercise Construct an assessment to evaluate the ability of final year students to interpret common X-Ray presentations that they are likely to see as primary care physicians. Design this assessment keeping in mind assessment design principles.

26. THANK YOU!

27. Introduction to Objective forms of assessment ´ Objectivizing assessment in medical education ´Essay…> MCQs ´Newer forms of objective assessment ´OSCE ´OSPE ´OSAV ´MCQs: most common form of objective written assessment Used in All fields

28. Debate on MCQs

29. Strengths of MCQs ´ Valid, reliable, high educational impact, acceptable ´ If properly constructed- higher cognitive domains ´ Greater sampling ´ Quick and easy marking ´ Objective assessment ´ Familiar to students and teachers ´ Factors irrelevant to assessing knowledge (handwriting, presentation, grammar)- eliminated ´ Amenable to modification- item analysis ´ Difficulty level and discriminating power- precise calculation ´ Question banking

30. Scanning MCQs through assessment criteria Valid: ´ Learning objectives: content and level ´ Post-validity: item analysis Reliable : 100% as objective Educational/ learning impact: high as students tend to focus and read in depth Acceptability Cost-effective

31. Why the Criticism? Always pick answer C and you’ll be right about 25% of the time

32. Weaknesses of MCQs ´ Recognition of answer ´ Susceptible to guessing ´ Recall of facts ´ Misinterpretation by students ´ Time consuming to construct ´ Only cognitive domain

33. Salient Features of NeXT Regulations

34. NExT Regulations 27 th June 2023

35. NExT Regulations- 27 th June 2023

36. General Principles of MCQ Formulation Recap- rBCW / BCME

37. Learning objectives Level of cognitive domain Level of learner/ test taker Must Know/ Desirable/ Nice to know Determine the type of MCQ General rules Specific component-wise rules Guidelines in writing flawless MCQs

38. General rules Determine the Content area (Learning objectives) Identifying the most important and relevant topics Avoiding testing the knowledge of medical trivia Decide the Level of cognitive domain… preferably higher Source of questions 1 concept at a time Vertical formatting Guidelines in writing flawless MCQs

39. Taxonomy of cognitive domain Level 1: Knowledge/ Recall of facts Level 2: Understanding/ interpretation of data Level 3: Problem solving abilities

40. COMPONENTS OF MCQs DIRECTION ITEM STEM ALTERNATIVES / RESPONSES / CHOICES - KEY DISTRACTORS

41. 1. The organism that causes trichomonal infection is a A. bacteria B. fungus C. protozoa D. spirochete Each of the questions below is followed by four alternatives. Select the best or the most appropriate answer and encircle the alphabet corresponding to it.

42. Each of the questions below is followed by four alternatives. Select the best or the most appropriate answer and encircle the alphabet corresponding to it. 1. The organism that causes trichomonal infection is a bacteria B. fungus C. protozoa D. spirochete

43. Each of the questions below is followed by four alternatives. Select the best or the most appropriate answer and encircle the alphabet corresponding to it. 1. The organism that causes trichomonal infection is a bacteria B. fungus C. protozoa D. spirochete

44. Each of the questions below is followed by four alternatives. Select the best or the most appropriate answer and encircle the alphabet corresponding to it. 1. The organism that causes trichomonal infection is a bacteria B. fungus C. protozoa D. spirochete

45. Each of the questions below is followed by four alternatives. Select the best or the most appropriate answer and encircle the alphabet corresponding to it. 1. The organism that causes trichomonal infection is a bacteria B. fungus C. protozoa D. spirochete

46. Components of a MCQ: Stem – the information for the question Lead-in – the question/ what to answer Alternatives/ Options: 1. Key/ the correct alternative 2. Distracters -- the incorrect alternative Entire MCQ: Item! Anatomy / Structure of Scenario based MCQ

47. Shape of the MCQ

48. For determining whether maternal malnutrition is a risk factor for low birth weight, 100 low birth weight and 100 normal birth weight babies were selected and their mothers were interviewed about their diet during pregnancy. Identify the study design. a. Cross-sectional study b. Case-control study c. Cohort study d. Randomized controlled trial Anatomy / Structure of MCQ STEM LEAD-IN DISTRACTERS KEY

49. Specific directions for writing different components of MCQs

50. Should be long relative to options Clarity in words and direction. Unambiguous Not tricky or misleading- deceive the student Complete information about the question asked Construction of STEM

51. Well-constructed single best response questions satisfy the “cover-the-options” rule. The questions could be administered as write in questions. E.G. A 52-year-old man has had increasing dyspnea and cough productive of purulent sputum for 2 days. He has smoked one pack of cigarettes daily for 30 years. Temperature is 37.2°C (99.0°F). Breath sounds are distant with a few rhonchi and wheezes. Leukocyte count is 9000/mm3 with a normal differential. Gram stain of sputum shows numerous neutrophils and gram-negative diplococci. X-ray of the chest show hyperinflation. Which of the following is the most likely diagnosis? Construction of STEM

52. Short format versus Clinical scenario (Vignettes) Higher cognitive domain Critical thinking, decision making Contextual/ real-life situation Integration of various disciplines Construction of STEM

53. Construction of STEM Phrase repeated in each option Which of the following would decrease radiation dose by half? A. Decreasing mA by ¼ B. Decreasing mA by 1/3 C. Decreasing mA by ½ D. Decreasing mA by ¾ Place all common elements in the stem By what fraction would mA need to be decreased to lower the radiation dose by half? A. ¼ B. 1/3 C. ½ D. ¾

54. Construction of STEM Use negatives sparingly Negative mindset…… negative educational impact NOT/ EXCEPT Which of the following is not a characteristic CT finding of small airway disease? Which one of the following is NOT a characteristic feature of osteoporosis? (a) Decreased bone density (b) Frequent bone fractures (c) Raised body temperature (d) Lower back pain If inevitable, BOLD / Underline the negative phrase “Which of the following is NOT a characteristic CT finding of small airway disease?” “Which of the following best distinguishes small airway disease from interstitial lung disease on chest CT?”

55. No double negatives: Negative options after a negative stem E.g. Typhoid fever is characterized by all except : Lymphopenia absence of l eucocytosis … Stepladder fever Pea-soup diarrhoea Typhoid fever is characterized by all EXCEPT : Lymphocytosis Leucocytosis Stepladder fever Pea-soup diarrhoea Construction of STEM Which procedure is NOT recommended for the victim of an accident? Do not leave the victim alone. Do not move the victim. Notify the police. Do not talk to the victim.

56. Verbosity, Window Dressing, and Red Herrings: unnecessary, superfluous information ‘verbosity’ ( ie , extra words),“window dressing” ( ie , extraneous material), and “red herrings” ( ie , information designed to mislead the examinee). You see in your office a 24 year old woman for her routine postpartum visit, at 6 weeks. Her baby is irritable and cries “all the time’ (including during her visit.) As she is changing the diaper during her visit, you notice that the child has a very red bum but upon questioning, she tells you she did not notice anything abnormal. Her partner is away in Afghanistan and she does not have a lot of friends, as they just recently were posted in Kingston. She states that she has a hard time getting dressed in the morning, cries easily and feels like she does not love her child. She denies any medical or psychiatric history. Her exam is essentially normal. She weighs 2 kg less than at her first antenatal visit at 10 weeks. You most likely clinical diagnosis is: Postpartum hypothyroidism Hyperthyroidism Postpartum blues Postpartum depression Psychotic depression Construction of STEM

57. Avoid grammatical clues in stem Some options start with a vowel Not all options follow from the stem Plurals in stem/options e.g. A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of A. examination of cerebrospinal fluid B. glucose with vitamin B1 (thiamine) C. CT scan of the head D. phenytoin E. diazepam Construction of STEM

58. Construction of STEM Avoid asking two questions/ concepts at the same time Example: Which one of the following occurs in nephrotic syndrome and small cell carcinoma? (a) Hyponatremia (b) Hypernatremia (c) Hypokalemia (d) Hyperkalemia

59. Construction of STEM Do not use eponyms and abbreviations Example: Which one of the following is NOT a complication of MS? (a) Vomiting (b) Weakness (c) Pulmonary oedema (d) Optic neuritis

60. Construction of STEM Avoid clues suggesting the correct answer Example: Which one of the following investigations is a measure of thyroid function? (a) Serum bilirubin (b) Thyroid stimulating hormone (c) Adrenocorticotrophic hormone (d) Prostate specific antigen

61. Write a question for the one-best-answer Do NOT use incomplete statements or opinions Be specific, avoid frequency terms (usually, often) Confirm answer to lead-in can be found from stem Which of the following is – abnormal? – most likely? – the most likely cause? – Defective/ deficient/ nonfunctioning? If ‘best answer’ format- most likely/most common/most probable/most appropriate/ First Drafting the lead-in

62. Write the key (the one best answer) first Keep option length fairly consistent Place them vertically Logical or numerical sequence Independent/ Mutually Exclusive/ Non-overlapping Positively phrased Developing the alternatives/ options General rules

63. One and only one non-ambiguous answer for the only one correct answer format Balance your key Placement of correct answer in the list of options C,c,c,c ,…. Acbd … acbd … acbd … Solution- list MCQs in alphabetical order Rules for developing the key (correct option)

64. Rules for the writing the distracters Easy/ difficult???

65. Use plausible distractors: Avoid obvious wrong/easy to eliminate options Use common student errors Closer the distracters to the key… more plausible… more chances of it getting selected over the right answer. Don’t go too close to the key Rules for writing the distracters Which one of the following type of RBCs is commonly seen in iron deficiency anaemia ? (a) Microcyte (b) Macrocyte (c) Normocyte (d) Lymphocyte

66. Rules for the writing the distracters Use only internally consistent alternatives Example: The predominant anion of intracellular fluid is (a) Cl - (b) HPO4- (c) HCO3- (d) Fat

67. AOTA and NOTA x Most commonly used and abused in MCQs AOTA: Strongly discouraged Recognition of 2 right options identifies the answer Recognition of 1 wrong option eliminates it NOTA: May be asked , specially if answer requires some degree of computation Rules for the writing the distracters

68. Absolute terms Never/ always: should not be used Relative terms: May/ Usually Relative frequency- often/ rarely: caution No abbreviations Rules for the writing the distracters In patients with advanced dementia, Alzheimer’s type, the memory defect A. can be treated adequately with phosphatidyl -choline (lecithin) B. could be a sequela of early parkinsonism C. is never seen in patients with neurofibrillary tangles at autopsy D. is never severe E. possibly involves the cholinergic system

69. Rules for the writing the distracters Ensure that there is one correct response Example: A raised serum level of LDH indicates the involvement of which one of the following organs? (a) Heart (b) Liver (c) Brain (d) Ovary

70. Rules for the writing the distracters Avoid overlapping of options Example: Which one of the following ranges is the normal value for serum bilirubin level (mg/ dL ) in an adult? (a) 0.2 – 0.8 (b) 0.6 – 2.0 (c) 1.0 – 3.0 (d) 2.0 – 3.6

71. Place numerical options in numerical order/ Logical order Normal haemoglobin of an Indian adult, non-pregnant female ranges between: Developing the alternatives/ options General rules 10-13 gm % 12-15 gm % 14-16 gm % 16-18 gm % 10-12 gm % 12-14 gm % 14-16 gm % 16-18 gm % 14-16 gm % 10-12 gm % 12-14 gm % 16-18 gm %

72. Developing the alternatives/ options General rules Specify units of lab investigations correctly Example: Jaundice is clinically detectable when the serum bilirubin concentration is (a) 0.5-1 (b) 1-1.5 IU/L (c) 1.5-2 KAU/L (d) >2 mg/ dL

73. Types of MCQs One MCQ test can have a combination of various types of MCQs

74. Time allocated for different types of MCQs SBR type 40 sec Multiple true/false type 45 -50 sec Relationship/analysis type 50 sec Multiple response type 50 sec Case history type 60-90 sec

75. Recap of MCQ formulation

76. TYPES OF MCQS Single Best Response Simple Completion Type Question Type Negative Form Type Case History Type Multiple Completion Type Independent True-False Type Matching Type Assertion Reason (Relationship Analysis) Type Pictorial Type

77. SIMPLE COMPLETION TYPE 1. The organism that causes trichomonal infection is a bacteria B. fungus C. protozoa D. spirochete Directions : Each of the questions below is followed by four alternatives . Select the best or the most appropriate answer and encircle the alphabet corresponding to it.

78. QUESTION TYPE Which stratum of epidermis contains keratin? A. Spinosum B. Granulosum C. Lucidum D. Corneum Keratin contains in _____ stratum of epidermis

79. NEGATIVE FORM TYPE A patient receiving aminophylline intravenously to relieve asthmatic attack may show all the following reactions EXCEPT A. Hypotension B. Vomiting C. Visual disturbances D. Hyperventilation

80. Higher order thinking Assessment drives learning Nifedipine was once drug of choice in hypertensive emergencies. However, Nifedipine is not preferred in hypertensive emergencies in a cardiac disorder patient. Which of the following best describes the reason for this? Heart blocks Reflex tachycardia Diastolic cardiac arrest Systolic cardiac arrest

81. Example : A 35-year-old male who presented with fracture of the shaft of right humerus, developed inability to extend his right wrist and also had a loss of sensation over the back of right forearm and lateral half of dorsum of right hand. The most likely nerve injured would be A. Musculocutaneous B. Median C. Radial D. Ulnar CASE HISTORY TYPE/ SCENARIO BASED

82. Directions for framing MCQs Purpose “IF YOU ARE NOT CERTAIN WHERE YOU ARE GOING, YOU MAY END UP ELSEWHERE”

83. Taxonomy of cognitive domain Level 1: Knowledge/ Recall of facts Level 2: Understanding/ interpretation of data Level 3: Problem solving abilities

84. Directions for framing MCQs Purpose Content Important part of syllabus

85. Directions for framing MCQs Purpose Content Stem Based on a single central problem Positive stem in a simple and clear language Negative stem: NOT or EXCEPT

86. Directions for framing MCQs Purpose Content Stem Choices Distractors : Plausible and grammatically consistent with the stem Order of numerical alternatives Cautious use of ‘ all of the above ’ & ‘ none of the above’ Any element common to all the distractors should be incorporated in the stem

87. Which procedure is NOT recommended for the victim of an accident? Do not leave the victim alone. Do not move the victim. Notify the police. Do not talk to the victim.

88. Which one of the following type of RBCs is always seen in iron deficiency anaemia ? Microcyte Normocyte Lymphocyte All of the above None of the above

89. Which one of the following type of RBCs is always seen in iron deficiency anaemia ? Microcyte Normocyte Lymphocyte All of the above None of the above

90. Triple Cs Concept Construction Corroboration

91. Concept of Scenario based SBR MCQ

92. Triple Cs Concept Construction Corroboration

93. Whom do you want as your competent IMG

94. Recall Vs Application of knowledge/Problem solving Recall items Seem clinically backwards “Which of the following findings is most likely to be seen in postsurgical patients with pulmonary embolism?” is an item structured similarly to most textbook questions. - recall isolated facts Problem-solving items Contextual and authentic Requires test-takers to read an item and identify relevant information, interpret that information in a certain context, integrate that information with what they already know, and then answer the question posed Does not depend on memory alone

95. Benefits of Application of Knowledge Item Type 1. The authenticity of the examination is greatly enhanced by using items that require test-takers to integrate information to “solve” clinical problems. 2. The items are more likely to focus on important information, rather than trivia. 3. These items help to identify those test-takers who have memorized a substantial body of factual information but are unable to use that information effectively in clinical situations. 4. Test-takers need to differentiate relevant from irrelevant information in the item.

96. Recall Versus Problem-solving MCQs A woman is diagnosed with venous thromboembolism. Which of the following is the most appropriate treatment? This is an example of a recall of a fact item, which requires examinees to simply recall the treatment of Venous thromboembolism

97. Recall Versus Problem-solving MCQs A 47-year-old woman comes to the emergency department because of shortness of breath and left lower extremity pain with ambulation. Yesterday, she returned from Europe after a 10-hour flight. She has no remarkable medical history and takes only an oral contraceptive daily. Vital signs are within normal limits. Physical examination shows asymmetrical swelling of the left calf of greater than 2 cm compared to the right side. The remainder of the physical examination discloses no abnormalities. This vignette provides more clinical context, requiring the examinee to recognize the historical and physical presentation of VTE. From this, a variety of questions could be asked that require the examinee to interpret data, including the following: Diagnosis Next step in determining the diagnosis (which requires clinical suspicion of diagnosis and the next test to order) Treatment (which requires clinical suspicion of diagnosis and determination of the most appropriate next step in management) Mechanism (which requires clinical suspicion of diagnosis, the most appropriate treatment, and the mechanism of that treatment)

98. Example: Recall Based Acute intermittent porphyria is the result of a defect in the biosynthetic pathway for Collagen Corticosteroid Fatty acid Glucose *Heme Thyroxine (T4)

99. Example: Problem Solving An otherwise healthy 33-year-old man has mild weakness and occasional episodes of steady, severe abdominal pain with some cramping but no diarrhea. One aunt and a cousin have had similar episodes. During an episode, his abdomen is distended, and bowel sounds are decreased. Neurologic examination shows mild weakness in the upper arms. These findings suggest a defect in the biosynthetic pathway for Collagen Corticosteroid Fatty acid Glucose *Heme Thyroxine (T4)

100. Recall Versus Problem-solving MCQs

101. Push you to think about the topic Nifedipine was once a drug of choice in hypertensive emergencies. However, Nifedipine is not preferred in hypertensive emergencies in a cardiac disorder patient. Which of the following best describes the reason for this? Heart blocks Reflex tachycardia Diastolic cardiac arrest Systolic cardiac arrest. In this question, Students need to think about the concept to answer. It is not a straightforward recall. Qualifies to be included as a scenario‑based MCQ It assesses the application of knowledge If an MCQ forces a learner to reach a conclusion, make a prediction, or select a course of action However, if the MCQ assesses only the rote memory of isolated facts without any application, then it is a simple recall‑based MCQ. There is a scope for using scenario‑based MCQs to provide feedback to students in formative assessment.

102. Recall Vs Application of knowledge/Problem solving First step - Develop an appropriate stimulus/context to introduce the topic, such as a clinical or experimental vignette If there is no such context, the resulting item will assess only knowledge recall. Use actual patient scenarios as vignette Avoid relying on or adhering too closely to patient cases because these often have atypical features that may divert from a typical or representative case and lead to confusion. Problem-solving MCQs- medium-to high-stakes summative examination

103. Pseudo scenario Understanding that scenario‑based MCQ means increasing the length of the question’s stem Assessors try to make a scenario by adding redundant words/sentences or doing unnecessary window dressing

104. Recall-based MCQ Which of the following is not characteristic of a reflex? Reflexes can be modified by impulses from various parts of the CNS Reflexes may involve simultaneous contraction of some muscles and relaxation of others Reflexes are chronically suppressed after spinal cord transection Reflexes involve transmission across at least one synapse Pseudo-scenario based MCQ While exercising, a 42‑year‑old female developed a sudden onset of tingling in her right leg and an inability to control movement in that limb. A neurological exam showed a hyperactive knee-jerk reflex and a positive Babinski sign. Which of the following is not characteristic of a reflex? Reflexes can be modified by impulses from various parts of the CNS Reflexes may involve simultaneous contraction of some muscles and relaxation of others Reflexes are chronically suppressed after spinal cord transection Reflexes involve transmission across at least one synapse

105. Recall-based MCQ Which of the following is not characteristic of a reflex? Reflexes can be modified by impulses from various parts of the CNS Reflexes may involve simultaneous contraction of some muscles and relaxation of others Reflexes are chronically suppressed after spinal cord transection Reflexes involve transmission across at least one synapse Pseudo-scenario based MCQ While exercising, a 42‑year‑old female developed a sudden onset of tingling in her right leg and an inability to control movement in that limb. A neurological exam showed a hyperactive knee-jerk reflex and a positive Babinski sign. Which of the following is not characteristic of a reflex? Reflexes can be modified by impulses from various parts of the CNS Reflexes may involve simultaneous contraction of some muscles and relaxation of others Reflexes are chronically suppressed after spinal cord transection Reflexes involve transmission across at least one synapse

106. Recall-based MCQ Which of the following is not characteristic of a reflex? Reflexes can be modified by impulses from various parts of the CNS Reflexes may involve simultaneous contraction of some muscles and relaxation of others Reflexes are chronically suppressed after spinal cord transection Reflexes involve transmission across at least one synapse Pseudo-scenario based MCQ While exercising, a 42‑year‑old female developed a sudden onset of tingling in her right leg and an inability to control movement in that limb. A neurological exam showed a hyperactive knee-jerk reflex and a positive Babinski sign. Which of the following is not characteristic of a reflex? Reflexes can be modified by impulses from various parts of the CNS Reflexes may involve simultaneous contraction of some muscles and relaxation of others Reflexes are chronically suppressed after spinal cord transection Reflexes involve transmission across at least one synapse Redundant sentences in the example Here, the use of a scenario seems unnecessary. The question which follows the scenario actually does not need it. Even without the scenario, this question can be asked. In fact, here, the time of the student is getting wasted in reading the scenario. The use of scenario should have a rationale and not just for the sake of adding it. Hence, this type of window dressing has to be avoided.

107. If not this way – Then how to frame the Vignette

108. Triple Cs Concept Construction Corroboration

109. Steps in Designing Scenario‑Based Multiple Choice Question

110. Starting Point

111. Step 1: The starting point Step 1: Choose a competency and learning objective from the curriculum on which you wish to frame MCQ. Is it possible that you have an objective in mind or you have experienced a clinical situation that led to the generation of ideas about MCQ. Any of the above can be the starting point for MCQ, for example, competency in Pharmacology PH 1.52‑Describe management of common poisoning, insecticides, common sting, and bites.

112. Stem

113. Step 2: Frame a stem The stem is the main body of the MCQ and provides the stimulus, trigger, or information needed to answer that MCQ. At the end of the stem, write a specific question, also known as a lead‑in. The lead‑in poses a specific question that tells the learner what is the task for the learner. The lead‑in should be specific, clearly asking the task to be done by learner. Avoid the use of negatives in the lead‑in (e.g., not, except).

114. Guidelines for clinical vignette content ´ Test application of knowledge using clinical vignettes to pose medical decisions in patient care situations ´ Focus items on common or potentially catastrophic problems ´ Pose clinical decision-making tasks that would be expected of a successful test-taker ´ Pose/craft clinical situations that would be handled by a provider/specialist ´ Focus on specific tasks that the successful test-taker must be able to undertake at the next stage of training or upon commencement of specialty practice Focus on areas in which clinical reasoning mistakes are often made

115. Guidelines for clinical vignette content The following can be used as a template for a patient vignette Not all of the following components are necessary , but when present should be in the order indicated: ´ Age, gender (eg, 45-year-old woman) (add option to indicate self-identified) ´ Site of care (e.g. the emergency department) ´ Presenting symptoms (e.g. headache) ´ Duration of symptoms (e.g. 2 days) ´ Patient history, including past medical history, family history, psychosocial history, and review of systems if important and plausible for the scenario ´ Physical findings ´ Results of diagnostic studies ´ Initial treatment, subsequent findings

116. Discrimination power of vignettes

117. Discrimination power of vignettes

118. Discrimination power of vignettes

119. Discrimination power of vignettes ´ “long vignette” - still relatively short in length ´ Clinical knowledge and science exams - test-takers to demonstrate proficiency in sorting through patient information , synthesizing the important findings , and reaching a conclusion . ´ As a result, these items may have extraneous information as well as the essential information to answer the question. ´ If there is concern about the vignette length, it is possible to synthesize findings with a statement such as, “The family history is noncontributory.”

120. Patient chart format

121. Patient chart format

122. In clinical subjects, the Stem can be written in the form of a clinical case scenario or clinical vignette. Patient vignettes can include information related to patient/ subject such as: Age, gender, sign/symptoms, any relevant history Physical examination findings, laboratory investigations.

123. Example for stem of MCQ A 30‑year man from nearby village is bought to the emergency department in an unconscious state. There is a history of domestic problems. Patient’s clothes are wet and are soiled with urine and faeces. On examination, frothy secretion present in the mouth, pupil are constricted both side, blood pressure = 80/60 mmHg, electrocardiogram showed bradycardia (heart rate 52 beats/min).

124. In basic/paraclinical subjects, the scenario can be patient vignette or a concept‑based scenario followed by the lead‑in, for example., MCQ‑Which of the following term best describes the decreased effects of long‑term use of beta‑adrenergic agonists in bronchial asthma? a. Pharmacokinetic tolerance b. Pharmacodynamic tolerance c. Tachyphylaxis d. Cumulation.

125. While doing hip surgery, all the arteries taking part in cruciate anastomosis of upper thigh are to be ligated. Which of the following artery will maintain the blood supply? a. Lateral circumflex b. Superior gluteal c. Inferior gluteal d. First perforating artery.

126. In pharmacology, questions can often be constructed around the basics for developing more application/ clinically orientated therapeutics by creating the option lists to require more detailed knowledge of pharmacology e.g., of adverse drug reactions. Avoid giving a stem which is unnecessarily long or not aligning with lead‑in. Therefore, add a relevant stem and make sure lead‑in needs a stem.

127. Lead-ins

128. The item lead-in should be focused, closed, and clear; the test-taker should be able to answer the item based on the vignette and lead-in alone.

129. Lead-ins Next step in item writing - to phrase the question with the use of a lead-in focused on the patient The lead-in should be a single, closed, clear question. √ “Which of the following is the most appropriate next step in management?” or √ “Which of the following is the most likely diagnosis?” X Avoid an open ended lead-in such as- “The diagnosis in the patient is Ideally, after reading the vignette and the lead-in, a test-taker should be able to answer the item without seeing the options. Another reason to use a closed lead-in is because it helps to avoid certain item flaws, such as grammatical cueing

130. Lead ins for health promotion/prevention of disease Which of the following immunizations…. most appropriate screening test? tests would have predicted these findings? most appropriate intervention? patient at greatest risk? most appropriate next step in management to prevent [morbidity/mortality/disability]? Supplementation with which of the following is most likely to have prevented this condition? A 15-year-old boy has had two episodes of severe anaphylactic shock following bee stings. Which of the following is the most appropriate intervention? A. Administration of corticosteroids during the summer B. Long-term prophylactic antihistamine therapy C. Protective clothing *D. Desensitization with bee venom extract E. Restrict him to the house during the summer months

131. Lead ins for Pathophysiology/mechanics of the disease Which of the following is the most likely explanation for these findings ? Which of the following is the most likely location of the patient’s lesion? Which of the following is the most likely pathogen ? Which of the following findings is most likely to be increased/decreased ? A biopsy is most likely to show which of the following? A 10-year-old girl develops gross hematuria 14 days after a sore throat. She has a blood pressure of 170/100 mm Hg and 2+ pedal and pretibial edema. The serum urea nitrogen (BUN) level is 3.2 mg/ dL . Which of the following is the most likely cause? A. Coarctation of the aorta B. Decreased production of endothelial-derived relaxant factor C. Increased production of aldosterone D. Increased production of catecholamine *E. Intravascular volume of expansion

132. Lead ins for diagnosis of the disease Which of the following is the most likely diagnosis? Which of the following is the most appropriate next step in diagnosis? Which of the following is most likely to confirm the diagnosis? A 52-year-old man has had increasing dyspnea and cough productive of purulent sputum for 2 days. He has smoked one pack of cigarettes daily for 30 years. His temperature is 37.2 C (99 F). Breath sounds are distant with a few rhonchi and wheezes. His leukocyte count is 9000/mm3 with a normal differential. Gram’s stain of sputum shows numerous neutrophils and gram-negative diplococci. X-ray films of the chest show hyperinflation. Which of the following is the most likely diagnosis? A. Asthma B. Bronchiectasis *C. Bronchitis D. Pulmonary embolism E. Streptococcal pneumonia

133. Lead ins for management of the disease Which of the following is the most appropriate initial or next step in patient care? Which of the following is the most effective management? Which of the following is the most appropriate pharmacotherapy? Which of the following is the first priority in caring for this patient? ( eg , in emergency department) A previously healthy 15-year-old boy has cramping periumbilical pain; after several hours, the pain shifts to the right lower quadrant and becomes constant. He vomits several times and is brought to the emergency department. The abdomen is tender on deep palpation of the right lower quadrant. Findings on chest and abdominal x-ray films are normal. Leukocyte count is 15,000/mm3. Urinalysis shows 3 leukocytes/ hpf . Which of the following is the most appropriate initial management? A. Supportive treatment at home; return at once if the pain increases B. Barium enema C. CT scan of the abdomen D. Intravenous pyelography and cystography *E. Surgical exploration of the abdomen

134. Distractors

135. Step 4: Construct options or choices. This includes: The correct answer, also known as the key or correct option The distractors, i.e., three distractors.

136. Each MCQ should have only one correct answer, i.e., one key. The other three options are known as distractors. If a student does not know the answer, the distractors should be able to distract the student away from the correct answer. Choices must be: Homogeneous so that they can be rank‑ordered as needed Approximately of similar length.

137. While constructing the choices, avoid the use of none of the above/All of the above and avoid the use of choices giving clear hint to correct answer. Add distractors, which is considered to be challenging as distractors decide the quality of MCQ. Distractors should be of high quality to get authentic MCQ. Your experience of teaching can guide you to many examples of distractors in that topic. Hence, try to generate distractors from your day‑to‑day teaching experience.

138. Assessors find it is easier to add the first distractor, the second distractor takes a bit more effort and time and the third distractor becomes a challenge and takes more time. By the time assessor reaches the third distractor, they get tired or do not get a choice, or they put something like rhyming words, which are easily cracked by learner.

139. Aligning MCQ to Bloom’s taxonomy Aligning MCQ to Bloom’s taxonomy makes the assessment authentic. There should be a blend of both lower‑order and higher‑order thinking. The questions should be designed in such a manner that it leads the students to think, analyze, and solve the problem. While designing the MCQs, Bloom’s taxonomy should be kept in mind and try to include the questions which can test all the levels. To test the higher levels, the complexity of the question can be increased and not the difficulty of it. Make the student think on how to act in the given scenario so that higher levels are tested rather than just the recall questions.

140. Corroboration or Prevalidation of Scenario Based SBR MCQ

141. Triple Cs Concept Construction Corroboration

142. Is the direction clear? Is each item independent? Are all unrelated data eliminated? Is the language simple? Is it brief? Time appropriate? Are there abbreviations, eponyms, acronyms? Negative stems? Is this the best format? CHECKLIST : ITEM AS A WHOLE

143. Is it clear, concise, unambiguous? Does it ask an opinion? Does it contain one word? Vague expressions?…..fairly high, considerably greater…….. All common elements placed in the stem? Double negatives? CHECK LIST : STEM

144. Are they logical and plausible? Parallel in form? Numerical order? Do all options complete grammatically? Distractors functional? None of the above? All of the above? Mutually exclusive? Synonyms? CHECK LIST : OPTIONS

145. Undeniably the only correct answer? Are letters corresponding to the key equally distributed? Is key to an item provided by data in another item? Clues in keys: Synonyms? CHECKLIST : KEY

146. Post-validation of MCQs- Item Analysis

147. Item Analysis Process of evaluating a single MCQ (item) after its administration to the students. Based on students’ response to each item Various indices/values are calculated and interpreted Identify a defective question.

148. How is Item Analysis Done? Mark the papers Arrange them in rank order, with student scoring highest marks at the top. Select upper 1/3 rd students with high scores (high achiever group: HAG ) Select lower 1/3 rd students with less scores (low achiever group: LAG ) middle third are kept aside. If the student no. appearing for test is large, 100 from higher group and 100 from lower group suffice.

149. For each item a table is prepared as follows: Item No. 1 Stem of MCQ: Which one of the following benzodiazepines can be safely used in patients with liver disease? Alternatives No. of Responses A - Oxazepam B – Diazepam C – Chlordiazepoxide D - Nitrazepam No response *A = Key High achiever group Low achiever group Total responses (T)

150. Item No. Stem of MCQ: Which one of the following benzodiazepines can be safely used in patients with liver disease? Alternatives No. of Responses A - Oxazepam B – Diazepam C – Chlordiazepoxide D - Nitrazepam No response *A = Key High achiever group IIIII IIIII IIIII IIIII III - 23 IIIII II - 7 II - 2 IIIII IIIII IIIII - 15 III - 3 Low achiever group IIIII - 5 IIIII IIIII IIIII IIIII - 20 IIIII IIIII IIIII - 15 IIIII - 5 IIIII - 5 Total responses (T) 50 50 For each item a table is prepared as follows: