QUESTIONNAIRE DEVELOPMENT new.pptx

QUESTIONNAIRE DESIGN AND DEVELOPMENT FOR MEDICAL RESEARCH

1. Developing questionnaires for medical research By: Dr Hanaa Bayomy Assistant professor of Community Medicine

2. Introduction Questionnaire is a measuring device used to query a population/sample in order to obtain information for analysis. A questionnaire is simply a list of questions that is completed by or for a respondent. (Health Research Methodology 2nd ed. 2001 WHO)

3. Introduction Questionnaires are one of the most popular survey methods. It is the most convenient way to collect data from large numbers of often geographically diverse individuals on a wide range of issues. Questionnaire may be administered in a variety of ways, including face to face, telephone, post and online.

4. Introduction Questionnaires are good for gathering data about constructs that are not directly observable e.g., student satisfaction with a new curriculum, patients’ ratings of their physical discomfort, etc. Such constructs do not have a commonly agreed upon metric. A survey scale comprises series of questions designed to capture the different facets of a given construct.

5. Introduction The questionnaire can influence the response rate achieved in the survey, the quality of responses obtained, and consequently the conclusions drawn from the survey results. A well-designed questionnaire is the one ensuring the collection of worthwhile data about the phenomenon under study using a set of items that every respondent will interpret the same way , respond to accurately, and be willing and motivated to answer.

6. Introduction Definitions: A construct is A hypothesized concept or characteristic that a survey or test is designed to measure. Key variables are those which describe the construct, for example if the study is relative to cancer, key variables will be relative to cancer. Explicative variables are all those variables which might be correlated to the key variables, for example in the case of cancer it could be the environment, stress, food, etc. Structure variables are variables which are usually used to describe the sample of the study such as age, sex, education, and profession.

7. Types of survey questionnaires Exploratory questionnaires If the data to be collected is qualitative for the purpose of better understanding or generation of hypotheses on a subject. It may be that no formal questionnaire is needed . 2. Formal standardized questionnaires: If the researcher is looking to test and quantify hypotheses and the data is to be analyzed statistically. A formal standardized questionnaire is needed.

8. Forms of questionnaires Both structured and semi-structured forms can be used in quantitative surveys.

9. Questionnaire design It should be borne in mind: 1. A well-designed questionnaire should meet the research objectives. 2. It should obtain complete and accurate information. 3. It should make it easy for respondents to give the necessary information and for the interviewer to record the answer, do sound analysis, and interpretation. 4. It should be brief , to the point, and so arranged that the respondent(s) remain interested throughout the interview.

10. Questionnaire design for medical education research

11. AMEE (Association of Medical Education in Europe) seven-step design process: Conduct a literature review, Conduct interviews and/or focus groups, Synthesize the literature review and interviews/ focus groups, Develop items, Conduct expert validation, Conduct cognitive interviews. Conduct pilot testing.

12. AMEE seven-step survey design process Step 1: Conduct a literature review To define the construct and identify existing survey scales that might be used. Previously validated scales are sensitive to target population, local context, and intended use of the scale.

13. AMEE seven-step survey design process Step 2: Conduct interviews and/or focus groups When a new scale is used, it is necessary to learn how the population of interest conceptualizes and describes the construct of interest

14. AMEE seven-step survey design process Step 3: Synthesize the literature review and interviews/focus groups By merging the results of the literature review and interviews/focus groups to ensure that the conceptualization of the construct makes theoretical sense to scholars in the field and uses language that the population of interest understands.

15. AMEE seven-step survey design process Step 4: Develop items To write survey items that adequately represent the construct of interest in a language that respondents can easily understand and are written in accordance with current best practices in survey design .

16. Types of questions Closed-ended question – A survey question with a finite number of response categories from which the respondent can choose . Open-ended question – A survey question that asks respondents to provide an answer in an open space (e.g., a number, a list or a longer in-depth answer).

17. Ideal qualities of a question Yields a truthful, accurate answer . Asks for one answer on one dimension . Accommodates all possible responses . Uses specific, simple language . Has mutually exclusive and response options. Produces variability in response. Minimizes social desirability . Pretested

18. Avoiding five common pitfalls in survey design. Artino et al. 2011. Lippincott Williams and Wilkins/Wolters Kluwer Health Pitfalls 1- Creating a double-barreled item . e.g., How often do you talk to your nurses and administrative staff when you have a problem? 2- Creating a negatively worded item . e .g., The chief resident should not be responsible for denying admission to patients. 3- Using statements instead of questions . e.g., I am confident I can do well in this course (not at all to completely true). 4- Using agreement response anchors . e.g., The high cost of health care is the most important issue in America today. (strongly disagree to strongly agree). 5- Using too few or too many response anchors Solutions Create multiple items for each questions. Wording items positively. Should the chief resident be responsible for admitting patients? Formulate survey items as questions. How confident are you that you can do well in this course? Use construct-specific response anchors (not at all to extremely important). Use 5-7 response anchors to achieve stable participant responses.

19. Avoiding four visual-design pitfalls in survey development. Artino & Gehlbach (2012). Lippincott Williams and Wilkins/Wolters Kluwer Health Pitfalls 1- Labeling only the end points of your response options. E.g., How interesting did you find this clinical reasoning course? Not at all Extremely interesting interesting 2- Labeling response options with both numbers and verbal labels. E.g., How much did you learn in today’s workshop? -2 -1 0 1 2 almost a little some quite a great nothing bit a bit amount 3- Unequally spacing your response options . E.g., How much did you learn from your peers in this course? almost a little some quite a great nothing bit a bit amount 4. Placing non-substantive response options together with substantive response options. E.g., How satisfied are you with the quality of the libr ary services? not at all slightly moderately quite extremely not Satisfied Satisfied Satisfied Satisfied Satisfied applicable Solutions Verbally label each response option. How interesting did you find this clinical reasoning course? Not at all slightly moderately quite Extremely interesting interesting interesting interesting interesting Use only verbal labels. How much did you learn in today’s workshop? almost a little some quite a great nothing bit a bit amount Maintain equal spacing between response options. How much did you learn from your peers in this course? almost a little some quite a great nothing bit a bit amount Use additional space to separate non-substantive response options from the substantive options. How satisfied are you with the quality of the library services? not at all slightly moderately quite extremely not Satisfied Satisfied Satisfied Satisfied Satisfied applicable

20. Different Likert-type response options Construct being assessed Five-point, unipolar response scales Seven-point, bipolar response scales Confidence Not at all confident Slightly confident Moderately confident Quite confident Extremely confident Completely unconfident Moderately unconfident Slightly unconfident Neutral Slightly confident Moderately confident Completely confident Interest Not at all interested Slightly interested Moderately interested Quite interested Extremely interested Very uninterested Moderately uninterested Slightly uninterested Neutral Slightly interested Moderately interested Very interested Effort Almost no effort A little bit of effort Some effort Quite a bit of effort A great deal of effort

21. Different Likert-type response options Construct being assessed Five-point, unipolar response scales Seven-point, bipolar response scales Importance Not important Slightly important Moderately important Quite important Essential Satisfaction Not at all satisfied Slightly satisfied Moderately satisfied Quite satisfied Extremely satisfied Completely dissatisfied Moderately dissatisfied Slightly dissatisfied Neutral Slightly satisfied Moderately satisfied Completely satisfied Frequency Almost never Once in a while Sometimes Often Almost always

22. Putting questions into a meaningful order and format Opening questions should be easy, not embarrassing , and within knowledge and experience of respondents to encourage them to continue. Questions flow logically , items on one aspect are grouped together, flow from general to specific, from least sensitive to more sensitive, from factual and behavioral to attitudinal and opinion questions. Question variety , using open-ended questions here and there, showing cards/pictures to increase interest of respondents. Closing questions: important questions should be in the earlier part of the questionnaire and sensitive questions should be left to the end.

23. Choose the methods of reaching target respondents Face-to-face interview Pros: high response rates, can clarify questions, suitable for longer questionnaires. Cons: high costs, time-consuming, need training of interviewers, transportation, and respondents may give socially acceptable answers.

24. Choose the methods of reaching target respondents Telephone interviews Pros: high response rates, fast, suitable for short and not complex questionnaire. Cons: interruptions, unsuitable timing of telephone calls, require interviewers training, might be difficult to target specific geographical locations.

25. Choose the methods of reaching target respondents Mail questionnaires Pros: high response rates, easy, low cost, can cover geographical area and large samples, and avoid interviewer bias. Cons: difficult to control respondents and responses, and time consuming.

26. Choose the methods of reaching target respondents Internet questionnaire Pros: easy, fast, avoid interviewers bias and distortion, low cost, and avoid socially influenced responses. Cons: poor control over respondents' selection, difficult to follow up, and need special design.

27. AMEE seven-step survey design process Step 5: Conduct expert validation (content validation) This step involves collecting data from content experts to establish that individual survey items are relevant to the construct of interest and that key items or indicators have not been omitted. Both quantitative and qualitative assessment methods are used to improve the content validity of a new questionnaire/scale.

28. AMEE seven-step survey design process Step 6: Conduct cognitive interviews (cognitive pre-testing/ response process validity) An evidence-based qualitative method specifically designed to investigate whether a survey question satisfies its intended purpose. To ensure that respondents interpret items in the manner that survey designer intends .

29. Cognitive processes when responding to a survey

30. Cognitive interview The think-aloud technique requires respondents to verbalize every thought that they have while answering each item. The verbal probing is a more active form of data collection where the interviewer administers a series of probe questions designed to elicit specific information.

31. Commonly used verbal probes. Willis & Artino (2013). J grad Med Edu.

32. AMEE seven-step survey design process Step 7: Conduct pilot testing The data obtained from the pilot testing are used to ascertain the internal structure of the questionnaire (scale uni-dimensionality) using factor analysis, and reliability (internal consistency) using the Cronbach’s alpha coefficient.

33. Reliability

34. Reliability Reliability – The extent to which the scores produced by a particular measurement procedure or instrument (e.g., a survey) are consistent and reproducible. Reliability is a necessary but insufficient condition for validity. If a questionnaire is unreliable, then it can’t be valid.

35. Aspects of reliability

36. Test-retest reliability It is used to assess the consistency of response to the items in a questionnaire from one time to another. Test administered twice to the same participants at different times. Disadvantages: Practice effect Too short intervals Some traits may change with time.

37. Test-retest reliability The reliability coefficient is the correlation coefficient (Pearson; r) Ranges from -1 to +1 Correlation coefficient 0.7-1.0 is considered strong, 0.3-0.69 is moderate. The test-retest reliability can be evaluated by using intra-class correlation coefficient (ICC). ICC ranges between 0 and 1. The value of ICC is considered appropriate if it is at least 0.70

38. Internal consistency reliability Here one should judge the reliability of the tool by estimating how well the items that reflect the same construct yield similar results i.e., how consistent the results are for different items for the same construct. Internal consistency reliability is expressed using: Split-half reliability Coefficient alpha

39. Split-half Reliability Split the contents of the questionnaire into two equivalent halves; either odd/even number, first/second half, or randomly selected contents. Then, we treat the halves as alternate forms. Correlation between the two halves indicates the reliability of one half. Problems with how we split the test.

40. Spearman-Brown formula Spearman-Brown formula is used to express reliability of the entire test.

41. Cronbach’s alpha Most commonly used. It is acceptable if value is 80% or above. The result says to what extent all items represent the same construct

42. Inter-rater reliability Assesses how two or more scorers differ in their evaluation of a test. Intraclass Correlation Coefficient (ICC): used for continuous data and captures both correlation and agreement among raters Kappa : used for categorical data and takes into account possibility of chance agreement between examiners

43. VALIDITY

44. Concept of Validity The degree to which a questionnaire measures what it was intended to measure. Degree to which the researcher has measured what he has set out to measure. (Smith, 1991) Are we measuring what we think we are measuring? (Kerlinger, 1973)

45. Types of Validity

46. 1- Face validity The extent to which a measuring instrument appears valid on its surface Each question must have a logical link with the objective E.g.: Questionnaire about domestic violence should have questions related to that issue. Not a validity in technical sense because it does not refer to what is being measured rather what it appears to measure. It has more to do with rapport and public relations than with actual validity.

47. Face validity Evaluate in terms of: Face validity can be done using 2 methods: Interview/prob method: The investigator discuss with participants each item, and whether it is appropriate to elicit accurate responses. Bilingual method: It is employed if the tool is translated into a regional language. A bilingual expert is employed to assess the face validity.

48. 2- Content validity The degree to which the items on the instrument are representative of the knowledge being tested or the characteristic being investigated. Expert judgment is the primary method used to determine whether a test has content validity. Coverage of issue should be balanced Each aspect should have similar and adequate representation in questions. No statistical test is employed here.

49. How do experts evaluate validity Average Congruency Percentage (ACP) Average percentage of relevant questions as determined by different experts. If the value >90 …… valid. Content Validity Index (CVI) Content Validity Index for Individual items (I-CVI). Content Validity Index for the Scale (S-CVI).

50. CVI: a panel of 3-10 experts are invited to review the relevance of each question.

51. Content Validity Index I-CVI Five or fewer experts: all must agree (I-CVI = 1.0) Six or more: (I-CVI should not be less than 0.78) S-CVI The proportion of items on an instrument that achieved relevance by all the content experts S-CVI/UA – Universal agreement S-CVI/Ave - Average

52. 3- Criterion validity The instrument’s capacity to predict behaviour or ability in a given area. The measuring instrument is called ‘criteria’. The responses on the questionnaire being developed are checked against a gold standard tool which is direct and independent measure of what the new questionnaire is designed to measure. In the absence of such a gold standard one, can use proxy measures like clinical examination or direct questions to respondents.

53. Criterion validity Concurrent validity : the measurement and the criterion refer to the same point in time. E.g.: visual inspection of a wound for evidence of infection validated against bacteriological examination of a specimen taken at the same time. Predictive validity : If the test is used to predict future performance. Academic aptitude test that is validated against subsequent academic performance.

54. 4- Construct validity Most important type of validity . It refers to the extent to which the new questionnaire accurately measures an existing ideas/hypothesis concerning the concepts/ constructs that are being measured. Construct validity can further be subdivided into: Convergent validity : It is a general agreement between an item and its own scale. Discriminate validity : It is a general disagreement between an item and other scales.

55. Construct validity Factor analysis For newly developed tools. Factor analysis is a complicated statistical procedure used to estimate where each item in the questionnaire is correctly reflecting the corresponding construct. If the tool has high construct validity, then shows increased correlation with the corresponding domains.

56. Construct validity Factor Analysis Various items are gathered into common factors Common factors are synthesized into fewer factors and then relation between each item and factor is measured Unrelated items are eliminated

57. Questionnaire translation A questionnaire translation process should focus in achieving the conceptual equivalence instead of achieving linguistic equivalence. In view of this, the forward –backward-forward translation technique should be applied

58. Questionnaire translation

59. Questionnaire translation VALIDATION STUDY It is highly recommended to conduct a validation study on the translated questionnaire to examine its psychometric properties such as the validity and reliability of the questionnaire. The appropriate study design for validation study is cross-sectional with at least 100 participants. These participants should be selected from various social demographic as well as socio economic background, so that the sample is more representative of the population.

60. Conclusion It is necessary that data collection tools are constructed systematically based on a sound scientific method since research outcome is directly dependent on the quality and the completeness of the data collected. Reduced response rates and incomplete responses to self administered questionnaires make it mandatory that the tool be developed as simple and respondent friendly as possible.

61. References Artino Jr, A. R., La Rochelle, J. S., Dezee, K. J., & Gehlbach, H. (2014). Developing questionnaires for educational research: AMEE Guide No. 87. Medical teacher, 36(6), 463-474. Trivedi, C. (2020, December 16). Reliability coefficients. ConceptsHacked. Retrieved from https://conceptshacked.com/reliability-coefficients Morrison Jo. Assessing questionnaire validity. 2022. Select statistical services Ltd. https://select-statistics.co.uk/blog/assessing-questionnaire-reliability/ Morrison Jo. Assessing questionnaire validity. 2022. Select statistical services Ltd. https://select-statistics.co.uk/blog/assessing-questionnaire-validity/

62. Thank you