Nominal group technique and Delphi method (Consensus methods)

Presentation on consensus methods and its applications. Nominal group technique and Delphi method

1. Nominal Group Technique and Delphi method Presenter: Dr Girish J, M.D. AIIMS, New Delhi

2. Introduction The Nominal Group Technique (NGT) and Delphi Technique are consensus methods used in research that is directed at problem-solving, idea-generation, or determining priorities Aim= to achieve a general agreement or convergence of opinion around a particular topic 1. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm [Internet]. 2016 [cited 2020 Dec 27];38:655–62. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4909789/

3. The NGT is a highly structured face-to-face group interaction, which empowers participants by providing an opportunity to have their voices heard and opinions considered by other members It was designed by Delbecq and Van de Ven and comprises four key stages: silent generation, round robin, clarification and voting (ranking or rating) Nominal Group Technique 1. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm [Internet]. 2016 [cited 2020 Dec 27];38:655–62. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4909789/

4. FGD vs Consensus methods Focus groups are useful for investigating an issue in-depth, including the identification of problems, questions or significant issues. Consensus methods= raise potential solutions or answers to a question, which can then be prioritised or agreed upon. A key strength of consensus methods= balanced participation from group members In focus group- the facilitator must control for, and minimise the risk of, a dominant participant influencing the discussion.

5. 1. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm [Internet]. 2016 [cited 2020 Dec 27];38:655–62. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4909789/

6. Nominal Group Technique Silent generation of ideas in writing: Welcome statement – importance of the task and of each member’s contribution Leader reads out the question – usually open-ended question that calls for a list of items e.g. the elements of a specified problem Each member (including leader) take 5 minutes to write their ideas in response to the question Discussion is not permitted 1. Survey methods: Abramson and Abramson

7. Nominal Group Technique ‘Round-robin’ feedback of ideas: Round-robin – “taking turns” Leader asks each member to contribute one of the ideas they have written (ideas not duplicated) Each idea is numbered and written on a board or flip pad Completed sheets of which are taped or pinned where they are visible to all members Members encouraged to add ideas; they may ‘pass’ in one round and contribute in a later one Process until no further ideas are forthcoming Discussion is not permitted during this stage 1. Survey methods: Abramson and Abramson

8. Nominal Group Technique Serial discussion of ideas: Each of the ideas listed on the board or flip pad is discussed in turn Discussion: whether any questions, clarification, needs explanation of logic behind it, or express a view about its relative importance The object is to obtain clarity on points of view, but not to resolve differences of opinion Modify the list if overlap present after the serial discussion Modest rewording may be undertaken if the group wishes to refine the list 1. Survey methods: Abramson and Abramson

9. Nominal Group Technique Preliminary vote: • Each participant is asked to select a specified number (five to nine) of ‘most important’ items from the total list and copy them on cards. • If six are to be chosen, each participant is asked to write ‘6’ on the ‘most important’ card, then ‘1’ on the least important. • The leader also ranks the items. • The cards are then collected and shuffled to maintain anonymity, and the votes are read out and recorded on a tally-chart. 1. Survey methods: Abramson and Abramson

10. Nominal Group Technique Discussion of preliminary vote: • Brief discussion of the voting pattern is now permitted. • Purpose of this discussion is additional clarification, and not to pressure to change their votes. Final vote: • Voting is then repeated Alternate to ranking: scale from 0 (unimportant) to 10 or 100 (very important) 1. Survey methods: Abramson and Abramson

11. Nominal Group Technique For many small groups: • Voting done in small groups • In break, group leaders meet to prepare a master list of items (aggregated votes), including the top five to nine priorities identified by each group • Where necessary, items are reworded or combined • All the participants then gather in a single large group and discuss each item in the master list in turn, for clarification • At any member’s request, items not included in the master list can be added • A final vote is then conducted

12. Nominal Group Technique Uses of NGT: • Involve participants from different background • Balances the influence of individuals • Confront issues through constructive problem solving • Prioritize ideas democratically Disadvantages of NGT • Requires preparation • Lends itself only to a single-purpose and minimizes discussion

13. Delphi method Delphi Technique are consensus methods used in research that is directed at Problem-solving Idea-generation Determining priorities The Delphi method assumes that group judgments are more valid than individual judgments 1. Survey methods: Abramson and Abramson

14. Delphi method The Delphi technique uses a multistage self-completed questionnaire with individual feedback , to determine consensus from a larger group of expert’s Uses interactions between group (called panel) members via questionnaires rather than face-to-face communication. This means that it preserves participant anonymity , if that is relevant Delphi technique is commonly used to develop guidelines with health professionals. 1. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi Method: Strengths and Weaknesses. 2019 Nov 1 [cited 2021 Jan 6];2:1–19.

15. First round questionnaire Present a series of statements that the respondent is asked to rate on a clearly defined Likert scale. Respondents are asked both to rate the item and to write free-text comments Second-round questionnaire The responses to the first-round questionnaires collated Reminders are sent to non-responders Presents the same statements, the individual respondent’s rating and the median rating from the entire panel. A selection of the free-text responses is given, to represent the breadth of opinion. After considering the group median and free-text comments, respondent’s re-rate the statements. Reaching a consensus 1. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi Method: Strengths and Weaknesses. 2019 Nov 1 [cited 2021 Jan 6];2:1–19.

16. Three main variants of Delphi: “Conventional” Delphi - an open-ended exploratory phase followed by multiple consensus-seeking rounds “real-time” Delphi, multiple rounds are temporally compressed to a single meeting “Policy” Delphi - forum in which ideas are presented to decision-makers, who choose a solution

17. Example of delphi method A Delphi method of consensus development was used, comprising three survey rounds in the context of obesity research. In Round 1, participants were asked to rate agreement/disagreement with 77 statements across seven domains relating to definitions of, and approaches to, using big data Participants were also asked to contribute further ideas in relation to these topics, which were incorporated as new statements ( n  = 8) in Round 2. In Rounds 2 and 3 participants re-appraised their ratings in view of the group consensus

18. Advantages of Delphi No proximity or a face-to-face meeting and thus allows for expert’s independent thought There is less opportunity to (be forced to) conform with the dominant view Through the feedback given in Delphi, an individual expert may enrich his/her in-sight into empirical factors or theoretical assumptions allowing them to correct any misconceptions 1. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi Method: Strengths and Weaknesses. 2019 Nov 1 [cited 2021 Jan 6];2:1–19.

19. Advantages of Delphi (Contd..) Enables group communication free from geographical constraints Saves money and time; it also reduces travel costs By straddling the divide between the qualitative and quantitative methods it is able to provide a more complete picture 1. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi Method: Strengths and Weaknesses. 2019 Nov 1 [cited 2021 Jan 6];2:1–19.

20. Disadvantages There is a lack of guidance and agreed standards on how to interpret and analyse the results. Delphi is quite time-consuming work for both researchers and participants, so vulnerable to drop-outs. Participants might also drop out due to the long process. Use of monetary payments or moral persuasion to convince the participants may introduce bias into the results. Methodological limitations include the difficulty of generalising the results to a wider population due to the sample size . 1. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi Method: Strengths and Weaknesses. 2019 Nov 1 [cited 2021 Jan 6];2:1–19.

21. The response rate among the invited experts, limited views, the uneven spread of expertise among the participants or the participants’ specific agenda and/or geographic/cultural location. As the procedure depends on the quality of the feedback provided, the careful analysis of the responses is a big responsibility of the re-searcher. It is difficult to determine what constitutes a sufficient consensus The experts’ responses in the Delphi process might not be truly independent, especially when the experts involved are in contact with each other While in the case of e-Delphi, entering data into computer-based data screens is an advantage, it might also be an inconvenience for some experts. Disadvantages 1. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi Method: Strengths and Weaknesses. 2019 Nov 1 [cited 2021 Jan 6];2:1–19.

22. Applications of consensus methods Health systems research Insufficient data to make decisions Prioritise health problems Opinions of other health professionals / lay persons / carers Development of quality & safety indicators Prescribing research Guidelines development Enhance clinical decision making Include patient / carer perspectives in decisions Qualitative research Building theories Participatory action and community engaged research Diagnostics Tools or questionnaire development Medical education Comparison of new / revised curriculum Assessment tools development Development of educational resources

23. THANK YOU

24. History of delphi The Delphi Technique was developed by the Rand Corporation in 1953 The Delphi method was developed at the beginning of the Cold War to forecast the impact of technology on warfare. In 1944, General Henry H. Arnold ordered the creation of the report for the U.S. Army Air Corps on the future technological capabilities that might be used by the military.

25. Consensus methods Advantage: Communication with others – opportunity to reach a modified judgement after appraising the opinions of others (unlike interviews or questionnaire surveys) Disadvantage of group techniques: Decisions may be heavily influenced by this interaction, unduly affected by: Chairperson’s bossiness or ineffectiveness, Dominance by forceful speakers Deference to authority, power, prestige or age, or friendships or antagonisms between participants