Delphi technique

Delphi technique

1. DELPHI TECHNIQUE Mahmoud Sobeh Supervisor: Dr. Khaled Al Hallaq June, 2021

2. outline Introduction Definitions Quick example Advantages Disadvantages Situations to use Delphi Delphi Procedures Types of Delphi’s and main characteristics

3. Introduction Delphi Technique is a way to achieve agreement among a group of experts on a certain issue. It based on the assumption that Decisions made from a structured setting is going to be more accurate. (two heads are better than one) It was first used in technology forecasting studies initiated by the RAND (Research and Development) Corporation for the American military in 1944 . The Delphi technique has been used widely in health research within the fields of technology assessment, education and training, priorities and information and in developing nursing and clinical practice. Delphi as key word generates about 188,000 records on Google scholar search and over 3600 on the EBSCO academic electronic search. The numbers indicate that it is well-accepted across many disciplines

4. What is the Delphi Technique? Delphi Technique (PMBOK 4 th edition ) An information gathering technique used as a way to reach a consensus of experts on a subject. Experts on the subject participate in this technique anonymously . A facilitator uses a questionnaire to solicit ideas about the important project points related to the subject. The responses are summarized and are then recirculated to the experts for further comment. Consensus may be reached in a few rounds of this process. The Delphi technique helps reduce bias in the data and keeps any one person from having undue (unjustified) influence on the outcome.

5. Simple Definition Delphi technique is a research approach used to gain consensus through a series of rounds of questionnaire surveys , where information and results are fed back to panel members between each round.

6. Quick example (corn count) Std.Dev Average High Low Round 37,406 24 , 083 100 , 000 1 , 000 1 11,602 758 10,750 12,000 10,000 2 620 11 , 158 11 , 900 10 ,500 3 185 11 , 658 11 , 900 11 , 500 4 162 11 , 641 11 , 850 11 ,500 5  

7. Advantages of Delphi Bring geographically dispersed panel experts together. Anonymity and confidentiality of responses Quiet, thoughtful consideration Focus is on “ideas” rather than “individuals” Opportunities for participants to reconsider their opinions. Focused, avoids unnecessary sidetracking for panelists Ties together the collective wisdom of participants Effective and flexible/adaptable technique Addresses three main problems with Focus Groups: • Dominant personalities • Group pressure • Noise

8. Disadvantages of Delphi No right or wrong answers, only expert opinion. feedback mechanism may lead to conformity rather than consensus The bandwagon effect of a majority opinion. N o direct contact, No dialogue Lack of universal guidelines for determining consensus, sample size and sampling techniques Requires time/participant commitment Time delays between rounds in data collection process Drop-outs, response rates

9. Situations to use Delphi Develop priorities. Develop policy. Forecast about the future. Where ethical or social dilemmas dominate economic or technical dilemmas.

10. Delphi Procedures Define the problem Select experts Round 1 Qualitative Analysis Next Round Assess Consensus Repeat until consensus obtained Rank and Inference yes No

11. Define the problem Identify the problem that you want to work on, writing it down in a clear way that is easy to understand. This can be in various forms , form a questionnaire to a broad and open question. You can work on one problem and you can work on several problems at once. The constraint is usually the bandwidth and expertise of the people in the Delphi group

12. Select experts 8-12 Experts suggested (diminishing returns if more are added). Requirements for Experts : Knowledge and experience with the issues under investigation Capacity, willingness and to sufficient time participate Effective communication skills Note that many panels are not true ' experts' but rather 'informed advocates '

13. Round 1 QUESTIONNAIRE In your opinion, what technical innovations are likely to be important for on-scene disaster management within the next ten years? This may be existing technology, or your predictions for new technology. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open ended questions. Best to provide a limited number of questions (5 – 10). The goal is to generate ideas. Starting point for the remaining rounds.

14. Qualitative Analysis Amount of information from first round can be overwhelming . Often ideas are worded differently by participants and need to be grouped . Attempt to not change wording if possible. Identify statements that are similar and group Retain unique statements Use these to create closed (ranking) questions

15. Round 2 QUESTIONNAIRE The following is a list of technologies that respondents of the first round have listed as potentially important within the next ten years. Please rate your impression of the importance of each of the following will take for on-scene disaster management in the next 10 years using the following scale. 1 2 3 4 5 6 7 8 9 Not at all Very Important Important Please circle your response A. Ultrasound 1 2 3 4 5 6 7 8 9 B. Internet Access 1 2 3 4 5 6 7 8 9 C. Heads up Display 1 2 3 4 5 6 7 8 9 D. Smart Phones 1 2 3 4 5 6 7 8 9 E. Personal Robots 1 2 3 4 5 6 7 8 9 Closed ended questions are formed from information of round 1 • Participants rank importance/agreement of answers from First Round • Generally yes/no or rating scales

16. What is Consensus?? Indicates whether the expert panel agree with one another, Not whether they agree or disagree with the statement. Example: If all experts disagree with a statement, this is consensus

17. Assess Consensus There is no universal agreement of what is sufficient consensus in a Delphi study. Recommendation vary from 51% to 80% This MUST be decided before any data is obtained. Stability between rounds may be a better indicator.

18. Assess Consensus For Categorical Variables: from 51% to 100% agreement For Continuous Variable: For 3 point scale  IQR<= 1 For 4 point scale  IQR<1.0 OR IQR=1.0 and >60% of respondents are generally positive or negative.

19. Consensus

20. Round 3 QUESTIONNAIRE Below is a prioritized list that respondents in the second round rated as most important. The mean response from the previous round is indicated, as is your choice from the second round. In this round you are given a chance to reconsider your response based on the responses of others. Please rate your impression of the importance of each of the following will take for on-scene disaster management in the next 10 years using the following scale. 1 2 3 4 5 6 7 8 9 Not at all Very Important Important Please circle your response Inform participants of the results 0f previous round Remind participants what they indicated Allow them a chance to revise If there is a large number of statements, remove those that already have consensus in previous round   Your response round 2 Your response round 2 Group mean response Your revised response A Smart phone 8 8.3 1 2 3 4 5 6 7 8 9 B Ultrasound 1 6.7 1 2 3 4 5 6 7 8 9 C Internet access 9 3.4 1 2 3 4 5 6 7 8 9 D Personal robots 9 1.2 1 2 3 4 5 6 7 8 9

21. Classical Delphi Subsequent Rounds Rounds continue until "consensus“ is obtained. • At least 70% response rate needed to maintain rigor.

22. Ranking Normally those statements that have consensus are then ranked from highest to lowest. Statements that fail to reach consensus: eliminate or add to discussion

23. Inference Friendomization is common Remember experts are NOT a random sample. Cannot infer about the general population. Use of confidence intervals and p-values is questionable

24. Types of Delphi’s and main characteristics

25. Types of Delphi’s and main characteristics

26. References Highly recommended for anyone doing a Delphi Study