Evolution of the American Trends Panel

Here is my presentation from the 2020 AAPOR 75th Annual Conference

1. Evolution of the American Trends Panel Nick Bertoni Courtney Kennedy Panel Manager Director of Survey Research AAPOR 75 th Annual Conference

2. 2 Pew Research Center’s path to an online panel For 20 years, we were an RDD telephone polling organization We created our online “American Trends Panel” in 2014 and just finished fielding our 68 th panel survey This talk will draw on our experiences with the ATP and highlight some important changes over the years

3. 3 National probability-based sample of adults who take surveys for us Created in 2014 100% online survey administration Non-internet households are provided tablets and data plan We conduct 1 to 2 surveys per month, either 8- or 15-min long $5 to $20 for each survey Primary platform for Center domestic data collection American Trends Panel Overview

4. Building the ATP May 21, 2020 4

5. 5 ATP Establishment, Enhancement and Expansion

6. 6 ATP Establishment, Enhancement and Expansion

7. 7 ATP Establishment, Enhancement and Expansion

8. 8 ATP Establishment, Enhancement and Expansion

9. 9 ATP Establishment, Enhancement and Expansion

10. 10 ATP Establishment, Enhancement and Expansion

11. Recruitment mode – from rdd to abs May 21, 2020 11

12. 12 Changing recruitment from RDD to ABS ATP was initially recruited at the end of RDD polls in 2014 and 2015 One limitation of this approach is that the cumulative response rate to panel surveys was 2% In 2017 we had a dedicated RDD recruitment survey Starting in 2018 we switched to ABS recruitment 10% Response rate to recruitment surveys X 50% Agreement to join panel among recruitment respondents X 55% Panelists active at the start of the wave X 77% Wave-level response rate 2% Cumulative response rate for ATP wave

13. 13 Changing recruitment from RDD to ABS When we switched to recruiting via ABS, we saw notable improvement Recruitment Recruitment survey AAPOR RR3 % of recruitment survey respondents agreeing to join the panel 2017 RDD Design 10% 50%

14. 14 Changing recruitment from RDD to ABS When we switched to recruiting via ABS, we saw notable improvement Recruitment Recruitment survey AAPOR RR3 % of recruitment survey respondents agreeing to join the panel 2017 RDD Design 10% 50% 2018 ABS Design 12% 94%

15. 15 Changing recruitment from RDD to ABS When we switched to recruiting via ABS, we saw notable improvement Lower response rate and panel agreement rate in 2019 attributed to targeted recruitment of Hispanic and non-internet households Recruitment Recruitment survey AAPOR RR3 % of recruitment survey respondents agreeing to join the panel 2017 RDD Design 10% 50% 2018 ABS Design 12% 94% 2019 ABS Design 11% 80%

16. 16 Changing recruitment from RDD to ABS Now our cumulative response rate looks like this 10% 11% Response rate to recruitment surveys X 50% 79% Agreement to join panel among recruitment respondents X 55% 85% Panelists active at the start of the wave X 77% 77% Wave-level response rate 2% 6% Cumulative response rate for ATP wave

17. 17 Changing recruitment from RDD to ABS Now our cumulative response rate looks like this To be sure, it’s still low. And, yes, Montaquila and Brick saw this pattern 10 years ago in NHES! But as the ABS-recruits become a larger and larger share of the panel and as we refine the ABS protocol and improve the recruitment response rate… 10% 11% Response rate to recruitment surveys X 50% 79% Agreement to join panel among recruitment respondents X 55% 85% Panelists active at the start of the wave X 77% 77% Wave-level response rate 2% 6% Cumulative response rate for ATP wave

18. 18 Changing recruitment from RDD to ABS In time it may be possible to get a cumulative response rate more likely 15% even without an in-person recruitment stage This would provide a clearer, more compelling distinction between probability-based versus opt-in online polls Even more importantly, ABS yielded a more representative sample 10% 11% 25% Response rate to recruitment surveys X 50% 79% 94% Agreement to join panel among recruitment respondents X 55% 85% 85% Panelists active at the start of the wave X 77% 77% 77% Wave-level response rate 2% 6% 15% Aspirational cumulative response rate

19. 19 Changing recruitment from RDD to ABS The ABS recruited sample was closer to population benchmarks on virtually every dimension we measured It’s important to note that ABS oversampled based on commercial flags for black, Hispanic, and age 18-24 (%) unweighted profile of ABS-recruited versus existing, RDD-recruited panelists

20. interview mode – from mail to web May 21, 2020 20

21. Why USE mail mode to begin with? May 21, 2020 21

22. May 21, 2020 22 Coverage Not everyone uses the internet Those who do not use the internet are different than those who do Prefer to include non-internet households

23. 23 Covering those without internet access There are three main approaches: Provide the technology and interview online Interview by telephone Interview by mail Selecting the optimal approach is a multi-dimensional problem We see at least six important dimensions

24. 24 Covering those without internet access Dimension Provide technology Interview by telephone Interview by mail Cost Very expensive Less expensive(?) Relatively inexpensive

25. 25 Covering those without internet access Dimension Provide technology Interview by telephone Interview by mail Cost Very expensive Less expensive(?) Relatively inexpensive Representation of non-internet adults Weaker with potential to change panelists Strong Strong

26. 26 Covering those without internet access Dimension Provide technology Interview by telephone Interview by mail Cost Very expensive Less expensive(?) Relatively inexpensive Representation of non-internet adults Weaker with potential to change panelists Strong Strong Mode effects risk? None Yes Minimal

27. 27 Covering those without internet access Dimension Provide technology Interview by telephone Interview by mail Cost Very expensive Less expensive(?) Relatively inexpensive Representation of non-internet adults Weaker with potential to change panelists Strong Strong Mode effects risk? None Yes Minimal Possible to do timely surveys? Yes Yes No

28. 28 Covering those without internet access Dimension Provide technology Interview by telephone Interview by mail Cost Very expensive Less expensive(?) Relatively inexpensive Representation of non-internet adults Weaker with potential to change panelists Strong Strong Mode effects risk? None Yes Minimal Possible to do timely surveys? Yes Yes No Can do extensive skips, fills, rotations? Yes Yes No

29. 29 Covering those without internet access Dimension Provide technology Interview by telephone Interview by mail Cost Very expensive Less expensive(?) Relatively inexpensive Representation of non-internet adults Weaker with potential to change panelists Strong Strong Mode effects risk? None Yes Minimal Possible to do timely surveys? Yes Yes No Can do extensive skips, fills, rotations? Yes Yes No Added risk of programming error No Yes Yes

30. 30 Covering those without internet access Each solution has its limitations On the ATP, we started with the mail approach but switched in 2016 to providing the technology For us concerns about timeliness and mode effects are greater than concerns about cost and somewhat reduced representation of the non-internet population But this remains a difficult issue

31. Converting the panelists – logistics and RESULTS May 21, 2020 31

32. May 21, 2020 32 Conversion Protocol Advance letter to current mail mode panelists (N=574) Conversion call Inform panelists of the conversion from mail to web Determine internet access and/or device needs Ask to convert Obtain email if available, provide tablet with internet connection if necessary Tablet gets configured and shipped by vendor Follow up call after panelists receive tablet

33. May 21, 2020 33 Conversion Stats Initial results of the Conversion Call were encouraging Mail to Web Conversion Call % of mail panelists that…       Percent   Agreed to convert 66     Refused to convert 25     Non-contact 8     N 574 Source: Mail to Web Conversion Call survey conducted 4/11/2016-1/4/2017. Results do not add up to 100 due to rounding. PEW RESEARCH CENTER  

34. May 21, 2020 34 Conversion Stats If only it ended at the Conversion Call Mail to Web Conversion % of mail panelists that…       Percent   Refused after initially agreeing to convert 3     Refused after receiving the tablet 12     Received the tablet but never participated 9     Participated at first but then disappeared 2 Note: Percent shown is of total attempted to convert (N=574) PEW RESEARCH CENTER  

35. May 21, 2020 35 Conversion Stats When all of the dust settled Mail to Web Conversion % of mail panelists that…       Percent   Agreed to convert 41     Refused to convert 59         N 574 Source: Mail to Web Conversion. PEW RESEARCH CENTER  

36. May 21, 2020 36 Composition of the Panel: No Observed Differences We observed no statistically significant difference in sample composition of the panel pre- and post-conversion for several key areas Sex Political ideology Race/Ethnicity Follow politics Party ID Follow news Voter registration Religious attendance

37. May 21, 2020 37 Composition of the Panel: Internet Use A smaller proportion of the panel are non-internet users after the conversion (N=4,563) (N=4,248)

38. May 21, 2020 38 Composition of the Panel: Education A smaller proportion of the panel has HS or less education after the conversion % (N=4,563) (N=4,248)

39. May 21, 2020 39 Contact Information Nick Bertoni Panel Manager [email protected]