UX Research Moderation Best Practices & Using Your Psychologist Voice

The time you have with study participants is precious and short, so it is up to study moderators to maximize the amount of data collected during the interview.

A key part of this is how the moderator interacts with study participants to keep them comfortable and talking about useful topics.

This talk provides UX Practitioners with tips on:
- Extracting high-quality data from participants
- Detail on how you can leverage your “Psychologist Voice” to put participants at ease and keep them talking about the topic at hand

1. UX Research Moderation Best Practices & Using Your Psychologist Voice Prepared by Dan Berlin, Watch City Research

2. Hi! I’m Dan Berlin 2 Seven years tech support for VOD and other digital cable TV systems BA, Psychology MBA + MS in Human Factors in Information Design Ten years at Mad*Pow where I formed and managed the research team Started Watch City Research and edited “97 Things Every UX Practitioner Should Know” 1999 2011 2001 2008 2021 2006 2023 Started program to earn PhD in Business – Experience Design

3. 3 COMPANIES I’VE WORKED WITH

4. Sampling of what I’ve worked on Usability studies of: Websites Software Mobile apps Medical devices Infotainment systems Kiosks Ethnographic observations with: Professors in their homes or offices Consumers in-store Trainmasters at a train yard Most fun study: Trainmaster Most different: Museum exhibit hall experience* Most heart-breaking story: HIV patient *I didn’t personally do this study, but helped a bunch

5. Today’s Chat

6. User Interview Questions 6

7. Writing Good Interview Questions Open-Ended Questions Who, What, When, Where, Why, How Sticking to the “6 W’s” is the best way to avoid asking leading questions Start broad & use follow-up questions to narrow in Examples: “Tell me about how you prepare printed materials for your classes.” “How do you know when it is time to refill your prescriptions?” “Describe what you do to get ready to go watch a soccer match.” “What healthy changes do you want to make or have been making?” 7

8. Writing Good Interview Questions Follow a broad question with a series of probing questions While moderating, you’ll follow the participants’ lead But writing down anticipated probing questions will remind you to ask about important topics that participants don’t mention Example: “Tell me about sharing the printer and copier with others.” What works? What doesn’t work? What are the steps you take to print something from your computer? How do you feel about the costs to print? What goes into your decision-making process when it comes to printing something vs. relying on a digital version? 8

9. Writing Good Interview Questions Question Style Don’t be too direct , it may put the participant on the defensive This is particularly important with sensitive topics Don’t directly ask if a participant understands something Instead, ask “what is your understanding of…” Example: Participant’s goal is to go to gym 3/week Poor question: “Why didn’t you go to the gym the last two weeks?” Better question: “What has kept you from going to the gym the last two weeks?” 9

10. What to Listen For User’s picture of the world (mental model) Their perception of how things work What things are grouped together as similar, or are different? Stories of how things usually happen Probe on instances of things happening – “ Tell me about the last time you…” Stories of exceptions – when things go really good or bad Contradictions in stories With same participant From one participant to another Barriers preventing them from acting 10

11. Questions to Avoid Don’t ask participants how something should be designed , they aren’t designers – the goal is to learn about their current behaviors and informational needs Don’t ask participants to predict their future behavior , they aren’t psychics “ How would you use this new feature?” Instead: “What could the product do to ease your workflow?” Don’t ask multiple questions in a single breath – keep questions simple Break compound questions into multiple questions 11

12. The Most Important Question “Is there anything I didn’t ask you about today that you think is important for me to know?” 12

13. Additional User Interview Resources Understanding Your Users: A practical guide to user requirements. Catherine Courage & Kathy Baxter, Morgan Kaufmann, 2005. Chapter 7 Interviewing Users: How to uncover compelling insights. Steve Portigal , Rosenfeld Media, 2013. Mental Models: Aligning Design Strategy with Human Behavior. Indi Young, Rosenfeld Media, 2008. Chapter 7 13

14. Moderation Best Practices 14

15. Building Trust and Comfort 15 Trust Comfort Collect Data Build Rapport Neutral Observation & Follow-up Questions Define Social Interaction Rules Comfortable Conversation Be Accepting Thanks Greeting Introduction Interview / Evaluation Wrap-Up

16. Building Rapport Start building rapport during the recruiting process Smooth processes Understandable instructions Clear communication Be a friendly person Smile Use their name, but only before you start recording Be a good listener Make the other person feel important – and do so sincerely Ask a standard question when greeting the participant “Did you have any trouble accessing the online session?” “How is your day going so far?” Be empathetic Apologize if they had trouble dealing with an online platform or finding the office 16

17. Define Social Interaction Rules Put participants at ease by setting expectations Describe the session List the activities Describe the roles Collecting open & candid feedback You’re not emotionally involved in the design/project There are no wrong answers Your job is to get honest opinions Research with human subjects, so you must tell them about… Recording Data usage Voluntary participation Ability to withdraw without penalty 17

18. Be Accepting Watch your reactions – don’t show surprise May make them think that they are giving a wrong answer Don’t overly agree May make them think they are giving an answer you want to hear Don’t be negative Watch your tone – stay neutral and accepting Use your judgment with laughter & responding to humor Make them feel comfortable, but don’t invite jokes from them Show you understand their point of view “I can understand that…” “I can see that…” “That does sound very frustrating…” 18

19. Be Accepting Be yourself, mostly No one is perfectly neutral Interject some rapport-building comments when needed Quiet or uncomfortable participants “I can imagine that was challenging,” etc. Again, show that you are human like them Body Language Head nodding while participant is speaking  interviewees speak 50% longer No one is an expert moderator in their first few sessions Invite critique from others 19

20. Keep it Consistent and Conversational Try not to read from the guide like a script – be more natural Reading from the guide is unnatural and not conversational Doing so takes your mind off active listening & thinking about probing questions You can be consistent across participants, but still keep it conversational You’ll feel dumb at first, asking the same exact question, in the same way, ten times in a row It’s acting, it’s an acquired skill, you’ll get more natural over time Can also say, “I’m going to read this, so everyone hears the same thing.” 20

21. Neutral Observation Some utterances may introduce bias “Oh!”, “Interesting” – indicating unexpected answer “Yes”, “Perfect”, “Great” – indicating agreement “Hmmm.”, “Really?” – indicating disagreement Notice that tone is key Neutral is best “ Mhmm ”, “Uh huh”, “Continue”, “Tell me more”, “OK” “ Mhmm ” or “Uh huh” vs. silence  interviewees saying 31% more phrases Don’t suggest answers to your questions It’s a natural impulse for us, but biases the participant Really listen Pay attention – stay in the moment Look at the participant Take notes if you can Be quiet – give them time to say what they need to 21 “You need to not only be ready to hear your participant’s take on things, but you should also be hungry for it. This willingness to embrace is an active, deliberate state.” - Steve Portigal , Interviewing Users

22. Probing Never let a participant get away with a simple answer to a question Always be probing! “Tell me more about that…” “How does that make you feel?” “It sounds like you said X, what else about X?” “What makes you say X?” “You said X, why not Y?” <silence> If a participant asks you a question, ask the question right back Ask what caused the participant to ask the question If the participant gives a one-word answer and you move on to the next question, you’ve missed an opportunity to probe deeper 22

23. Non-verbal Communication Posture – slouching indicates disinterest Facial expressions – appropriately responsive Distance – optimum conversational distance Voice tone – conversational tone Body movements – convey interest by squarely facing the participant and leaning in where appropriate Mannerisms – distracting mannerisms may convey disinterest 23

24. Increasing Study Validity Evaluator effect: Moderators have different worldviews and will observe different things Different moderators will uncover different research findings Increasing the number of moderators will help mitigate this effect Every other day Every two participants Multiple studies Confirmation bias: we hear what we want to hear Be the neutral third-party! Be aware of your own cognitive biases Geographic spread to avoid localizing to one area 24

25. Notes on Taking Notes Good note taking is one of the most critical aspects of qualitative research It’s how we capture data Complete data = useful findings; missing data = incomplete findings Note taking helps gets the data into your mind Transcriptions are good as a second read and for tagging Focus on actions, mannerisms, change in tone, comments, and happenings Did they say something confusing? Something great? Did the moderator have to explain anything? Ideally, you capture as much as possible verbatim Multitasking: typing what you just heard, while putting the current conversation in an auditory buffer If you can’t catch verbatim comments, try to capture the main points 25

26. Notes on Taking Notes Highlight the good stuff If you hear a good quote, put it in quotes or CAPS – will help in finding verbatim text Write down observed themes Note taking is more taxing than you think Your mind will easily wander – don’t let it! You are in a dark, quiet room; you will get tired – wake up! Visually highlight as you go, when possible Use bold, underline, or CAPS to draw attention to very important findings Write down the top-three findings at the end of each session 26

27. Notes on Taking Notes Tell the participant you will be taking notes Try to still maintain normal eye contact , even though you’re feverishly typing away Learn to touch-type If in person, keep the laptop at a lower level, so it doesn’t get between you and the participant too much During the session, don’t be afraid to tell the participant you are catching up on your notes , especially when remote Gives the participant a moment to think too 27

28. Books on Moderation 28

29. Your Psychologist Voice 29

30. Your Psychologist Voice Research best practices focus on methodologies and moderation Preparation, documentation, & technique Asking non-leading questions Types of questions to elicit different data Participant activities We don’t often hear about how to best carry your voice while moderating research 30

31. Your Psychologist Voice Matching the participant’s tone and pace Can be unnatural May think moderator is mocking them Just be yourself We don’t pay enough attention to our voice “Yourself” may be a booming, intimidating voice or a meek voice that doesn’t command authority Instead, use your psychologist voice 31

32. Your Psychologist Voice 32

33. Your Psychologist Voice 33 Back in high school, I participated in group meetings with the school psychologist, Dr. Serkin Over the years, Dr. Serkin always talked to us in a regular voice, just like you and I would during a conversation But then, one day, I walked into Dr. Serkin’s office, and he was talking in a much calmer, slower, softer voice I asked, “Dr. Serkin, what’s up with your voice?” He replied, “I’m practicing my psychologist voice.”

34. Your Psychologist Voice Pace – even, calm pace Power – talk from your belly, not your mouth/head Volume – audible, but not too loud Timbre – warm and soothing Emphasis – ensure to stress the important words Pause – stop to emphasize key points Depth – where you are comfortable, to a point Inflection – not monotone 34

35. Your Psychologist Voice Listen to listen and to understand Ask “Why” multiple times without using the word “why” Turn questions back onto participants Confirm that you understood them correctly Keep quiet – keep the participant talking Never sound disappointed in an answer Always be interested in what they have to say Convey that the participant is the expert 35

36. Julian Treasure: How To Speak So That People Want to Listen 36

37. Practice Voice Mindfulness “Mindfulness” = active attention on the here and now “Voice mindfulness” = active attention about how you convey your words Tough to transcend unconscious behavior Warm up your voice Practice, practice, practice 37

38. 97 UX Things Book and Podcast 38 Podcast website: https://97uxthings.buzzsprout.com Book sections: Career Strategy Research Design Content