[ChatGPT Research] First time interaction with ChatGPTReport.pptx

ChatGPT report as assignment for User Research position in the Microsoft interviewing process.

1. ChatGPT Research Report 🔄Regenerate report Findings, insights, recommendations

2. Executive Summary Purpose of the study This executive summary presents the key findings and insights from our research conducted through surveys and interviews on ChatGPT usage. The main objective of this study was to identify the significant challenges new users encounter during their initial interaction with ChatGPT. Method to gather data The study utilized both qualitative and quantitative data. Surveys (37 participants) we approached first, to get insights from diverse users, validating assumptions and identifying new user roadblocks. In-depth interviews (6 participants) and usability testing (2 participants) further revealed and comprehended these challenges. Key Findings Familiarity with AI-powered chatbots is widespread among interviewed ChatGPT users. Users primarily use ChatGPT for job assistance, including tasks like writing improvements and tailored searches. Users showed lower familiarity with the tool's possibilities and benefits, affecting their motivation to use it. Many users are unsure of what they want to achieve when first using ChatGPT. First-time usage satisfaction is high, with users finding it interesting and valuable 4/5 . Challenges include defining queries and obtaining desired answers, leading to some users abandoning the chat. No major technical issues was found during initial usage. Concerns include data privacy, reliability, accuracy, and information quality, shared by both users and non-users of ChatGPT. Users reported higher efficiency in English compared to Serbian usage. Recommendations Increase tool reliability Enhance privacy and trust Educate users on ChatGPT applications Proactive and interactive query guidance Interactive learning loop & personalization of the tool More interactive signup process Increase convenience

3. Contents What is the goal of the research and main research hypothesis. 01. Intro Describe used methodology and collected data. 02. Methodology Summarize data in presentable findings and insights. 03. Findings & Insights Make a list of recommendations for identified roadblocks. 04. Recommendations

4. Main research hypothesis Below are the assumptions on what areas users are facing challenges when using the tool: Understanding Possibilities and Limitations of ChatGPT Effective Engagement Query Guidance Data Privacy Assurance Handling Misunderstandings or Errors Trust and Reliability of the tool Technical Support Language Support 01. Intro Main Research Question: What are the most significant challenges new users encounter when they initially interact with ChatGPT?

5. Participants : random sample of ChatGPT users 37 responses Survey questions Link to a survey Online surveys Participants : Sometimes, every-day and domain expert users 6 interviewees Interview Script In-depth interviews Participants : Fresh New Users 2 interviewees Usability Testing Script Usability Testing 02. Methodology The research approach involved a combination of qualitative and quantitative data. We started with surveys to gather insights from a diverse range of users, validating our research assumptions and identifying potential roadblocks for new users. Building on this, we conducted in-depth interviews and usability testing to thoroughly identify and understand these challenges.

6. 03. Findings and Insights Survey Results: Detailed survey findings can be accessed here and listed answers can be found in this spreadsheet . Interview transcripts: The transcripts of the interviews are available here , Usability Testing: The transcript of the usability testing interviews is available here . Collected Data

7. Data Overview Total of 45 participants were involved in this study: 37 surveyed users, 6 interviewed and 2 for usability testing. 73% of the survey participants were in the age range of 25-34 years old. Out of all participants (surveyed and interviewed) 35 were already users of the tool , 8 have never used the tool before and only 2 were not familiar with the ChatGPT concept at all. General data Familiarity with the concept of AI-powered chatbots is widespread among the participants. Tool is perceived as extension of Google and it’s mostly use the tool for job assistance (repetitive and operational tasks, writing Improvement, tailored search, problem-solving, and creative perspective). Interviewed users showed low familiarity with the tool possibilities and benefits compared to surveyed users - however all participants needs guidance on this. Common takeaways

8. Data Overview Overall satisfaction with the first-time usage of the tool was marked as very good both with surveyed and interviewed users - 4 out of 5 . The first-time experience with ChatGPT is positive, with users finding it interesting and valuable . Both surveyed and interviewed users, share concerns about data privacy, reliability, accuracy, and information quality. Non-users also approve of this by raising their concerns regarding the same topic of using ChatGPT. Surveyed non-users of the tool expressed a higher lack of need for its usage. Additionally, they demonstrated a limited understanding of the tool's possibilities and benefits. Both interviewed and surveyed users face challenges in defining queries and often struggle to get the desired answers, leading some users to abandon queries if they are not successful within a few interactions. No serious technical issues were noted during any of the first usages The interviewed users also appreciate the tool's potential and use it in both English and Serbian languages, with higher efficiency reported in English.

9. Identified entry barriers Lack of knowledge on ChatGPT usage often means lack of need to use it. This can be a blocker for first users that deeply affects motivation and interest to use the tool. Understanding the purpose 01. The sign-up process is perceived as lengthy and requires unnecessary steps , including email and phone verification. Onboarding process also needs to be more engaging, interactive, and valuable. Long signup 02. The first query thread can be time-consuming and mentally demanding for users - usually 4 to 5 iterations. Some users quit during the process. Demanding detailed queries 03.

10. When we don't understand each other over queries, I try two more times to rephrase to get ideas, and in the end, I write my own, but his suggestions still mean a lot to me. Sometimes, I give up after 2-3 attempts." - ChatGPT User, 29, SDR

11. Identified entry barriers No information on references or logic used for given answers. No real-time or up-to-date data. Lack of reliability of the tool 04. Users are concerned about data privacy. They don’t feel safe sharing sensitive data with the tool which limits the number of use-cases and decrease their motivation. Data privacy concerns 05. Expecting too much without knowing what ChatGPT is capable of. Not knowing what user is looking for. Both can bring new users to disappointment . Big or unclear expectations 06.

12. 04. Recommendations

13. Display references and sources so users can verify provided information. Provide more up-to-date or real-time data to increase the quality and reliability of answers. Increase reliability of the tool Provide transparent information on data handling. Make arrangements with different companies so their employees can use the tool safely and freely. Increase privacy and level of trust

14. Even for these easy tasks I delegate to it, I double-check to ensure accuracy. I approach every piece of information with caution, because it might type faster if I'm in a time crunch, but the accuracy is around 80%. - ChatGPT User, 28, Software developer

15. "I haven't shared sensitive data so far. I received instructions from my employer that we are not allowed to share sensitive data with it, and that we can’t use it. So I use it from my own account." - ChatGPT User, 29, SDR

16. Provide guidance on how ChatGPT can be used in everyday life and for some professional roles. Relate users with similar interests and usage patterns to recommend how they can best utilize the tool. Create an open statistics feature to showcase how others are using ChatGPT. Educate the market on potential applications of ChatGPT Create proactive user guidance to teach and lead users to clearly convey the specific input required to obtain desired answers. The tool should ask questions to specify unclear queries the user got and ask for additional inputs to get more satisfying results - to anticipate users' needs and suggest relevant information. Proactive query guidance

17. I miss feature 'Did you mean this?' so it guide me on how to talk to it when it already sees that it repeats some answers even though the questions asked are different. Like a conversation template to better engage and obtain better information. - ChatGPT First-time user, 29, p sychology student

18. I would like to use it more often - but I don't have an idea of all the things it can be used for. I can Google it, but I'm not that interested - I believe I could use it more for work, but I'm just not enthusiastic about it. - ChatGPT User, 29, SDR

19. Make a tool interactive and personalized so it can connect the data and assume its own suggestions on use cases from which users can benefit (Did you mean…? I noticed you searched hotels in Dubai, do you need transportation as well?, You’ve been using my help with ___, you might need help with ____ as well, etc.) Interactive learning loop & personalization Provide a more playful onboarding flow. Make an interactive process where you can get familiar with the interface, try it out before the signup, learn about the interactions, and ask queries. Interactive signup process

20. Perhaps if it could understand the sequence of my next questions and start suggesting things I might need - like a place to stay, tickets, etc., without me having to type it all out. To act like some sort of assistant and recommend everything I need . - ChatGPT First-time user, 28, software developer

21. Enhance the tool's performance in multiple languages, especially for translation purposes, to ensure more accurate and relevant results. Language optimization Allow users to take notes on given answers and save searches with it, and synchronize the tool with their mobile devices for convenient access. Note-taking and mobile integration

22. Thank you !