Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera

The performance of ChatGPT© and other LLMs has improved tremendously, and in online environments, they are increasingly likely to be used in a wide variety of situations, such as ChatBot on web pages, call center operations using voice interaction, and dialogue functions using agents. In the offline environment, multimodal dialogue functions are also being realized, such as guidance by Artificial Intelligence agents (AI agents) using tablet terminals and dialogue systems in the form of LLMs mounted on robots. In this multimodal dialogue, mutual emotion recognition between the AI and the user will become important. So far, there have been methods for expressing emotions on the part of the AI agent or for recognizing them using textual or voice information of the user’s utterances, but methods for AI agents to recognize emotions from the user’s facial expressions have not been studied. In this study, we examined whether or not LLM-based AI agents can interact with users according to their emotional states by capturing the user in dialogue with a camera, recognizing emotions from facial expressions, and adding such emotion information to prompts. The results confirmed that AI agents can have conversations according to the emotional state for emotional states with relatively high scores, such as Happy and Angry.

1. Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera Tokushima University Hiroki Tanioka

2. MOTIVATION 2

3. Office Moving I nformation ★ Here!! HOW TO SOLVE? I have a question... ① ★ Here!! Map Not user friendly as she/he have to move around. 3

4. HOW TO SOLVE? Office Moving I nformation ★ Here!! Online Support ② It would be better solution, but the fact that you actually visited the site means it may not be an online solution. I have a question... 4

5. HOW TO SOLVE? Office Moving I nformation ★ Here!! Can I help you? ③ The best solution would be to recognize the urgency and emotional state of the user, as online responses do not adequately convey this. I have a question... 5 Can I help you? Using a camera like Duckietown

6. FACIAL EXPRESSION RECOGNITION The same facial expression does not yield the same emotional state. Faces (upper left: normal, upper right: smiling, lower left: angry, and lower right: sad) of Japanese woman who visiting for consultation. 6

7. HOW TO BUILD? FER (Facial expression recognition in Python) gpt-3.5-turbo (a model of ChatGPT) Adding recognized emotion information in JSON format to the ChatGPT prompt FER 7

8. HOW TO TEST? Two types of dialogue scenario CASE A : ”Hello” + Emotion CASE B : Counseling for a friend + Emotion FBot prototype system is developed in Python script on Google Colaboratory 8

9. RESULT (CASE A): “Hello.” + Emotion Response (beginning of example) Understand Worry- ing Encouraging Hello! It’s nice to hear from you. How are you feeling today? If there’s something on your mind or if you’d like to chat --- about anything, I’m here to listen and support you. Remember, your emotions are valid, and it’s okay to express how you feel. Take your time, and feel free to share whatever you’re comfortable with. Hello. It’s great to see you feeling happy ! How can I assist you today? ✔︎ Hello! I’m here to listen and support you. If you’d like to talk about what’s on your mind or if there’s anything I can do to help, feel free to share. Remember, it’s okay to feel a range of emotions, and I’m here to offer assistance in any way I can. Take a deep breath, and know that you’re not alone. ✔︎ ✔︎ ✔︎ Hello! I’m here for you. If you’d like to talk about what’s been bothering you or if there’s anything specific you need help with, please feel free to share. Remember, it’s important to express your emotions and seek support when you’re feeling down . I’m here to listen and offer my assistance in any way I can. Take your time, and know that you are not alone. ✔︎ ✔︎ ✔︎ 9

10. RESULT (CASE B): “Thank you.”+ Emotion Response (beginning of example) Understand Worry- ing Encouraging You’re welcome! I’m here to support you. If you ever need to talk more about comforting your friend or if there’s anything else on your mind, feel free to reach out. Remember, your compassion and care for your friend are truly valuable. Take care of yourself too during this time . Sending you positive thoughts and warm wishes. ✔︎ You’re welcome! I’m glad to see you feeling happy . If you have any more questions or need further assistance in the future, feel free to reach out. Remember, your kindness and support can truly brighten someone’s day. Keep spreading positivity and joy. Take care and stay happy! ✔︎ ✔︎ You’re welcome! It’s important to acknowledge your feelings and know that it’s okay to experience a range of emotions. If you ever need someone to talk to or seek comfort, don’t hesitate to reach out. Remember, you are not alone, and there are people who care about you and want to support you. Take care of yourself and remember to be kind to yourself during this time. Sending you positive thoughts and strength. ✔︎ ✔︎ ✔︎ You’re welcome! I’m here to support you and offer guidance whenever you need it. Remember that you are a caring friend, and your efforts to comfort and support others are truly admirable. If you ever want to talk more or share your feelings, feel free to reach out. Keep spreading kindness and positivity, even during challenging times. Take care, and stay strong. Sending you virtual hugs and support! ✔︎ ✔︎ ✔︎ 10

11. SUMMARY AND REMAINING ISSUES 11