HCI_PROJECT_USING_HANDGESTURE_AND_CONTROL

HCI-project

1. ENHANCING HUMAN-COMPUTER INTERACTION: USING MEDIAPIPE AND OPENCV IN PYTHON FOR HAND TRACKING 21BPS1006- NIKHIL N IYER 21BPS1215- KASANGOTTU PRANEETH 21BPS1355- S PRANAY 21BPS1359- B V B J SANTOSH ARYAN

2. The presentation will delve into the integration of MediaPipe and OpenCV within Python to achieve accurate hand tracking capabilities. By harnessing the potential of these tools, we aim to enhance human-computer interaction by enabling precise hand gesture recognition and tracking. Through this exploration, we seek to showcase the practical applications and benefits of utilizing MediaPipe and OpenCV for advanced hand tracking tasks, emphasizing their role in improving user experiences and interaction possibilities in various domains. Introduction

3. Detailing the sequential process from library installation to input stream preprocessing involves: Installing necessary libraries. Preprocessing the input stream. Implementing MediaPipe's hand detection model and employing OpenCV algorithms for tracking entails: Implementing MediaPipe's hand detection model. Utilizing OpenCV algorithms for tracking. Integrating MediaPipe and OpenCV components for seamless real-time hand tracking involves: Integrating MediaPipe components. Combining with OpenCV for real-time tracking. Methodology: Approach

4. Full Code import cv2 import mediapipe as mp import math import numpy as np from ctypes import cast, POINTER from comtypes import CLSCTX_ALL from pycaw.pycaw import AudioUtilities, IAudioEndpointVolume import speech_recognition as sr numbers_dict = {'volume': 0, 'brightness': 1, 'two': 2, 'three': 3, 'four': 4, 'five': 5, 'six': 6, 'seven': 7, 'eight': 8, 'nine': 9, 'ten': 10} r = sr.Recognizer() with sr.Microphone() as source: print("Speak the word volume or brightness : ") while True: try:

5. audio = r.listen(source) text = r.recognize_google(audio) if text.lower() in numbers_dict: number = numbers_dict[text.lower()] print("You said: {text} ({number})") break else: print("Sorry, that wasn't a number. Please try again.") except sr.UnknownValueError: print("Sorry, I could not understand audio") print(number,type(number)) if number==0: print("You said Volume") else: print("You said Brightness") number = 1 if number == 0: mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles mp_hands = mp.solutions.hands

6. # Volume Control Library Usage devices = AudioUtilities.GetSpeakers() interface = devices.Activate(IAudioEndpointVolume._iid_, CLSCTX_ALL, None) volume = cast(interface, POINTER(IAudioEndpointVolume)) volRange = volume.GetVolumeRange() minVol , maxVol , volBar, volPer= volRange[0] , volRange[1], 400, 0 # Webcam Setup wCam, hCam = 640, 480 cam = cv2.VideoCapture(0) cam.set(3,wCam) cam.set(4,hCam) # Mediapipe Hand Landmark Model with mp_hands.Hands( model_complexity=0, min_detection_confidence=0.5, min_tracking_confidence=0.5) as hands:

7. while cam.isOpened(): success, image = cam.read() image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) results = hands.process(image) image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) if results.multi_hand_landmarks: for hand_landmarks in results.multi_hand_landmarks: mp_drawing.draw_landmarks( image, hand_landmarks, mp_hands.HAND_CONNECTIONS, mp_drawing_styles.get_default_hand_landmarks_style(), mp_drawing_styles.get_default_hand_connections_style() ) # multi_hand_landmarks method for Finding postion of Hand landmarks lmList = [] if results.multi_hand_landmarks: myHand = results.multi_hand_landmarks[0]

8. for id, lm in enumerate(myHand.landmark): h, w, c = image.shape cx, cy = int(lm.x * w), int(lm.y * h) lmList.append([id, cx, cy]) # Assigning variables for Thumb and Index finger position if len(lmList) != 0: x1, y1 = lmList[4][1], lmList[4][2] x2, y2 = lmList[12][1], lmList[12][2] # Marking Thumb and Index finger cv2.circle(image, (x1,y1),15,(255,255,255)) cv2.circle(image, (x2,y2),15,(255,255,255)) cv2.line(image,(x1,y1),(x2,y2),(0,255,0),3) length = math.hypot(x2-x1,y2-y1) if length < 50: cv2.line(image,(x1,y1),(x2,y2),(0,0,255),3) vol = np.interp(length, [50, 220], [minVol, maxVol]) volume.SetMasterVolumeLevel(vol, None) volBar = np.interp(length, [50, 220], [400, 150]) volPer = np.interp(length, [50, 220], [0, 100])

9. # Volume Bar cv2.rectangle(image, (50, 150), (85, 400), (0, 0, 0), 3) cv2.rectangle(image, (50, int(volBar)), (85, 400), (0, 0, 0), cv2.FILLED) cv2.putText(image, f'{int(volPer)} %', (40, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0, 0, 0), 3) cv2.imshow('handDetector', image) if cv2.waitKey(1) & 0xFF == ord('q'): break cam.release() elif number == 1: import screen_brightness_control as sbc import mediapipe as mp import cv2 import numpy as np from math import hypot cam_clip = cv2.VideoCapture(0) my_Hands = mp.solutions.hands hands = my_Hands.Hands() Hand_straight_line_draw = mp.solutions.drawing_utils

10. while True: success, img = cam_clip.read() imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) results = hands.process(imgRGB) lmList = [] if results.multi_hand_landmarks: for handlandmark in results.multi_hand_landmarks: for id, lm in enumerate(handlandmark.landmark): h, w, _ = img.shape cx, cy = int(lm.x * w), int(lm.y * h) lmList.append([id, cx, cy]) Hand_straight_line_draw.draw_landmarks(img, handlandmark, my_Hands.HAND_CONNECTIONS) if lmList != []: x1, y1 = lmList[4][1], lmList[4][2] x2, y2 = lmList[8][1], lmList[8][2] cv2.circle(img, (x1, y1), 4, (255, 0, 0), cv2.FILLED) cv2.circle(img, (x2, y2), 4, (255, 0, 0), cv2.FILLED) cv2.line(img, (x1, y1), (x2, y2), (255, 0, 0), 3)

11. length = hypot(x2 - x1, y2 - y1) bright = np.interp(length, [15, 220], [0, 100]) # print(bright, length) sbc.set_brightness(int(bright)) # cv2.rectangle(img, (50, 150), (85, 400), (0, 0, 0), 3) # cv2.rectangle(img, (50, int(bright)), (85, 400), (0, 0, 0), cv2.FILLED) cv2.putText(img, f'{int(bright)} %', (40, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0, 0, 0), 3) cv2.imshow('Image', img) if cv2.waitKey(1) & 0xff == ord('q'): break

12. Gesture Volume Control Gesture Volume Control, which fundamentally transforms the way users interact with audio systems. By harnessing the power of hand gestures, users can seamlessly adjust the volume levels of their devices without the need for physical buttons or controls. This innovative technology utilizes real-time hand tracking capabilities, enabled by advanced computer vision algorithms. Users simply make intuitive gestures in the air, such as moving their hand up or down, to increase or decrease the volume respectively.

13. Gesture Brightness Control Gesture Brightness Control, which redefines the way users manage the brightness levels of their devices. By leveraging intuitive hand gestures, users can effortlessly adjust the brightness of screens and displays without the need for traditional buttons or sliders. This cutting-edge technology utilizes real-time hand tracking capabilities, powered by advanced computer vision algorithms. Users can simply make natural gestures, such as swiping up or down with their hand, to increase or decrease the brightness respectively.

14. Voice-Assisted Gesture Control for Seamless Audio and Display Management Voice-Assisted Gesture Control, designed to seamlessly integrate voice commands with intuitive hand gestures for enhanced audio and display management. With this innovative system, users can effortlessly adjust both volume and brightness levels using a combination of voice commands and hand gestures. When utilizing voice commands to adjust volume, our system intelligently recognizes the user's intent and seamlessly transitions to Gesture Volume Control, allowing users to fine-tune audio levels with intuitive hand gestures. Similarly, when assisting with brightness settings, the system seamlessly activates Gesture Brightness Control in response to voice commands, enabling users to adjust display brightness effortlessly using natural hand movements.

15. we aim to explore the tangible applications of precise hand tracking facilitated by MediaPipe and OpenCV. By delving into real-world scenarios, we intend to illustrate the transformative potential of this technology in reshaping human-computer interaction across diverse fields. we will examine how accurate hand tracking can revolutionize interactions between humans and computers. Through this exploration, we will highlight the wide-ranging impact of this technology on industries such as gaming, virtual reality, augmented reality, education, healthcare, and more. Strengthening Human Computer Intraction

16. Envisioning the anticipated results entails foreseeing precise and dynamic hand detection and tracking capabilities in real-time scenarios. This involves not only accurately identifying hand movements but also adapting to dynamic changes swiftly and effectively. When discussing the vast array of potential applications, we're exploring the wide-reaching possibilities across various domains. Gesture recognition stands out as a key area where this technology could be applied, allowing users to interact with devices and interfaces using intuitive hand gestures. Impact and Potential Uses

17. The core objective of the project is to develop a functional system capable of real-time hand tracking using MediaPipe and OpenCV within Python. This involves creating a software solution that can accurately detect and track hand movements in real-time scenarios. By utilizing MediaPipe and OpenCV, the project aims to leverage advanced computer vision techniques to achieve this goal efficiently. The project strives for high accuracy and robustness in hand detection and tracking mechanisms. Highlights of the Project

18. Video Demonstration: Exploring Intuitive Gesture Control in Action https://drive.google.com/file/d/1PjK8_JlwFt6v8EZWJfGHutMCXkiwJAEh/view?usp=drivesdk

19. The emphasis is placed on significant discoveries and advancements in the realm of real-time hand tracking, highlighting notable progress and breakthroughs achieved thus far. These advancements signify strides made in improving the accuracy, efficiency, and versatility of hand tracking technologies. Conclusion

20. References MediaPipe: A Framework for Building Perception Pipelines. Retrieved from: https://mediapipe.dev/ OpenCV Documentation. Retrieved from: https://docs.opencv.org/4.x/index.html

21. THANK YOU