Arduino controlled Robot using hand gestures using Python

Presentation by Suyash Joshi showing how to use hand gestures using machine learning to control Arduino based robot using Python, Arduino C, Mediapipe and also saving the ML detections to influxDB.

Source code on GitHub: https://github.com/suyashcjoshi/HandGestureBot/tree/main

1. | © Copyright 2023, InfluxData1
Building a gesture
controlled bot with
Arduino & Python


Suyash Joshi
sjoshi@influxdata.com

PyData London Aug 2023

2. | © Copyright 2023, InfluxData2
WhoAmI
●Senior developer advocate at InfluxData.
●10+ years of software engineering ar Oracle & RingCentral in
the SF Bay Area.

●MA in Game Design from Goldsmiths, Univ. of London,
●BS in Computer Science from University of Michigan

●Meetup Organizer: Creative Code & AI
●STEM Evangelist & Magician ??????

●@suyashcjoshi on social media

3. | © Copyright 2023, InfluxData3
Bill of Material


1.Arduino Uno R4, 1 DC motor
with shield, 5 Servo Motors, 9V
Battery and connector*

2.MeArm Robot (Laser Cut,
assembled by hand)

3.Walking Robot (Laser cut,
assembled by hand) with 1 DC
Motor and Battery soldered on

4.Miscellaneous: Screwdriver,
Soldering Iron, Solder, jumper
wires etc

4. | © Copyright 2023, InfluxData4
Why Arduino based Robotics ?
-Your entry to robotics!
-Simulates real world use cases : Crane operation, Robot Dog
-Real robots are expensive!
-Open source code, DIY plans, Affordable robot kits are easily available
Boston Dynamic Spot Robot Industrial Crane

5. | © Copyright 2023, InfluxData5
Software


1.Open Source Arduino & Python
program using multiple libraries

2.GestureRecognizer.py: Using
MediaPipe and pre-trained model to
detect hand gestures and showing
using OpenCV

3.MotorController.py: Connects to
Arduino and motor using PySerial and
sends positional data to servo
motor(s)

4.InfluxDBWriter.py : Connectina and
writing values at time intervals very
quickly to the DB

6. | © Copyright 2023, InfluxData6
write()

query()*
pyserial


MediaPipe
for Gesture
Recognition



InfluxDB


Arduino Uno R4
(WiFi, Serial)


hand gesture rotation*/name is sent to the servo motor

7. | © Copyright 2023, InfluxData7
Live Code Demo
&
Code Walkthrough!
7

8. | © Copyright 2023, InfluxData8
Why Hand Motion and Gesture?
-Natural User Interface, alternate would game controller, keypads
-Easy to detect with Machine Learning - MediaPipe
-Can be trained for custom gestures, movements etc and deployed
on web, mobile, IoT devices for free!
-Trendy - AR, VR etc

9. | © Copyright 2023, InfluxData9
Why Arduino ?
-Affordable specially ESP-32 boards
-Accessible : Open source, lot’s of peripherals like motors, sensors etc
-Huge community online sharing Code, guides answering questions etc
-More fun!
-Can talk to Rapsberry Pi, Web, computer, Mobile etc

10. | © Copyright 2023, InfluxData10
Why a Time Series Database ?
-Optimized for ingestion of large quantities of data rapidly
-InfluxDB is a leading Open Source & Cloud time series database
available to get started for free! (no credit card required)
-For Logging & Monitoring (Observability)
-For saving state of the motor

11. | © Copyright 2023, InfluxData11
User Interface : Hand landmarks and
gesture detection to control robot

12. | © Copyright 2023, InfluxData12
Natural Human Interface
Robot Arm
Human
hand
AI

13. | © Copyright 2023, InfluxData13 | © Copyright 2023, InfluxData13
References
1.Google MediaPipe: https://github.com/google-ai-edge/mediapipe
2.Robot Arm: https://www.meetup.com/creativecode/
3.InfluxDB: https://www.influxdata.com/
4.Arduino Uno: https://store.arduino.cc/products/uno-r4-wifi
5.WebSerial: https://developer.mozilla.org/en-US/docs/Web/API/Web_Serial_API
6.PySerial: https://github.com/pyserial/pyserial
7.OpenCV: https://opencv.org/
8.Code on GitHub: https://github.com/suyashcjoshi/HandGestureBot

14. | © Copyright 2023, InfluxData14 | © Copyright 2023, InfluxData14
Thank you!
Suyash Joshi

@suyashcjoshi
sjoshi@influxdata.com