Real time conversion of sign language to text and speech presentation

Sign language is a rich and expressive form of language used by the deaf and hard hearing community.It involves the use of hand gestures to convey meaning and emotions. However not everyone is familiar with the sign language making it challenging for them to respond to sign language conversations. In this project, we basically focus on producing a model which can recognise hand gestures in order to form a complete word by combining each gesture and also to break the communication barrier.

1. Real-time Conversion of
Sign Language to Text
Members:
Geo Nibson K. B VAS19EE026
Hariprasad K. R. VAS19EE029
Harsh K Satheesh VAS19EE030
Jeevan Chacko VAS19EE033
Guide: Arun Xavier
Assistant Prof. Vidya Academy of
Science and Technology
Co-guide: Lasitha Mohandas
Assistant Prof. Vidya Academy of
Science and Technology

S8 EEE, Dept. of Electrical and Electronics Engineering
Vidya Academy of Science and Technology
22/06/2023 1

2. Contents
⮚Introduction
⮚Objectives of Work
⮚Literature review
⮚Methodology
⮚Result
⮚Budget
⮚References
22/06/2023 2

3. Introduction
⮚Our model, which can recognize fingerspelling
based hand gestures in order to form a complete
word by combining each gesture.
⮚Gestures are the nonverbally exchanged messages
and these gestures are understood with vision.
⮚This nonverbal communication of deaf and dumb
people is called sign language.
22/06/2023 3

4. Introduction
22/06/2023 4
ASL Gestures

5. Objectives of Work
⮚To convert ASL(American Sign Language) signed
hand gestures into text as well as speech
⮚Eliminate communication barrier with the hearing
impaired and as well as provide teaching aid for sign
language.
22/06/2023 5

6. Literature Review
Sl.
No
Title Year Author Summary
1. Using Deep
Learning in Sign
Language
Translation to
Text

2022 Mary Jane C, Carl
Jose M. Guingab, Ron
Andrew Relayo, Mark
Joseph C. Sheng, John
Ray D. Tamayo School
of Information
Technology, Mapua
University.
The systematic review paper
successfully determined the
factors that are present in most
sign language to text methods,
focusing on deep learning features
and its success in identifying
human gesture input and output
the accurate translations.
2. Conversion of
Sign Language
into Text

2018 Mahesh Kumar N B
Assistant Professor
Bannari Amman
Institute of
Technology, Erode,
India.
Using various concepts of image
processing and fundamental
properties of image and
developed this system. By using
LDA algorithms recognition of
gesture has done successfully.
22/06/2023 6

7. Literature review
Sl.
No
Title Year Author Summary
3. Sign Language
to Speech
Conversion-An
Assistive System
for Speech
Impaired.
2020 Nagesh Kumar D N,
Archana V Marathe,
Madhukar M, Manoj,
Adarsh Prabhakara, Shyam
S Bharadwaj Dept of ECE,
Jyothy Institute of
Technology, Bengaluru.
It is done using identifying the
edges and corners of image
using image processing
techniques. Image processing is
done using Python OpenCV
4. A Review on
Conversion of
Sign Language
into Text and
Audio
2022 Nishchal Gowda H S,
Preksha V L, Prathibha M,
Prathiksha N V,Dr. Surekha
T P .Student, Electronics
and Communication
Engineering,
Vidyavardhaka College of
Engineering, Karnataka,
India
Different techniques like
sensor-based technology, image
processing techniques, vision-
based technology, discrete
wavelet transformation
technique, and machine
learning using python for
conversion of sign language
into text and speech.
22/06/2023 7

8. Methodology
Input by
the user
Output
Hand
Gestures Capturing the image
and Identifying the
text corresponding
to the hand gesture
Text and
speech
22/06/2023 8

9. Methodology
Image
Acquisition
Hand Detection and
Tracking
Landmark
detection
Data Extraction Training
Training model Prediction
Example Output
(text and speech)
Camera
Sign
language
22/06/2023 9

10. MediaPipe Library
⮚The MediaPipe Hand Landmarker task lets you
detect the landmarks of the hands in an image.
⮚The task to localize key points of the hands and
render visual effects over the hands.
⮚This task operates on image data with a machine
learning (ML) model as static data or a continuous
stream and outputs hand landmarks in image
coordinates.



22/06/2023 10

11. Hand-landmarks
22/06/2023 11

12. Random Forest Classifier
⮚Ensemble learning algorithm used for classification tasks.
⮚Combines multiple decision trees.
⮚Each tree is trained on a random subset of features and
data.
⮚Predictions are made by aggregating the predictions from
all the decision trees using majority voting.
⮚Handles large datasets, high dimensionality, noisy data,
and missing values effectively.
⮚Provides robust and accurate predictions.
⮚Versatile and widely used in various applications.
⮚Examples of applications include spam email detection,
sentiment analysis, and medical diagnosis.
22/06/2023 12

13. Random Forest Classifier

22/06/2023 13

14. Circuit Diagram
22/06/2023 14

15. Simulation result
Image acquisition Hand-landmarks
detection
Prediction
B
22/06/2023 15

16. Simulation Result
22/06/2023 16

17. Simulation Result
22/06/2023 17

18. Hardware Components
Raspberry Pi Board






Micro computer contains
programmable GPIO pins.
Programmed using Python

22/06/2023 18
Camera





Has a 5-megapixel
camera with adjustable
focus.
High resolution and
video recording up to
1080p 30fps

19. Hardware Components
Speaker and Amplifier




Maximum power
output of 3 Watts
The speaker has an impedance
of 4 ohms.
22/06/2023 19
I2C LCD





Consists of an LCD panel, a
driver IC and an I2C interface
module
I2C is a serial communication
protocol that allows multiple
devices to communicate with
each other using just two wires

20. Result

22/06/2023 20

21. Budget
Components Price
(Rs)
Raspberry Pi 3 5600
LCD Display 250
Speaker and circuit board 600
Other expenses 700
Total expenses 7150
22/06/2023 21

22. Conclusion and Future Scope
⮚This project is about conversion of American Sign
Language to text and speech in real time.
Raspberry Pi is used and Python language used to
simulate the project. The output is given as text
and speech.
⮚This deep learning model is trained over the
American sign language dataset, this model can be
diversified by training over the Indian sign
language dataset, British sign language dataset,
etc.

22/06/2023 22

23. References
⮚Priyanka R. Potdar, Dr. D. M. Yadav,” Innovative
Approach for Gesture to Voice Conversion”, International
journal of innovative research and development 2014
⮚Arslan Arif, Syed Tahir Hussain Rizvi, Iqra Jawaid,
Muhammad Adam Waleed, Techno-Talk: An American
Sign Language (ASL) Translator, at CoDIT’16 - April 6-8,
2016, Malta
⮚Vi N. T. Truong, Chuan-Kai Yang, Quoc-Viet Tran, ”A
translator for American sign language to text and speech”,
2016 IEEE 5th Global Conference on Consumer
Electronics.

22/06/2023 23

24. Reference
⮚Sathwick, Sign Language Recognition using Deep
Learning 2022, https://towardsdatascience.com/sign-
language-to-text-using-deep-learning-7f9c8018c593
⮚Vibhu Gupta, Mansi Jain, Garima Aggarwal, ”Sign
Language to Text for Deaf and Dumb”, 2022 12th
International Conference on Cloud Computing, Data
Science Engineering.

22/06/2023 24

25. Thank you
22/06/2023 25