Sign Language Delection using Machine Learning 21.pptx

PPT on sign language

1. RAJIV GANDHI COLLEGE OF ENGINEERING, RESEARCH & TECHNOLOGY, CHANDRAPUR Department of Computer Science and Engineering Mini Project On “ SIGN LANGUAGE DETECTION “ Semester V Mini Project - I Submitted By Mohit Titarmare Gaurav Wankar Gaurav Thapliyal Yash Raut Guided By Dr Rahila Sheikh Mini project Incharge Prof. Madhavi Sadu H.O.D,CSE Dr. Nitin Janwe

2. Sign Language Detection using Machine Learning

3. Introduction The world is hardly living without communication, no matter whether it is in the form of text, voice or visual expression. The communication among the deaf and dumb people is carried by text and visual expression. Hands and facial parts are immensely influential to express the thoughts of human in confidential communication.

4. Primary goals Sign language is learned by deaf and dumb, and usually it is not known to normal people, so it become a challenge for communication between a normal and hearing impaired person. It strike two over mind to bridge the between hearing impaired and normal people to make the communication easier. Sign language detection system takes an input expression from the hearing impaired person given output to the normal person in the form of text.

5. Data Description We analyze 4800 images of signed image which is ISL of the English alphabet which have a spread of 26 class labels assigned to them. each class label is a set of sign image of the English alphabet. All the image are size 300 and 360 pixel and we perform both the model optimization and prediction on these downscale image .

6. Below figures show an example of every class of sign images dataset .

7. Technical work Creating the dataset for sign language detection It is fairly possible to get the dataset we need on the internet but in this project, we will be creating the dataset on our own. We will be having a live feed from the video cam and every frame that detects a hand in the ROI (region of interest) created will be saved in a directory (here gesture directory) that contains one folder image, each containing 26 folders containing images captured using the create_gesture_data.py 7

8. Inside of train (test has the same structure inside) 8

9. Now for creating the dataset we get the live cam feed using OpenCV and create an ROI that is nothing but the part of the frame where we want to detect the hand in for the gestures. The red box is the ROI and this window is for getting the live cam feed from the webcam.

10. In the above example, the dataset for 1 is being created and the thresholded image of the ROI is being shown in the next window and this frame of ROI is being saved in ..image/ A /example.jpg 10 For the train dataset, we save 701 images for each number to be detected, and for the test dataset, we do the same and create 40 images for each number.

11. Training CNN Now on the created data set we train a CNN. First, we load the data using ImageDataGenerator of keras through which we can use the flow_from_directory function to load the train and test set data, and each of the names of the number folders will be the class names for the imgs loaded. from function import * from sklearn.model_selection import train_test_split from keras.utils import to_categorical from keras.models import Sequential from keras.layers import LSTM, Dense from keras.callbacks import TensorBoard label_map = { label:num for num , label in enumerate(actions)} # print( label_map ) sequences, labels = [], [] for action in actions: for sequence in range( no_sequences ): window = [] for frame_num in range( sequence_length ): res = np.load ( os.path.join (DATA_PATH, action, str(sequence), "{}. npy ".format( frame_num ))) window.append (res) sequences.append (window) labels.append ( label_map [action]) X = np.array (sequences) y = to_categorical (labels). astype (int) X_train , X_test , y_train , y_test = train_test_split (X, y, test_size =0.05) log_dir = os.path.join ('Logs') tb_callback = TensorBoard ( log_dir = log_dir ) model = Sequential() model.add (LSTM(64, return_sequences =True, activation=' relu ', input_shape =(30,63))) model.add (LSTM(128, return_sequences =True, activation=' relu ')) model.add (LSTM(64, return_sequences =False, activation=' relu ')) model.add (Dense(64, activation=' relu ')) model.add (Dense(32, activation=' relu ')) model.add (Dense( actions.shape [0], activation=' softmax ')) res = [.7, 0.2, 0.1] model.compile (optimizer='Adam', loss=' categorical_crossentropy ', metrics=[' categorical_accuracy ']) model.fit ( X_train , y_train , epochs=200, callbacks=[ tb_callback ]) model.summary () model_json = model.to_json () with open(" model.json ", "w") as json_file : json_file.write ( model_json ) model.save ('model.h5')

12. 12

13. Predict the gesture In this, we create a bounding box for detecting the ROI and calculate the accumulated_avg as we did in creating the dataset. This is done for identifying any foreground object. Now we find the max contour and if contour is detected that means a hand is detected so the threshold of the ROI is treated as a test image. We load the previously saved model using keras.models.load_model and feed the threshold image of the ROI consisting of the hand as an input to the model for prediction.

14. Sign Language Detection Output

15. Advantages Enhanced education and learning opportunities Promotes understanding and cultural awareness Increased accessibility and inclusion Improved customer service and interaction

16. Disadvantages Limited vocabulary and sign variations Potential for misinterpretation and bias Accuracy and reliability concerns Privacy and security issues

17. Conclusion In this report a functional real time vision based American Sign Language recognition for D&M people have been developed for asl alphabets. We achieved final accuracy for 98.0% on our data set. we are able to improve our prediction after implementing two layers of algorithm in which we verify and predict symbols which are more similar to each other .

18. (PDF) Real Time Sign Language Recognition and Speech Generation | Prof. Dr. Subarna Shakya - Academia.edu (PDF) SIGN LANGUAGE RECOGNITION (researchgate.net) (PDF) Language mediators as professional bilinguals (researchgate.net) (PDF) Sign Language Recognition, Generation, and Modelling: A Research Effort with Applications in Deaf Communication (researchgate.net) (PDF) Real-Time Sign Language Recognition Using a Consumer Depth Camera (researchgate.net) REFERENCE

19. Thank you