PATHFINDER FOR Mid Term Presentation.pptx

AI Pathfinder presentation

1. TLM3001 Design Project [2023/24 T3] Aw Jun Min [email protected] Project Title: PathFinder May 2024 Mid Term Presentation ‹#›

2. Agenda TLM300 1 Design Project Presentation Overview ‹#› ‹#› Introduction Problem Statement Proposed Solution Software and Libraries Hardware Implementation Overview Website Development Depth Net Preprocessing Unit Testing Challenges Faced Current Progress Remaining Project Tasks Conclusion

3. Problem Statement TLM3001 Design Project Introduction CHALLENGES FACED BY THE VISUALLY IMPAIRED ISSUES WITH CURRENT SYSTEMS NEED FOR INNOVATION Navigation and Mobility Limited Access to Information Limited Accuracy and Reliability High costs and availability Enhanced precision and real-time feedback Affordable and accessible solutions User-Friendly and Comfortable ‹#›

4. Proposed Solution TLM3001 Design Project Introduction Overview: Develop a comprehensive monitoring system to provide real-time insights into the visually impaired surroundings, fostering a sense of security and connection for friends and family. Key Deliverables: Website Development: Design and deploy a user-friendly website hosted on Firebase, enabling secure login and account creation for users. Edge Computing Integration: Implement edge computing capabilities using the Jetson Nano to process live video feeds captured by a wearable camera vest. Depth Sensing Algorithms: Develop and deploy depth estimation algorithms to process live video feeds, providing valuable insights into the user's surroundings. Objectives: Real-Time Monitoring: Enable real-time monitoring of the user's surroundings by processing live video feeds for depth sensing and providing access to processed images on the website. User Engagement: Foster user engagement and satisfaction by providing friends and family with valuable insights into the user's whereabouts and surroundings in real-time. ‹#›

5. Software and Libraries TLM3001 Design Project Architecture Python Numpy OpenCV Pyttsx3 Flask Threading Cuda Firebase TensorRT ‹#›

6. Hardware TLM3001 Design Project Architecture 1x Jetson Nano Developer Kit B01 1x 64GB MicroSD 1x 5V 4A DC Jack Universal Power Adaptor 1x Raspberry Pi 8MP Camera Module 1x AC8265 Wireless Module 1x Monitor for Testing 1x Wifi Dongle 1x Portable Power Source 1x Binocular Dual IMX219, 8MP Existing Hardware Pending Hardware ‹#›

7. Hardware TLM3001 Design Project Architecture ‹#›

8. Implementation Overview TLM3001 Design Project Architecture ‹#› Data Collection Preprocessing Parsing Transformation Visualisation Frames are retrieved from camera Jetson nano receives frames from camera Performs preprocessing of frames Queues preprocessed frames for further processing Algorithm uses depth commands, and deep learning to parse images. Upon parsing, we pass to Pysstx3 for audio and send images to the firebase and flask server. Webpage hosted locally, while being connected to cloud storage is able to show frames of the visually impaired point of view and live streams the visually impaired current location.

9. Website Development TLM3001 Design Project Implementation Overview ‹#› HTML & JavaScript Utilised HTML for structuring web content. Implemented JavaScript for dynamic functionality and interactivity. Connection to Firebase Storage Successfully integrated Firebase Storage with the website. Enabled storage and retrieval of data using Firebase. Hosting Limitations Unable to host the website solely on a local server. Required solutions to deploy the website for broader accessibility. Database Storage Integration Successfully connected the database storage to the website. Ensured smooth data transactions between the site and Firebase Storage.

10. Website Development TLM3001 Design Project Implementation Overview ‹#› Users are able to sign up and sign in, with its own unique ID, this makes the project scalable as long as the user has a Jetson Nano Authentication Within each user UID, every frame and log are able to be retrieved for viewing, and are kept and stored These URLs are references to current or past images captured Realtime Database

11. Website Development TLM3001 Design Project Implementation Overview ‹#› Upon authentication, live captured frames of the whereabouts can be seen. Captured Frames in Real time A further option to view the live feed of the user of PathFinder was also implemented. Live Video Stream

12. Depth Net Preprocessing TLM3001 Design Project Implementation Overview ‹#› The Algorithm is working on this principle [1,0,1], in the format of [left, centre, right]; “1” means clear “0” means not clear

13. Unit Testing TLM3001 Design Project System Test CSI-Camera Jetson Inference Pre-Testing Setup Checks Run gstreamer pipeline to check if camera is capturing video correctly Error handling to review error logs for issues in video streaming Perform testing on images for feasibility before implementing library ‹#›

14. Challenges Faced TLM3001 Design Project Dependency Issues: Compatibility problems with required Python versions and libraries. Need to downgrade certain libraries to be compatible with the Firebase Admin SDK. Firebase Admin SDK Compatibility: The Firebase Admin SDK is not inherently suitable for Jetson Nano. Jetson Nano's architecture and default Python version (3.6.9) require specific adjustments. Python Version Constraints: Resolved by using an older Python version to ensure compatibility with necessary libraries. Dependency Issues Concurrency Handling Storage Limitations ‹#›

15. Challenges Faced TLM3001 Design Project Challenge Managing concurrent video processing and web streaming. Both tasks must run in parallel, not in series. Solution: Threading Threads enable simultaneous video streaming and frame processing. Allows continuous listening and processing of video frames without interruption. Problem with Sequential Execution: If processes run in series: Flask server handles initialization, streaming, and processing one after the other, which causes delays and interrupts continuous streaming. Dependency Issues Concurrency Handling Storage Limitations ‹#›

16. Challenges Faced TLM3001 Design Project Storage Limitations Limited retention period of images URLs in Firebase Storage up to 7 days Dependency Issues Concurrency Handling Storage Limitations ‹#›

17. Current Progress TLM3001 Design Project Implementation Overview Accomplishments: Development of site, firebase storage and authentication. Jetson Nano software implementation for camera, and depth processing, and wifi module. Design and assembly of system hardware. Optimising algorithm to identify obstructions and returning an audio output. ‹#›

18. TLM3001 Design Project Fall Detection using Object Detection: Objective: Deploy, Train and Integrate trained ML model for real-time feedback for fall factors. Tasks: Train and optimise ML model in a closed environment. Implementation of second camera, Model: IMX219-83 Handling potential power issues Integrate System Ensure system is portable and functional for testing in real world environment. ‹#› Remaining Project Tasks Implementation Overview

19. Conclusion TLM3001 Design Project Project Recap: Achievements Next Steps Questions? Aimed to develop a cost-effective and robust visual assistance for the visually impaired Leveraging machine learning algorithms and Jetson Nano Successful development of Firebase, Jetson Nano dependencies and libraries, and hardware development. Focus on second Phase of project: training the model for detection of steps. Achieve real-time, reliable system based on machine learning Embrace Continuous Improvement ‹#›