“Optimized Vision Language Models for Intelligent Transportation System Applications,” a Presentation from Nota AI

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/optimized-vision-language-models-for-intelligent-transportation-system-applications-a-presentation-from-nota-ai/

Tae-Ho Kim, Co-founder and CTO of Nota AI, presents the “Optimized Vision Language Models for Intelligent Transportation System Applications” tutorial at the May 2024 Embedded Vision Summit.

In the rapidly evolving landscape of intelligent transportation systems (ITSs), the demand for efficient and reliable solutions has never been greater. In this presentation, Kim shows how an innovative approach—optimized vision language models—can dramatically enhance the accuracy and robustness of computer vision solutions for ITSs.

Kim also illustrates how optimized vision language models can be implemented in real time at the edge, enabling intelligent decision-making for applications such as traffic management, vehicle recognition and pedestrian safety. Finally, he explains how Nota AI is utilizing optimized vision language models to revolutionize numerous ITS applications, leading to safer, more efficient and environmentally friendly transportation systems.

1. Optimized Vision Language
Models for Intelligent
Transportation System
Applications
Tae-Ho Kim
CTO & Co-Founder
Nota Inc.

2. •In this talk, we explorethe challenges in ITS.
•How vision language model (VLM)can solve these challenges.
•Future work will also be addressed.
Introduction
2© 2024 Nota Inc.

3. Identity of NotaAI
3© 2024 Nota Inc.Artificial Intelligence Semiconductors (System)
Computer
Vision
Natural
Language
Processing
LLMs
GenAI
CPU
NPU
TPU
GPU
Embedded
BoardSTT/TTS
Edge
Hardware-aware AI Model
Optimization
Nota AI bridges the gap between AI & semiconductors.

4. Nota AI’s Main Product: NetsPresso
4© 2024 Nota Inc.
NetsPresso
®
simplifies AI model optimization for target devices with automated processes.

5. Platform& EdgeSolutions: ElevatingExcellence
5© 2024 Nota Inc.Solution SolutionPlatform
Using NetsPresso, Nota AI also has created a solution business in ITS

6. NotaAI's Expertise
6© 2024 Nota Inc.
Nota AI also specializes in GenAI compression

7. 7© 2024 Nota Inc.
Intelligent Transportation System (ITS)

8. EnhancingTrafficFlow: Real-Time LightweightITS Solutions
8© 2024 Nota Inc.VRU SafetySolutions
AI-based real-time hazardous situation
screening and control system
AutomaticIncident
Management System
Real-timeincidentdetectionandanalysis
forsafe road condition
AI SmartParking
Real-timeparkingoccupancyand
parkingfacilityutilizationanalysis
SmartIntersectionSystem
Real-time incident detection and
analysis for safe road condition

9. Use Case 1:SmartIntersectionSystem
9© 2024 Nota Inc.
○Daejeon metropolitan city ITS
construction project
○Intersection CCTV AI video analysis
(600 ch)
○98% accuracy in traffic volume
counting in night and rainy conditions

10. Use Case 2: AI Safe Crossing
10© 2024 Nota Inc.

11. Use Case 3: VRU SafetySolutions
11© 2024 Nota Inc.

12. Use Case 4: AI Smart Parking
12© 2024 Nota Inc.
○UK Milton Keynes stadium
outdoor parking management
○USA San Diego outdoor parking
management (Caltrans)

13. •Ill-posed problem: How can we define “road debris”?
•Contextual problem: How can we define “accidents” with object detection?
•Rare dataset: How can we obtain dataset?
Challenges in ITS
13© 2024 Nota Inc.

14. Ill-posed Problem: Road Debris
14© 2024 Nota Inc.
Can it be detected by legacy AI models?

15. Contextual Problem: Accident Detection
15© 2024 Nota Inc.
Can it be detected by legacy AI models?

16. •Ill-posed problem: How can we define “Road debris”?
•Contextual problem: How can we define “Accidents” with Object detection?
•Rare dataset: How can we obtain dataset?
Challenges in ITS
16© 2024 Nota Inc.
Requires super-generalized model: Foundation Model

17. Foundation Model on the Edge: Challenges in Industrial AI
17© 2024 Nota Inc.
Deep Learning Model
Logic Layer
(Kalman filtering…)
As-is Problem
•Rule-based algorithm is fragile.
•Logic added for new requirements.
•Errors on the object detection/tracking propagate to logic layers.
•On-site calibration.
•Data drift.
•Data acquisition on rare events is hard.
•Sophisticated model composition.
•On-site calibration.
Input Image
Inference Result

18. •VLM is capable of comprehending complex scene.
•VLM already contains various logic.
•VLM is robust on data drift.
•VLM is aware of rare events.
•VLM needs less or no calibration.
•Still, VLM is not understanding video.
Foundation Model
(VLM)
Input Image
Inference Result
Foundation Model on the Edge: Challenges in Industrial AI
18© 2024 Nota Inc.
To-be Features

19. Vision Language Model: LLaVA
19© 2024 Nota Inc.
LLaVA Live LLaVA
Source: jetson-ai-lab.com

20. Working Prototype of LLaVA on Accidents Detection
20© 2024 Nota Inc.

21. Benchmark on Models
21© 2024 Nota Inc.
Source: jetson-ai-lab.com

22. •More advanced optimization required
•VLM needs to comprehend temporal consistency
•Domain adaptation might be required for user specific scenario
•Interface for product required
•Prompt engineering is required for higher performance
Future Work
22© 2024 Nota Inc.

23. •Industrial AI is already a widely used technology, but technology is limited when
the problem is complex and underdetermined.
•Among GenAI models, a Vision Language Model (VLM) can understand complex
scenes, so it could analyze complex queries and events.
•For example, in ITS, road debris and car accidents are severe problems, but this
couldn’t be solved by legacy AI models.
•Using VLMs, these problems can be solved as well.
•However, VLMs are still computationally expensive, so lightweight VLMs are
required for the next step.
Conclusion
23© 2024 Nota Inc.

24. Visit Nota AI @318 !
24© 2024 Nota Inc.

25. 25© 2024 Nota Inc.
Thank you for your attention!