“Entering the Era of Multimodal Perception,” a Presentation from Connected Vision Advisors

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/09/entering-the-era-of-multimodal-perception-a-presentation-from-connected-vision-advisors/

Simon Morris, Serial Tech Entrepreneur and Start-Up Advisor at Connected Vision Advisors, presents the “Entering the Era of Multimodal Perception” tutorial at the May 2024 Embedded Vision Summit.

Humans rely on multiple senses to quickly and accurately obtain the most important information we need. Similarly, developers have begun using multiple types of sensors to improve machine perception. To date, this has mostly been done with “late fusion” approaches, in which separate ML models are trained for each type of sensor data, and the outputs of these models are combined in an ad hoc manner. However, such systems have proven difficult to implement and disappointing in their perception performance.

We are now witnessing a transition away from this siloed sensor approach. Recent research shows that superior perception performance can be realized by training a single ML model on multiple types of sensor data. In this talk, Morris explains why this new approach to multimodal perception will soon dominate and outlines the key business challenges and opportunities that are emerging as a result, including challenges and opportunities in frameworks, tools, databases and models.

1. Entering the Era of
Multimodal Perception
Simon C. Morris
Serial Tech Entrepreneur and Start-Up Advisor
Connected Vision Advisors

2. Generative AI –All the Buzz
2© 2024 Connected Vision Advisors
Forbes 2Oct 2023: The 10 Biggest Generative AI Trends For 2024
Generate content, like:
1.Writing articles
2.Composing music
3.Designing graphics
4.Generating code
5.Etc…

3. There’s Another Opportunity
3© 2024 Connected Vision Advisors
Reasoning is perhaps the
ultimate goal of AI, but
with human evolution it
was both multisensory
integration and
reasoning that lead to
life saving decisions.
Obvious examples:
•Sight + sound
•Smell + taste

4. Multisensory –Better Decisions –Safer
4© 2024 Connected Vision Advisors
This Photoby Unknown Author is licensed under CC BY-SA

5. •Sensor fusion -the process of combining sensor data or data derived
from disparate sources
•The result -less uncertainty or a better decision compared to when the
sensor sources were used individually.
•Edge perception today -combination of camera, lidar, time of flight
(ToF), radar, ultrasonic, GPS sensor data
Multisensory Integration –Fusion on the Edge
5© 2024 Connected Vision Advisors

6. In the last three years
alone, there have been
over 1.7 million patents
filed and granted in radar-
camera fusion
Proof of the Importance of Fusion Opportunity
6© 2024 Connected Vision Advisors
Source: GlobalData’sreport on Artificial Intelligence in Automotive: Radar Camera Fusion
0
2
4
6
8
10
Range x FOV
Range Resolution
Elevation Resolution
Angular Resolution
Velocity
Discrimination
Processing
Overhead
Device Cost
Low Light
Performance
Dust/Fog/Snow
Robustness to
interference
Colour Detection
Object Classification
Object
Segmentation
Object Tracking
VisionRadar Lidar

7. Fusion Patent Volume 2020-2022 –Automotive
7© 2024 Connected Vision Advisors
Bubble size = patent
volumes 2020-2022
Source: GlobalDataPatent Analytics

8. Fusion for Edge Perception
8© 2024 Connected Vision Advisors
Camera Radar Fusion -YouTube
Ref: JunhyukHyun, Yonsei University
Korea, Computational Intelligence Lab
Camera Ready Fusion

9. Fusion for Edge Perception
9© 2024 Connected Vision Advisors
High-level fusion (decision fusion) -incumbent
• GPS + inertial navigation systems, or
• Fusing objects and trajectories of objects

10. Low-level fusion (early fusion of the raw data)
• Fuse lidar point cloud data + RGB camera data train AI model
• Fuse radar RAD data + RGB camera data train AI model
Fusion for Edge Perception –Where’s It Going?
10© 2024 Connected Vision Advisors
IEEE Transactions on Intelligent Vehicles ( Volume: 9, Issue: 1, January 2024)
Mid-level fusion (fusing features)
• Radar or lidar point cloud + vision object detection
Latest research shows that AI models trained on early fusion data will provide better detection

11. 1.Huge raw data bandwidth (raw radar data 30 Gbps +) to process –needs
very high AI performance, but low cost, power effective for edge
•Incumbents exceeding 20 TOPs: e.g. QCOM, TI, Ambarella, Renesas
•Challenging start-ups exceeding 20 TOPs: e.g. Hailo, Blaize, Brainchip, EdgeCortix,
untether, DeepX,
2.Lack of early-fusion training data to train AI models and non-public
Challenges to Adoption of Mid & Low-Level Fusion
11© 2024 Connected Vision Advisors

12. 3.Need for a perception processing framework for calibrating, synchronizing
different data sources
4.An easier (lower cost, time) means of collecting calibrated data sources in
the field
5.An easier (lower cost, time) means to annotate fused data, auditing
annotation and train an edge AI model based on this data
Challenges to Adoption to Mid & Low-Level Fusion
12© 2024 Connected Vision Advisors

13. Anywhere where multi-mode perception sensing is required such as:
•Automotive (ADAS to AD, driver monitoring)
•Off-road (agriculture, construction, mining, material movement,
defense)
•Factory, warehouse mobile robotics
•Drones (autonomy without pilot or GNSS)
•Security/surveillance
•Etc…
The Untapped Opportunity
13© 2024 Connected Vision Advisors

14. Semiconductor
•Incumbents exceeding 20 TOPs: e.g.
•Challenger start-ups exceeding 20 TOPs: e.g. Tens Torrents, SiMa.ai, Rebellions,
Picks & Shovels: Positioned for Edge Fusion Perception
14© 2024 Connected Vision Advisors

15. Edge AI Fusion Perception Frameworks -very few
Picks & Shovels: Positioned for Edge Fusion Perception
15© 2024 Connected Vision Advisors
AI Models Optimized for the Edge -focus primarily on vision only, not fusion
Data Annotation Tools -focus primarily on image and video annotation

16. 1.Perception based on data fusion yields superior perception
2.Challenges of “early fusion” are being overcome
3.Fusion will move this decade from incumbent “late fusion” designs to
“early fusion” implementations
4.Opportunities exist to differentiate throughout the value-chain to
address the demands of early fusion
Take Aways
16© 2024 Connected Vision Advisors

17. References/Resources
17© 2024 Connected Vision Advisors
GlobalDataPatent Analytics: Key players in radar camera fusion
JunhyukHyun, Yonsei University Korea, Computational Intelligence Lab
GlobalData’sreport onArtificial Intelligence in Automotive: Radar Camera Fusion
Forbes 2Oct 2023: The 10 Biggest Generative AI Trends For 2024
IEEE Transactions on Intelligent Vehicles ( Volume: 9, Issue: 1, January 2024) : Radar-Camera Fusion for
Object Detection & Semantic Segmentation in Autonomous Driving: A Comprehensive Review