“Addressing Tomorrow’s Sensor Fusion and Processing Needs with Cadence’s Newest Processors,” a Presentation from Cadence

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/addressing-tomorrows-sensor-fusion-and-processing-needs-with-cadences-newest-processors-a-presentation-from-cadence/

Amol Borkar, Product Marketing Director at Cadence, presents the “Addressing Tomorrow’s Sensor Fusion and Processing Needs with Cadence’s Newest Processors” tutorial at the May 2024 Embedded Vision Summit.

From ADAS to autonomous vehicles to smartphones, the number and variety of sensors used in edge devices is increasing: radar, LiDAR, time-of-flight sensors and multiple cameras are more and more common. And, as sensors have improved, the data rates associated with them have also increased. Traditionally, a dedicated processor has been utilized to process data from each sensor independently. Today, however, there is a growing need for a single, unified processor capable of processing multimodal sensor data utilizing both classical and AI algorithms and implementing sensor fusion for robust perception.

In this talk, Borkar introduces the new Vision 341 DSP and Vision 331 DSP from Cadence. These cores provide a versatile single-DSP solution for various workloads, including image sensing, radar, LiDAR and AI tasks. He explores the architecture of these new processors, highlights their performance and efficiency and outlines the associated developer tools and software building blocks.

1. Addressing Tomorrow’s Sensor
Fusion and Processing Needs
with Cadence’s Newest
Processors
Amol Borkar
Product Marketing Director
Cadence Design Systems

2. •The number of sensors feeding data is increasing
•Combination of vision, radar, lidar, and Time-of-Flight (ToF) in use
•Automotive is a good example of a large number of sensors per vehicle
Major Market Trends
2
2
© 2024 Cadence Design Systems

3. Camera
Radar
Lidar
Rear
collision
avoidance
Blind spot
detection
Narrow Front
200m +
Main Forward
100m
Wide Forward
60m+
Forward Side
50m-100m
Rear Side
50m-100m
Rear 50m
Short Range
Radar 40m
Long Range
Radar200m+
Med. Range
Radar 200m
Med. Range
Radar 200m
Lidar mapping
coverage: 1-4
sensors
Blind spot
detection
More Sensors Better Perception ADAS Autonomy
Front
collisionavoidance
Traffic Jam assist
Highway Pilot assist
Auto Steer/ ACC
Acceleration/Deceleration
for Lane change &
Collision avoidance
>40 sensors
(source: Yole)
The Path to Autonomy (Growing Number of Sensors)
3
© 2024 Cadence Design Systems

4. Automotive Market Trends and Segmentation
4
Customers decide on platforms: Same basic architecture for different segments
Central Compute
Radar/Lidar/Vision/
Voice/Audio + more
Sensor processing,
AI + pre/post-proc.
Digital Cockpit/Infotainment
Radar/Lidar/Vision/
Voice/Audio
Sensor processing,
AI + pre/post-proc.
Edge/Domain Controllers
Radar/Lidar/Vision
Sensor processing
Low-end AI
MCU and CPU
ASIC or ASSP
Central Compute
SoC
$-
$10,000
$20,000
$30,000
20232024202520262027202820292030
Central Compute ($M) MPU ($M)
MCU ($M) Special Purpose Logic ($M)
2023 to 2033 CAGR
Automotive SoC
MPU: 8.95%
Central Compute: 29.28%
MCU: 12.14%
Special Purpose Logic: 15.14%
Source: IBS Aug 2022
© 2024 Cadence Design Systems

5. Major Automotive Market Trends
5
Increasing number of sensors
Different types of sensors such as camera, 4D radar, lidar, and Time-of-Flight
Looking both inside and outside the vehicle
Automotive vendors designing for feet-off to hands-off to eventually mind-off
ADAS and IVI functions moving to singular SoC “Central Compute”
Heterogenous compute and sensor fusion
Industry moving towards Software-Defined-Vehicles (SDV)
© 2024 Cadence Design Systems

6. Evolution of the 4D Imaging Radar
2D Radar 3D Radar 4D Radar 4D Imaging Radar
Azimuth (Horizontal Angular Information), Relative velocity (Doppler)
Range (Relative Distance) Range (Relative Distance), 60-77 GHz
Brings richness of data,
Higher field of view
Higher angular resolution
Camera-level 3D information
Elevation (Vertical Angular information)
Legacy Radar
Use of Radar/Vision/Lidar
Radar
Module
Camera
Central Compute ECU
Raw sensor
data
Need for sensor fusion
Sensor processing
3D point cloud processing
AI
6
© 2024 Cadence Design Systems

7. AI Inference Trend
Traditional DSP-
Based Algorithms –
HOG, HAR Classifier,
and Keyword
Detection
CNN
RNN
LSTM
Transformer
Language
Graph Neural
Network (NN)
Audio Codecs
Transformer
Vision (VIT)
Noise Control
Until 2014 2012 to 2018 2017 to now 2020 to now Future?
Heavy Convolution
Deep NN
DepthwiseConvolution
Attention
Matrix Multiplier
Softmax
LayerNorm
•AI workloads are changing rapidly
•SoCs have lengthy design/development cycles
•Silicon designed today, needs to stay relevant two or more years from now
Multiple VIT
Changing attention
SWIN Transformers
Need for future readiness in a rapidly evolving world of AI
7
© 2024 Cadence Design Systems

8. •What is sensor fusion?
•Combine information from multiple sensors
•E.g., left eye and right eye
•How does it help?
•Redundancy: If one fails other works
•Better quality: Error compensation and seeing more
•Utilize each sensor’s strength and minimize their weakness
•Leverage classical approaches + AI for the best combination!
Need For Sensor Fusion?
8
© 2024 Cadence Design Systems

9. Sensor Fusion Examples
9
Depth Perception (2 or more cameras) SLAM (Camera + IMU)
ADAS (Radar + Camera) ADAS (Camera + LiDAR + AI)
© 2024 Cadence Design Systems

10. •So many sensors feeding data
•So many different processing blocks
•Need for heterogeneous compute:
•Different architectures? Or common architectures
•Scalability issue
•Code reuse
Typical Perception System
10
AXI4 Interconnect (NoC)
Domain
Specific
DSP
AXI4 Interconnect (NoC)
L2 Memory
Domain-
Specific
Accelerator
NPU
Domain
Specific
DSP
Domain-
Specific
Accelerator
NPU GPU CPUCPU
© 2024 Cadence Design Systems

11. •Move toward central compute architecture
•Majority of compute will come to the central compute block
•Central SoC will be a critical component for computing
•Technology nodes are shrinking but manufacturing costs are going
up
•Scalability of processing unit/compute unit/IP will be necessary to
reduce costs
•Can I get more out of my IPs?
•Common architecture helps scale performance
•One IP can process multiple input modalities
(camera, radar, lidar, etc.)
•Increase performance by adding more instances or multi-
core
•Cadence has the answer for you
Racing Towards Central Compute
11
Multiple Different
Compute Units
Multiple Instances of
similar Compute Units
© 2024 Cadence Design Systems

12. •New 512b and 1024b SIMD DSPs to address vision, radar, lidar, AI,
and sensor fusion needs
•Combining the proven Tensilica® ConnXand Vision ISAs to
deliver the best PPA for high-end multi-sensor processing
•Everything of Tensilica® Vision Q7 and Vision Q8 plus more!
•Key improvements over previous TensilicaVision DSPs:
•More datatypes: BFloat16, Complex floating point
•CV filter improvements: 1D filters up to 2X better and
NMS up to 2.5X better
•New ISA to support fixed and floating point-based
•Fixed point and floating point support for radar operations, FFT, etc.,
and up to 6x performance improvement
•Improvements to neural network quantization and depthwiseseparable convolution performance
•Same libraries, software, ecosystem, and backward code compatibility
•Automotive grade and ASIL-certified IP
TensilicaVision 341/331 DSP IP: Vision/Radar/AI
12
© 2024 Cadence Design Systems

13. •Accelerates 2D and 1D Fast Fourier Transform (FFT)
commonly used in radar processing
•16/24-bit fixed-point processing
•AXI-based IP with multiple DMA (128/256-bit) for fast
data movement
•System software to work as FFT offload engine for
Vision 331/341
•ISO 26262 ready for automotive market
Vision 4DR Accelerator
Compared to a Vision 341 DSP:
4Xgreaterperformance
6Xgreater performance/area
13
© 2024 Cadence Design Systems

14. CPU
Complex
Sensor
Fusion
Hub
IO
Hub
Memory
AXI4 Interconnect (NoC)
Vision DSP
Inst
RAM
Data
RAM
AXI4 Interconnect (NoC)
L2 Memory
Vision DSP
Inst
RAM
Data
RAM
Vision 4DR
Acc
Inst
RAM
Data
RAM
Neo
NPU
Inst
RAM
Data
RAM
Neo
NPU
Inst
RAM
Data
RAM
•Multiple
•Tensilica
®
VisionDSPs
•Vision 4DR Accelerator for 4D Imaging Radar
•NPUs
Vision 4DR
Acc
Inst
RAM
Data
RAM
Sensor Fusion Hub for Automotive
14
© 2024 Cadence Design Systems

15. Output
Single DSP capable of processing multi sensor work loads saves area, power and increases performance + efficiency on an SOC
Unified DSP cores
Vision, Radar, LiDAR, ToF
Unified DSP cores
Vision, Radar, LiDAR,
ToF, and AI
Neo NPU
(NN Models)
R1
C2
Radar heads
R2
Vision PCL
Radar head
Camera
Fused Point cloud
Radar PCL
CV accelerators
FFT AcceleratorsC1
Camera
Unsupported layers
Unified DSP cores
Vision, Radar, LiDAR, ToF,
and AI
CV accelerators
FFT Accelerators
Sensor Fusion: Vision + Radar + LiDAR
15
© 2024 Cadence Design Systems

16. OS Layer (XTOS, XOS, ThreadX, and FreeRTOS)
Embedded
C/C++
HalideOpenCL
ONNX/TensorFlow/
PyTorch
Tensilica® XtensaC/C++ Compiler (LLVM)
OpenCL Compiler (LLVM)
Halide Compiler NeuroWeave™
XAF
TensorFlow
Micro Lite
CV Lib/
SLAM Lib/
DSP Lib/
Eigen Lib/
Simulink Lib/
Radar Lib
OpenCL
Runtime
OpenCL BIFL
Library
NN Library
Audio Lib/
NN Library
XRP
E
c
o
s
y
s
t
e
m
iDMA
Memory
Manager
XIPC
Tensilica® Xtensa
and TIE
Vision Radar/Lidar/CommsAudio/Voice
Cadence Compiler / Tool
Cadence SW library / Runtime
User Code
Tensilica® DSP
and Accelerators
Cadence Low level
SW Components
HAL
AI Processor
Cadence Tensilica: Comprehensive Software Solutions
16
© 2024 Cadence Design Systems

17. Cadence
®
Neo
™
Neural Processing Units (NPUs)
•Target broad range of on-device and edge AI applications from
smart speakers to autonomous driving
•Deliver wide range of AI performance in a low-energy footprint
•Efficiently offload AI inferencing from any host processor
Cadence NeuroWeave
™
Software Development Kit (SDK)
•Provides unified support across Cadence AI and TensilicaIP
products
•Streamlines productdevelopment and enables an easy
migration as design requirements evolve
AI Product Announcement –September 13, 2023
17
http://www.cadence.com/go/NPU
© 2024 Cadence Design Systems

18. Object Seg/
Localization
Object
Classification
Always
ON
Wake
Word
Face
Detection
Sound
Analytics
Speech
Recognition
Emotion
Detection
ADAS
Systems
One Cadence Tensilica AI Software Compiler Toolchain
Few GOPS 100s of TOPS
NeuroWeave™ SDK
Common Simulator,
Testbench, and KPI f/w
Configurable and
Scalable AI Compiler
Quant
bypass
© 2024 Cadence Design Systems

19. Putting It All Together
19
Radar Vision + AI Radar + Vision + AI Performance Markets
Turbo
Boost
Base
ConnX230
ConnX120
Tensilica®
ConnX230
ConnX110
Vision 230
Vision 130
Vision 240
ConnX110
4D Imaging Radar
L2 –L5
Military Application
Auto: In-cabin radar,
Park assist,
Trunk opening
Medical: Elderly care,
Childcare
Consumer: Person detection
4D Non-Imaging Radar
(L2/L3) -BSD
Industrial-Robotics
GPR's
3D Radar
4D Radar
4D Imaging Radar
2DRadar
Vision 331
Vision 341
Vision 230
Vision
4DR
Neo NPU
Neo NPU
Neo NPU
© 2024 Cadence Design Systems

20. Summary
Automotive Market
•Automotive SoC architectures driven by software-defined vehicles, large number of diverse sensors, and higher throughput
Tensilica
®
Vision 331 DSP and Vision 341 DSP: single DSP for sensor fusion
•A single DSP eliminates the need for multiple DSPs for different sensors: camera, radar
•Both DSPs come with large software ecosystems; existing vision and radar software, vision, and radar library
•Enable fast TTM with the same SIMD and VLIW architecture and instruction set used by their highly successful Vision
DSPpredecessors
Vision 4DR Accelerator: Hardware accelerator for growing 4D imaging radar
•When paired with the new DSPs for 4D imaging radar applications, the Vision 4DR accelerator offers:
•Greater performanceand performance/area advantagecompared to a DSP alone
© 2024 Cadence Design Systems

21. •The Role of Centralized Storage in the Emerging
Zonal Automotive Architecture
•Multi-Modal Sensor Fusion-Based Deep Neural
Network for End-to-End Autonomous Driving With
Scene Understanding
Resources
21
2024 Embedded Vision Summit
•Talk: SLAM for Embedded Systems:
An introduction and its challenges
May 23
rd
, 4:50 pm
•Cadence Booth
#518 Exhibition Hall
© 2024 Cadence Design Systems