“Meeting the Critical Needs of Accuracy, Performance and Adaptability in Embedded Neural Networks,” a Presentation from Quadric

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/07/meeting-the-critical-needs-of-accuracy-performance-and-adaptability-in-embedded-neural-networks-a-presentation-from-quadric/

Aman Sikka, Chief Architect at Quadric, presents the “Meeting the Critical Needs of Accuracy, Performance and Adaptability in Embedded Neural Networks” tutorial at the May 2024 Embedded Vision Summit.

In this presentation, Sikka explores the challenges of accuracy and performance when implementing quantized machine learning inference algorithms on embedded systems. He explains how the thoughtful use of fixed-point data types yields significant performance and efficiency gains without compromising accuracy. And he explores the need for modern SoCs to not only efficiently run current state-of-the-art neural networks but also to be able to adapt to future algorithms.

This requires industry to shift away from the approach of adding custom fixed-function accelerator blocks adjacent to legacy architectures and toward embracing flexible and adaptive hardware. This hardware flexibility not only allows SoCs to run new networks, but also enables ongoing software and compiler innovations to explore optimizations such as better data layout, operation fusion, operation remapping and operation scheduling without being constrained by a fixed hardware pipeline.

1. Meeting the Critical Needs
of Accuracy, Performance
and Adaptability in
Embedded Neural Networks
Aman Sikka
Chief Architect
Quadric

2. Agenda
2
•Challenges in current NPUs (Neural Processing Units)
•Trends in neural networks
•Increase need for DSP like ops but DSP cannot be a fallback
•Back to basics
•Fixed point vs. floating point
•Designing a flexible architecture
•Conclusion
•Q/A
© Quadric

3. 3
Traditional NNs and Hardware

4. Traditional CNN: Mainly MAC dominated
4
NPU Ops (matrix, pooling, activation)
Classic Algorithm & Control Ops
CPU/DSP
Hardwired
Convolution & Matrix
Co-processor
Pooling & Activations
(selected, pre-wired)
Conv
Conv
Pool
FC
Softmax
Conv
Conv
Pool
Conv
Conv
Conv Conv
Conv
Conv
Conv
Concat
FC
Softmax
© Quadric

5. Transformers: Can be heavy on DSP-like compute
5
Every data transfer between NPUblock
and CPU-DSPdecreasesperformance
and adds power
NPU operations (matrix, pooling,
activation)
Classic Algorithm & Control Ops
Layer Norm
Softmax
Shape xform
Matmul Matmul Matmul
Shape xformShape xform
Matmul
Matmul
Layer Norm
Matmul
Linear
Matmul
Attention block
Energy Cost of 32bdata element
transferfrom ALU/MAC to
Reg File 1
LRM 2-3X
L2 MEM 70X
Off-chip DDR225X
© Quadric

6. Non-NPU ops are here to stay
6
•SWIN network requires ~77% of the workload to be executed
on a programmable device
“When a SWIN network is executed on an AI computational
block that includes an AI hardware accelerator designed for
an older CNN architecture, only ~23% of the workload might
run on the AI hardware accelerator’s fixed architecture. In
this one instance.”
Cadence says:
https://www.cadence.com/en_US/home/resources
/white-papers/why-a-dsp-is-indispensable-in-the-
new-world-of-ai-wp.html
© Quadric

7. 7
•Reshape
•Transpose
•Shifted
window
•Patch
creation
•Embeddings
lookup
•…
•Softmax
•Layer norm
•Group norm
•Instance norm
•Pixel Co-relation
•Positional encodings
•2 input matmuls
•Look up tables
•…
•NMS
•ROIAlign
•Noise reduction
•FFT/RFFT
•Equation solver
•Mean subtract
•…
Data Transformations Inference Pre/Post Processing
Non-NPU ops are here to stay
© Quadric

8. •Floating-point operations are
getting more common during
inference and can take a large
part of compute
•Future designs would have
multiple DSP cores, CPUs, AI
accelerators, vision accelerators
…
Where are we headed?
8© Quadric

9. 9
Back to basics: Float32 vs Fixed Point

10. Float32 representation
10
<23bits><8bits>1
Mantissa(fraction)ExponentSign
Float32
−1
&#3627408480;
∗2
????????????&#3627408477;&#3627408476;&#3627408475;??????&#3627408475;&#3627408481;
∗&#3627408474;??????&#3627408475;&#3627408481;??????&#3627408480;&#3627408480;??????
•Range: refers to the span of values that can be represented.
Exponentprovides the dynamic range.
•Precision: refers to the ability of a format to distinguish between
two close values. Mantissaprovides the precision within a range.
IEEE 754
© Quadric

11. •Floating point offers
•Better dynamic range.
•Ease of development, as a user doesn’t need to adjust for
precision and range
•But at a significant cost –power consumption!
What does floating point offer?
11© Quadric

12. •Known ranges of input and output: In all quantized neural nets the
input and output ranges are well known. One can use a calibration
dataset to identify all that.
•Fixed ranged ops: Sin, cos, sigmoid, softmax, norm, etc.
•Per operation range estimate: In almost all neural nets the data ranges
across layer and operations can easily be gathered with a calibration
data set
Do we really need floating point?
12© Quadric

13. •Dynamic fixed point : Based on the operations done internally one can analyze the
math and change fixed point precision per calculation.
Do we really need floating point?
13© Quadric

14. Fixed point32 representation
14
<p-bits><31-p bits>1
Mantissa(fraction)Integer BitsSign
Fixed32<FracBits(p)>
•Range and Precision: can be controlled by the developer. Precision can be
represented in for 31 fractional bits.
•Example:
•FixedPoint32<24>.. Base int32with 24bits to represent fractional value
•FixedPoint16<11>.. Base int16with 11bits to represent fractional value
© Quadric

15. •Quadric already supports 60+ networks (transformers, detector,
segmentation, classifier …) within <1%top1 accuracy loss compared to
floating-point models.
Fixed point accuracy numbers
15
http://quadric.io/evs24
© Quadric

16. 16
Back to basics: Designing an architecture from lessons learned

17. Chimera GPNPU
DMA
Neighbor
Access
Processing Element
Pipeline Control
32bit ALU
(Full C++ Target)
32 Entry Arch
Reg File
LRM –Local Register Memory
(4KB SRAM)
MACs
8 ,16, 32
8b x 8b / 4b x 8b
17
© Quadric

18. Chimera GPNPU
Scalable: 1 TOPS to 64 TOPS single core
Up to 512 TOPS Multi core
PE
18
© Quadric

19. Existing solutions
19
Offers very little programmability
•Code partitioning/ programming
complexity
•System complexity / power
•Accelerator brittleness
•No ability to modify hardware after
tapeout
•Leads to lower-performance
“fallback” onto the DSP or CPU
•Shortens market lifetime of SoC
❌
Hardwired
Convolution & Matrix
Co-processor
Programmable
Scalar + Vector
DSP
Shared Memory
AXI
Local
Mem &
Caches
Pooling & Activations
(selected, pre-wired)
Buffer Memory
© Quadric

20. Chimera GPNPU —A code powered AI engine
20
100% of GPNPU is end user programmable
•Dramatically easier software programming
model with ability to program in C++/python
•Simpler SOC architecture
•Long SoC lifespan –easy ML operator support
✅
Programmable Architecture
Memory
© Quadric

21. Programming model
21
Machine Learning Training Frameworks
ONNX Runtime/ Relay
Chimera GraphCompiler (CGC)
User Application
Code (C++)
Chimera LLVM C++ Compiler
Target Silicon
with Chimera
GPNPU
Chimera ISS
(Cycle approximate
/ System C)
IR
C++
C++
Trained NN Graph Float32/int8
ONNX
Quadric
Tool/libraries
© Quadric

22. Instruction-based simulation gives detailed bandwidth,
power, and performanceinsights​
22© Quadric

23. •Custom implementationof nodes/subgraphs
•e.g., NMS, proprietary layers,custom operators
Custom operatorsupport
23© Quadric

24. 24
Conclusions

25. •Floating-point unit can easily be replaced with fixed-point integer math equally well.
Same accuracy with lower power & higher performance.
•Fixed operation units (ASICs) only work in niche applications. In today’s world with AI
algorithms changing every 3 weeks, one needs a very flexible architecture which is easy
to program.
•Operations requiring non-mac compute are becoming very common. Having multiple
DSP/special cores is not the right fallback....
Conclusions
25
Need a unified architecture to handle all workloads…
© Quadric

26. 26
Q&A

27. About Quadric: http://quadric.io/evs24
Pure play Semiconductor IP Licensing
•Processor IP& Software Tools
Edge / device AI/ML Inference + DSP
processing
Silicon Proven Test Chip
HQ: Silicon Valley –Burlingame CA
Total Venture Capital Raised: $48M
May-2023: First IP delivery, DevStudio Online
Patents: 25Granted
Successful silicon in 2021
Visit us at Booth: 717
© Quadric
27