“Transforming Enterprise Intelligence: The Power of Computer Vision and Gen AI at the Edge with OpenVINO,” a Presentation from Intel

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/transforming-enterprise-intelligence-the-power-of-computer-vision-and-gen-ai-at-the-edge-with-openvino-a-presentation-from-intel/

Leila Sabeti, Americas AI Technical Sales Lead at Intel, presents the “Transforming Enterprise Intelligence: The Power of Computer Vision and Gen AI at the Edge with OpenVINO” tutorial at the May 2024 Embedded Vision Summit.

In this talk, Sabeti focuses on the transformative impact of AI at the edge, highlighting the role of the OpenVINO tool kit in streamlining the AI solution life cycle on Intel hardware. This includes the development of energy-efficient computer vision and generative AI models suitable for edge computing.

Sabeti showcases cutting-edge AI applications, such as multimodal LLMs for document understanding and YOLO object detection for smart retail solutions. She addresses the entire edge compute ecosystem, discussing how to optimize AI processes from training to inference across various computing platforms, including Intel GPUs. Additionally, she explores how businesses can seamlessly transition between edge and cloud environments and how Intel’s portfolio of solutions unlock the advantages of edge computing, such as data protection and AI acceleration.

1. Transforming Enterprise Intelligence:
The Power of Computer Vision and
Gen AI at the Edge with OpenVINO
Leila Sabeti
Americas AI Technical Sales Lead
Intel

2. •Edge AI applications
•The OpenVINOtoolkit: An open-standard for building AI at the edge, in
the cloud, or locally
•Deep-dive: Enterprise intelligence and Intel®’sportfolio
•Flexible edge and cloud computing paradigms
•Q&A
Agenda
2© 2024 Intel

3. ▪Real time data processing​
▪Wider reach
▪Data sovereignty
▪Cost efficiency
Edge
IPCamera
Depth Camera
NVR
Sensor
Frictionless Retail
Traffic Monitoring
Defect Detection
AI is Everywhere – from Edge to Cloud
© 2024 Intel 3

4. Enterprise Intelligence at the Edge with Intel®
Node
Fine-tuning, Inference
Cluster
Light Training, Tuning, Peak Inf.
© 2024 Intel 4

5. Medium AILight AI Heavy AI
Efficiency for sub-100 W designs
CPU AI, built-inGPU, built-inNPU
Scale up perf/W for diverse system designs
Discrete GPU & built-in CPU AI
Optimize for peak perf and density
Optimize with high-end Discrete GPU
Choose Your Compute
© 2024 Intel
Small
Gen AI
5

6. OpenVINO: An Open Standard for Building AI at the Edge
© 2024 Intel
Optimized Performance
FPGACPU NPUGPU
6

7. Supports myriad AI use
cases
Wide range of AI
performance
Real-time and offline
execution
Flexible video streams
Ranges from few to many
streams, and TOPS in low-
end NVRs to TOPS for on-
prem servers
Temperature range
Ranges from -5 to +105
degree C
TDP/Power
Ranges from sub 10 W to
over 300 W
Software Benefits
with OpenVINO
Hardware Benefits
with Intel®’s Edge AI
Portfolio
Unlocking Software Optimizations with Hardware
Using the Intel® Edge AI Portfolio
Open-source for AI, DL,
Inference
Performance-optimized Cross-platform Support
OpenVINO Toolkit
Value Differentiators
© 2024 Intel 7

8. Compute for AI: Intel®’s Platforms
▪Single NN Pipeline
▪Multi NN Pipeline
▪Data Fusion Pipeline
Seamlessly Integrated into Existing
Camera and Video Deployments
Intel® Edge AI BoxEdge AI Platforms
Partner edge platforms using Intel® Arc GPU
© 2024 Intel 8

9. 9© 2024 Intel
Challenges and Opportunities
with AI at the Edge

10. Objective: Optimize the queuing process
and reduce wait times via object detection.
Challenges:
•Real-time scalability
•Device setup and calibration
•Model performance
•Low-power
Intel’s solution:
•Fast and efficient inference with
optimized YOLOv8 models using
OpenVINO
Intelligent Queue Management at the Edge with
OpenVINO and YOLOv8
© 2024 Intel 10

11. Mobile Multi-modal Assistant with MobileVLM and
OpenVINO
© 2024 Intel
Objective: Use a mobile chatbot
to answer questions about images
Challenges:
•Fast token generation
•Memory-efficiency
•Model size
Intel’s solution:
•Compress and quantize LLM
models for faster, efficient
local inference
GPU
11

12. Snapshot of MobileVLM Output
© 2024 Intel 12

13. Document Visual Question Answering
with Pix2Struct and OpenVINO on CPU
© 2024 Intel
Objective: Use a low-power
chatbot to answer questions
about documents on the fly.
Challenges:
•Visual understanding
•Memory-efficiency
•Model size
Intel’s solution:
•Compress and quantize multi-
modal models for faster,
efficient local inference
Intel® Core Ultra 9 processor CPU
13

14. Snapshot of Pix2Struct Output
© 2024 Intel 14

15. Enterprise Intelligence with LLMs using RAG
15© 2024 Intel
Connect knowledge bases to LLMs with Retrieval Augmented Generation
(RAG)

16. Running LLM + RAG with OpenVINO and LangChain on
iGPU for the edge
Enable enterprise intelligence through knowledge-based search
© 2024 Intel 16

17. Enterprise Data Protection at the Edge
17
Intel® Software Guard Extensions (Intel®
SGX)
Secure Access Service Edge (SASE)
Intel® QuickAssist Technology (Intel® QAT)
© 2024 Intel

18. 18© 2024 Intel
Edge to Cloud Paradigms

19. Edge to Cloud: Flexibly Using Compute
19
Edge Cloud
Network
Pros
Cost efficiency, increased
control, etc.
Cons
Low-power constraints
Pros
Large amount of data,
limitlesscompute on demand​
Cons
Privacy risks, high latency, etc.
© 2024 Intel

20. Cloud Edge
Edge to Cloud with OpenVINO Model Server
Move workloads across the edge and cloud
Powered by OpenVINO Runtime
© 2024 Intel 20

21. Deploying a Quantized Tiny-llama model across client and server
LLM Assistants:
OpenVINO Model Server with INT8 Compression
© 2024 Intel
21

22. Intel®’s AI Hardware Portfolio
22© 2024 Intel
Cloud PlatformsEdge AI Platforms Client Platforms
Desktops
Laptops
Partner edge platforms using Intel® Arc GPU

23. •AI at the edge is transforming enterprise intelligence
•But not without several challenges: scalability, setup, AI performance, etc.
•At Intel®, we see the full end-to-end stack as key for optimizing AI at the
edge, and across the cloud to edge
•OpenVINO is an open standard, ready-to-use for building AI and Gen AI
•Try It Yourself: openvino.ai
Conclusion
© 2024 Intel 23

24. Resources
Resources
•openvino.ai
•intel.com/edgeai
•Demos: intel.com/openvinonotebooks
•Enterprise Security Solutions at the
Edge with Intel
2024 Embedded Vision Summit
May 23
rd
(12:00 pm – 12:30 pm)
“Identifying and Mitigating Bias in AI”
May 23
rd
(1:30 pm – 2:00 pm)
“Intel’s Approach to Operationalizing AI in
the Manufacturing Sector”
24© 2024 Intel

25. AI: The New Age
Solving the World’s Toughest
Challenges, Together.
Opt-In for Early Access When
Registration Opens!
Calling All Developers & Technologists!
From front-end, web, app devsto back-end, full-stack, database &
DevOps to data scientists & researchers, and more:
Learn, collaborate, and solve at Intel Innovation –
an event for developers by developers.
www.intel.com/innovation
Save the Date:
September 24-25, 2024
San Jose Convention Center, CA
Hear from leading industry luminaries, technologists & start-up
entrepreneurs in the field of AI.
Learn the breadth of future technology advancements in AI through
keynotes, sessions, birds of a feathers, and hands-on labs.
Get the latest AI development tools, hands-on experience & join on-site
Hackathons to optimize your AI code & workflows.
Share unique ideas and perspectives and collaborate with your peers.

26. Join us at CVPR!
Hackster OpenVINO Challenge
Ends June 1
st

https://www.hackster.io/contests/ope
nvino2024/
OpenVINO at CVPR
Tutorial Date: June 17
th
https://paularamo.github.io/cvp
r-2024/