“Build a Tiny Vision Application in Minutes with the Edge App SDK,” a Presentation from Midokura, a Sony Group Company

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/07/build-a-tiny-vision-application-in-minutes-with-the-edge-app-sdk-a-presentation-from-midokura-a-sony-group-company/

Dan Mihai Dumitriu, Chief Technology Officer at Midokura, a Sony Group company, presents the “Build a Tiny Vision Application in Minutes with the Edge App SDK” tutorial at the May 2024 Embedded Vision Summit.

In the fast-paced world of embedded vision applications, moving rapidly from concept to deployment is crucial. In this presentation, Dumitriu introduces the Edge App Runtime and SDK—groundbreaking tools designed to streamline and accelerate the development process for edge computing solutions. Leveraging a pre-built app skeleton, the SDK simplifies the development journey, allowing developers to focus on customizing event handlers using popular high-level languages such as JavaScript and Python. This approach not only democratizes edge application development, but also significantly reduces the time to market.

With an integrated local tool that supports development, testing, building and deployment, the transition from a local environment to cloud deployment becomes seamless. Dumitriu explores how the Edge App Runtime and SDK are enabling the creation and deployment of edge applications in a matter of minutes, making edge application development more accessible and efficient than ever before.

1. © 2024 Midokura
Build a Tiny Vision
Application in Minutes with
the Edge App SDK
Dan Mihai Dumitriu
Chief Technology Officer
Midokura (Sony Group)

2. © 2024 Midokura
Sony’s AITRIOS™
Challenges of Embedded Development
Case for Polyglot Development
Dev Tools & SDK
Example Applications
Conclusion
* AITRIOS and AITRIOS logos are the registered trademarks or trademarks of Sony Group
Corporation or its affiliated companies.
Outline
2

3. © 2024 Midokura
Vision
Advanced vehicle
assistance
Smart
Manufacturing
Smart Cities
Retail storesSmart Home
Bring to market easy-
to-use sensing devices
Low barrier of entry for
solution developers
Agile development of
sensing applications
Low operational cost of
vision sensing apps
Polyglot DevelopmentMarketplace to connect
AI Developers &
Solution Developers
Targeting solution developers for various vertical applications.
3

4. © 2024 Midokura© 2024 Midokura
Challenges of Embedded Development
4

5. © 2024 Midokura
Safety on hardware without MMU (i.e. MCUs)
●Ensure memory safety
●Dynamic loading and linking
●Replacing modules at runtime
Enable solution developers
●Reduce the development effort
●Enable portability across hardware platforms
●Provide a channel for distribution via marketplace
Enable OEM ecosystem using Android model
●Give away the device OS and/or runtime
●Lower the R&D costs of device manufacturers
●Enable marketplace model by application standardization
●Access to marketplace raises value of hardware product
5
Goals

6. © 2024 Midokura
Development
●Embedded development is difficult in C
●No standard component model, so customization cost is high
Maintenance
●Difficult or impossible to change functionality after deployment
Security
●Many instances of compromised IoT devices
●Vulnerabilities cannot all be found before shipping
●#1 source of security vulnerabilities is memory bugs
6
Pain points of traditional embedded/IoT devices

7. © 2024 Midokura© 2024 Midokura
Why Polyglot Development?
7

8. © 2024 Midokura
Most popular programming, scripting, and markup
languages
Data extracted from https://survey.stackoverflow.co/2023
JavaScript and Python are far more
popular than C++. C, and Rust
Embedded systems are primarily
written in C
There is a mismatch…
8

9. © 2024 Midokura
Most popular AI frameworks are Python
Most frameworks related to data or AI
are Python-specific
●NumPy
●Pandas
●PyTorch
9

10. © 2024 Midokura
Decouple from language, target architecture & OS
WebAssembly System Interface (WASI): standardized and secure way to access system resources
10

11. © 2024 Midokura© 2024 Midokura
EVP Edge Stack
11

12. © 2024 Midokura
●Like Kubernetes, but for tiny IoT devices
○Lifecycle management of workloads on IoT devices
●EVP agent is like Kubelet
○Leverages WebAssembly Micro Runtime (WAMR)
○Strong isolation of modules
○Secure even on RTOS
●Communicates with EVP backend via MQTT
EVP (Edge Virtualization Platform)
12

13. © 2024 Midokura
EVP device stack
AITRIOS Edge App Services API
OS
Abstraction
Layer
WASI
Web Assembly Micro Runtime (WAMR)
Native Libraries & Device Drivers
OS (Linux, NuttX, etc)
HW (amd64, arm64, rv32, xtensa)
Module 1
. . .
Module 2 Module 3 Module 4 Module N
13

14. © 2024 Midokura
Edge app SDK: APIs
WASI-Sensor
●Read Image
●Configure (e.g., frame rate)
Communication
●Telemetry to EVP backend
○via MQTT
●Device to device
○Sockets
WASI-NN
●Load model
●Run inference
Data Storage
●Local DB
●Blob storage (HTTP
PUT/GET/POST)
14

15. © 2024 Midokura© 2024 Midokura
AITRIOS Edge App Dev Tools
15

16. © 2024 Midokura
MQTT/HTTP
Create
Build
Deploy
LUCID Vision Labs™
SENSAiZ
Smart Camera using
Sony IMX500
Intelligent Vision
Sensor & ESP32
processor
Host PC
WEdge Agent
Reify
DeloymentEVP Backend
Deployment/
life-cycle
management
On-the-fly
optimization
for a variety
of devices
REST
AITRIOS
CLI
16
Local developer setup

17. © 2024 Midokura
17
AITRIOS developer tools
●EVP-in-a-Box (Backend)
●EVP CLI
○Create module project
○Build module
○Register module
●EVP GUI
○Manage device
○Create edge application
○Do test inference

18. © 2024 Midokura
18
EVP GUI

19. © 2024 Midokura
Visual programming
19

20. © 2024 Midokura© 2024 Midokura
Example Edge App
20

21. © 2024 Midokura
IMX500 EVP
Neural network (NN) for
license plate detection
(LPD)
Detection boxes
Raw frame data
●Creates high-resolution crop of the
detection
●Applies 2nd NN to detect
characters/numbers of license plate
(LPR network, in tf-lite runtime)
●Reads license plate content from
character detections
Kawaguchi 328
yu 58-54
IMX500 Type 2
prototype
1 2
3
License plate content
1
2
3
Shows the
license
plate
content in
the IoT
platform
Demo viewer
AP (RaspPi)
License plate reading (LPR) edge app
21

22. © 2024 Midokura
IMX500
License plate reading data flow
PPL
Crop
using
OpenCV
Senscord
Sink
Character
Detection
using
WASI-NN
LPR logic
Image
Sensor
License
Plate
Detection
on DSP
ISP
Image
Bounding
Box
Bounding
Boxes of
Characters
License Plate
String
22
Metadata
Cropped
Image

23. © 2024 Midokura
23
License plate reading data flow

24. © 2024 Midokura
24
License plate reading demo result

25. © 2024 Midokura
●Polyglot development for embedded systems is important
○AI developers use Python
○Embedded developers use C
●WebAssembly is a great abstraction layer
○High performance
○Multi-platform
○Scales up and down
●Edge Apps can be built by anyone
Conclusions
25

26. © 2024 Midokura
Further information
Midokura
https://www.midokura.com
AITRIOS
https://www.aitrios.sony-semicon.com/en/
IMX500
https://developer.sony.com/develop/imx500/
Demo video
https://bit.ly/mido-vsa-demo
26

27. © 2024 Midokura© 2024 Midokura
Questions
27

28. © 2024 Midokura
28