“Modernizing the Development of AI-based IoT Devices with Wedge,” a Presentation from Midokura, a Sony Group Company

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/07/modernizing-the-development-of-ai-based-iot-devices-with-wedge-a-presentation-from-midokura-a-sony-group-company/

Dan Mihai Dumitriu, Chief Technology Officer of Midokura, a Sony Group Company, presents the “Modernizing the Development of AI-based IoT Devices with Wedge” tutorial at the May 2023 Embedded Vision Summit.

IoT device development has traditionally relied on a monolithic approach, with all firmware developed by a single vendor using a rigid waterfall model, typically in C, and infrequently updated. This paradigm is insufficient. For AI-enabled IoT devices to reach their potential, developers must be able to easily program and update low-cost, low-power AI-capable sensors, such as Sony’s IMX500.

In this talk, Dumitriu presents Wedge—a combination of runtime, device agent and cloud service. Wedge automates software life-cycle management for devices, provides isolation and enables agile development. Wedge is based on WebAssembly, a binary instruction format for a stack-based virtual machine. Wedge’s runtime and memory overhead are within 2x of native C/C++ code, and it supports languages such as Python and AssemblyScript. Dumitriu also presents the vision sensing pipeline, a sensor data processing layer developed on top of Wedge, which is accessible via REST API and a visual interface.

1. © 2023 Midokura (Sony Group)
Modernizing the
Development of AI-
Based IoT Devices with
Wedge
Dan Mihai Dumitriu
CTO
Midokura(Sony Group)

2. © 2023 Midokura (Sony Group)
•Sony Semiconductor Solutions is the largest vendor of CMOS image sensors
•AITRIOS is a new business to develop sensingsolutions based on the sensor
portfolio
•Embedded Edge AI and IoT are key ingredients
•Midokurais a subsidiary of Sony Semiconfocused on software infrastructure
Introduction
2

3. © 2023 Midokura (Sony Group)
Sony AITRIOS
3
Address privacy concerns
by elaborating metadata-only
Low power and
cost friendly
Faster and optimized
on-sensor AI processing
Less bandwidth needed
and very low latency
AI-enabled
image sensor
(IMX500)
AI/device
management
cloud service
New type of
AI/computer vision
hardware platform
IMX500

4. © 2023 Midokura (Sony Group)
An Accessible Platform that Enables Intelligent
Solutions Based on Distributed Visual Sensing
4
Advanced vehicle
assistance
Smart
Manufacturing
Retail stores
Retail stores
Smart Home
Bring to market a
plethora of low cost,
easy-to-use sensing
devices
Low barrier of entry for
solution developers
Agile development of
sensing applications
Low operational cost of
vision sensing apps
Polyglot Development Marketplace to connect
AI Developers & Solution
Developers
Targeting solution developers for various vertical applications.

5. © 2023 Midokura (Sony Group) 5
Problems of Development on Tiny
IoT Devices

6. © 2023 Midokura (Sony Group)
•Operating system (RTOS) is effectively like a library
•Application and OS must be tested together
•Waterfall development
•Heavy testing process late in development cycle
•Cannot continuously deploy applications
Development is Not Agile
6

7. © 2023 Midokura (Sony Group)
•Tiny IoT devices are based on MCUs (microcontroller units)
•MCU typically cannot do virtual memory
•Without memory isolation, cannot do agile development
•Dynamic deployment of applications is unsafe
No Safety and Isolation
7

8. © 2023 Midokura (Sony Group)
•High barrier of entry for application developers
•Apps are typically written in C
•Poor code reuse across device types
•HW specific interfaces and drivers
•AI developers typically don't know C well
•Develop mostly in Python
Application Development Is Difficult
8

9. © 2023 Midokura (Sony Group) 9
Introducing WEdge

10. © 2023 Midokura (Sony Group)
•Like Kubernetes, but for tiny IoT devices
•Automated lifecycle management of workloads
•Strong isolation of modules
•Leverages WebAssemblyMicro Runtime (WAMR)
•WEdgecloud does automatic optimization for target device
•Polyglot SDK enables developer productivity
Intro to WEdge
10

11. © 2023 Midokura (Sony Group)
•WebAssembly(Wasm) is a low-level bytecode format that runs in a sandboxed
environment
•Designed as a portable target for compiling high-level languages like C, C++, and Rust
•Runs on various platforms, including browsers, servers, and now even on tiny IoT devices
•Compact size, fast execution, and high security compared to other runtime environments
•AoT(Ahead of Time) compilation supports multiple target ISAs via LLVM backends
•High runtime performance -within 2x of native
Intro to WebAssembly
11

12. © 2023 Midokura (Sony Group)
•Combination of language-level and runtime-level protections
•Linear memory model
•All memory accesses are bounds-checked to prevent buffer overflows or
underflows
•Enforces type safety at both compile-time and runtime
•Control Flow Integrity (CFI) prevents control flow hijacking attacks
•WebAssemblySystem Interface (WASI) for standardized and secure way to
access system resources
•Fine grained control
WebAssembly for Sandboxing
12

13. © 2023 Midokura (Sony Group)
WEdge Device Stack
13
WEdge Services API
OS
Abstraction
Layer
WASI
Web Assembly Micro Runtime
Native Libraries & Device Drivers
OS + BSP
HW
Module 1
. . .
Module 2 Module 3 Module 4 Module N

14. © 2023 Midokura (Sony Group)
•Sensors
•Read image
•Configure (e.g. frame rate)
•Communication
•Send telemetry to cloud
•HTTP PUT/GET/POST
•Other protocols (e.g. CAN)
WEdge Services API
14
•AI/ML
•Load model
•Run inference
•
Data storage
•Local DB
•Distributed DB cache

15. © 2023 Midokura (Sony Group)
SDKs in various languages
Polyglot Development
15

16. © 2023 Midokura (Sony Group)
Decouple from Target Architecture & OS
16

17. © 2023 Midokura (Sony Group)
•Enables more flexible and dynamic development cycles
•Devs can break down complex software into smaller, more
manageable modules that can be developed, tested, and
deployed independently
•Polyglot development helps devscode in their language of choice
•For AI developers, that is Python
Agile Development Enabler
17

18. © 2023 Midokura (Sony Group) 18
Vision Sensing Applications

19. © 2023 Midokura (Sony Group)
•Programming sensing applications is complex
•Code reuse is typically poor and applications are monolithic
•VSA models a complex application as a series of simple nodes
•Promotes code reuse
•Enables low-code development via API or GUI
•VSA nodes are deployed as Wasmmodules
Vision Sensing Applications (VSA)
19

20. © 2023 Midokura (Sony Group)
Goal: Make an intelligent sensor perform a complex task
How: Model this complex task as a sequence of individual small tasks
Complex task => VSA
Small tasks (partial) => Nodesof the VSA
Why?
●Reusable nodes -can have different creators
●Allow to customize an existing application to a new use case by modifying some nodes
●Security: Use Webassemblytechnology with its sandboxing approach to allow custom
code without impact on the device security
Vision Sensing Application (VSA)
20

21. © 2023 Midokura (Sony Group) 21
Example Application: People Counting

22. © 2023 Midokura (Sony Group)
People Counting VSA
22

23. © 2023 Midokura (Sony Group)
VSA of People Counting Application
23

24. © 2023 Midokura (Sony Group)
•WebAssemblyis a great choice for tiny IoT devices
•High performance and low overhead
•WEdgeenables high development productivity and agility
•Vision Sensing Applications (VSAs) are a convenient paradigm for
programming sensing applications
•Promoting component reuse
•Enabling low-/no-code development
Conclusion
24

25. © 2023 Midokura (Sony Group)
Further Information
25
Midokura
https://www.midokura.com
AITRIOS
https://www.aitrios.sony-semicon.com/en/
IMX500
https://developer.sony.com/develop/imx500/
Demo Video
https://bit.ly/mido-vsa-demo
Please visit our booth!

26. © 2023 Midokura (Sony Group) 26
Thank you