Building End-user Applications on Embedded Devices with WPE

The Web engine is the most important component of a Web Browser, enabling
developers to harness the power of the Web Platform to build their
applications. However, Web Browsers are not the only type of applications that
can be built with Web Engines, which can also be used to develop other types of
applications using the same Web-based technologies, but for a different type of
use cases other than "browsing the Web".

These use cases can cover a wide range of situations outside of the traditional
desktop or mobile environments, such as the ones embedded systems are usually
used for (e.g. set-top-boxes, smart home appliances, GPS navigation devices, or
in-car/in-flight infotainment systems, to name a few). And in those situations
it is very common to be running on boards with SoCs and a particular set of HW
capabilities that make it crucial for the Web Engine to be able to tightly
integrate with them.

In this session we will focus on how WPE, a fully Open Source port of the
WebKit Web engine for Linux-based embedded devices, can be used to adapt to the
different challenges that embedded devices pose to develop end-user
applications, using the power of the Web Platform underneath.


(c) Embedded Open Source Summit 2024
April 16-18, 2024
Seattle, Washington (US)
https://events.linuxfoundation.org/embedded-open-source-summit/
https://eoss24.sched.com/event/1aNTr/building-end-user-applications-on-embedded-devices-with-wpe-mario-sanchez-prada-igalia

1. Building end-user applications
on embedded devices with WPE
Mario Sánchez Prada
[email protected]
1 / 36

2. About me
CS Engineer, partner of Igalia
Involvement in some Open Source communities
e.g. Chromium, WebKit, GNOME
Other work done in the past:
Linux-based OS’s (i.e. Endless OS, Litl OS)
Maemo (Hildon Application Manager)
Samsung SmartTV platform
Currently coordinating Igalia's WebKit team
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20242 / 36

3. About Igalia
Specialized Open Source consultancy, founded in 2001
Fully remote, HQ in A Coruña (Spain). Flat structure.
Top contributors to all the main Web Rendering Engines
WebKit, Chromium, Gecko and Servo
Active contributor to other areas and OSS projects
V8, SpiderMonkey, JSC, LLVM, Node.js, GStreamer, Mesa, Linux Kernel...
Members of several working groups:
W3C, WHATWG, WPT, TC39, OpenJS, Test262, Khronos...
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20243 / 36

4. Outline
Web rendering engines
What is WebKit?
What is WPE
Integrating WPE in your product
Conclusion
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20244 / 36

5. Web rendering engines
(aka Web engines)
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20245 / 36

6. What is a Web rendering engine?
A Web rendering engine is the software component that enables
you to leverage the power of the Web Platform in your product:
Fetchs HTML/CSS/JavaScript content from multiple sources.
Interprets the Web content to produce an internal representation.
Produces a result that users can interact with.
It's a extremely flexible platform. Some examples:
Textual and non-textual content
Multimedia playback
Fully-fledge applications
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20246 / 36

7. What is a Web rendering engine?
Web rendering engine ≠ Web browser
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20247 / 36

8. Most popular Web engines


Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20248 / 36

9. What is WebKit?
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 20249 / 36

10. What is WebKit?
Open Source Web rendering engine
Started as a fork of KHTML and KJS in 2001.
Forked again become Blink in 2013.
Main goals:
Performance, portability, stability, compatibility, standards compliance,
security, hackability and embeddability.
Support for different platforms:
Desktop & Mobile: Mac, iOS and Linux
Embedded: set-top-boxes, video game consoles, smart home appliances,
in-vehicle/inflight entertainment, GPS devices, digital signage...
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202410 / 36

11. Advantages of WebKit
?????? Complete implementation of the Web Platform
?????? Embeddable as top priority
?????? Flexible and modular architecture
?????? Privacy and security by design
?????? Performance and stability
?????? Independent Linux-based flavours
?????? Not controlled by any big corporation
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202411 / 36

12. WebKit Architecture
Application:
What end-users interact with.
WebKit:
Exposes an API to applications and
implements the split-process model.
WebCore:
Layout, rendering, network,
multimedia, accessibility...
JavaScriptCore:
The JavaScript engine.
Platform:
Platform-specific hooks.
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202412 / 36

13. WebKit Ports
WebKit port: adaptation of WebKit to a specific platform.
 
Official WebKit Ports (upstream ports)
Mac: Safari, Apple Mail, iTunes, App Store...
iOS: every browser on iOS devices (including Chrome).
WinCairo: Microsoft Playwright, Playstation SDK
Playstation: Playstation s4 & Playstation 5
WebKitGTK: GNOME Web, Evolution, Shotwell...
WPE: Cog and other custom-made "browsers" for embedded devices.
https://docs.webkit.org/Ports/Introduction.html
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202413 / 36

14. WebKitGTK and WPE
WebKit ports targeting Linux-based systems
Common parts: GLib, libsoup (networking), GStreamer (multimedia)...
Key differences: graphics stack, input handling. Different use cases.
WebKitGTK:
Go-to solution to embed Web content in GTK applications.
Integration with GNOME components. Supports GTK3 and GTK4.
WPE:
Lower level, aimed at embedded devices.
Requires graphics and input backend to work.
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202414 / 36

15. WebKit Ports: WebKitGTK
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202415 / 36

16. What is WPE?
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202416 / 36

17. What is WPE?
WebKit port optimized for Linux embedded devices
Fully operational JavaScript engine (64-bit, also 32bit).
Focus on flexibility, security and performance.
Minimal set of dependencies
Backends-based architecture for input and output.
Low memory and storage footprint.
Support for HW-accelerated graphics and multimedia.
ℹ️ https://wpewebkit.org/
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202417 / 36

18. What is not WPE?
NOT a general purpose Web Browser
Provides just the building blocks for Web-based applications.
Doesn't implement all the APIs found on other WebKit ports.
Does not rely on any particular UI Toolkit (i.e. backends).
Can also be useful for less conventional use cases
    e.g. server-side rendering, headless mode...
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202418 / 36

19. WPE Architecture
Application:
The end application, which can use WPE
directly or via the provided Cog launcher.
WebKit:
The actual WebKit port, including the API
layer to link against from applications.
Backend:
Platform-specific implementation of the
graphics and/or the input layers.
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202419 / 36

20. WPE components
WPEWebKit:
The actual WebKit port.
Relies on the backends for page display and input.
libwpe:
Provides rendering-related callbacks implemented by the graphical backend.
Allows the input backend to rely events from the application to WebKit.
WPEBackend-FDO:
The reference FreekDesktop.Org-based backend (i.e. Wayland).
Supports several architectures plus regular PC architectures.
Can be replaced by a device-specific backend
Cog:
Small single “window” launcher for WPE, with no user interface.
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202420 / 36

21. WPE-based products
Some examples of use cases we are aware about:
Set-Top-Boxes (both RDK and non RDK based)
Smart Home Appliances
HiFi audio/sound systems & music streaming
Digital Signage
GPS navigation devices
Video/Audio conference
Headless server-side rendering
QA and testing
...
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202421 / 36

22. Demos
??????
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202422 / 36

23. 23 / 36

24. Integrating WPE in your project
??????️
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202424 / 36

25. 1. Carefully choose your hardware
Determine the specifics of the HW you'll rely on.
It might impose specific constraints (e.g. linear-only direct scanout)
Determine the input devices needed:
e.g. Physical keyboards, touch-based devices
Determine the output devices needed:
e.g. Standard screens, custom displays, no displays at all
Figure out whether custom backends are needed
Otherwise WPE provides Wayland and DRM already
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202425 / 36

26. 2. Assemble the required components
WPE libraries:
Current API: WPEWebKit, libwpe, backends
New API: WPEWebKit, WPEPlatform (includes built-in platforms)
WPE dependencies:
2D/3D rendering: cairo, OpenGL, EGL, GLES...
Multimedia pipeline: GStreamer
Networking: libsoup
Other: glib, atk, wayland...
Other libraries:
Anything else your application needs
Launcher application:
Cog (uses the old API)
Your own application
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202426 / 36

27. Example application (new API)
#include <wpe/webkit.h>
int main(int argc, const char *argv[]) {
g_autoptr(GMainLoop) loop = g_main_loop_new(NULL, FALSE);
g_autoptr(WebKitWebView) view = webkit_web_view_new(NULL);
webkit_web_view_load_uri(view,
(argc > 1) ? argv[1] : "https://wpewebkit.org");
g_main_loop_run(loop);
return EXIT_SUCCESS;
}
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202427 / 36

28. 3. Develop your product
Prototype/Write your Web application:
Plain HTML/CSS/JS and/or common frameworks (e.g. React, Angular, Vue...)
Prototype your application (e.g. on desktop device)
Avoid poorly performing features on your platform:
Certain constrained devices might impose specific limitations
Ideally avoid particularly expensive operations (e.g. blur)
Integrate with the rest of your platform:
Leverage the Web Platform APIs to implement most features
Possible to extend functionality through custom adaptation
Testing & QA:
Run tests on target HW (e.g. i.MX6, i.MX8, Broadcom...)
Testing automation frameworks: WebDriver, PlayWright
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202428 / 36

29. 4. Bundle up and ship
Distribution packages available:
Debian:
Raspbian:
Arch Linux:
Source code:
Tarballs available at
Git repositories:
WebKit:
LibWPE:
FDO Backend:
Cog:
https://packages.debian.org/source/stable/wpewebkit
https://archive.raspbian.org/raspbian/pool/main/w/wpewebkit
https://archlinux.org/packages/extra/x86_64/wpewebkit
https://wpewebkit.org/release
https://github.com/webKit/WebKit/
https://github.com/WebPlatformForEmbedded/libwpe
https://github.com/Igalia/WPEBackend-fdo
https://github.com/Igalia/cog
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202429 / 36

30. 4. Bundle up and ship
Yocto-based images:
Layer available at
Building an image with WPE:
Recommended way of building WPE-based images
Buildroot-based images:
Support available at:
Updated on a best-effort basis
https://github.com/Igalia/meta-webkit
https://github.com/Igalia/meta-webkit/wiki/WPE
https://github.com/buildroot/buildroot/blob/master/package/wpewebkit/wpewebkit.mk
https://github.com/buildroot/buildroot/blob/master/package/libwpe/libwpe.mk
https://github.com/buildroot/buildroot/blob/master/package/wpebackend-fdo/wpebackend-fdo.mk
https://github.com/buildroot/buildroot/blob/master/package/cog/cog.mk
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202430 / 36

31. Last but not least: collaborate upstream!
WPE WebKit website:
API documentation:
Security Advisories:
Mailing list:
Matrix:
Bug tracker:
See also
https://wpewebkit.org
https://wpewebkit.org/reference/stable
https://wpewebkit.org/security
https://lists.webkit.org/mailman/listinfo/webkit-wpe
https://matrix.to/#/#wpe:matrix.org
https://bugs.webkit.org
https://webkit.org/reporting-bugs
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202431 / 36

32. Conclusion
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202432 / 36

33. Conclusion
Web Rendering Engines useful for more than Web Browsers.
WPE leverages WebKit's strengths and makes it ideal for
embedded devices, providing the extra flexibility needed.
Integrating WPE in your project requires carefully choosing your
Hardware and assembling the required components.
Products can be developed as regular Web applications, and
adaptations are still possible thanks to WPE's modular design.
WPE and WebKit fully Open Source, contributions are welcome!
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202433 / 36

34. Questions?
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202434 / 36

35. Thanks!
Building end-user applications on embedded devices with WPE
Mario Sánchez Prada, Embedded Open Source Summit 202435 / 36

36. 36 / 36