Pharo foreign function interface (FFI) by example by Esteban Lorenzano

Since Pharo 5 we have been developing a new framework for communicate with external libraries called UnifiedFFI (or UFFI) that has opened the doors to perform a huge step in our support for mainstream necessities. In this talk I will talk about the intrinsic problems of developing such frameworks and put it in practice through the example of the newest tool in the Pharo family: Iceberg (the git client).

1. UFFI
by Example

2. About me
Pharo architect since 2012
Owned a company to develop in
Pharo back in 2008
Java senior architect for 7 years (and
15 years overall Java experience)
Web, microprocessors, etc., etc., etc.
JavaScript, C++, ObjC, C#, Delphi,
ASM and lots of languages no longer
exist or have been long-time
forgotten
24 years (!) programming experience
(yes… I’m becoming old)

3. Why FFI*?
Why we want it or even need it?
*FFI: Foreign Function Interface

4. –Dan Ingalls
“an operating system is a collection of things that
don't fit inside a language; there shouldn't be one”

5. –John Lennon
“Life is what happens to you while you're busy
making other plans..”

6. FFI in Pharo
FFI Plugin
Alien FFI
NativeBoost FFI

7. FFI plugin
Compiler build (no extensions available).
Basic C types and structures
No callbacks
Not very easy to understand
void pango_cairo_update_layout (cairo_t *cr, PangoLayout *layout)
CairoCanvas >> pangoUpdateCairo: anAddress1 layout: anAddress2
<cdecl: void 'pango_cairo_update_layout' (void* void*) module: '/opt/
local/lib/libpangocairo-1.0.0.dylib'>

8. AlienFFI plugin
Object per method (“aliens”)
Very low level
Callbacks!
Not very easy to understand
void pango_cairo_update_layout (cairo_t *cr, PangoLayout *layout)
CairoCanvas >> pangoUpdateCairo: anAlien1 layout: anAlien2
^ (Alien
lookup: 'pango_cairo_update_layout'
inLibrary: '/opt/local/lib/libpangocairo-1.0.0.dylib')
primFFICallResult: (result := Alien newGC: 4)
with: anAlien1
with: anAlien2

9. NativeBoost FFI
Runtime build
Easy to create OO models with it (look Athens)
Very easy to understand
Very low level and hard to modify
void pango_cairo_update_layout (cairo_t *cr, PangoLayout *layout)
CairoCanvas >> pangoUpdateLayout: layout
^ self
nb Call: #(void pango_cairo_update_layout (self, PangoLayout *layout))
module: '/opt/local/lib/libpangocairo-1.0.0.dylib'

10. Why we removed
NativeBoost?
It was not working on Spur (or in 64bit)
It was hard to maintain (and we need a different
implementation for each architecture)
Since we were using NB exclusively for FFI, we decided
to replace it with a different one, using the VM plugin

11. What is UFFI?
A front-end to express FFI calls in Pharo.
Uses ThreadedFFIPlugin, but should be able to plug
others in the future.
It shares same philosophy as NativeBoost:
‣keep as much as possible in the image
‣no need to modify VM to add functionality
But it is not ASM: Just plain Smalltalk.

12. Goals
Keep NativeBoost syntax
‣Because is cool :)
‣Provide backward compatibility for most use-cases
Enhance documentation and self-documentation
Be the unified base for future FFI backends
implementations

13. Meta-goals
Expand the “universe of possible”:
-Be able to use existing libraries our there instead
needing to reproduce them (run away of the “not
invented here” syndrome).
-Do not need plugins to expand functionality
Provide an easy and efficient way to interact with the
outside world.

14. How does a call looks like?
char *getenv(const char *)
getEnv: variable
^ self
ffiCall: #( String getenv( String variable ))
module: LibC
(People who know NativeBoost will find this very familiar….)

15. How does a call looks like?
char *getenv(const char *)
getEnv: variable
^ self
ffiCall: #( String getenv( String variable ))
module: LibC
A regular Pharo method with one argument

16. How does a call looks like?
char *getenv(const char *)
getEnv: variable
^ self
ffiCall: #( String getenv( String variable ))
module: LibC
A literal array to represent C function

17. How does a call looks like?
char *getenv(const char *)
getEnv: variable
^ self
ffiCall: #( String getenv( String variable ))
module: LibC
Types annotation used to generate marshalling code

18. How does a call looks like?
char *getenv(const char *)
getEnv: variable
^ self
ffiCall: #( String getenv( String variable ))
module: LibC
The value to be passed when calling out

19. How does a call looks like?
char *getenv(const char *)
getEnv: variable
^ self
ffiCall: #( String getenv( String variable ))
module: LibC
The library to lookup C function

20. FFILibrary
A very simple abstraction to define module names that
can be different each platform.
Can be used also as a place to store C function
definitions (like a real library :) ).

21. Insights to UnifiedFFI
getEnv: variable
^ self
ffiCall: #(void pango_cairo_update_layout (self, PangoLayout *layout))
module: PangoLibrary
1. Generate bytecodes for marshalling
2. Re-send the method execution
void pango_cairo_update_layout (cairo_t *cr, PangoLayout *layout)

22. How does a call looks like?
49 <20> pushConstant: <cdecl: void 'pango_cairo_update_layout' (void*
void*) module: '/opt/local/lib/libpangocairo-1.0.0.dylib'>
50 <05> pushRcvr: 5
51 <10> pushTemp: 0
52 <76> pushConstant: 1
53 <E1> send: instVarAt:
54 <8A 82> pop 2 into (Array new: 2)
56 <E2> send: invokeWithArguments:
57 <87> pop
58 <78> returnSelf
void pango_cairo_update_layout (cairo_t *cr, PangoLayout *layout)

23. Types
Support for standard C types: int, float, etc.
Support for type aliases (map a name to one of the defined types)
Complex types:
-FFIExternalObject: External addresses (objects)
-FFIOpaqueObject: Opaque C types/structures
-FFIExternalStructure
-FFIExternalArray, FFITypeArray
-FFIExternalEnumeration
-FFIExternalValueHolder: Buffers (to pass referenced data, e.g. “double *”)
-FFIConstantHandle: Windows HANDLE (constant addresses)

24. So… and the example?
Demo time :)

25. Thanks!
Smalltalk quitSession.