Android graphic system (SurfaceFlinger) : Design Pattern's perspective

SurfaceFlinger is a vital system service in Android system, responsible for the composting all the application and system layer and displaying them. In this slide,we looked in detail how surfaceFlinger was designed from Design Pattern's perspective.

1. Android UI Subsystem Design Bin (Pierr) Chen http://pierrchen.blogspot.com Pattern’s Perspective

2. Android View System Libgui SufaceFlinger HWC EGL Components Apps 3D

3. Perspective How Why

4. Combined Libgui SufaceFlinger + How / Why

5. Pattern

6. Design Pattern is a general reusable solution to a commonly occurring problem

7. Observer/ Publish -Subscribe

8. UML A oo way of saying callback in c.

9. Examples You subscribe to an RSS to get notification for new article Controller subscribe to View for Event MediaPlayer notify user the status of current player ConsumerBase subscribe to BufferQueue for enqueue event SurfaceFlinger subscribe to HWC for hotplug / vsync

10. Proxy In China, we need a ________ to visit YouTube .

11. Proxy - UML

12. Examples Sp implementation - Smart Reference Binder - Remote Proxy

13. Bp / Bn

14. Bp / Bn Example

15. Producer Consumer Sushi shop

16. Simplest form put get producer consumer

17. Not that simple, Actually put/get usually in different thread ->Need synchronization ->Need define wait/wake policy for empty/full

18. Evolution Evolves (Resource)Queue Evolves Producer Evolves Consumer

19. Questions? Isn’t put and get enough to describe the model? Why we need dequeue? is dequeue an consumer action or producer action? dequeue acquire enqueue release

20. Sushi shop, example Each Plate cost $1000 So for each customer they only provide you 3 Plate Y ou have to return the used Plate to the trail so that servant can make next sushi.

21. +Resource Pool dequeue acquire enqueue release

22. Resource State free dequeued queued acquired dequeue queue cancel acquire release

23. Participants 店员 - Producer - SurfaceTextureClient/ANativeWindow 顾客 - Consumer - SurfaceTexture/FramebufferService 转盘 - Resource Pool - BufferQueue 盘子 - Resource - Graphic Buffer 寿司 - Resource Content - Content of Graphic Buffer

24. When to consume? +Observer dequeue acquire enqueue release Observer

25. Scheduled consumer dequeue acquire enqueue release Observer Scheduler Message trigger

26. Producer in another Process + Proxy BQ dequeue acquire enqueue release Observer Scheduler Message Proxy trigger

27. +Sync- er When a dequeue a resource but actually you can not use it immediately. You have to wait until that resource is free. Then, why not dequeue block? BQ dequeue acquire enqueue release Observer Scheduler Message Proxy trigger sync- er Sync- er

28. SurfaceFlinger VSYNC BQ dequeue acquire enqueue release Observer Scheduler Message Proxy surfaceTextureClient BufferQueue SurfaceTexture SF main thread trigger fence n ew fence trigger

29. Surface INVALIDATE SurfaceTexturClient ::queue -> BufferQueeu :: QueueBuffer - > SurfaceTexture got notified -> Layer:: onFrameQueuedLayer -> SignalLayerUpdate -> requestNextVsync VSYNC come -> dispatchMessageInvalidate SF main thread -> handleMessageInvalidate -> acquireBuffer -> createIMGKHR -> postMessageRefresh 1.Producer thru proxy 2.Listener get notified 3.Schedule Consumer 4.It is time to consume! (thru ActiveObject ) 5.Consume

30. Connecting Producer/Consumer You can understand Compositor SurfaceTexture SurfaceTexture FrameBufferSurface REFRASH happened after INVALIDATE

31. REFRESH Draw each layer’s latest onscreen buffer as Texture to FBS - prepare eglSwapBuffer for FBS EGL queue current buffer - produce dequeue next buffer of FBS FBS’s listener update fb handle - listener notified HWC commit will display the updated fb handle - consume

32. INVALIDATE and REFRESH

33. More

34. Thread Pool

35. Goal: Increase throughput and Improve responsiveness ServerSocket socket = new ServerSocket (); While(true){ Socket con = socket.accept (); handleRequest (con); } Sequential execution Thread– Per -Task service client client client client client client

36. Thread pool Pattern is where a number of threads are created to perform tasks submitted to task queue client Task submission Task execution Submit task r etrieve result

37. Let’s return to Producer/Consumer a while… Task Queue is a Producer/Consumer , multiply Consumer, one or more Producer BufferQueue is really a Resource Pool

38. Binder Server

39. Those Binder Thread will submit the task to SurfaceFlinger main thread. Which is Achieved though….

40. Active Object

41. The active object design pattern decouples method execution from method invocation for objects that each reside in their own thread of control.

42. Why not Passive Object Object called in the client thread, result in more complex synchronization Schedule/ MessageQueue sequential the request to servant object

43. Active Object: Participants

44. SF main thread Message queue Client postMessage surfaceFlinger Active Object: Used in SurfaceFlinger

45. How message are submitted from client thread to server thread?

46. Command Pattern Purpose: Decouple the Invoker and the Receiver The command can be queue, transferred, stored and/or execute at different time.

47. Command Pattern

48. SF in Action SF main thread SF MessageCreateSurface Client

49. Combined: Thread Pool + Command + Active Object + Proxy sp < ISurface > Client:: createSurface ( ISurfaceComposerClient :: surface_data_t * params , const String8& name, uint32_t w, uint32_t h, PixelFormat format, uint32_t flags) { class MessageCreateLayer : public MessageBase { public : MessageCreateLayer (SurfaceFlinger* flinger, ISurfaceComposerClient :: surface_data_t * params , const String8& name, Client* client, uint32_t w, uint32_t h, PixelFormat format, uint32_t flags) : flinger(flinger), params ( params ), client(client), name(name), w(w), h(h), format(format), flags(flags) { } sp < ISurface > getResult () const { return result; } virtual bool handler() { result = flinger-> createLayer ( params , name, client, w, h, format, flags); return true; } };   sp < MessageBase > msg = new MessageCreateLayer ( mFlinger.get (), params , name, this, w, h, format, flags); mFlinger -> postMessageSync ( msg ); return static_cast < MessageCreateLayer *>( msg.get () )-> getResult (); }  

50. Architecture Integrated, Connected Patterns

51. Recap Observer Proxy Producer/Consumer Thread Pool Command Pattern Active Object

52. BnSurfaceTexture SF Main Thread BufferQueue SurfaceTextureClient FBS BufferQueue invalidate invalidate refresh SurfaceComposerClient Client SurfaceFlinger P roxy Thread Pool Producer/Consumer Command Active Object Observer

53. Android View System Libgui SufaceFlinger HWC EGL Apps 3D Android UI Framework: View ,Usage & Patterns Android App Development 101 Android SurfaceFlinger: Architecture explained by User case Android UI: Patterns Perspective Other Topics Android Graphic Integration