Mobile Graphics (part1)

Marco Agus (CRS4)

1. Mobile Graphics Marco Agus Marcos Balsa June 2015 1

2. Outline Part 1: Evolution of mobile graphics Part 2 : Graphics development for mobile systems Part 3: Mobile graphics trends and real time visualization of massive models on mobile systems 2

3. Part 1 Evolution of mobile graphics 3

4. Mobile evolution (1/3) 4

5. Mobile evolution (1/3) Color display 5

6. Mobile evolution (2/3) 6

7. Mobile evolution (2/3) Smartphones , OS 7

8. Mobile evolution (3/3) 8

9. Mobile evolution (3/3) High resolution displays 9

10. Mobile evolution (3/3) Similar evolution for PDAs , and tablets 10

11. Mobile evolution…. in movies Motorola DynaTac Nickname “brick phone” Weight: over 2 pounds Cost: thousands of dollars B attery life: around 35 minutes. Money never sleeps…..This is your wake-up call, pal… GO TO WORK Wall Street , 1987 Michael Douglas in Gordon Gekko 11

12. Mobile evolution… in movies Hello Neo… Do you know who this is? The Matrix , 1999 Laurence Fishburne in Morpheus Nokia 8110 Nickname “banana phone” 145g, display monochrome, Smart SMS It costed 1000 eur 12

13. Mobile evolution 13

14. Mobile evolution 14

15. Mobile evolution 15

16. Mobile evolution… in movies Skyfall , 2012 Daniel Craig in James Bond Sony Xperia T Smartphone Android Display 4.6” 1280x720 It costed 6 00 eur 13 Mpixel camera + position sensors 16

17. Mobile evolution… in movies Iron Man 3 , 2013 Robert Downey Jr in Tony Stark Future devices? Transparent and foldable high resolution screens Gesture interfaces Wearable / integrated to body 17

18. Mobile evolution 18

19. Mobile evolution... in games Nokia Snakes From 1997 an estimated 350 Mdevices, making it one of the most widely distributed games ever created. Installed on Nokia devices until 2007 19

20. Mobile evolution… in games Angry Birds (Rovio) first released for Apple's iOS in December 2009 2 billion downloads across all platforms widespread diffusion end popolarity Adventure parks (Finland and Malaysia) 20

21. Mobile evolution… in games Unreal Engine (GDC & Google I/O 2014) running on an  Nvidia  Tegra K1 processor will support Google Tango and Samsung Gear VR easy porting of games sophisticated 3d effects 21

22. Mobile connectivity evolution Bandwidth is doubling every 18 months Mobile internet users overcame desktop internet users 2017 smartphone traffic expected at 2.7 GB per person per month 22

23. Displays and User Interface Before 2007 – old days PDA  Palm OS/ Windows Pocket / Windows CE Stylus interaction (touch screens at early stages) Touch era 2007 – iOS /iPhone 2008 – Android / HTC Dream or G1 Touch-enabled devices (no stylus required) Nowadays Wearables  <2” Smartphones  3-6” Tablets  >7-10” DLP projectors integrated 23

24. Display characteristics 500 ppi 5.1”/5.5” 2560x1440 (>500 PPI) 10.1” 3840x2160 (438 PPI) 24

25. Chip evolution (1/2) 25

26. Chip evolution (2/2) 26

27. Modern smartphones (tablets) are compact visual computing powerhouses DIFFUSION: more than 4.6 billion mobile phone subscriptions [Ellison 2010] NETWORKING: High speed internet connection (typical 1GB/month plan) 3G - < 0.6-3Mbps ~ 100KB/s - 400KB/s (latency ~ 100-125ms) 4G – < 3-10Mbps ~ 400KB/s - 1MB/s (latency ~ 60-70ms) 5G - 1Gbps (from 2016?) MEMORY: Increasing RAM and storage space RAM 1-3GB Storage 8-64GB COMPUTING: Increasing processing power CPU 4-8 core @ 2.5Ghz GPU 72-192 cores (~ALUs) Scenario 27

28. Scenario More than 4.6 billion mobile phone subscriptions [Ellison 2010] High speed internet connection (typical 1GB/month plan) 3G - < 0.6-3Mbps ~ 100KB/s - 400KB/s 4G – < 3-10Mbps ~ 400KB/s - 1MB/s Increasing RAM and storage space RAM 1-3GB Storage 8-64GB Increasing processing power CPU 4-8 core @ 2.5Ghz GPU 72-192 cores (~ALUs) http://unity3d.com/es/unity/multiplatform/mobile 28

29. Where are we going? Powerful devices for acquiring, processing and visualizing information Accessibility of information (anybody, any time, anywhere) Immense potential (integration of acquisition, processing, visualization, cloud computing, and collaborative tasks) 29

30. Mobile Graphics Marco Agus Marcos Balsa June 2015 Development 30

31. Mobile Graphics OS architecture programming languages 3D APIs IDEs Heterogeneity 31

32. Mobile Graphics OS architecture programming languages 3D APIs IDEs Heterogeneity 32

33. Mobile Graphics OS Programming Languages Architectures 3D APIs Cross-development X86 (x86_64): Intel / AMD ARM (32/64bit): ARM + (Qualcomm, Samsung, Apple, NVIDIA,…) MIPS (32/64 bit): Ingenics , Imagination. Android iOS Windows Phone Firefox OS, Ubuntu Phone, Tizen … C++ Obj -C / Swift Java C# / Silverlight Html5/JS/CSS OpenGL / GL ES D3D / ANGLE Metal / Mantle / Vulkan (GL Next) Qt Marmalade / Xamarin / Muio Monogame / Shiva3D / Unity / UDK4 / Cocos2d-x 33

34. Operating Systems 34

35. Operating Systems Linux based ( Qt …) Ubuntu, Tizen , BBOS… Web based (HTML5) ChromeOS , FirefoxOS , WebOS (deceased?)… Windows Phone iOS (~ unix + COCOA) Android (JAVA VM) http://www.theregister.co.uk/2014/08/04/android_beats_ios_for_first_time/ 2014 35

36. Operating Systems arm x86 mips 3D driver OpenCL support C/C++ Main API android v v v GLES 3 wary / dev drivers - renderscript yes Java ios v GLES 3 only for sys yes, through obj-c objective-C Winphone 8 v v D3D / ANGLE DirectCompute yes .NET Silverlight tizen v GLES 3 yes yes C++ firefox OS v v GLES 3 no no HTML5/JS/CSS ubuntu touch v v GLES 3 ? yes HTML5, QML 36

37. Operating Systems Windows Phone 8 Windows development – Visual Studio Good debugging / compiler / integration Great integration and deployment API access C#, VB.NET, and C++ 3D API D3D OpenGL access through ANGLE Advantages Visual Studio HW is quite selected/homogeneous Disadvantages ~OpenGL wrapper just recently! 37

38. Operating Systems iOS Development under MacOS Xcode – good IDE/debug Clang compiler! API access Objective-C, swift Library programming C++ support Advantages: Homogeneous hardware (biggest issues are resolution related) State-of-the-art CPU/GPU (PowerVR SGX 54X/554, G6400) Good dev tools (Xcode + Clang 5.1) Inconvenients: Closed platform Requires iDevice for development/shipment (mostly) 38

39. Operating Systems Android Development in Eclipse Java-based – integrated debugging (non-trivial for NDK) GCC / clang compilers Advantages Wide variety of hardware configurations (CPU/GPU) Java based + C++ as dynamic library (JNI or NDK+NativeActivity) Open source Toolchain provided for Windows/Linux/MacOS (GCC 4.9 + Clang 3.4) Faster access to new hardware / functionality! Inconvenients Heterogeneous device base (hard to target all configurations) Not so integrated IDE -- ~mixed pieces 39

40. Operating Systems App development -- publishing WinPhone & iOS requires less effort for distribution Easy to reach the whole user base Android has a wide variety of configuration that require tuning User base is typically reached in an incremental way (supporting more configs) Many HW configurations (CPU/GPU) give more options to explore  WinPhone has not yet the same market share Variety of configurations Research A matter of taste ? Dev. Platform ? Linux  Android (OpenGL support!) Mac  iOS (OpenGL support!) Windows  Windows Phone (OpenGL support?) / Android ??  40

41. Operating Systems More in depth iOS provides good tools for development/profiling Big codebase in Linux (porting issues are not big) Tied to Xcode IDE  not bad at all! Clang compiler is pretty good OpenGL access is fine and drivers are homogeneous Mac + iDevice are required!  bad!  friends ? Safari WebGL not yet compliant Android provides toolchain to be integrated on various IDEs Eclipse/Qt Creator/Visual Studio/… Easy to develop/debug with GDB ? Not so well-integrated Simpler access – devices/market/dev. Platform Windows provides the best IDE Devices are always one step behind API now getting close to… (OpenGL) 41

42. Operating Systems Brief comparison (focus is research here  ) Windows phone 8 + Best IDE – Visual Studio (2013) - Windows development (porting Linux code?) - Market is quite restrictive (100$ /year. – 5 free uploads/year) Certification + review - OpenGL support ? Through ANGLE (over D3D) iOS + Best devices ? (homogeneity, at least) + Good IDE – Xcode + clang - Market is rather restrictive (100$/year) Review + OpenGL support – PowerVR GPUs only Android + Best platform ? (more open, more devices , more flexible ?) - Many IDEs – integration is not great (~tricky) + Visual Studio / Eclipse / QtCreator + With GCC / clang compiler + Market is very accessible (25$) + OpenGL support / OpenCL support Monetization? Research? Monetization?? 42

43. Programming Languages http://www.tops-int.com/blog/which-programming-languages-are-used-for-web-desktop-and-mobile-apps/ 43

44. Programming Languages C/C++ Classic, performance, codebase, control Objective C Bit different style (message based), well-documented API for iOS, mainly COCOA/iOS Java Android is VM/JIT based, ~portability (API), well-known, extended, codebase C# VM based, ~Java evolution, MONO (Win, Android, iOS) Swift Apple new language, simplicity, performance, easy, LLVM-based compilers HTML5/JS Web technologies, extended, compatibility Perl, Python, Ruby, D, GO (Google), Hack (facebook), … More options, not so popular ? 44

45. Programming Languages C/C++ Classic, performance, codebase, control Objective C Bit different style (message based), well-documented API for iOS, mainly COCOA/iOS Java Android is VM/JIT based, ~portability (API), well-known, extended, codebase C# VM based, ~Java evolution, MONO (Win, Android, iOS) Swift Apple new language, simplicity, performance, easy, LLVM-based compilers HTML5/JS Web technologies, extended, compatibility Perl, Python, Ruby, D, GO (Google), Hack (facebook), … More options, not so popular ? Typically light front-end against main platform API programming language, loading codebase in whatever language through dynamic code loading 45

46. Programming Languages C/C++ Classic, performance, codebase, control Objective C Bit different style (message based), well-documented API for iOS, mainly COCOA/iOS Java Android is VM/JIT based, ~portability (API), well-known, extended, codebase C# VM based, ~Java evolution, MONO (Win, Android, iOS) Swift Apple new language, simplicity, performance, easy, LLVM-based compilers HTML5/JS Web technologies, extended, compatibility Perl, Python, Ruby, D, GO (Google), Hack (facebook), … More options, not so popular ? Swift 2 Open-source  Performance (LLVM)  Portability ?  46

47. Architectures MIPS ARM x86 47

48. Architectures CPU architectures X86 – ARM – MIPS 48

49. Architectures x86 (CISC 32/64bit) Intel Atom Z3740/Z3770 Bay Trail (2W) AMD Mullins (not yet in the market) 4.5W -Power consumption +Performance PartOf (desktop class GPU!) +compatibility with old SW ? 49

50. Architectures x86 (CISC 32/64bit) Intel Atom Z3740/Z3770 Bay Trail (2W) AMD Mullins (not yet in the market) 4.5W ARM RISC 32/64bit MIPS RISC 32/64bit Acquired by Imagination , Inc. @ 2014 -Power consumption +Performance PartOf (desktop class GPU) +compatibility with old SW ? +Power efficiency +Performance/watt +Smaller area (RISC)  lower cost +demonstrated its capacities on consoles (PS/PS2/PSP/N64/Wii…), also on SGI  50

51. Architectures – RISC vs. CISC but… CISC (Complex Instruction Set Computer) Fast program execution (optimized complex paths) Complex instructions (i.e. memory-to-memory instructions) RISC (Reduced Instruction Set Computer) Fast instructions (fixed cycles per instruction) Simple instructions (fixed/reduced cost per instruction) FISC (Fast Instruction Set Computer) Current RISC processors integrate many improvements from CISC: superscalar, branch prediction, SIMD, out-of-order Philosophy  fixed/reduced cycle count/instr. (SIMD?) Discussion (Post-RISC): http://archive.arstechnica.com/cpu/4q99/risc-cisc/rvc-5.html 51

52. Architectures RISC Integrate complex instructions ARM / MIPS CISC Reduce instruction complexity Intel Atom MMX/SSE/Out-of-Order 52

53. Architectures Tegra X1 8-core CPU 256 core Maxwell GPU 53

54. Architectures 54

55. Architectures A comparison of the share of revenue generated by Intel and ARM based manufacturers. Sources:  EW / iSuppli 55

56. Architectures A comparison of the share of revenue generated by Intel and ARM based manufacturers. Sources:  EW / iSuppli Are desktop computers stalling ??? Average user – mobile/tablet/docking station Power user – desktop/supercomputer 56

57. Architectures – X86 Intel (32/64 bit) Competitive with Bay Trail Atom Z3470 ~4W Pursuing low power consumption instead of performance GPU: Intel HD graphics for Bay Trail ~ GF 8600M GT | GF210 Present in many tablets (i.e. Surface) with Windows Phone/Android Present in a few smartphones AMD Not yet competitive in low power > 4W  Good GPU performance (GCN 192 core) No known smartphone/tablet shipped Supported on Android, Windows Phone, Tizen, Firefox OS, Ubuntu Touch,… 57

58. Architectures – X86 Typically paired with integrated GPU Intel HD Graphics Radeon APU General strategies Well known from desktop Mostly  Cache coherence temporal/spatial ordering ~Typically lower frequency  exploit multi-core SIMD  MMX/SSE/SSE3/SSE3 Desktop entry-level GPU performance Although cut down  58

59. Architectures – ARM ARM Ltd. RISC processor (32/64 bit) – getting to 64bit ~ 2014/15 IP (intellectual property) – Instruction Set / ref. implementation CPU / GPU (Mali) Licenses (instruction set OR ref. design) Instruction Set license -> custom made design (SnapDragon, Hummingbird in iPhone 4 & Galaxy S) Optimizations (particular paths, improved core freq. control,…) Reference design (Cortex A9, Cortex A15, Cortex A53/A57…) Licensees (instruction set OR ref. design) Apple, Qualcomm, Samsung, Nvidia, MediaTek, AMD @<2014 … Few IS licenses, mostly adopting reference design Manufacturers Contracted by Licensees GlobalFoundries, United Microelectronics, TSM, and Intel (@2013)  59

60. Architectures – ARM… Suppported on Android, iOS, Win Phone, Tizen, Firefox OS, BlackBerry, Ubuntu Phone, … Biggest mobile market share Typically paired with mobile GPUs: Adreno 2x0/3x0/4x0 – Qualcomm PowerVR SGX 54X / V64x0 (Rogue) – Imagination Mali 400/450/T628 – ARM GeForce ULP / K1 – Nvidia Vivante GC4000 General strategies: Cache coherence – week sequential code guarantees on multithreading!! Heavy dependence on compiler  optimize instruction scheduling Operation dependencies , loop unrolling, etc… Use SIMD extensions   NEON 60

61. Architectures - MIPS MIPS RISC processor (32/64 bit) IP (intellectual property) – licensing Recently acquired by Imagination, Inc. Can provide full solution ( SystemOnChip , SoC ): wireless/ cpu / gpu Performance/watt should be comparable to that of ARM GPU from Imagination have demonstrated its value iDevices have always included its PowerVR SGX / Rogue cores Good integration with CPU and other components on SoC could provide a very competitive solution (i.e. Qualcomm) Supported on Android, Mer (fork from MeeGo ) Knowledge from previous HW (PSP, PS, PS2, WII…) Pretty much the same with ARM HW 61

62. Some notes on multithreading SMP (Symmetric Multi Processor) – shared memory Multithreading – two threads on the same core Code reordering matters (out of order) We expect All memory operations are executed in order All operations on a single processor appear as executed in order But X86 provides processor consistency Loads are not reordered against other loads (same happens with Stores) Load and Store instructions are not guaranteed in order ARM provides even weaker sequential guarantees Load and Stores can be reordered 62

63. ARM CPUs – code example Solutions: Memory barriers CPU dependent Atomic operations CPU dependent PTHREAD mutex Uses atomic operations in a portable way http://developer.android.com/training/articles/smp.html 63

64. Architectures GPU architectures 64

65. Tessellation & Geom. Proc. Architectures - GPU Primitives Vertex Processing Primitive Assembly Rasterization Pixel Processing Framebuffer Operations Vertex Shader Geometry Shader Fragment Shader Tessellation Eval./Control Shader Image courtesy of: http://rnd.azoft.com/fluid-dynamics-simulation-on-ios/ Simplified OpenGL 3D pipeline (ES 2.0  3.0) Fixed hardware: Vertex + Pixel Shaders GPU Unified Shaders: Vertex/Pixel -- Compute Desktop 65

66. Architectures - GPU Dedicated shaders Vertex Shader + Pixel Shader Unified architecture (modern) Shader core Vertex/Geometry/?/Pixel shader Texture access Triangle Assembler Rasterizer  (#ALU/MADDs / core) 66

67. Architectures - GPU Talking about performance… What is a GPU core ? Mobile  typically full core/ proc Replication of HW at high level Desktop Compute Units ? ALUs ?... We’ll try to count ALUs Nowadays less hard to find info Mali 400MP4 -> full 4 cores (~ proc )! PowerVR SGX544MP3  3 full cores! 67

68. Architectures - GPU Device CPU GPU Gflops Gflops FP32 Cores/ALUs Samsung Glxy Nexus (2011) TI OMAP 4460 2-core A9 PowerVR SGX 540 4,8 0,01 8 cores Adv. chinese dev (2014) MediaTek MT6589T 4-core A7 PoverVR SGX 544MP2 9,6 0,03 16 cores Apple iPhone 5 (2012) Apple A6 2-core (armv7) PowerVR SGX 543 MP3 28,8 0,08 48 cores 2013 Android flagship (HTC One, Galaxy S4) Snapdragon 600 4-core Krait 300 Adreno 320 51,2 0,14 64 cores Avg Tegra 4 device (2014) (4+1)-core A15 GeForce ULP 96,8 0,20 24VS + 48PS= 72 cores Apple iPad 4 (2012) Apple A6x 2-core (armv7) PowerVR SGX554 MP4 76,8 0,21 128 cores Apple iPhone 5s (2014) Apple A7 2-core (armv8) PowerVR G6430 115,2 0,21 128 cores 2014 Android flagship (Galaxy S5) Samsung Exynos 5 4-core A7 + 4 core A15 ARM Mali T628 MP6 102,4 0,28 96 cores 2014 Android flagship (Galaxy S5) SnapDragon 801 4-core Krait 400 Adreno 330 166,5 0,46 128 cores Sony PS 3 (2006) PowerPC 1 core + 7 SPE Nvidia G70 228,8 0,63 8VS + 24PS=136 cores XBOX 360 (2005) 3-core x86_64 ATI R500 Xenos 240,0 0,66 192 cores MiPad, Project Tango, HTC Volantis (2014) 4+1 cores A15 Nvidia Tegra K1 326 1,00 192 cores One Plus 2 SnapDragon 810 64bit 4xA53 + 4xA57 Adreno 430 324-388 1,0 256 core Core ~ #ALU/MADDs 68