Android Multimedia Framework

This slide shows how the StageFright framework works on the Android.

1. Android Multimedia
Framework

on Jelly Bean

Author: Picker

2. Introduction to OpenMAX

3. Platform Media Framework

PD penMAX (IL peines mt conventos
Audio Video

Media Engines - CPUs, DSP, Hardware Accelerators etc.

independently from the other layers

OpenMAX layers can be none together or

Resource from Khronos::OpenMAX

4. Application Application

Application

Multimedia Framework / Middleware

E
System Media Driver

OpenMAX Integration Layer

Component Component Component
Interface Interface Interface

Codec

OpenMAX Development Layer

DL Primitives DL Primitives

Resource from Khronos::OpenMAX

5. from
Multimedia Framework

to
OpenMAX

6. Application Application

Application

Multimedia Framework / Middleware

E
System Media Driver

OpenMAX Integration Layer

Component Component Component
Interface Interface Interface

Codec

OpenMAX Development Layer

DL Primitives DL Primitives

Resource from Khronos::OpenMAX

7. Application Application
Application
Multimedia Framework / Middleware
| E
OpenMAX Integration Layer
Component Component Component |
Interface Interface Interface !
Codec Codec E
|
for cet of compen ie —— |
OpenMAX Development Layer
DL Primitives DL Primitives

Resource from Khronos::OpenMAX

8. Go through the OpenMAX

Operating System

9. APPLICATIONS

Se ae eee .__ AN

APPLICATION FRAMEWORK

Manager Manager Providers System Manager

Manager

LIBRARIES ANDROID RUNTIME

Surface Media 3 e T EN

Manager Framework Libraries

Dalvik Virtua

OpenGL|ES FreeType
Machine

LINUX KERNEL

Display Camera Bluetooth Flash Memory Binder (IPC)
Driver Driver Driver Driver Driver

USB Keypad WiFi Audio Power
Driver Driver Driver Drivers Management

10. The simple stack architecture

Developer API (MediaPlayer)

EventHandler SurfaceHolder

11. Big picture of the StageFright

StageFright Framework
TimedEventQueue| OMXCodec | OMXClient

Renderer == DO EEE —]

AudioPlayer MediaExtractor

12. The Simple Workflow

Applications
Media Player
Application
Application Framework
Mediaplayer ————>_android_media_MediaPlayer
Libraries
(mediaplayer.cpp)__ Mediaplayerservice —> StageFrightPlayer
M layer |
AwesomePlayer
AudioPlayer MediaExtractor OMXCodec TimedEvent Queue — AwesomeLocalRenderer
MPEGAEXtractor omx SoftwareRenderer

;

ColorConverter

13. The StageFright Workflow

2. Get the Audio Track &
Video Track.

Developer API (MediaPlayer)

1. Create an extractor for the Media File.

5. Decoded video data

AwesomeRenderer

4. Return the decoded data

5. Sync the audio &
video time

Y

TimedQueueEvent

3. Put the media source into the codec

5. Decoded audio data

AudioPlayer

14. Path of All the Components

15. Path of the Source Files

APPLICATI
frameworks/base/media/java/android/media

APPLICATION FRAMEW

frameworks/base/media/jni

LIBRARIES ANDROID RUNTIME

frameworks/av/media/libmedia
frameworks/av/media/libmediaplayerservice

frameworks/av/media/libstagefright

LINUX KERNEL

16. Big Picture

Class Diagram

17. Il

18. High-Level Applications Framework

Applications Framewarl

+ Mediaplayer (MediaPlayerClien

[reeibatasourced frequent

+ Mediatiayertiort

[leretibranegy + |
H Ea
a
+ android. media MediaPlayer rentransact (sequential ‘FaetiediaPlayerSenieed troquet

[vaetOatasouree)) (eequentialy
prepared) (sequential
[+astart0 {sequential

be = edapTeyarservee,
[Forester] (sequential)
Fseidatasocen Teee een DE
‘Toervideosuracetesture() (sequential) get OM {sequential
prepareasyned tsequertal}
status 10 (sequential FastDatasoureen Teequertaly

start {sequent

TEMER GAS {prepared secuen

Hs Estan {sequential

+prepareñsircl (sequent
Ranas pool ÉsotLoopng fsoquertal)
‘Ssetvalomed) sequentaly

ER
= WaciaPrayarseniee FAUODUpUE
‘eeibataSoureel) ear
+ Client | FrerestePlayert (sequently
Ésotoatasource) fsoquential
DOUTE estan (sequently

LEE TE ETUI

19. Il

20. + Riatwewndowa] [7 SufareTeurechent

StageF right Framework

ad

ce Cocos FestgeureeO fsequenta
stan (coqueta + Mediasours
SevetTed [sequental

[#paus00 (soquortia)
resume {sequential

amen fia

0530 raqueta

atada

F Re] [EA] [VE
ended Gener PEGAR E

rs ocean

ee: |

Tr pc il RL

A ]

asivatasourced Tauern AL LENS
Fonade sale)

repare) fesquerta)
Biss) troquet F0 raqueta
Éprepareñaynet) {sequential}

“Stand teequential}
Stpestudeotvert 10 {cequential)

nee indus

Frodo
A st0p0 (sequential)
postEvend (sequen

amic FORE

Freannea Teequertan)
+ dacomectí [sequential

a a

e
+ Hediatodeetat

geimstancel (eequentany
Éinécodecsyname0 {sequential
indcodecéyrpe0 (sequential)
-getsuppertedtypest) {eequentialy

22. EL

FaRemoreoMe
[+miocalona

+getomxd {sequentialy
J} +lsiiiodost) (sequential)
+senccommando) (sequential)
+sotParamotart) (sequential)
L+getParametent) {sequential}

om

FOMINodein sance

-lisiNodest) feequertial)
4+senccommand,) (sequential)
“+getParameter() {sequential}

OpenMAX Interface | =

-=AddVendorPlugin) {sequential b
+addPlugind (sequential)
/+makeComponentinstanceO {sequertial}

OPI + SoROWICompanent
[-makecomponentnstance0 (sequential) HB Handied) (sequential
+ dostroyComponentinstance() (sequential)

SimplesoROMicomponent

‘FonwessageRaceived) (sequently

ade

= SORMPEGA] E AGE

23. Four Steps for Playing the Media File

Step 1: MediaPlayer mp = new MediaPlayer();
Step 2: mp.setDataSource(mediaPath);

Step 3: mp.prepare();

Step 4: mp.start();

24. reset)

E —

prepareAsyncQ,

onEiror(),
—

OnPreparedlistene onPrepared)

prepareAsync)

Looping — true &&
playback completes

seekTo(/pause()

Looping — false &&
onCompletion() invoked on

stopQ \\ OnCompletionlist

stan)
(note: fim beginning)

seckTo()

25. How to set the data source?

26. The workflow of the setDataSource

Media Player
Application

setDataSource()

Application Framework

MediaPlayer android_media_ MediaPlayer

setDataSource()

setDataSource()

a j mom) |

create) setDataSource() setDataSource()

Return the type of the MediaExtractor Get the Track and MetaData

Create the extractor

27. The sequence flow of the setDataSource

28. seDataSourceo | OnBrrorListener ESC D

prepare Async()

OnPreparedListener. onPreparedO

prepareAsync()

Looping = true &&
playback completes

ho, seckTo(/fpause()

Looping — false &&
onCompletion() invoked on

stop. \ OnCompletionlist

start)
(note: lpm beginning)

seckTo()

29. How to prepare to play?

30. The workflow of the prepare

Media Player
Application

prepare()

Application Framework

MediaPlayer android_media_MediaPlayer

prepare()

prepareAsync()

Create the AwesomeEvent for

preparing the synchronization postEvent()

31. The sequence flow of the prepare

Meda

Peer gemir Ame OS

setvideosutaceTedtured. |

prepareñsynct)

32. seDataSourceQ | OnBrrorListener u)

prepareAsync()

OnPreparedListener.onPreparedQ)

Looping = true &&
playback completes

bo seckToOfpause)

Looping = false &&

onCompletion( invoked on
OnCompletionList an

(note: beginning)

seekToQ

33. How to start the playing?

34. The workflow of the start

Media Player
Application

MediaPlayer

Application Framework

android| media_MediaPlayer

postVideoEvent_I()

35. The sequence flow of the start

mea

pare

Ar

tat

Fern
>

36. Synchronization Architecture of StageFright

Em Ea

37. Synchronization Flow of StageFright

5. onVideoLagUpdate.
Call the postVideoLagEvent_I() to

3. onVideoEvent handle the delay.

1. Call the start) to
start audio player

2. Synchronize playing.
Catch the timing of the video & audio

Y

| TimedQueueEvent

AudioPlayer

2. in the fillBuffer(), the memepy() will be
called in order to copy the data buffer
for renderer

4, Call the postVideoEvent( for rendering

38. The Synchronization Formula of StageFright

sizedone
mFrameSize )
mSampleRate

mNumF ramesP layed +

mPositionTimeRealsUs =( Xx 1000000

mTimeSourceDeltaUs =mPositionTimeRealUs —mPositionTimeMediaUs
nowUs= RealTimeUs— mTimeSourceDeltaUs

latenessUs= now Us — timeUs

mPositionTimeRealsUs: the time in real playing
mPositionTimeMediaUs: the time which is defined in the media source

39. The Synchronization Conditions

Conditions of latenessUs
1. > 5000001!

2. >40000
3. <-10000 1750 if (latenessUs > 5000001!
1751 && mAudioPlayer != NULL
1752 && mAudioPlayer->getMediaTimeMapping(
1753 &realTimeUs, &mediaTimeUs)) {
1757 mVideoBuffer->release();
1758 mVideoBuffer = NULL;
1763 postVideoEvent_10);
1764 return;
1765 }
1767 if (latenessUs > 40000) {
1768 // We're more than 40ms late. 1793 if (latenessUs < -10000) {
à 1794 II We're more than 10ms early.
ee Te aire 1795 postVideoEvent_I(10000);
mydeoBuflen= d 1796 return;
1787 2007
1788 postVideoEvent_I();
1789 return;

1791s}

40. Conclusion

*Easy for maintaining rather than OpenCore
eSupport Software/Hardware Codecs
-Unstable Product
«Rough Mechanisms
° Create the NuPlayer for playing the
multimedia streaming