A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems

This tutorial introduces modern performance and energy-aware video coding and content delivery solutions and tools, focusing on popular video streaming applications, i.e., VoD and live streaming. In this regard, after introducing fundamentals of modern video encoding and networking paradigms, we introduce modern solutions systems, using per-title encoding, per-scene encoding, virtualized and software networks, edge computing, overlay networks such as Content Delivery Networks (CDNs) and/or Peer-to-Peer (P2P) paradigms to provide latency and energy-efficient VoD and live HAS streaming. Furthermore, the tutorial also presents our tools, software, datasets, and testbeds to demonstrate our latest achievements and share practical insights for researchers, engineers, and students who want to improve conversational streaming or even test such techniques with a focus on latency, economic cost, and energy.

1. A Tutorial on Latency- and Energy-Aware
Video Coding and Delivery Streaming Systems
—
Dr. Reza Farahani, Dept. of Information Technology (ITEC), University of Klagenfurt, Austria
Dr. Vignesh V Menon, Video Communication and Applications Dept., Fraunhofer HHI, Germany

09.02.2024 © FraunhoferSeite 1
EUVIP 2024 - Geneva, Switzerland Sept 2024

2. MHV’24
Topics to Cover
Part I: Introduction to Video Streaming

Part II: Video Coding
●Introduction
●Fundamentals of Video Coding
●Video Complexity Analysis
●Per-Title and Per-Scene Encoding for Adaptive Streaming
●Latency-aware Video Coding for Adaptive Streaming

Part III: Video Delivery
●Fundamentals of Video Computing and Networking Paradigms
●Large-scale Testbed Design
●Latency-and QoE-aware Adaptive Video Streaming Delivery




A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

3. Part 1: Video Streaming
—
Introduction
—

4. MHV’24
Introduction
Video Traffic
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

●Video is dominating today’s Internet traffic
○Video streaming includes 66% (fixed) and 64% (mobile) of the total
Internet traffic
○Video-on-Demand (VoD) with 54% and live streaming with 14%
have become significantly popular video streaming applications

Sandvine Global Internet Phenomena Report (Jan 2024)

5. MHV’24
Introduction
Progressive Download
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
One Request
One Response
video

6. MHV’24
Introduction
Video Streaming
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●Continues content transmission by server and simulatinus consumption by client

7. MHV’24
Introduction
Video Stream Metrics
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●Bitrate: The amount of data transmitted per second in the video stream
●Resolution: The dimensions of the video frame (width x height)
●Framerate: The number of frames displayed per second (e.g., 24, 30, 60 FPS), influencing the
smoothness of the video

8. MHV’24
Introduction
Popular Streaming Services
https://www.wowza.com/blog/what-is-low-latency-and-who-needs-it
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

9. MHV’24
Introduction
Video Delivery over HTTP Protocols
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●Based on a concept called "segmentation"

○Content split up in numerose fixed- duration segments
○All segments can be decoded individually
○Allows to seamlessly switch to different encodings at chunk boundaries

●Live Content
○Manifest and chunk files created and delivered in real time
○Manifests need to be polled periodically

●Most used implementations

○Apple's HTTP Live Streaming (HLS)
○MPEG Dynamic Adaptive Streaming over HTTP (DASH)
○Microsoft's HTTP Smooth Streaming (HSS)

10. MHV’24
Introduction
Video Delivery over HTTP Protocols
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
1
2
3
4
5
5
5
6
6
6
7
Encoder
Origin
Packager
CDNs
HTTP ResHTTP Req
Reza Shokri Kalan, Reza Farahani, Emre Karsli, Christian Timmerer, and Hermann Hellwagner. Towards Low Latency Live Streaming: Challenges in a
Real-world Deployment. 13th ACM MMSys, 2022.
Quality
Bandwidth
TimeTime

11. MHV’24
Introduction
Video Delivery over HTTP Protocol
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●HTTP pseudo-streaming

○Like progressive download
○But seeking is supported (jump 2 time)

● RTMP Streaming
○Proprietary Adobe protocol

●HTTP adaptive streaming

○Server provides many small files
○In different resolutions / bitrates
○Client glues files together for full video

12. MHV’24
Introduction
Different HAS Protocols (MPEG DASH Manifest Structure)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●Describes part of the content
●Has a start time and a duration
●e.g. used as scenes or chapters as well as ads

13. MHV’24
Introduction
Different HAS Protocols (Apple HLS Manifest Structure)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

14. MHV’24
Introduction
Bitrate Ladder
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

15. MHV’24
Introduction
Video Streaming Challenges and Tradeoffs
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

Quality
Content Time
Quality
TimeContent
Energy

16. MHV’24
Introduction
Latency and Energy Optimization
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Video Encoding Video Delivery Video Decoding Video Quality Assessment

17. Part 2: Video Coding
—
Introduction
—

18. MHV’24
Video Coding Introduction
Overview of Topics Covered
●Fundamentals of video coding
○Basic principles of video compression, including intra-frame and inter-frame compression.
○Key video codecs (H.264, HEVC, VVC) and how they achieve efficiency.
●Video complexity analysis
○How video complexity affects encoding efficiency and quality.
○Techniques to analyze and optimize video encoding based on content.
●Per-title and per-scene encoding for adaptive streaming
○Overview of per-title encoding, where content is optimized based on its unique characteristics.
○Introduction to per-scene encoding, which dynamically adjusts encoding settings based on the complexity of
individual scenes within a video.
●Latency-aware Video Coding for Adaptive Streaming
○Dynamically adjusting encoding settings based on the coding latency.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

19. MHV’24
Video Coding Introduction

Video Coding
What is video coding?
●Video coding (also known as video compression) is the process of
reducing the file size of video content while maintaining visual quality.
●It achieves this by removing redundancies in the video data (both
spatially within frames and temporally across frames) and by
representing the video in a more efficient way.

Importance of video compression in modern applications
●Efficiency: Compression allows large video files to be stored,
transmitted, and streamed with much lower bandwidth and storage
requirements.
●Scalability: Enables video streaming services (like Netflix and
YouTube) to deliver content to a vast number of users across a variety
of devices and network conditions.
●Cost savings: Reduces data usage and storage requirements for
providers and consumers.
●Improved user experience: Faster load times, smoother playback, and
the ability to stream in high resolution (4K, 8K) even over constrained
networks.
Figure: Significant demand for bandwidth and video in the connected
home of the future. Source: Cisco Annual Internet Report, 2018-2023.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

20. MHV’24
Video Coding Introduction


Use Cases of Video Coding
●Streaming services
○Video coding is the backbone of platforms where
content is streamed on-demand across multiple
devices.
○Adaptive bitrate streaming adjusts video quality in
real-time based on the viewer’s network conditions,
ensuring uninterrupted playback.
●Video conferencing
○Efficient video compression is crucial for real-time
video communication tools.
○It minimizes latency and ensures that high-quality
video is maintained even under bandwidth constraints.
Figure: Number of subscribers for various VoD streaming service providers. Source:
Statista Market Insights
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

21. MHV’24
Video Coding Introduction

Use Cases of Video Coding
●Broadcasting
○Traditional television and live broadcasting use video compression
to transmit high-quality video over limited bandwidth.
○Formats like MPEG-2 and H.264 are widely used in TV
broadcasting.
●Adaptability across devices and network conditions
○Video coding enables the delivery of content that adjusts to varying
screen sizes (from smartphones to 4K TVs) and varying network
speeds (from slow 3G connections to fiber broadband).
○This adaptability is crucial in providing a consistent user experience.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

22. MHV’24
Video Coding Introduction

Evolution of Video Coding Standards
●From MPEG-2 to HEVC, AV1, VVC
○MPEG-2 (1995): One of the first widely-used standards for video
compression. Still used in DVD, digital TV broadcasting, and satellite TV.
○H.264/AVC (2003): A significant improvement in compression efficiency over
MPEG-2, leading to wide adoption in streaming, Blu-ray, and web video.
○HEVC/H.265 (2013): Offers about 50% better compression than H.264 while
maintaining similar video quality. Used for 4K UHD streaming.
○AV1 (2018): An open-source codec developed by the Alliance for Open
Media (AOMedia). Designed for web video and streaming services, with
better efficiency than HEVC.
○VVC (Versatile Video Coding, 2020): The next step in video compression,
providing up to 50% bitrate savings compared to HEVC, designed for 8K
video, VR, and next-generation streaming.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

23. MHV’24
Video Coding Introduction

Evolution of Video Coding Standards
●Key milestones in video compression technologies
○Transition from analog to digital video broadcasting.
○The rise of high-definition (HD), 4K, and now 8K video resolutions.
○Development of adaptive streaming technologies for real-time delivery across various devices and network
conditions.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

24. Part 1: Video Coding
—
Fundamentals of Video Coding
—

25. MHV’24
Fundamentals of Video Coding
What is Video Compression?
●Definition of video compression and encoding
○Video compression is the process of reducing the size of video files by removing redundant or unnecessary
data.
○Encoding refers to converting video data into a specific format that can be easily stored, transmitted, and
decoded for playback.
●Lossless vs. lossy compression
○Lossless compression: Retains all original data, resulting in no loss of quality (e.g., PNG, FLAC). However, the
compression rate is limited.
○Lossy compression: Reduces file size by discarding some data, leading to a reduction in quality. This method
achieves much higher compression ratios and is used in most video codecs (e.g., H.264, H.266).
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

26. MHV’24
Fundamentals of Video Coding
Why Compression is Essential
●File sizes without compression
○Raw, uncompressed video files can be enormous, especially in high resolutions (e.g., 1 minute of
uncompressed 4K video can exceed 100 GB).
●Storage and bandwidth savings
○Compression dramatically reduces file sizes, making it feasible to store and transmit video over networks.
○Streaming services like Netflix or YouTube would be impractical without video compression, as bandwidth
and storage costs would be excessive.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

27. MHV’24
Fundamentals of Video Coding
Components of a Video Frame
●Luma and Chroma Components (YUV Format)
○Luma (Y) represents brightness in the image.
○Chroma (U, V) represents the color information (hue and
saturation).
○Video is typically stored and transmitted in the YUV format
because human eyes are more sensitive to brightness than
color, enabling further compression.
●Color Subsampling (4:4:4, 4:2:2, 4:2:0)
○4:4:4: No subsampling, all color and brightness information
retained.
○4:2:2: Horizontal chroma resolution is halved, saving bandwidth.
○4:2:0: Both horizontal and vertical chroma resolution is halved,
offering substantial savings with minimal perceived quality loss.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: https://encyclopedia2.thefreedictionary.com/chroma+subsampling

28. MHV’24
Fundamentals of Video Coding
Temporal and Spatial Redundancy
●Spatial redundancy (within frames)
○Refers to repeating patterns or areas of similar color/texture within a single frame.
○Compression techniques like Discrete Cosine Transform (DCT) aim to reduce this redundancy by encoding
repeated information more efficiently.
●Temporal redundancy (across frames)
○Many consecutive frames in video contain similar or identical elements (e.g., background, stationary objects).
○Compression reduces redundant information between frames by referencing previous or future frames instead
of encoding the same data repeatedly.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: https://erg.abdn.ac.uk/future-net/digital-video/mpeg2.html

29. MHV’24
Fundamentals of Video Coding
Intra-frame (Spatial) Compression
●How spatial compression works within a single frame
○Intra-frame compression removes redundant information from a single frame, without reference to other
frames.
●Key techniques: DCT (Discrete Cosine Transform), Quantization
○DCT: Converts image data into frequency components, making it easier to compress by representing flat
regions with fewer bits.
○Quantization: Rounds off less significant details, reducing precision to save bits, but slightly reducing quality.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

30. MHV’24
Fundamentals of Video Coding
Inter-frame (Temporal) Compression
●Compressing across frames: P-frames and B-frames
○I-frames (Intra-coded frames): Full frames with
no reference to others, used as anchor points.
○P-frames (Predictive frames): Encoded based on
differences from previous I-frames or P-frames,
reducing file size by eliminating redundancies.
○B-frames (Bidirectionally-predictive frames):
Encoded using both previous and subsequent
frames for even more compression.
●Motion estimation and compensation
○Identifies how objects move between frames and
encodes the movement instead of re-encoding
entire objects, significantly reducing the amount
of data.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

31. MHV’24
Fundamentals of Video Coding
Inter-frame (Temporal) Compression
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

32. MHV’24
Fundamentals of Video Coding
Intra vs. Inter Compression
●Differences in use cases
○Intra compression: Used when every frame must be self-contained, such as in video editing or random access
scenarios (e.g., keyframes in video streaming).
○Inter compression: Used in continuous playback scenarios, such as streaming and broadcasting, where frames
can reference others to save data.
●Encoding trade-offs
○Intra: Higher quality, but larger file size and bandwidth usage.
○Inter: Smaller file sizes but requires more computational power for decoding and may introduce more visual
artifacts if not carefully tuned.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

33. MHV’24
Fundamentals of Video Coding
Block-based Encoding
●Macroblocks and CTUs (Coding Tree Units)
○Macroblocks (in H.264): Fixed-size blocks (16x16 pixels) used to represent video data.
○CTUs (in HEVC): Larger, more flexible units that can vary in size from 16x16 to 64x64 pixels, allowing for
more efficient encoding of complex scenes.
●Partitioning frames into blocks for more efficient compression
○Frames are divided into blocks, and each block is encoded individually, allowing the codec to optimize
encoding based on local scene complexity.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: https://www.elecard.com/page/video_encoding_in_simple_terms

34. MHV’24
Fundamentals of Video Coding
Quantization in Video Coding
●Reducing precision to achieve compression
○Quantization reduces the precision of less important
data (e.g., subtle color changes) to save space.
●Trade-off between quality and compression ratio
○Lower quantization values maintain quality but result in
larger file sizes.
○Higher quantization values create smaller files but may
introduce artifacts such as blockiness and color
banding.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

35. MHV’24
Fundamentals of Video Coding
Entropy Coding (CABAC and CAVLC)
●Techniques to further compress data after quantization
○Entropy coding is applied after quantization to
remove statistical redundancy, further reducing the
size of the data stream.
●Overview of entropy coding algorithms
○CABAC (Context-Adaptive Binary Arithmetic
Coding): More efficient but computationally
complex, used in H.264 and HEVC for high
compression gains.
○CAVLC (Context-Adaptive Variable-Length Coding):
Simpler, faster, but less efficient, typically used in
H.264 for real-time applications.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: CABAC encoding in AVC.
Source: D. Marpe, H. Schwarz, T. Wiegand, Context-based adaptive binary arithmetic
coding in the H.264/AVC video compression standard. Circuits and Systems for
Video Technology, IEEE Transactions on. 13. 620 - 636.
10.1109/TCSVT.2003.815173.

36. MHV’24
Fundamentals of Video Coding
Video Codecs Overview
●MPEG, H.264/AVC, HEVC, AV1, VVC comparison
○MPEG-2: Older codec, used in DVDs and digital TV, offering lower
compression efficiency.
○H.264/AVC: Widely adopted, significant improvement in compression, used
in streaming and Blu-ray.
○HEVC/H.265: 50% better compression than H.264, used for 4K and higher
resolutions.
○AV1: Designed for web and streaming, more efficient than HEVC.
○VVC: Next-generation codec, designed for 8K and immersive formats,
offers up to 50% better compression than HEVC.
●Strengths and weaknesses of each codec
○Trade-offs in compression efficiency, computational complexity, and
licensing costs.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

37. MHV’24
Fundamentals of Video Coding
H.264/AVC Compression Structure
●Overview of how the H.264 codec works
○Uses block-based encoding, inter- and intra-frame compression,
and CABAC/CAVLC entropy coding to achieve high efficiency.
●Why it’s widely used
○Versatility: Supports a wide range of devices, resolutions, and
bitrates.
○Efficiency: Offers a good balance between compression, quality,
and complexity.
○Ubiquity: Supported by nearly all modern devices and video
platforms.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: AVC encoding

38. MHV’24
Fundamentals of Video Coding
HEVC/H.265 Improvements
●Differences between H.264 and H.265
○HEVC uses larger CTUs, more flexible block structures,
improved motion estimation, and more advanced prediction
techniques compared to H.264.
●Efficiency improvements in coding tools
○Improved Motion Estimation: More accurate prediction
reduces redundant data.
○Higher Compression Ratios: Saves bandwidth and storage,
especially for 4K and HDR content.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: HEVC encoding

39. MHV’24
Fundamentals of Video Coding
Introduction to VVC (Versatile Video Coding)
●Next-generation compression: VVC
○Successor to HEVC, providing up to 50% better compression
efficiency.
●Tools and improvements for future video streaming
○Increased block flexibility: Supports larger and more complex
block structures
■Recursive quad-tree (QT) partitioning
■Nested multi-type tree (MTT) using binary splits and
ternary splits.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

40. MHV’24
Fundamentals of Video Coding
Introduction to VVC (Versatile Video Coding)
●A few improved Intra-picture prediction tools
○Refined angular and planar prediction
○Cross-component linear model (CCLM)
○Matrix-based Intra-prediction (MIP)
●A few improved Inter-picture prediction tools
○New merge mode candidates
○Improved motion vector difference coding
○Decoder side motion vector derivation
Figure: MIP. Proposed as a neural-network-based prediction; simplified
in VVC to linear matrix multiplication.
Figure: Decoder side motion vector derivation. Source: JVET-S2002
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

41. MHV’24
Fundamentals of Video Coding
VVC open-source tools
VVC VTM reference software:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

42. MHV’24
Fundamentals of Video Coding
Comparison of VVC with HEVC encoding
Figure: PSNRYUV BD-rate gain and relative encoder runtime for VVenC in
comparison to HM-17.0 and VTM (JVET HD and UHD test sequences, MCTF
enabled for HM-17.0 and VTM-19.2). Results are given for the 5 preset options:
faster, fast, medium, slow and slower. VVenC is running multi-threaded using 6
threads for version <= 0.2 and 8 threads for version >=1.0. Lower PSNR YUV
BD-rate values mean a better compression for the same objective quality in terms of
PSNRYUV.
Figure: PSNRYUV BD-rate gain and relative encoder runtime in comparison to
HM-17.0 for VVenC and x265 running with 8 threads. Lower PSNR YUV BD-rate
values mean a better compression for the same objective quality in terms of
PSNRYUV.
Source: https://github.com/fraunhoferhhi/vvenc/wiki/Encoder-Performance
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

43. MHV’24
Fundamentals of Video Coding
Comparison of VVC with HEVC encoding
Figure: MS-SSIM YUV BD-rate gain and encoder runtime in comparison to HM-16.24 for VTM
and VVenC with perceptually optimized quantization parameter adaptation enabled for HD4K
sequences (MCTF enabled for HM-16.24 and VTM-19.2). VVenC results are given for the 5
preset options: faster, fast, medium, slow and slower. VVenC is running multi-threaded using 6
threads for version <= 0.2 and 8 threads for version >=1.0. Lower MS-SSIM YUV BD-rate
values mean a better compression for the same quality in terms of MS-SSIMYUV.
Figure: MS-SSIMYUV BD-rate gain and encoder runtime in comparison to HM-16.24 for
VVenC with QPA enabled and x265 with --tune=ssim. Both VVenC and x265 are
running with 8 threads. Lower YUV MS-SSIM YUV BD-rate values mean a better
compression for the same quality in terms of MS-SSIMYUV.
Source: https://github.com/fraunhoferhhi/vvenc/wiki/Encoder-Performance
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

44. MHV’24
Fundamentals of Video Coding
Rate-Distortion Optimization (RDO)
●Balancing compression and video quality
○RDO is a technique used to find the best balance between minimizing bitrate
(compression) and maximizing visual quality.
●Why rate-distortion optimization is essential
○Ensures that compression algorithms maintain acceptable video quality while
reducing file size as much as possible.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

45. MHV’24
Fundamentals of Video Coding
Encoding Profiles and Levels
●What are profiles and levels in video codecs?
○Profiles: Define a subset of features that the codec supports (e.g., Baseline, Main, High profiles in H.264).
○Levels: Define limitations such as resolution, bitrate, and framerate (e.g., Level 4.1 in H.264 supports 1080p
video at up to 60 fps).
●Examples from H.264 and H.265
○H.264’s Baseline Profile is commonly used in video conferencing, while High Profile is used for high-definition
content.
○HEVC includes Main and Main10 profiles for SDR and HDR content.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

46. MHV’24
Fundamentals of Video Coding
Codec Complexity vs. Quality Trade-off
●How more complex encoding methods improve quality but increase encoding time
○Advanced techniques like CABAC and larger block sizes offer higher compression efficiency but require more
processing power and time to encode.
●Trade-offs
○Real-time applications may require simpler compression techniques to reduce latency, while offline encoding
can use more complex methods for better quality and smaller file sizes.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: M. Uhrina, L. Sevcik, J. Bienik, and L. Smatanova, “Performance Comparison of VVC, AV1, HEVC, and AVC for High Resolutions,” Electronics, vol. 13, no. 5, Art. no. 5, Jan. 2024, doi:
10.3390/electronics13050953.

47. MHV’24
Fundamentals of Video Coding
Overview of Video Containers
●MP4, MKV, MOV, etc.
○MP4: The most widely used container for streaming and storage, supports
H.264, H.265, and audio tracks.
○Audio Video Interleave (AVI): Introduced by Microsoft; older container format
popular for its compatibility with legacy systems.
○Matroska Video (MKV): Open-source, supports advanced features like
chapter navigation and metadata.
○MOV: Apple’s proprietary format, often used in video editing and QuickTime.
●How containers package video, audio, and metadata
○Video containers bundle video, audio, and metadata (e.g., subtitles, chapters)
into a single file for easy playback and transport.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

48. MHV’24
Fundamentals of Video Coding
Real-Time Encoding vs. Offline Encoding
●Differences in real-time applications
○Real-time encoding: Used in video conferencing, live streaming, and gaming. Prioritizes low latency and fast
encoding over high compression efficiency.
○Offline encoding (VOD): Used for on-demand streaming and post-production. Can take more time to optimize
for quality and compression efficiency.
●Use cases
○Real-time encoding for platforms like Zoom or Twitch, where speed is critical.
○Offline encoding for services like Netflix, where video is pre-encoded for optimal quality and delivery efficiency.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

49. Part 1: Video Coding
—
Video Complexity Analysis
—

50. MHV’24
Video Complexity Analysis
What is Video Complexity?
●Definition of video complexity
○Video complexity refers to the level of difficulty in encoding video
data due to various content factors. It directly affects the amount
of data required to represent the video accurately after
compression.
●Factors that influence complexity
○Resolution: Higher resolutions (e.g., 4K, 8K) contain more pixels
and therefore require more data to compress effectively.
○Motion: Fast-moving objects and frequent camera movements
increase complexity because more changes occur between
frames.
○Texture: Videos with intricate details (e.g., leaves, water, fabric
patterns) are harder to compress since more data is needed to
preserve these fine textures.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Kennedy, Martin & Ksentini, Adlen & Hadjadj-Aoul, Yassine & Muntean, Gabriel-Miro. (2012). Adaptive Energy Optimization in Multimedia-Centric Wireless Devices: A Survey.
IEEE Communications Surveys &amp Tutorials. PP. 1-19. 10.1109/SURV.2012.072412.00115.

51. MHV’24
Video Complexity Analysis
Measuring Video Complexity
●How to measure complexity
○Motion vectors: Measure the amount of movement between frames. More complex motion leads to larger
and more frequent motion vectors [1].
○Bitrate: Videos with high complexity require a higher bitrate to maintain quality, as more data needs to be
preserved during compression.
○Frame rate: Higher frame rates increase complexity because more frames per second need to be
processed, each with potentially new information.
[1] L. Eichermuller, G, Chaudhari, I. Katsavounidis, Z. Lei, H. Tmar, C. Herglotz, A. Kaup, ‘SVT-AV1 encoding bitrate estimation using motion search information’, https://arxiv.org/pdf/2407.05900.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

52. MHV’24
Video Complexity Analysis
Measuring Video Complexity
●SITI
○A state-of-the-art open-source tool that assesses spatial and temporal complexity in video content.
Each frame F (p) is subjected to a Sobel filter [Sobel(F(p))], followed by the computation of the standard deviation (std)
for each Sobel-filtered frame:
TI is the maximum temporal variance observable between consecutive frames in a video sequence. First, pixel
difference between successive frames, denoted as D(p), is calculated. The standard deviation [std[D(p)]] is computed
for each distinct difference D(p). TI is then calculated as the maximum value from the array of standard deviations
[std[D(p)]] across the entirety of the video sequence:
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Github: https://github.com/slhck/siti

53. MHV’24
Video Complexity Analysis
Spatial Complexity in Video
●How texture and detail impact compression and processing
○High texture/detail: Videos with lots of fine details (e.g., foliage, water ripples) require more bits to encode
because there's little redundancy to exploit. Compression must work harder to preserve this detail.
○Low texture/detail: Scenes with flat or uniform regions (e.g., blue sky, solid walls) are easier to compress
because these areas can be represented with fewer bits.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

54. MHV’24
Video Complexity Analysis
Temporal Complexity in Video
●Impact of fast motion, scene changes, and transitions
○Fast motion: Videos with high-speed action (e.g., sports, action movies) generate more changes between
frames, making it harder to compress without increasing the bitrate.
○Scene changes: Abrupt changes from one scene to another break the temporal redundancy that compression
algorithms rely on, forcing encoders to refresh entire frames (I-frames).
○Transitions: Gradual transitions, like crossfades or wipes, increase complexity as multiple scenes blend
together and need careful encoding to avoid artifacts.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

55. MHV’24
Video Complexity Analysis
VCA
Brightness:
Spatial texture complexity:
Temporal texture complexity:
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Vignesh V Menon, Christian Feldmann, Klaus Schoeffmann, Mohammad Ghanbari, and Christian Timmerer. 2023. Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming.
In Proceedings of the First International ACM Green Multimedia Systems Workshop (GMSys 2023). Association for Computing Machinery, New York, NY, USA, 259–264.
https://doi.org/10.1145/3593908.3593942

56. MHV’24
Video Complexity Analysis
Effect of Complexity on Compression
●How complexity affects bitrate and compression efficiency
○Higher bitrate for high complexity: Videos with more motion, texture, or rapid changes require more bits to
preserve quality, increasing the file size.
○Reduced compression efficiency: When complexity is high, standard compression techniques (like inter-frame
compression) may not be as effective, requiring higher bitrates or lower quality.
Figure: PCC between the spatial complexity features (SI and EY) and bitrate in All Intra
configuration with medium preset of x264 and x265 encoders for the VCD dataset.
Figure: PCC between the spatial complexity features (SI and EY) and temporal features (TI and h) with
bitrate in the Low Delay P picture (LDP) configuration with various presets of x265 encoder for the VCD
dataset.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Vignesh V Menon, Christian Feldmann, Hadi Amirpour, Mohammad Ghanbari, and Christian Timmerer. 2022. VCA: video complexity analyzer. In Proceedings of the 13th ACM
Multimedia Systems Conference (MMSys '22). Association for Computing Machinery, New York, NY, USA, 259–264. https://doi.org/10.1145/3524273.3532896

57. MHV’24
Video Complexity Analysis
Scene Change Detection
●Algorithms for detecting scene changes
○Scene change detection algorithms analyze frame
differences and identify when a significant shift in
content occurs (e.g., from one camera shot to
another).
○Techniques include histogram analysis, pixel
comparison, and motion vector analysis.
●How to optimize encoding for different scenes
○Use I-frames at scene changes to refresh content
and ensure smooth transitions.
○Adjust bitrate allocation dynamically based on
scene complexity, providing more bits to complex
scenes and fewer to simpler ones.
Figure: EY per frame computed for the initial 1500 frames of the Tears of Steel sequence. Black
circles denote the regions of scene transitions, determined manually.
Figure: snow_mnt frames 255 to 269 (A, B, C) and FoodMarket4 frames 498 to 556 (D, E, F).
V. V. Menon, H. Amirpour, M. Ghanbari and C. Timmerer, “Efficient Content-Adaptive Feature-Based Shot Detection for HTTP Adaptive Streaming,” in 2021 IEEE International Conference on Image
Processing (ICIP), 2021, pp. 2174-2178.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

58. MHV’24
Video Complexity Analysis
Impact on Encoding Time and Resources
●Higher complexity increases encoding time
○Encoding complex videos requires more
computational resources and time due to the
increased need for precise motion estimation,
handling fine details, and managing rapid scene
transitions.
○Complex scenes may require multiple encoding
passes to fine-tune parameters, significantly
increasing the overall processing time.
●GPU/CPU usage during complex encodes
○High complexity stresses both CPU and GPU
resources. GPU-based encoders are often used
for faster real-time processing, but they may still
struggle with very complex video content.
○Complex encoding tasks often benefit from
multi-core CPUs and specialized hardware
accelerators.
Figure: PCC between the spatial complexity features (SI and EY) and temporal features (TI and
h) with encoding time in the Low Delay P picture (LDP) configuration with various presets of x265
encoder for the VCD dataset.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

59. MHV’24
Video Complexity Analysis
Example of Video Complexity Analysis
●Case study: Sports vs. News vs. Films
○Sports: High motion and fast camera changes make sports content highly complex. Encoding needs to handle
fast-moving players, crowd noise, and quick scene shifts.
○News: Typically lower complexity with static shots, minimal motion, and simple backgrounds, allowing for
easier compression with fewer bits.
○Films: Complexity varies depending on genre. Action films have rapid motion and special effects, while dramas
may feature static or slowly changing scenes, impacting bitrate and compression differently.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

60. MHV’24
Video Complexity Analysis
Using Complexity Analysis for Better Encoding
●Optimizing encoding parameters based on video
complexity
○Bitrate adaptation: Adjust the bitrate to allocate
more bits to complex scenes (e.g., fast motion or
high texture) and fewer to simpler scenes (e.g.,
static shots).
○Scene-based encoding: Use per-scene encoding
to optimize compression settings based on the
specific complexity of each scene.
○Adaptive motion estimation: Dynamically adjust
the precision of motion estimation depending on
the complexity of the scene, saving processing
power for simpler scenes while preserving quality
in more complex ones.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

61. Part 2: Video Coding
—
Per-Title and Per-Scene Encoding for Adaptive Streaming
—

62. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Introduction to Adaptive Streaming
●What is Adaptive Streaming?
○Adaptive streaming is a technique that dynamically adjusts the quality of video playback based on real-time
network conditions.
○It divides video into small chunks (usually a few seconds long) that can be encoded at different bitrates and
resolutions, allowing seamless transitions between quality levels.
●Why it’s crucial for modern video delivery
○User experience: Ensures smooth playback, avoiding buffering even when network speeds fluctuate.
○Device adaptability: Enables content to be played across various devices, from smartphones to 4K TVs, by
delivering appropriate resolution and bitrate for each.
○Network efficiency: Optimizes bandwidth usage, reducing strain on both user networks and content delivery
networks (CDNs).
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

63. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
HTTP Adaptive Streaming (HLS, DASH)
●Overview of Adaptive Streaming Protocols
○HLS (HTTP Live Streaming): Developed by Apple, it is widely used in
streaming services and supported on most platforms, particularly iOS.
○DASH (Dynamic Adaptive Streaming over HTTP): An open standard with
widespread support. It works similarly to HLS but is codec-agnostic, making it
more flexible.
●How these Protocols work
○Both HLS and DASH use manifest files (e.g., M3U8 for HLS) that list available
video chunks at different bitrates and resolutions.
○The video player selects the appropriate bitrate based on real-time network
conditions and switches between chunks seamlessly.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

64. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Benefits of Adaptive Streaming
●Delivering optimal quality across devices and networks
○Scalability: Content is delivered at varying resolutions
(e.g., 360p, 720p, 4K) to match device capabilities.
○Network adaptability: Adjusts video quality in real time
to match the available bandwidth, providing smooth
playback even on slower connections.
○Improved user retention: Less buffering and faster
startup times improve the viewer experience, reducing
the chances of users abandoning streams.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: https://bitmovin.com/blog/adaptive-streaming/

65. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Challenges in Adaptive Streaming
●Bandwidth constraints
○Viewers with slower internet connections may still experience buffering or low-quality streams if the network
can't support even the lowest bitrate.
●Latency
○Switching between quality levels can introduce latency if not handled properly, affecting the overall viewing
experience.
●Device capability differences
○Streaming platforms must accommodate a wide range of devices with different screen sizes, processing
power, and codec support, making it challenging to ensure uniform quality.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

66. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Per-Title Encoding Overview
●What is Per-title encoding?
○A method where each video is encoded
uniquely based on its specific complexity.
Rather than using a one-size-fits-all bitrate
ladder, the encoding settings are tailored to
each title.
●How it differs from traditional encoding
○Traditional encoding uses a fixed bitrate
ladder for all content, regardless of
complexity.
○Per-title encoding analyzes the content (e.g.,
a fast-action movie vs. a slow drama) and
selects the optimal encoding parameters.
Figure: Conceptual plot to depict the bitrate-quality relationship for any video source encoded at different
resolutions. Source: https://netflixtechblog.com/per-title-encode-optimization-7e99442b62a2
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: https://streaminglearningcenter.com/wp-content/uploads/2021/03/Per_Title_Encoding_update.pdf

67. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Why Per-Title Encoding is Important
●Optimize each video file based on content complexity
○Lower complexity content: Simple scenes (e.g.,
cartoons or static news) can achieve good quality
at lower bitrates.
○Higher complexity content: Action movies or sports
require higher bitrates to maintain quality due to
fast motion and scene changes.
○Per-title encoding ensures each video is encoded
as efficiently as possible, balancing quality and file
size.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: Some titles reach very high PSNR (45 dB or more) at bitrates of 2500 kbps or less.
On the other extreme, some titles require bitrates of 8000 kbps or more to achieve an
acceptable PSNR of 38 dB.
Source: https://netflixtechblog.com/per-title-encode-optimization-7e99442b62a2

68. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Per-Title Encoding Process
●Step-by-step breakdown of Per-title encoding workflow
○Content analysis: The encoder analyzes the complexity of the video (e.g., motion, detail, scene transitions).
○Bitrate ladder selection: A custom bitrate ladder is created based on the content complexity, determining the
appropriate resolution and bitrate combinations.
○Encoding: The video is encoded at multiple bitrates and resolutions based on the custom ladder.
○Validation: The encoded streams are validated to ensure they meet quality thresholds (e.g., using VMAF or
PSNR).
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

69. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Adaptive Bitrate Ladder Optimization
●Building a bitrate ladder specific to content characteristics
○Low complexity videos: Require fewer bitrate tiers and can have lower
bitrates without noticeable quality loss.
○High complexity videos: Require more bitrate tiers, with higher bitrates
allocated to preserve quality during fast motion or detailed scenes.
●Dynamic adjustment of the ladder
○The ladder is adjusted not only based on the content but also based on
the expected viewing devices (e.g., 4K TVs vs. mobile phones).
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: An example bitrate ladder.

70. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Examples of Per-Title Encoding
●Case studies
○Fast-action movies: Higher bitrates are required to maintain quality due to the frequent changes in motion and
detail.
○Animated films: Often less complex than live-action films, they can be encoded at lower bitrates while
maintaining excellent visual quality.
○Dialogue-heavy content: Content with minimal motion or static backgrounds can use significantly lower bitrates
without affecting quality.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

71. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
How Per-Title Encoding Reduces Bandwidth
●Optimizing file sizes without compromising on quality
○By tailoring bitrate to content, per-title encoding reduces
unnecessary data usage. Low-complexity videos can be
encoded at lower bitrates, saving bandwidth while
maintaining quality.
○Reduces overall storage and bandwidth costs for
streaming platforms without compromising the viewing
experience.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
V. V. Menon, P. T. Rajendran, C. Feldmann, K. Schoeffmann, M. Ghanbari and C. Timmerer, "JND-Aware Two-Pass Per-Title Encoding Scheme for Adaptive Live Streaming," in IEEE
Transactions on Circuits and Systems for Video Technology, vol. 34, no. 2, pp. 1281-1294, Feb. 2024, doi: 10.1109/TCSVT.2023.3290725.

72. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Per-Scene Encoding Overview
●What is per-scene encoding?
○Per-scene encoding adjusts encoding settings
dynamically within a single video based on the
complexity of individual scenes.
○It fine-tunes bitrate allocation for each scene,
optimizing quality where necessary and reducing
bitrate in simpler scenes.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: https://bitmovin.com/blog/per-scene-adaptation-going-beyond-bitrate/

73. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Scene Complexity and Adaptive Encoding
●How scene complexity drives encoding decisions
○Complex Scenes (e.g., fast motion or high detail): Require higher bitrates and more advanced encoding
techniques to preserve quality.
○Simple Scenes (e.g., static shots or dialogue): Can be encoded at lower bitrates without noticeable quality loss,
optimizing file size.
●Dynamic adjustments
○During encoding, the video is analyzed frame-by-frame or scene-by-scene, adjusting the bitrate and encoding
parameters accordingly.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

74. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Per-Scene Encoding Use Cases
●Example: Encoding action scenes vs.
Dialogue-heavy scenes
○Action scenes: High-motion scenes with
rapid camera changes (e.g., car chases)
require more data to avoid blockiness or
artifacts.
○Dialogue-heavy Scenes: Static scenes with
minimal movement (e.g., a conversation)
can use significantly less data without
affecting the viewing experience.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: Combining shot encodes to produce optimal encodes; example Trellis paths show
fixed QP encoding, minimizing bitrate for a given average quality or maximizing quality for a
given average bitrate. Selected shot encodes have approximately equal slope in (R,D)
space.
Source:https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimiza
tion-framework-e19f1e3a277f

75. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Implementing Per-Scene Encoding
●How to integrate per-scene encoding in an encoding
pipeline
○Pre-encoding analysis: Use machine learning or
algorithms to detect scene complexity.
○Encoding settings: Automatically adjust the bitrate,
resolution, and compression algorithms for each
scene.
○Quality control: Use objective quality metrics
(VMAF, SSIM) to validate the final output and
ensure consistent quality.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: Encoding a shot using a set of parameters, such as resolution and QP, and obtaining a
single (R,D) point for it.
Source:https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization
-framework-e19f1e3a277f

76. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Per-Scene and Per-Title Combined
●Using both techniques together for maximum efficiency
○Per-title: Optimizes the encoding for the entire video based on its overall complexity.
○Per-scene: Further refines the bitrate allocation within each title, ensuring that each scene gets the right
amount of data for its complexity.
○Combined, these methods reduce file sizes, improve quality, and enhance streaming efficiency.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

77. Part 2: Video Coding
—
Latency-aware Video Coding for Adaptive Streaming
—

78. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Machine Learning in Per-Title/Per-Scene Encoding
●How ML can help identify optimal encoding parameters
○Content analysis: Machine learning models can
analyze video content to predict optimal bitrate
ladders and scene complexity.
○Real-Time adjustments: ML models can
dynamically adjust encoding parameters based on
the content and network conditions, improving
both efficiency and quality.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: AI-assisted per-title encoding architecture.
Source: https://websites.fraunhofer.de/video-dev/deep-encode-part-i/

79. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Optimizing Encoding Efficiency with AI
●Future trends in AI-driven video encoding
○AI can automate complex encoding decisions that previously required manual adjustment, improving both the
speed and efficiency of encoding workflows.
○AI can also predict user behavior (e.g., fast-forwarding) and pre-buffer certain scenes at higher or lower
qualities to optimize bandwidth usage.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
V. V. Menon, H. Amirpour, C. Feldmann, A. Ilangovan, M. Smole, M. Ghanbari, C. Timmerer, Live PSTR: Live Per-title Encoding for Ultra HD Adaptive Streaming, NAB 2022.

80. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Quality Metrics for Adaptive Streaming
●PSNR, SSIM, and VMAF metrics for quality
assessment
○PSNR (Peak Signal-to-Noise Ratio): A basic
metric for measuring the difference between the
original and encoded video. Higher values
indicate better quality.
○SSIM (Structural Similarity Index): Measures
perceived quality by comparing structural
information (e.g., luminance, contrast).
○VMAF (Video Multi-Method Assessment Fusion):
A more advanced metric used by Netflix, which
combines multiple models to better predict
human-perceived video quality.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: VMAF workflow

81. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Quality Metrics for Adaptive Streaming
●XPSNR
○VMAF scores does not correlate well with the
subjective MOS of VVC-coded UHD bitstreams.
○Designed as a psycho-visually inspired,
simplified spatio-temporal sensitivity model.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
C. R. Helmrich, M. Siekmann, S. Becker, S. Bosse, D. Marpe, and T. Wiegand, "XPSNR: A Low-Complexity Extension of the Perceptually Weighted Peak Signal-to-Noise Ratio for
High-Resolution Video Qua- lity Assessment," in Proc. IEEE Int. Conf. Acoustics, Speech, Sig. Process. (ICASSP), virt./online, May 2020. www.ecodis.de/xpsnr.htm

C. R. Helmrich, S. Bosse, H. Schwarz, D. Marpe, and T. Wiegand, "A Study of the Extended Perceptually Weighted Peak Signal-to-Noise Ratio (XPSNR) for Video Compression with
Different Resolutions and Bit Depths," ITU Journal: ICT Discoveries, vol. 3, no. 1, pp. 65 - 72, May 2020. http://handle.itu.int/11.1002/pub/8153d78b-en
Table: – Evaluation results for Spearman rank order correlation
with MOS values.

82. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Challenges in Implementing Per-Title/Per-Scene Encoding
●Hardware, software, and workflow challenges
○Hardware: Per-title and per-scene encoding requires significant processing power, especially for
high-resolution content.
○Software: Integration into existing workflows can be complex, requiring new tools and quality control
processes.
○Workflow: Encoding pipelines need to be flexible to handle dynamic content analysis and adaptive encoding
decisions.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

83. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
How Per-Title Encoding Impacts Viewers
●Benefits to user experience
○Faster start times: Optimized encoding reduces the amount of data that needs to be streamed, improving
startup times.
○Better quality on all devices: Ensures that viewers experience the best possible quality for their device and
network conditions, whether on mobile or 4K TV.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Source: https://bitmovin.com/blog/using-per-title-bitrate-ladder-optimize-encoding-try-new-benchmark-tool/

84. MHV’24
Latency-aware Video Coding
Latency-awareness
●Decoding time depends on the encoding resolution chosen for the video content. The number of pixels in each
frame significantly impacts the computational workload.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: Rate-XPSNR curves and decoding times of example video sequences from the
employed dataset, Inter-4K, encoded using VVenC v1.12 and decoded using VVdeC.
Figure: Quality-Rate-Time points for VVenC encodes and VVdeC decodes across six
spatial resolutions.

85. MHV’24
Latency-aware Video Coding

Reducing coding time- “greenifies” streaming
●Reducing encoding energy consumption (in data centers)
is critical in streaming applications since it contributes to
environmental sustainability [4].
●The streaming industry can reduce its carbon footprint and
energy consumption by minimizing coding time.
●Our prior experiments suggest a pseudo linear relationship
between coding time and coding energy consumption.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: Average encoding metrics for 7.5 fps, 15 fps, 24 fps, and 30 fps HLS CBR
encoding using the veryslow preset of the x264 [1] AVC [2] encoder. Source: [3]
Figure: Average encoding metrics for HLS CBR encoding at 30 fps using selected
presets of x264 [1] AVC [2] encoder. Source: [3]
[1] VideoLAN, “x264.” [Online]. Available: https://www.videolan.org/developers/x264.html
[2] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard,” in IEEE Transactions on Circuits and Systems for Video
Technology, vol. 13, no. 7, 2003, pp. 560–576.
[3] V. V. Menon, S. Afzal, P. T. Rajendran, K. Schoeffmann, R. Prodan, and C. Timmerer. [n. d.]. Content-Adaptive Variable Framerate Encoding Scheme for Green Live Streaming,
[Online]. Available: http://arxiv.org/abs/2311.08074
[4] A. Stephens, C. Tremlett-Williams, L. Fitzpatrick, L. Acerini, M. Anderson, and N. Crabbendam, “The Carbon Impacts of Video Streaming,” Jun. 2021.

86. MHV’24
Latency-aware Video Coding

Energy Consumption in Mobile Devices
●Higher power consumption in the devices reduces the battery lifetime.
●Scalable impact: When millions of users stream content globally, even a
small reduction in decoding time and energy consumption per user can
translate to significant overall energy savings.
●Lower device heating: Reducing the processing time also decreases the
heat generated, as continuous CPU/GPU activity generates heat, which
requires additional energy to cool the system.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: Smart-phone and its different components. Source: [1]
[1] Kennedy, Martin & Ksentini, Adlen & Hadjadj-Aoul, Yassine & Muntean, Gabriel-Miro. (2012). Adaptive Energy Optimization in Multimedia-Centric Wireless Devices: A Survey. IEEE
Communications Surveys &amp Tutorials. PP. 1-19. 10.1109/SURV.2012.072412.00115.

87. MHV’24
Latency-aware Video Coding

ML-Driven Video Encoding Parameter Selection
●Parameter selection agent: searches the space defined by the dominant features by maximizing a linear video
eco-quality cost function:
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: System architecture [1].
[1] Z. Azimi, R. Farahani, V. V. Menon, C. Timmerer and R. Prodan, "Towards ML-Driven Video Encoding Parameter Selection for Quality and Energy Optimization," 2024 16th International Conference
on Quality of Multimedia Experience (QoMEX), Karlshamn, Sweden, 2024, pp. 80-83, doi: 10.1109/QoMEX61742.2024.10598278.
Figure: RD curves and rate-encoding time graphs of a few
representative sequences [1].

88. MHV’24
Latency-aware Video Coding

QADRA open-source framework
●QADRA determines the encoding resolution and quantization
parameter (QP) for each target bitrate by maximizing XPSNR while
constraining the maximum encoding and/ or decoding time below a
threshold.
●Implements a JND-based representation elimination algorithm to
remove perceptually redundant representations from the bitrate
ladder.
●Open-source Python-based framework published under the GNU
GPLv3 license.
●Github: https://github.com/PhoenixVideo/QADRA
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Figure: QADRA framework [1].
[1] A. Premkumar, P. T. Rajendran, V. V. Menon, A. Wieckowski, B. Bross, and D. Marpe. 2024. Quality-Aware Dynamic Resolution Adaptation Framework for Adaptive Video Streaming. In Proceedings
of the 15th ACM Multimedia Systems Conference (MMSys '24). Association for Computing Machinery, New York, NY, USA, 292–298. https://doi.org/10.1145/3625468.3652172
Figure: RD curves and rate-encoding time graphs of a few
representative sequences [1].

89. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Future Trends in Adaptive Video Encoding
●What’s Next: VVC, and AI-Driven Approaches
○VVC: The next generation after HEVC, designed to handle 8K and immersive formats with even better
compression.
○AI and Machine Learning: AI-driven encoding workflows will continue to evolve, automating and improving
encoding efficiency for large-scale streaming services.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

90. MHV’24
Per-Title and Per-Scene Encoding for Adaptive Streaming
Summary
●Adaptive streaming: Ensures optimal video quality across diverse network conditions and devices.
●Per-title and Per-scene encoding: Offers significant improvements in bandwidth efficiency and visual quality by
tailoring encoding to content complexity.
●Future trends: AI-driven and next-gen codecs like VVC will shape the future of streaming, providing even more
efficient and high-quality delivery.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

91. Part 3: Video Delivery
—
Introduction
—

92. MHV’24
Introduction of Video Delivery

Overview of Topics Covered
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
https://bitmovin.com/dynamic-adaptive-streaming-http-mpeg-dash/

93. MHV’24
Introduction of Video Delivery

HAS clients
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

94. MHV’24
Introduction of Video Delivery


Adaptation Policies
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Bentaleb, etal, “A survey on bitrate adaptation schemes for streaming media over HTTP”, IEEE Communications Surveys & Tutorials, 2018

Reza Farahani , etal, "Towards AI-Assisted Sustainable Adaptive Video Streaming Systems: Tutorial and Survey", 2024, https://arxiv.org/abs/2406.02302
Non-AI-Based
AI-Based

95. MHV’24
Introduction of Video Delivery
Performance Metrics and Models

Network QoS
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●Server Load
●Cache Hit Ratio
●Backhaul Bandwidth Usage
●Fronthaul Bandwidth Usage
●CPU and Memory Usage
●Path Utilization
●Edge Server Utilization
●Delivery Delay
●Jitter
●Delivery Energy Consumption
●Fairness
●…
Application QoS (User QoE)
●Subjective
○MoS
●Objective
○Received Bitrate/Resolution
○Stall Number
○Stall Duration
○Quality Switch
○Fairness

P.1203 ITU https://www.itu.int › rec › dologin_pub

96. Part 3: Video Delivery
—
Fundamentals of Video Computing and Networking
Paradigms
—

97. MHV’24
Fundamentals of Computing and Networking Paradigms


Paradigm Section
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Content
Provider
Advertiser
CDN
Provider
ISP Player User

98. MHV’24
Fundamentals of Computing and Networking Paradigms


Transport Protocols
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
https://github.com/rmarx/h3-protocol-stack

99. MHV’24
Content Delivery Network (CDN)
●Stores frequently accessed content closer to users
●Handles high traffic volumes and adapts to demand changes efficiently.
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Fundamentals of Computing and Networking Paradigms

100. MHV’24
Fundamentals of Computing and Networking Paradigms


Multi-Access Edge Computing (MEC)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

101. MHV’24
Fundamentals of Computing and Networking Paradigms


Serverless Computing
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
An emerging paradigm in cloud computing
○Pay-per-use pricing model (Scale-down-to-zero)
○Provider does resource provisioning and scaling
○Event-driven

102. MHV’24
Fundamentals of Computing and Networking Paradigms


Peer-to-Peer (P2P) Delivery Network
●Platforms like BitTorrent
○distribute video content efficiently by sharing pieces of the video file among users.

●WebRTC:
○real-time video communication by allowing browsers to connect directly
○reduce the CDN-side load as well as stream start time
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

103. MHV’24
Fundamentals of Computing and Networking Paradigms


Hybrid P2P-CDN Delivery Network
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Tracker
●Seeders share content
●Leechers consume it
●Trackers coordinate the exchange

104. MHV’24
Fundamentals of Computing and Networking Paradigms


Software-Defined Networking
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Data Plane
Control Plane
https://opennetworking.org/sdn-definition/
●Control and data plan are decoupled
●Network intelligence is in the central controller

105. MHV’24
Fundamentals of Computing and Networking Paradigms


Software-Defined Networking
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●SO/IEC 23009 DASH part 5 Server And Network-assisted DASH
https://www.iso.org/obp/ui/es/#iso:std:iso-iec:23009:-5:ed-1:v1:en
●Common Media Client Data (CMCD)
standard (CTA-5004) and Common Media
Server Data (CMSD)

106. MHV’24
Fundamentals of Computing and Networking Paradigms


Network Virtualization and Slicing
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
https://stlpartners.com/research/cloud-2-0-network-functions-virtualisation-nfv-vs-software-defined-networking-sdn/
https://www.blueplanet.com/resources/what-is-network-slicing.htm
l

107. MHV’24
Fundamentals of Computing and Networking Paradigms


Virtual Video Network Functions (VNFs)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●Transcoder
○convert video inputs from one format, codec, bitrate, … to another
○higher to lower

●Super-resolution
○upscale and reconst finer details from lower-resolution inputs

●Cache
○caching content like CDNs

https://bitmovin.com/what-is-transcoding/

108. MHV’24
Fundamentals of Computing and Networking Paradigms


Service Function Chaining (SFC)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
VNF iVNF i+1 VNF n
VNF iVNF i+1 VNF n
SFC
Chains
Chain 1
Chain m
…
…
…
Orchestration
Placement

Scheduling
SFC
Definition
VNF
Definition

109. Part 3: Video Delivery
—
Large-scale Testbed Design

—

110. MHV’24
Realistic Testbed Design
CloudLab
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

111. MHV’24
Realistic Testbed Design


Performance Evaluation Setup (CloudLab Based Testbeds)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

112. MHV’24
Realistic Testbed Design


Performance Evaluation Setup (Local Computing Continuum)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Kubernetes-based Federated Computing Continuum Testbed (Edge cluster I)

113. MHV’24
Realistic Testbed Design


Performance Evaluation Setup (Local Computing Continuum)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Kubernetes-based Federated Computing Continuum Testbed (Edge cluster I)

114. MHV’24
Realistic Testbed Design


Kubernetes Orchestration
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

115. MHV’24
Realistic Testbed Design


Monitoring with Prometheus
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

116. MHV’24
Realistic Testbed Design


Monitoring Visualization with Grafana
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

117. Part 3: Video Delivery
—
Latency- and QoE aware Adaptive Video Streaming Delivery

—

118. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ARARAT System
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
✔MEC Servers:
◆Local Edge Server (LES)
◆Neighboring Edge Server (NES)

✔Virtualized Edge Functions:
◆Partial Cache (PC)
◆Video Transcoder (Tran)
Edge Layer
R. Farahani, M. Shojafar, C. Timmerer, F. Tashtarian, M. Ghanbari, H. Hellwagner. ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming. IEEE Transactions on Network and
Service Management (TNSM), 2022.

119. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ARARAT QoE Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
cache_map





















Requests, edge_map, comp_map





















R. Farahani, M. Shojafar, C. Timmerer, F. Tashtarian, M. Ghanbari, H. Hellwagner. ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming. IEEE Transactions on Network and
Service Management (TNSM), 2022.

120. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ARARAT QoE Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
ARARAT Action Tree
R. Farahani, M. Shojafar, C. Timmerer, F. Tashtarian, M. Ghanbari, H. Hellwagner. ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming. IEEE Transactions on Network and
Service Management (TNSM), 2022.

121. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ARARAT Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●The SDN controller run an MILP optimization model to respond:

○Where is the optimal place (i.e., LES, NESs, CSs, or the origin server) in terms of the following items for
fetching each client’s requested content quality level from?
■minimum serving time and minimum network cost (ARARAT)

○What is the optimal approach for responding to the requested quality level (i.e., fetch or transcode)?
Serving Latency Network Cost
ARARAT

122. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ARARAT Results (QoE)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
ASB: Average Segment Bitrate
AQS: Average Number of Quality
Switches
ANS: Average Number of Stalls
ASD: Average Stall Duration
APQ: Average Perceived QoE
calculated by ITU-T P.1203 mode 0 [1]

R. Farahani, M. Shojafar, C. Timmerer, F. Tashtarian, M. Ghanbari, H. Hellwagner. ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming. IEEE Transactions on Network and
Service Management (TNSM), 2022.

123. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ARARAT Results (QoE)
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
CHR: Cache Hit Ratio
ETR: Edge Transcoding Ratio
BTL: Backhaul Traffic Load
ANU: Average Network Utilization
ASL: Average Serving Time
NCV: Network Cost Value
ANC: Average Number of Communicated messages
from/to the SDN controller

R. Farahani, M. Shojafar, C. Timmerer, F. Tashtarian, M. Ghanbari, H. Hellwagner. ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming. IEEE Transactions on Network and
Service Management (TNSM), 2022.

124. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ALIVE System
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
R. Farahani, E. Cetinkaya, C. Timmerer, M. Shojafar, M. Ghanbari, H. Hellwagner. ALIVE: A Latency- and Cost-Aware Hybrid P2P-CDN Framework for Live Video Streaming. IEEE Transactions on Network and
Service Management (TNSM), 2023.

125. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ALIVE System
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
R. Farahani, E. Cetinkaya, C. Timmerer, M. Shojafar, M. Ghanbari, H. Hellwagner. ALIVE: A Latency- and Cost-Aware Hybrid P2P-CDN Framework for Live Video Streaming. Submitted to IEEE Transactions on
Network and Service Management (TNSM), 2023.

126. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ALIVE Energy Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
R. Farahani, E. Cetinkaya, C. Timmerer, M. Shojafar, M. Ghanbari, H. Hellwagner. ALIVE: A Latency- and Cost-Aware Hybrid P2P-CDN Framework for Live Video Streaming. Submitted to IEEE Transactions on
Network and Service Management (TNSM), 2023.

127. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ALIVE Network Utilization Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
ASB: Average Segment Bitrate
AQS: Average Number of Quality Switches
ANS: Average Number of Stalls
ASD: Average Stall Duration
ASL: Average Serving Time
APQ: Average Perceived QoE calculated by ITU-T
P.1203 mode 0 [1]




R. Farahani, E. Cetinkaya, C. Timmerer, M. Shojafar, M. Ghanbari, H. Hellwagner. ALIVE: A Latency- and Cost-Aware Hybrid P2P-CDN
Framework for Live Video Streaming. Submitted to IEEE Transactions on Network and Service Management (TNSM), 2023.

128. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
ALIVE Network Utilization Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
CHR: Cache Hit Ratio
ETR: Edge Transcoding Ratio
PTSR: Peer SR and TR Ratio
BTL: Backhaul Traffic Load
EEC: Edge Energy Consumption for running ETR
NCV: Network Cost Value


R. Farahani, E. Cetinkaya, C. Timmerer, M. Shojafar, M. Ghanbari, H. Hellwagner. ALIVE: A Latency- and Cost-Aware Hybrid P2P-CDN
Framework for Live Video Streaming. Submitted to IEEE Transactions on Network and Service Management (TNSM), 2023.

129. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
SARENA System
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
●OTT video
●Live video streaming
●Immersive multimedia
●Video Gaming
●Video analytics for security,
quality assurance, etc.

Increase in amount of video
generated and transported
Versatile QoE, QoS requirements
Resolution (4K, 8K)

Latency (LL,ULL)

Bitrate

130. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
SARENA System
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon

Virtual Proxy Function

Virtual Cache Function

Virtual Transcoding Function

CDN Cache

Origin Cache
















1
2
3
4
5
Multimedia VNFs

R. Farahani, A. Bentaleb, C. Timmerer, M. Shojafar, R. Prodan, H. Hellwagner. SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications. IEEE International Conference on
Communications (ICC), 2022.

131. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
SARENA QoE Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
ASB: Average Segment Bitrate
AQS: Average Number of Quality Switches
ANS: Average Number of Stalls
ASD: Average Stall Duration
APQ: Average Perceived QoE calculated by
ITU-T P.1203 mode 0 [1]
ASL: Average Serving Latency


R. Farahani, A. Bentaleb, C. Timmerer, M. Shojafar, R. Prodan, H. Hellwagner. SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications. IEEE International Conference on
Communications (ICC), 2022.

132. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
SARENA Network Utilization Results
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
NCV: Network Cost Value
ETR: Edge Transcoding Ratio
BTL: Backhaul Traffic Load
R. Farahani, A. Bentaleb, C. Timmerer, M. Shojafar, R. Prodan, H. Hellwagner. SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications. IEEE International Conference on
Communications (ICC), 2022.

133. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
HEFTLess System
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
HEFTLESS Architecture
ML-Based Video Processing Application
Reza Farahani, Narges Mehran, Sashko Ristov, Radu Prodan, "HEFTLess: A Bi-Objective Serverless Workflow Batch Orchestration on the Computing Continuum”. Accepted in IEEE CLUSTER 2024"

134. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery
HEFTLESS Results

A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
#functions#workflows
Reza Farahani, Narges Mehran, Sashko Ristov, Radu Prodan, "HEFTLess: A Bi-Objective Serverless Workflow Batch Orchestration on the Computing Continuum”. Accepted in IEEE CLUSTER 2024"

135. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery

AI-Assisted Video Streaming Systems
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Reza Farahani , Zoha Azimi, Christian Timmerer, Radu Prodan, "Towards AI-Assisted Sustainable Adaptive Video Streaming Systems: Tutorial and Survey", 2024, https://arxiv.org/abs/2406.02302

136. MHV’24
Latency- and QoE aware Adaptive Video Streaming Delivery

AI-Assisted Video Streaming Systems
A Tutorial on Latency- and Energy-Aware Video Coding and Delivery Streaming Systems
Dr. Reza Farahani, Dr. Vignesh V Menon
Reza Farahani , Zoha Azimi, Christian Timmerer, Radu Prodan, "Towards AI-Assisted Sustainable Adaptive Video Streaming Systems: Tutorial and Survey", 2024, https://arxiv.org/abs/2406.02302

137. Thank you for your attention
—
Reza Farahani ([email protected])
Vignesh V Menon ([email protected])