Enhancing Film Grain Coding in VVC: Improving Encoding Quality and Efficiency

This paper presents an in-depth analysis of film grain handling in open-source implementations of the Versatile Video Coding (VVC) standard. We focus on two key components: the Film Grain Analysis (FGA) module implemented in VVenC and the Film Grain Synthesis (FGS) module implemented in VVdeC. We describe the methodologies used to implement these modules and discuss the generation of Supplementary Enhancement Information (SEI) parameters to signal film grain characteristics in the encoded video sequences. Additionally, we conduct subjective and objective evaluations across Full HD videos to assess the effectiveness of film grain
handling. Our results demonstrate the capability of the FGA and FGS techniques to accurately analyze and synthesize film grain, thereby
improving the visual quality of encoded video content. Overall, our study contributes to advancing the understanding and implementation of film grain handling techniques in VVC open-source implementations, with implications for enhancing the viewing experience in multimedia applications.

1. Enhancing Film Grain Coding in VVC:
Improving Encoding Quality and Efficiency
—
Vignesh V Menon, Postdoctoral Researcher, Fraunhofer HHI
Sep. 15, 2024, International Broadcasting Convention (IBC), 2024

2. Introduction to Video Coding & Film Grain
—

3. MHV’24
Introduction
15.09.2024 © FraunhoferSlide 3
Film grain in video coding
•Filmgrain,aninherentcharacteristicofanalogfilm,contributestotheuniquevisual
aestheticsandcinematicexperienceinmovies[1].
•TheemergenceoftheVersatileVideoCoding(VVC)[2,3]standardbringsnew
opportunitiesandchallengesinefficientlyrepresentingfilmgrain,aimingtopreserveits
artisticvaluewhileensuringcompatibilitywithmodernvideocompressiontechniques.
•Compressionartifactslikeblockinessareintroducedwhengrainyvideocontentis
encodedatlowbitrates.Filmgrainpresentsachallengeinvideocodingduetoits
randomandnon-uniformnature,whichcanamplifycompressionartifactsifnot
adequatelyhandledduringencoding.
[1]InseongHwangetal.“EnhancedFilmGrainNoiseRemovalforHighFidelityVideoCoding”.In:2013InternationalConferenceonInformationScienceandCloudComputingCompanion.2013,pp.668–674.doi:
10.1109/ISCC-C.2013.69.
[2]BenjaminBrossetal.“OverviewoftheVersatileVideoCoding(VVC)StandardanditsApplications”.In:IEEETransactionsonCircuitsandSystemsforVideoTechnology.Vol.31.10.2021,pp.3736–3764.doi:
10.1109/TCSVT.2021.3101953
[3]AdamWieckowskietal.“ACompleteEndtoEndOpen-SourceToolchainfortheVersatileVideoCoding(VVC)Standard”.In:Proceedingsofthe29thACMInternationalConferenceonMultimedia.NewYork,NY,USA:
AssociationforComputingMachinery,2021,3795–3798.isbn:9781450386517.doi:10.1145/3474085.3478320.
Figure:Grainexample.

4. MHV’24
Introduction
15.09.2024 © FraunhoferSlide 4
Versatile Video Coding (VVC)
•DevelopedbyJVET(JointVideoExpertsTeam)
•Aimstoimprovecompressionefficiency,visualquality,and
supportforemergingmultimediaapplications.
•50%bettercompressionefficiencygainsoverHEVC.
[1]B.Brossetal.,"OverviewoftheVersatileVideoCoding(VVC)StandardanditsApplications,"inIEEETransactionsonCircuitsandSystemsforVideoTechnology,vol.31,no.10,pp.3736-3764,Oct.2021,doi:
10.1109/TCSVT.2021.3101953.
Figure:TypicalVVCencoder[1].

5. MHV’24
Introduction
15.09.2024 © FraunhoferSlide 5
VVC open-source toolchain

6. MHV’24
Introduction
15.09.2024 © FraunhoferSlide 6
Importance of Film Grain in Video Coding
Figure:IllustrationofgrainlossinOldTownCrosssequencewhenencodedwithVVenC.

7. Film Grain Handling in VVC
—

8. MHV’24
Film Grain Handling in VVC
15.09.2024 © FraunhoferSlide 8
▪Usesmotion-compensatedtemporalfiltering(MCTF)toextractfilmgrainfromvideocontent.
▪Quantificationoffilmgrainpropertieslikesize,intensity,anddistribution.
▪SEImessagestotransmitparametersforfilmgrain.
Film Grain Analysis (FGA) in VVenC

9. MHV’24
Film Grain Handling in VVC
15.09.2024 © FraunhoferSlide 9
▪ReplicatesfilmgrainduringdecodingusingSEIparameters
▪Allowsforthepreservationorsynthesisoffilmgrainevenin
compressedorfilteredcontent
Film Grain Synthesis (FGS) in VVdeC

10. MHV’24
Film Grain Handling in VVC
15.09.2024 © FraunhoferSlide 10
▪Toconveythegrainparameterstothedecoder,theencoderembedsitas
supplementalenhancementinformation(SEI)inthebitstream[1],asthe
"FilmGrainSEI.“.
▪FilmGrainSEIinheritsthesamesyntaxandsemanticsoftheAVCfilm
grainSEImessage[2].
▪Sinceweimplementafrequencyfilteringmodelforfilmgrainestimation,
film_grain_model_idissetto0.
▪Additiveblendingisusedwhenblending_mode_idissetto0.
▪Sinceourimplementationanalysesfilmgrainforonlythelumachannel,
comp_model_present_flag[0]issetto1.
▪FGCSEImessageisinsertedateachframe,whichisindicatedbysetting
thefilm_grain_characteristics_persistence_flagto0.ItalsomeanstheFGC
SEImessageonlyappliestothecurrentdecodedframe.
Film Grain SEI
[1]“RDD5:2006-SMPTERegisteredDisclosureDoc-FilmGrainTechnology—SpecificationsforH.264|MPEG-4AVCBitstreams,”RDD5:2006,pp.1–18,2006.
[2]VijayakumarGayathriRamakrishna,KaustubhShripadPatankar,andMukundSrinivasan,“Cloud-BasedWorkflowforAVCFilmGrainSynthesis,”ser.MHV’23.NewYork,NY,USA:AssociationforComputingMachinery,
2023,p.66–71.

11. Detailed Workflow
—

12. MHV’24
Film Grain Analysis in VVenC
15.09.2024 © FraunhoferSlide 12
Motion-compensated temporal filter (MCTF):
▪VVenCalreadyemploysadenoisingstage[5],basedonaframework
proposedinitiallyin[6]–motion-compensatedtemporal(pre-)filtering.
▪Thefilterperformsablockwisemotionsearchforeachfilteredframein
neighboringframestoremovethenoise.
▪Usinguptoeightpredictors,aweightedaverageofthecurrentframeblock
anditspredictorsisgeneratedandusedforfurtherencoding.
▪Improvedsearchstrategies,referencenumberreduction,andflexibleblock
sizewereintroduced,improvingthefilterruntimeandoperation[5].
[5]AdamWieckowski,TobiasHinz,ChristianR.Helmrich,BenjaminBross,andDetlevMarpe,“Anoptimizedtemporalfilterimplementationforpracticalapplications,”in2022PictureCodingSymposium(PCS),2022,pp.247–
251.
[6]JackEnhorn,RickardSjöberg,andPerWennersten,“ATemporalPre-FilterForVideoCodingBasedOnBilateralFiltering,”in2020IEEEInternationalConferenceonImageProcessing(ICIP),2020,pp.1161–1165.

13. MHV’24
Film Grain Analysis in VVenC
15.09.2024 © FraunhoferSlide 13
Mask Generation
▪Identifyandisolateflatandlow-complexityregionsoftheframefromthose
withsignificanttextureandcomplexity.
▪Edgedetectioncreatesaninitialbinarymaskwhereedgesaremarked.
Thegradientcalculation,non-maximumsuppression,doublethresholding,
andedgetrackingbyhysteresisareallemployedtogenerateadetailed
edgemap.
▪SuppressingLow-IntensityRegions:Aftergeneratingtheedgemap,
low-intensityregionsthatdonotcontributesignificantlytotheoverall
texturearesuppressed.
▪Morphologicaloperations,specificallydilation,areappliedtothemasks
tofillsmallholesandgaps,ensuringcontiguousregions.

14. MHV’24
Film Grain Analysis in VVenC
15.09.2024 © FraunhoferSlide 14
Grain Estimation
▪Differenceestimation:Differencebetweenoriginalandfilteredframes
▪Blockbasedanalysis
▪Sizedeterminedbasedonthepictureresolution
▪DCTtransformappliedtoeachblock
▪Variancecalculation
▪Meanintensitycalculation
▪Edgedetectionforlumacomponents

15. MHV’24
Film Grain Analysis in VVenC
15.09.2024 © FraunhoferSlide 15
Polynomial Fitting and Quantization
▪Modelsrelationshipbetweenintensityandvarianceintheestimatedfilmgrain
▪Prepareddatapointsareusedtofitann-thorderpolynomialfunction:
▪Setupmatrixequationsbasedondatapoints
▪Solvetheequationstofindthepolynomialcoefficients
▪Rescaletheparameterstomatchthedatarange
▪Extenddatapoints
▪Quantization
▪Llyod-maxquantizationtohaveaspecificrangeofresultingscalingfactors
▪Scalingestimatedparameters
▪Defineintensityintervalsandcorrespondingscalingfactors
▪Mergingsmallintervals
▪Scalingparametersto8bitrange
▪Settingthefinalparametersinthecomponentmodel

16. MHV’24
Proposed toolchain
15.09.2024 © FraunhoferSlide 16
•TheworkinSMPTE-RDD5[1]providesanin-depthlookatFGS,
whichispartofthedecodersideofthevideodistributionchain[2].
•ThisprocessisdefinedfortheH.264standard[3],iscompatible
withVVCwithoutmodificationssincebothsupportthesame
metadata[4].
•OurmethodoffersamorepreciseFGSspecificationthanVSEI[5,
6].
•Basedonfilteringinthefrequency/transformdomain,whichinvolves
filteringrandomnoisetosimulatethefilmgrainpattern.Inthis
model,filmgrainpatternsaregeneratedinthefrequencydomain
usingapairofcut-offfrequenciesthatdefinealow-passfilter.
Thesepatternsarethenscaledtotheproperintensitybefore
blendingintotheimage.
Film Grain Synthesis in VVdeC
[1] “RDD 5:2006 -SMPTE Registered Disclosure Doc -Film Grain Technology —Specifications for H.264 —MPEG-4 AVC Bitstreams,” RDD 5:2006, pp. 1–18, 2006.
[2]A.Wieckowski,G.Hege,C.Bartnik,C.Lehmann,C.Stoffers,B.Bross,andD.Marpe,“TowardsALiveSoftwareDecoderImplementationForTheUpcomingVersatileVideoCoding(VVC)Codec,”in2020IEEE
InternationalConferenceonImageProcessing(ICIP),Oct.2020,pp.3124–3128,ISSN:2381-8549.
[3]T.Wiegand,G.Sullivan,G.Bjontegaard,andA.Luthra,“OverviewoftheH.264/AVCvideocodingstandard,”IEEETransactionsonCircuitsandSystemsforVideoTechnology,vol.13,no.7,pp.560–576,Jul.2003.
[4]B.T.Oh,C.-C.J.Kuo,S.Sun,andS.Lei,“Filmgrainnoisemodelinginadvancedvideocoding,”inVisualCommunicationsandImageProcessing2007,vol.6508.SPIE,Jan.2007,pp.362–373.
[5]InternationalTelecommunicationUnion,“H.274:Versatilesupplementalenhancementinformationmessagesforcodedvideobitstreams,”Sep.2023.[Online].Available:https://www.itu.int/rec/T-REC-H.274
[6]MPEGvideotechnologies,“Part7:Versatilesupplementalenhancementinformationmessagesforcodedvideobitstreams,”inISO/IECDIS23002-7.[Online].Available:https://www.iso.org/standard/87644.html

17. Experimental Validation
—

18. MHV’24
Experimental setup
15.09.2024 © FraunhoferSlide 18
▪WerunexperimentsonanAMDEYPC7502Pprocessor(32cores)whereweruneachVVenCv1.12instanceusing
eightCPUthreads,withadaptivequantization.
▪fasterandmediumpresets
▪FullHD(1080p)resolution
▪Two-passratecontrol[1]atbitrates{1.0,1.5,2.4,3.4,4.5,5.8,7.8,9.0}Mbps.
▪ComparisonofFGSimplementedinVTMandVVdeC(usingfourCPUthreads)decoders.
[1]C.R.Helmrich,V.George,V.V.Menon,A.Wieckowski,B.Bross,andD.Marpe,“FastConstant-QualityVideoEncodingusingVVenCwithRateCappingbasedonPre-
analysisStatistics,”in2024IEEEInternationalConferenceonImageProcessing(ICIP),2024.

19. MHV’24
Quality assessment metrics
15.09.2024 © FraunhoferSlide 19
Observations:
▪TraditionalmetricslikePSNRandSSIMarenotsuitablefor
evaluatingtheperceptualqualityoffilmgraincodingowingtotheir
lackoftexturesensitivity.
▪PSNRandSSIMaresensitivetonoise,suchthattheypenalizethe
additionofsynthesizedfilmgrain.
▪VMAF[1],whilemoreadvanced,isnottrainedtoevaluatethe
perceptualqualityofVVC-codedvideos[2].
[1]ZhiLi,ChristosBampis,JulieNovak,AnneAaron,KyleSwanson,AnushMoorthy,andJanDeCock,“VMAF:Thejourneycontinues,”inNetflixTechnologyBlog,vol.25,2018.
[2]ChristianR.Helmrich,BenjaminBross,JonathanPfaff,HeikoSchwarz,DetlevMarpe,andThomasWiegand,“InformationonandanalysisoftheVVCencodersintheSDRUHDverificationtest,”inWG05MPEGJoint
VideoCodingTeam(s)withITU-TSG16,documentJVET-T0103,Oct.2020.
Take aways:
▪Giventheselimitations,specializedmetricsfocusingontextureenhancement,perceptionofcontrollednoise,andoverallfilm-likeappearance
wouldbemoreappropriateforevaluatingfilmgraincoding,subjecttofuturework.
▪Metricsthatincludehumanperceptionaspectsandconsidertexturefidelityalongsidenoisewouldofferabetterassessmentofthequality
enhancementsfilmgrainbringstovideocontent.

20. MHV’24
Subjective quality assessment
15.09.2024 © FraunhoferSlide 20
▪Origina
l
Figure: Cropped frame of BQTerrace encoded at 800 kbps.

21. MHV’24
Subjective quality assessment
15.09.2024 © FraunhoferSlide 21
Figure: Cropped frame of OldTownCross sequence encoded at 600 kbps at faster preset

22. MHV’24
Runtime Complexity
15.09.2024 © FraunhoferSlide 22
Encoding and Decoding
Encoding speed:
▪FGA contributes to the increased relative duration required for encoding
as the preset progresses towards fasterconfiguration.
▪The overall encoding time using the proposed toolchain reduces up to
11.59 % using slower preset.
Decoding speed:
▪Onaverage,VVdeC(FGS)isapproximately60timesfasterthanVTM
(FGS)andcanhandlereal-timedecoding.
▪TheoptimizationofFGSwithintheVVdeCdecoderisaworkin
progressandremainsafocusforfutureimprovements.
Table:EncodingruntimeincreasewithFGA.
Table:Decodingspeeds(infps)ofVTMandVVdeC.

23. Conclusions and Future Directions
—

24. MHV’24
Conclusions and future directions
15.09.2024 © FraunhoferSlide 24
Conclusions
▪WepresentedthemultifacetedimpactofFGAandFGSontheencoding,decoding,andsubjectiveperceptionofvideocontentforVVC-
basedimplementations.
▪Inscenariossuchaslow-bitrateencoding,FGSemergesasavaluabletoolinmitigatingcompressionartifactsbyintroducingcontrolled
noisethatmimicsnaturalfilmgraincharacteristics,effectivelycamouflagingcompressionartifacts.
▪OurobjectiveevaluationemphasizesthepotentialofFGStolowertherequiredbitratewhilemaintainingperceptualquality,underscoringits
significanceinvideocodingworkflows.
Future directions
▪TheoptimizationofFGSwithintheVVdeCdecoderisaworkinprogressandremainsafocusforfutureimprovements.
▪EffortsareongoingtoenhanceVVdeC'sefficiencyinhandlingFGStoachievebetterspeed,ensuringthattheperceptualbenefitsofFGS
canberealizedwithoutsubstantialcompromisesindecodingspeed.

25. MHV’24
Meet us
@
Booth B80 in Halle 8
15.09.2024 © FraunhoferSlide 25

26. Thank you for your attention
— ▪Vignesh V Menon ([email protected])
▪Adam Wieckowski ([email protected])
▪Christian Stoffers ([email protected])
▪Jens Brandenburg ([email protected])
▪Christian Lehmann ([email protected])
▪Benjamin Bross ([email protected])
▪Thomas Schierl ([email protected])
▪Detlev Marpe ([email protected])