Enhancing Film Grain Coding in VVC: Improving Encoding Quality and Efficiency

Enhancing Film Grain Coding in VVC:
Improving Encoding Quality and Efficiency
—
Vignesh V Menon, Postdoctoral Researcher, Fraunhofer HHI
Sep. 15, 2024, International Broadcasting Convention (IBC), 2024

Introduction to Video Coding & Film Grain
—

MHV’24
Introduction
15.09.2024 © Fraunhofer
Slide 3
Film grain in video coding
• Film grain, an inherent characteristic of analog film, contributes to the unique visual
aesthetics and cinematic experience in movies [1].
• The emergence of the Versatile Video Coding (VVC) [2, 3] standard brings new
opportunities and challenges in efficiently representing film grain, aiming to preserve its
artistic value while ensuring compatibility with modern video compression techniques.
• Compression artifacts like blockiness are introduced when grainy video content is
encoded at low bitrates. Film grain presents a challenge in video coding due to its
random and non-uniform nature, which can amplify compression artifacts if not
adequately handled during encoding.
[1] Inseong Hwang et al. “Enhanced Film Grain Noise Removal for High Fidelity Video Coding”. In: 2013 International Conference on Information Science and Cloud Computing Companion. 2013, pp. 668–674. doi:
10.1109/ISCC-C.2013.69.
[2] Benjamin Bross et al. “Overview of the Versatile Video Coding (VVC) Standard and its Applications”. In: IEEE Transactions on Circuits and Systems for Video Technology. Vol. 31. 10. 2021, pp. 3736–3764. doi:
10.1109/TCSVT.2021.3101953
[3] Adam Wieckowski et al. “A Complete End to End Open-Source Toolchain for the Versatile Video Coding (VVC) Standard”. In: Proceedings of the 29th ACM International Conference on Multimedia. New York, NY, USA:
Association for Computing Machinery, 2021, 3795–3798. isbn: 9781450386517. doi: 10.1145/3474085.3478320.
Figure: Grain example.

MHV’24
Introduction
Slide 4
Versatile Video Coding (VVC)
• Developed by JVET (Joint Video Experts Team)
• Aims to improve compression efficiency, visual quality, and
support for emerging multimedia applications.
• 50% better compression efficiency gains over HEVC.
[1] B. Bross et al., "Overview of the Versatile Video Coding (VVC) Standard and its Applications," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736-3764, Oct. 2021, doi:
10.1109/TCSVT.2021.3101953.
Figure: Typical VVC encoder[1].

MHV’24
Introduction
Slide 5
VVC open-source toolchain

MHV’24
Introduction
Slide 6
Importance of Film Grain in Video Coding
Figure: Illustration of grain loss in OldTownCross sequence when encoded with VVenC.

Film Grain Handling in VVC
—

MHV’24
Slide 8
▪ Uses motion-compensated temporal filtering (MCTF) to extract film grain from video content.
▪ Quantification of film grain properties like size, intensity, and distribution.
▪ SEI messages to transmit parameters for film grain.
Film Grain Analysis (FGA) in VVenC

MHV’24
Slide 9
▪ Replicates film grain during decoding using SEI parameters
▪ Allows for the preservation or synthesis of film grain even in
compressed or filtered content
Film Grain Synthesis (FGS) in VVdeC

MHV’24
Slide 10
▪ To convey the grain parameters to the decoder, the encoder embeds it as
supplemental enhancement information (SEI) in the bitstream [1], as the
"Film Grain SEI.“.
▪ Film Grain SEI inherits the same syntax and semantics of the AVC film
grain SEI message [2].
▪ Since we implement a frequency filtering model for film grain estimation,
film_grain_model_id is set to 0.
▪ Additive blending is used when blending_mode_id is set to 0.
▪ Since our implementation analyses film grain for only the luma channel,
comp_model_present_flag[0] is set to 1.
▪ FGC SEI message is inserted at each frame, which is indicated by setting
the film_grain_characteristics_persistence_flag to 0. It also means the FGC
SEI message only applies to the current decoded frame.
Film Grain SEI
[1] “RDD 5:2006 - SMPTE Registered Disclosure Doc - Film Grain Technology —Specifications for H.264 | MPEG-4 AVC Bitstreams,” RDD 5:2006, pp. 1–18, 2006.
[2] Vijayakumar Gayathri Ramakrishna, Kaustubh Shripad Patankar, and Mukund Srinivasan, “Cloud-Based Workflow for AVC Film Grain Synthesis,” ser. MHV’23. New York, NY, USA: Association for Computing Machinery,
2023, p. 66–71.

MHV’24
Film Grain Analysis in VVenC
Slide 12
Motion-compensated temporal filter (MCTF):
▪ VVenC already employs a denoising stage [5], based on a framework
proposed initially in [6]– motion-compensated temporal (pre-) filtering.
▪ The filter performs a blockwise motion search for each filtered frame in
neighboring frames to remove the noise.
▪ Using up to eight predictors, a weighted average of the current frame block
and its predictors is generated and used for further encoding.
▪ Improved search strategies, reference number reduction, and flexible block
size were introduced, improving the filter runtime and operation [5].
[5] Adam Wieckowski, Tobias Hinz, Christian R. Helmrich, Benjamin Bross, and Detlev Marpe, “An optimized temporal filter implementation for practical applications,” in 2022 Picture Coding Symposium (PCS), 2022, pp. 247–
251.
[6] Jack Enhorn, Rickard Sjöberg, and Per Wennersten, “A Temporal Pre-Filter For Video Coding Based On Bilateral Filtering,” in 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 1161–1165.

MHV’24
Slide 13
Mask Generation
▪ Identify and isolate flat and low-complexity regions of the frame from those
with significant texture and complexity.
▪ Edge detection creates an initial binary mask where edges are marked.
The gradient calculation, non-maximum suppression, double thresholding,
and edge tracking by hysteresis are all employed to generate a detailed
edge map.
▪ Suppressing Low-Intensity Regions: After generating the edge map,
low-intensity regions that do not contribute significantly to the overall
texture are suppressed.
▪ Morphological operations, specifically dilation, are applied to the masks
to fill small holes and gaps, ensuring contiguous regions.

MHV’24
Slide 14
Grain Estimation
▪ Difference estimation: Difference between original and filtered frames
▪ Block based analysis
▪ Size determined based on the picture resolution
▪ DCT transform applied to each block
▪ Variance calculation
▪ Mean intensity calculation
▪ Edge detection for luma components

MHV’24
Slide 15
Polynomial Fitting and Quantization
▪ Models relationship between intensity and variance in the estimated film grain
▪ Prepared datapoints are used to fit an n-th order polynomial function:
▪ Setup matrix equations based on datapoints
▪ Solve the equations to find the polynomial coefficients
▪ Rescale the parameters to match the datarange
▪ Extend data points
▪ Quantization
▪ Llyod-max quantization to have a specific range of resulting scaling factors
▪ Scaling estimated parameters
▪ Define intensity intervals and corresponding scaling factors
▪ Merging small intervals
▪ Scaling parameters to 8bit range
▪ Setting the final parameters in the component model

MHV’24
Proposed toolchain
Slide 16
• The work in SMPTE-RDD5 [1] provides an in-depth look at FGS,
which is part of the decoder side of the video distribution chain [2].
• This process is defined for the H.264 standard [3], is compatible
with VVC without modifications since both support the same
metadata [4].
• Our method offers a more precise FGS specification than VSEI [5,
6].
• Based on filtering in the frequency/transform domain, which involves
filtering random noise to simulate the film grain pattern. In this
model, film grain patterns are generated in the frequency domain
using a pair of cut-off frequencies that define a low-pass filter.
These patterns are then scaled to the proper intensity before
blending into the image.
Film Grain Synthesis in VVdeC
[1] “RDD 5:2006 - SMPTE Registered Disclosure Doc - Film Grain Technology —Specifications for H.264 — MPEG-4 AVC Bitstreams,” RDD 5:2006, pp. 1–18, 2006.
[2] A. Wieckowski, G. Hege, C. Bartnik, C. Lehmann, C. Stoffers, B. Bross, and D. Marpe, “Towards A Live Software Decoder Implementation For The Upcoming Versatile Video Coding (VVC) Codec,” in 2020 IEEE
International Conference on Image Processing (ICIP), Oct. 2020, pp. 3124–3128, ISSN: 2381-8549.
[3] T. Wiegand, G. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560–576, Jul. 2003.
[4] B. T. Oh, C.-C. J. Kuo, S. Sun, and S. Lei, “Film grain noise modeling in advanced video coding,” in Visual Communications and Image Processing 2007, vol. 6508. SPIE, Jan. 2007, pp. 362–373.
[5] International Telecommunication Union, “H.274 : Versatile supplemental enhancement information messages for coded video bitstreams,” Sep. 2023. [Online]. Available: https://www.itu.int/rec/T-REC-H.274
[6] MPEG video technologies, “Part 7: Versatile supplemental enhancement information messages for coded video bitstreams,” in ISO/IEC DIS 23002-7. [Online]. Available: https://www.iso.org/standard/87644.html

MHV’24
Experimental setup
Slide 18
▪ We run experiments on an AMD EYPC 7502P processor (32 cores) where we run each VVenC v1.12 instance using
eight CPU threads, with adaptive quantization.
▪ faster and medium presets
▪ Full HD (1080p) resolution
▪ Two-pass rate control [1] at bitrates {1.0, 1.5, 2.4, 3.4, 4.5, 5.8, 7.8, 9.0} Mbps.
▪ Comparison of FGS implemented in VTM and VVdeC (using four CPU threads) decoders.
[1] C. R. Helmrich, V. George, V. V. Menon, A. Wieckowski, B. Bross, and D. Marpe, “Fast Constant-Quality Video Encoding using VVenC with Rate Capping based on Pre-
analysis Statistics,” in 2024 IEEE International Conference on Image Processing (ICIP), 2024.

MHV’24
Quality assessment metrics
Slide 19
Observations:
▪ Traditional metrics like PSNR and SSIM are not suitable for
evaluating the perceptual quality of film grain coding owing to their
lack of texture sensitivity.
▪ PSNR and SSIM are sensitive to noise, such that they penalize the
addition of synthesized film grain.
▪ VMAF [1], while more advanced, is not trained to evaluate the
perceptual quality of VVC-coded videos [2].
[1] Zhi Li, Christos Bampis, Julie Novak, Anne Aaron, Kyle Swanson, Anush Moorthy, and Jan De Cock, “VMAF: The journey continues,” in Netflix Technology Blog, vol. 25, 2018.
[2] Christian R. Helmrich, Benjamin Bross, Jonathan Pfaff, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand, “Information on and analysis of the VVC encoders in the SDR UHD verification test,” in WG 05 MPEG Joint
Video Coding Team(s) with ITU-T SG 16, document JVET-T0103, Oct. 2020.
Take aways:
▪ Given these limitations, specialized metrics focusing on texture enhancement, perception of controlled noise, and overall film-like appearance
would be more appropriate for evaluating film grain coding, subject to future work.
▪ Metrics that include human perception aspects and consider texture fidelity alongside noise would offer a better assessment of the quality
enhancements film grain brings to video content.

MHV’24
Subjective quality assessment
Slide 20
▪ Origina
l
Figure: Cropped frame of BQTerrace encoded at 800 kbps.

MHV’24
Subjective quality assessment
Slide 21
Figure: Cropped frame of OldTownCross sequence encoded at 600 kbps at faster preset

MHV’24
Runtime Complexity
Slide 22
Encoding and Decoding
Encoding speed:
▪ FGA contributes to the increased relative duration required for encoding
as the preset progresses towards faster configuration.
▪ The overall encoding time using the proposed toolchain reduces up to
11.59 % using slower preset.
Decoding speed:
▪ On average, VVdeC (FGS) is approximately 60 times faster than VTM
(FGS) and can handle real-time decoding.
▪ The optimization of FGS within the VVdeC decoder is a work in
progress and remains a focus for future improvements.
Table: Encoding runtime increase with FGA.
Table: Decoding speeds (in fps) of VTM and VVdeC.

Conclusions and Future Directions
—

MHV’24
Conclusions and future directions
Slide 24
Conclusions
▪ We presented the multifaceted impact of FGA and FGS on the encoding, decoding, and subjective perception of video content for VVC-
based implementations.
▪ In scenarios such as low-bitrate encoding, FGS emerges as a valuable tool in mitigating compression artifacts by introducing controlled
noise that mimics natural film grain characteristics, effectively camouflaging compression artifacts.
▪ Our objective evaluation emphasizes the potential of FGS to lower the required bitrate while maintaining perceptual quality, underscoring its
significance in video coding workflows.
Future directions
▪ The optimization of FGS within the VVdeC decoder is a work in progress and remains a focus for future improvements.
▪ Efforts are ongoing to enhance VVdeC's efficiency in handling FGS to achieve better speed, ensuring that the perceptual benefits of FGS
can be realized without substantial compromises in decoding speed.

MHV’24
Meet us
@
Booth B80 in Halle 8
Slide 25

Thank you for your attention
— ▪ Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
▪ Adam Wieckowski (adam.wieckowski@hhi.fraunhofer.de)
▪ Christian Stoffers (christian.stoffers@hhi.fraunhofer.de)
▪ Jens Brandenburg (jens.brandenburg@hhi.fraunhofer.de)
▪ Christian Lehmann (christian.lehmann@hhi.fraunhofer.de)
▪ Benjamin Bross (benjamin.bross@hhi.fraunhofer.de)
▪ Thomas Schierl (thomas.schierl@hhi.fraunhofer.de)
▪ Detlev Marpe (detlev.marpe@hhi.fraunhofer.de)

Enhancing Film Grain Coding in VVC: Improving Encoding Quality and Efficiency

More Related Content

Similar to Enhancing Film Grain Coding in VVC: Improving Encoding Quality and Efficiency (20)

More from Vignesh V Menon (20)

Recently uploaded (20)

Enhancing Film Grain Coding in VVC: Improving Encoding Quality and Efficiency