![Introduction
Optimized encoding preset
Traditional open-source encoders like x264,4 x265, and VVenC5 have pre-defined sets of
encoding parameters (termed as presets), which present a trade-off between the encoding
time and compression efficiency.6
If the encoder preset is configured such that this higher encoding speed can be reduced while
still being compatible with the expected live encoding speed, the quality of the encoded
content achieved by the encoder can be improved.
Subsequently, when the content becomes complex again, the encoder preset needs to be
reconfigured to move back to the faster configuration that achieves live encoding speed.7
By employing efficient storage techniques and removing unnecessary representations, the energy
consumption associated with storing and transmitting redundant data can be minimized.8
4
VideoLAN. “x264”. In: url: https://www.videolan.org/developers/x264.html.
5
Adam Wieckowski et al. “VVenC: An Open And Optimized VVC Encoder Implementation”. In: Proc. IEEE International Conference on Multimedia Expo
Workshops (ICMEW). 2021, pp. 1–2. doi: 10.1109/ICMEW53276.2021.9455944.
6
Dieison Silveira, Marcelo Porto, and Sergio Bampi. “Performance and energy consumption analysis of the X265 video encoder”. In: 2017 25th European
Signal Processing Conference (EUSIPCO). 2017, pp. 1519–1523. doi: 10.23919/EUSIPCO.2017.8081463.
7
Sergey Zvezdakov, Denis Kondranin, and Dmitriy Vatolin. “Machine-Learning-Based Method for Content-Adaptive Video Encoding”. In: 2021 Picture Coding
Symposium (PCS). 2021, pp. 1–5. doi: 10.1109/PCS50896.2021.9477507.
8
Quortex. Mission: Emission. https://www.quortex.io/wp-content/uploads/2022/05/WhitePaper-Mission_-Emission.pdf. [Accessed May 2023]. 2022.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 4](https://image.slidesharecdn.com/jalepresentation-240213025806-5ffbfd6d/85/Optimal-Quality-and-Efficiency-in-Adaptive-Live-Streaming-with-JND-Aware-Low-latency-Encoding-4-320.jpg)
![Experimental validation
Experimental parameters
Table: Experimental parameters of JALE used in this paper.
Parameter Symbol Values
Set of representations
Resolution height [pixels]
R
360 432 540 540 540 720 720 1080 1080 1440 2160 2160
Bitrate [Mbps] 0.145 0.300 0.600 0.900 1.600 2.400 3.400 4.500 5.800 8.100 11.600 16.800
Set of presets [x265] P 0 (ultrafast) – 5 (medium)
Set of supported thread counts C 4 8 12 16 20 24
Total CPU threads N 96
Encoding speed threshold [fps] sT 30
Average target JND vJ 2 4 6
Maximum VMAF threshold vT 98 96 94
Benchmark schemes:
1 Default: ultrafast preset with eight threads for each encoding instance.12
2 Bruteforce: optimized thread count-preset pair with and without JND-based representation elimi-
nation when the models are fully accurate. This is accomplished by bruteforce encoding using all
thread count-preset pairs and selecting the optimized pair.13
Hence, it is suitable only for video-on-
demand applications.
3 CAPS14
determines the optimized preset for each representation for a target encoding speed of 30 fps.
We evaluate CAPS where c=4, 8, and 16, respectively.
12
Apple Inc., “HLS Authoring Specification for Apple Devices”.
13
Jan De Cock et al. “Complexity-based consistent-quality encoding in the cloud”. In: 2016 IEEE International Conference on Image Processing (ICIP). 2016,
pp. 1484–1488. doi: 10.1109/ICIP.2016.7532605.
14
Vignesh V Menon et al. “Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming”. In: 2022 Picture Coding Symposium (PCS). 2022,
pp. 253–257. doi: 10.1109/PCS56426.2022.10018034.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 10](https://image.slidesharecdn.com/jalepresentation-240213025806-5ffbfd6d/85/Optimal-Quality-and-Efficiency-in-Adaptive-Live-Streaming-with-JND-Aware-Low-latency-Encoding-10-320.jpg)
![Experimental validation
Results
Table: Average results of the encoding schemes compared to the default encoding.
Method vJ BDRP BDRV BD-PSNR BD-VMAF ∆S ∆N ∆E
[%] [%] [dB] [%] [%] [%]
Bruteforce
2 -23.85 -23.02 1.21 4.61 -51.32 - 1688.91
4 -24.54 -24.27 1.25 5.26 -62.13 - 1688.91
6 -26.30 -25.77 1.32 5.53 -72.03 - 1688.91
CAPS (c = 4) - -12.31 -14.98 0.82 2.91 -0.06 -50.00 8.65
CAPS (c = 8) - -20.26 -20.95 1.06 4.03 -0.09 0 39.68
CAPS (c = 16) - -28.13 -29.35 1.62 6.04 -0.35 100.00 96.84
JALE
2 -23.30 -22.80 1.19 4.47 -51.34 -43.13 -2.69
4 -24.41 -24.08 1.25 5.05 -62.95 -54.08 -22.70
6 -25.93 -25.47 1.32 5.38 -72.70 -63.83 -37.87
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 13](https://image.slidesharecdn.com/jalepresentation-240213025806-5ffbfd6d/85/Optimal-Quality-and-Efficiency-in-Adaptive-Live-Streaming-with-JND-Aware-Low-latency-Encoding-13-320.jpg)
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding
- 1. Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding Vignesh V Menon1 , Jingwen Zhu2 , Prajit T Rajendran3 , Samira Afzal4 , Klaus Schoeffmann4 , Patrick Le Callet2 , Christian Timmerer1 1 Video Communication and Applications Dept., Fraunhofer HHI, Berlin, Germany 2 Ecole Centrale Nantes, CNRS, LS2N, UMR 6004, Nantes Universite, Nantes, France 3 CEA, List, F-91120 Palaiseau, Université Paris-Saclay, France 4 Alpen-Adria-Universität, Klagenfurt, Austria 13 Feb 2024 Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 1
- 2. Outline 1 Introduction 2 JND-Aware Low latency Encoding (JALE) 3 Experimental validation 4 Conclusions Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 2
- 3. Introduction Introduction The cloud server can significantly streamline the encoding process by dynamically adjusting CPU thread counts based on resolution and bitrate, accommodating diverse video qualities within an adaptive streaming environment. 01 02 03 04 05 06 07 08 09 10 11 12 Representation ID 15 30 60 120 240 480 Encoding speed (in fps) 4 threads 8 threads 16 threads Figure: The encoding speed of each representation in HLS bitrate ladder1 for the Wood s000 sequence2 using ultrafast preset of x2653 using 4, 8, and 16 CPU threads for each representation. 1 Apple Inc. “HLS Authoring Specification for Apple Devices”. In: url: https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices. 2 Hadi Amirpour et al. “VCD: Video Complexity Dataset”. In: Proceedings of the 13th ACM Multimedia Systems Conference. New York, NY, USA: Association for Computing Machinery, 2022, 234–239. isbn: 9781450392839. doi: 10.1145/3524273.3532892. url: https://doi.org/10.1145/3524273.3532892. 3 VideoLAN. “x265”. In: url: https://www.videolan.org/developers/x265.html. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 3
- 4. Introduction Optimized encoding preset Traditional open-source encoders like x264,4 x265, and VVenC5 have pre-defined sets of encoding parameters (termed as presets), which present a trade-off between the encoding time and compression efficiency.6 If the encoder preset is configured such that this higher encoding speed can be reduced while still being compatible with the expected live encoding speed, the quality of the encoded content achieved by the encoder can be improved. Subsequently, when the content becomes complex again, the encoder preset needs to be reconfigured to move back to the faster configuration that achieves live encoding speed.7 By employing efficient storage techniques and removing unnecessary representations, the energy consumption associated with storing and transmitting redundant data can be minimized.8 4 VideoLAN. “x264”. In: url: https://www.videolan.org/developers/x264.html. 5 Adam Wieckowski et al. “VVenC: An Open And Optimized VVC Encoder Implementation”. In: Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW). 2021, pp. 1–2. doi: 10.1109/ICMEW53276.2021.9455944. 6 Dieison Silveira, Marcelo Porto, and Sergio Bampi. “Performance and energy consumption analysis of the X265 video encoder”. In: 2017 25th European Signal Processing Conference (EUSIPCO). 2017, pp. 1519–1523. doi: 10.23919/EUSIPCO.2017.8081463. 7 Sergey Zvezdakov, Denis Kondranin, and Dmitriy Vatolin. “Machine-Learning-Based Method for Content-Adaptive Video Encoding”. In: 2021 Picture Coding Symposium (PCS). 2021, pp. 1–5. doi: 10.1109/PCS50896.2021.9477507. 8 Quortex. Mission: Emission. https://www.quortex.io/wp-content/uploads/2022/05/WhitePaper-Mission_-Emission.pdf. [Accessed May 2023]. 2022. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 4
- 5. JND-Aware Low latency Encoding (JALE) JALE Figure: Live encoding using JALE envisioned in this paper. JALE is classified into three steps: 1 video complexity feature extraction, 2 joint thread count and preset prediction, 3 perceptually-redundant representation elimination. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 5
- 6. JND-Aware Low latency Encoding (JALE) Spatiotemporal complexity feature extraction Spatiotemporal complexity feature extraction Three DCT-energy-based features, the average luma texture energy EY, the average gradient of the luma texture energy h, and the average luminescence LYare used.9,10 (a) original frame (b) EY (c) h (d) LY Figure: Heatmap depiction of the luma texture information {EY, h, LY} extracted from the second frame of CoverSong 1080P 0a86 video of Youtube UGC Dataset. 9 Vignesh V Menon et al. “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming”. In: Proceedings of the First International Workshop on Green Multimedia Systems. 2023, 16–18. isbn: 9798400701962. doi: 10.1145/3593908.3593942. url: https://doi.org/10.1145/3593908.3593942. 10 Vignesh V Menon et al. “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”. In: IEEE Transactions on Circuits and Systems for Video Technology. 2023, pp. 1–1. doi: 10.1109/TCSVT.2023.3290725. url: https://doi.org/10.1109/TCSVT.2023.3290725. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 6
- 7. JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction Joint thread count and preset prediction Selection of the optimized thread count-preset pair for each segment per representation based on the video content complexity is decomposed into two parts: 1 train models to predict the encoding speed for each thread count-preset pair, 2 develop a function to obtain the optimized thread count- preset pair for each representation. The encoding speed of the tth representation of the input video segment (st) is modeled as: st = fS(EY, h, LY, rt, bt, nt, pt) (1) We use random forest models11 to predict the encoding speed for each thread count-preset pair. (PT × CT) models are trained, where PT and CT represent the number of encoding presets and the number of supported thread count per instance, respectively. 11 Leo Breiman. “Random Forests”. In: Machine Learning. Vol. 45. 2001. doi: 10.1023/A:1010933404324. url: https://doi.org/10.1023/A:1010933404324. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 7
- 8. JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction Joint thread count and preset prediction The optimized thread count-preset prediction function has a look-up table of (n̂t, p̂t) pairs. (4, medium) (8,medium) (12,medium) (16,medium) (20,medium) (24,medium) (4,fast) (8,fast) (12,fast) (16,fast) (20,fast) (24,fast) (4,faster) (8,faster) (12,faster) (16,faster) (20,faster) (24,faster) (4,veryfast) (8,veryfast) (12,veryfast) (16,veryfast) (20,veryfast) (24,veryfast) (4,superfast) (8,superfast) (12,superfast) (16,superfast) (20,superfast) (24,superfast) (4,ultrafast) (8,ultrafast) (12,ultrafast) (16,ultrafast) (20,ultrafast) (24,ultrafast) Figure: (n̂, p̂) look-up table used in the experimental validation of this paper. The priority of (n̂t, p̂t) pairs is decided based on the following constraints: 1 the achieved encoding speed ŝt of the tth representation must be greater than or equal to the target encoding speed sT, i.e., ŝt ≥ sT. 2 total number of CPU threads used for each representation is minimized. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 8
- 9. JND-Aware Low latency Encoding (JALE) Perceptually-redundant representation elimination Perceptually-redundant representation elimination Input: q : number of representations in R R = Sq t=1 {(rt , bt , n̂t , p̂t )}: representations with predicted thread count and preset v̂t ; 1 ≤ t ≤ q: predicted VMAF vT : maximum VMAF threshold vJ : average target JND Output: R̂ = (r, b, n̂, p̂): set of encoding configurations R̂ ← {(r1, b1, n̂1, p̂1)} u ← 1 if v̂1 ≥ vT then return R̂ t ← 2 while t ≤ q do if v̂t − v̂u ≥ vJ then R̂ ← R̂ ∪ {(rt , bt , n̂t , p̂t )} u ← t if v̂t ≥ vT then return R̂ t ← t + 1 return R̂ Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 9
- 10. Experimental validation Experimental parameters Table: Experimental parameters of JALE used in this paper. Parameter Symbol Values Set of representations Resolution height [pixels] R 360 432 540 540 540 720 720 1080 1080 1440 2160 2160 Bitrate [Mbps] 0.145 0.300 0.600 0.900 1.600 2.400 3.400 4.500 5.800 8.100 11.600 16.800 Set of presets [x265] P 0 (ultrafast) – 5 (medium) Set of supported thread counts C 4 8 12 16 20 24 Total CPU threads N 96 Encoding speed threshold [fps] sT 30 Average target JND vJ 2 4 6 Maximum VMAF threshold vT 98 96 94 Benchmark schemes: 1 Default: ultrafast preset with eight threads for each encoding instance.12 2 Bruteforce: optimized thread count-preset pair with and without JND-based representation elimi- nation when the models are fully accurate. This is accomplished by bruteforce encoding using all thread count-preset pairs and selecting the optimized pair.13 Hence, it is suitable only for video-on- demand applications. 3 CAPS14 determines the optimized preset for each representation for a target encoding speed of 30 fps. We evaluate CAPS where c=4, 8, and 16, respectively. 12 Apple Inc., “HLS Authoring Specification for Apple Devices”. 13 Jan De Cock et al. “Complexity-based consistent-quality encoding in the cloud”. In: 2016 IEEE International Conference on Image Processing (ICIP). 2016, pp. 1484–1488. doi: 10.1109/ICIP.2016.7532605. 14 Vignesh V Menon et al. “Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming”. In: 2022 Picture Coding Symposium (PCS). 2022, pp. 253–257. doi: 10.1109/PCS56426.2022.10018034. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 10
- 11. Experimental validation Results 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 0 1 2 3 4 5 Preset 4 8 12 16 20 24 Threads (a) average thread count-preset 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) CAPS (c=16) JALE (b) average encoding speed 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 Encoding speed (in fps) Default Bruteforce CAPS (c=8) CAPS (c=16) JALE (c) average VMAF Figure: Results for each representation in JALE. JND-based representation elimination is not considered in these plots. Faster encoding presets and more computational resources are needed to encode high- bitrate representations so that the encoding speed is above the threshold. JALE controls the encoding speed to be greater than sT but not significantly higher than the default encoding. This ensures higher CPU utilization when the encodings are carried out concurrently during a live feed. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 11
- 12. Experimental validation Results 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (a) Basketball s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (b) Characters s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (c) Dolls s001 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (d) Maples s000 Figure: Rate-distortion (RD) curves of representative sequences (segments) for Default encoding (blue line), CAPS (c = 8) encoding (red line), compared to JALE (vJ =6). Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 12
- 13. Experimental validation Results Table: Average results of the encoding schemes compared to the default encoding. Method vJ BDRP BDRV BD-PSNR BD-VMAF ∆S ∆N ∆E [%] [%] [dB] [%] [%] [%] Bruteforce 2 -23.85 -23.02 1.21 4.61 -51.32 - 1688.91 4 -24.54 -24.27 1.25 5.26 -62.13 - 1688.91 6 -26.30 -25.77 1.32 5.53 -72.03 - 1688.91 CAPS (c = 4) - -12.31 -14.98 0.82 2.91 -0.06 -50.00 8.65 CAPS (c = 8) - -20.26 -20.95 1.06 4.03 -0.09 0 39.68 CAPS (c = 16) - -28.13 -29.35 1.62 6.04 -0.35 100.00 96.84 JALE 2 -23.30 -22.80 1.19 4.47 -51.34 -43.13 -2.69 4 -24.41 -24.08 1.25 5.05 -62.95 -54.08 -22.70 6 -25.93 -25.47 1.32 5.38 -72.70 -63.83 -37.87 Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 13
- 14. Conclusions Conclusions We proposed JALE, a JND-aware low latency encoding scheme for adaptive live streaming applications. JALE jointly predicts the optimized encoder preset and CPU thread count for a given representation for each video segment based on the video content complexity features, target encoding speed, and the total number of available CPU threads. The JND-based representation elimination algorithm removes perceptually redundant rep- resentations in the bitrate ladder. JALE yields an overall average quality improvement of 0.98 dB PSNR and 4.41 VMAF points at the same bitrate, compared to the x265 ultrafast encoding of the reference HLS bitrate ladder using eight CPU threads for each representation. Considering a JND of six VMAF points, storage, thread count, and encoding time reductions of 72.70 %, 63.83 %, and 37.87 %, respectively, are observed. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 14
- 15. Q & A Q & A Thank you for your attention! Vignesh V Menon ([email protected]) Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 15