MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices

All rights reserved. ©2020
A Super-Resolution Based Approach for HTTP
Adaptive Streaming for Mobile Devices
ACM Mile-High Video 2022
March 03, 2022
Minh Nguyen, Ekrem Çetinkaya, Hermann Hellwagner, Christian Timmerer
Christian Doppler Laboratory ATHENA | Alpen-Adria-Universität Klagenfurt | Austria
ekrem.cetinkaya@aau.at | athena.itec.aau.at
1

Video Streaming on Mobile Devices
1 “YouTube by the Numbers: Stats, Demographics & Fun Facts”, Omnicore.
2
70% of YouTube watch time is
from mobile devices 1
70%
30%
2 “Experience Shapes Mobile Customer Loyalty”, Ericsson.
26% of smartphone users encounter
video streaming problem every day 2

ML-Benchmark GPU Scores of iPhones
3
ML-Benchmark GPU Scores, Source: https://browser.geekbench.com/ml-benchmarks
1797
1362
858
502
iPhone 13 (2021)
iPhone 11 (2019)
iPhone 8 (2017)
iPhone 6S (2015)

Super-Resolution
4
* Ahn, N., Kang, B., & Sohn, K. A. (2018). Fast, accurate, and lightweight super-resolution with cascading residual network.
In Proceedings of the European conference on computer vision (ECCV) (pp. 252-268)
Bilinear
CARN*
540p
1080p

5
SR-ABR Net WISH-SR
Why?
🔋 Mobile devices are becoming powerful
⏱ Execution time of SR-DNNs is still high
What?
🗂 ABR algorithm that considers throughput
cost, buffer cost, and quality cost.
🗂 An extension to WISH1 ABR. Trade-off
among different factors
Why?
💿 Reduce downloaded data while preserving
the QoE
🗂 ABR needs to consider when to apply SR
What?
🗂 Lightweight SR network that considers the
limitations of the mobile environment
🗂 Performance on-par with SoTA SR-DNNs
while running on real-time on mobile GPUs
Proposed Method
1M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021
IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.

SR-ABR
6

System Architecture
7
WISH-SR
Server
Client
SR Network Request X2 X3 X4
X2 X3 X4
HR LR
HTTP Get Request

SR-ABR Net
8
Convolution
ReLU
Add
Pixel
Shuffle
Convolution
ReLU
Add
Convolution
ReLU
Add
Convolution
ReLU
Convolution
Clip
ReLU
LR Frame HR Frame

WISH-SR ABR Algorithm
9
GET High Bitrate Segment
More transferred data
(higher throughput cost)
More download time
(higher buffer cost)
Higher Quality
(lower quality cost)

10
Throughput
Cost
Buffer
Cost
Conventional
Quality Cost
SR-Enabled
Quality Cost

11
Throughput Cost
Buffer Cost
Current bitrate
Estimated throughput
Download time of current segment
Current buffer - low threshold

12
Quality Cost Distortion penalty + Instability penalty
Conventional
Quality
Current bitrate
Maximum bitrate
SR
Quality
Improvement in quality level

13
Quality Cost
Throughput Cost Buffer Cost
M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021 IEEE
23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.

Evaluation Setup
14

Experimental Setup
15
Testbed
💻 Lenovo Thinkpad P1 (i7 / 16GB)
Ubuntu 18.04
📱 Xiaomi Mi 11 (Snapdragon 888)
Android 11 - ExoPlayer
Dataset - ABR
🗂 HEVC - Segment duration 4s
🗂{100, 145, 900, 2400, 4500} kbps
{270p, 360p, 540p, 720p, 1080p}
(i) Tears of steel - First 5 mins (ToS1) (Mix 🌍🗂 - 📉 SI 📉 TI)
(ii) Tears of steel - Last 5 mins (ToS2) (Mix 🌍🗂 - 📈 SI 📈 TI)
(iii) Gameplay - (Generated 🗂 - 📈 SI 📉 TI)
(iv) Rally (Natural 🌍 - 📉 SI 📈 TI)
🔗 Linux traffic control tool (tc)
4G Network trace1
Avg. 3787 kbps - Std.dev. 3193 kbps
RTT 20ms - Buffer 20s - Low threshold 4s
1D. Raca, J. J. Quinlan, A. H. Zahran, and C. J. Sreenan. “Beyond throughput: a 4G LTE dataset with channel and context metrics”. In Proceedings of the 9th ACM
Multimedia Systems Conference, pages 460–465. ACM, 2018.
2T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson. A buffer-based approach to rate adaptation: Evidence from a large video streaming service. In ACM
SIGCOMM Computer Communication Review, volume 44, pages 187–198. ACM, 2014.
3C. Wang, A. Rizk, and M. Zink. SQUAD: A spectrum-based quality adaptation for dynamic adaptive streaming over HTTP. In Proceedings of the 7th International
Conference on Multimedia Systems, pages 1–12, 2016.
4M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices. In 2021 IEEE 23rd Int’l.
Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
BBA-02, ExoPlayer, SQUAD3, WISH4

SR Network Training
16
Dataset
🗂 HEVC - Target Resolution 1080p
270p - X4, 360p - X3, 540p - X2
DIV2K Dataset 1
Frames from around ~ 100 Videos
Waterloo 2 - SJTU 3 - Tencent Video Dataset 4
1 Agustsson, Eirikur, and Radu Timofte. "Ntire 2017 challenge on single image super-resolution: Dataset and study." Proceedings of the IEEE conference on computer
vision and pattern recognition workshops. 2017.
2 M. Cheon and J.-S. Lee. Subjective and objective quality assessment of compressed 4K UHD videos for immersive experience. IEEE Transactions on Circuits and
Systems for Video Technology, 28(7):1467–1480, 2017.
3 L. Song, X. Tang, W. Zhang, X. Yang, and P. Xia. The SJTU 4K video sequence dataset. In 2013 Fifth International Workshop on Quality of Multimedia Experience
(QoMEX), pages 34–35, 2013. doi: 10.1109/QoMEX.2013.6603201.
4 X. Xu, S. Liu, and Z. Li. Tencent Video Dataset (TVD): A Video Dataset for Learning-based Visual Data Compression and Analysis. arXiv preprint arXiv:2105.05961, 2021
5 N. Ahn, B. Kang, and K.-A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on
Computer Vision (ECCV), pages 252–268, 2018.
Training
CARN-M5 - SR-ABR Net
Train on DIV2K - Finetune on encoded videos
Adam optimizer - Learning rate scheduler - MSE
Tensorflow-lite
Float16 quantization

Evaluation Metrics
17
Average Bitrate
# of Stalls and Stall Duration
QoE Score - ITU-T P.1203 Extension Mode 0
VMAF
VMAF/Bitrate

Results
18

SR-DNN Results
19
1 Ekrem Çetinkaya, Minh Nguyen, and Christian Timmerer. "MoViDNN: A Mobile Platform for Evaluating Video Quality Enhancement with Deep Neural Networks." arXiv
preprint arXiv:2201.04402 (2022).
Execution Speed (FPS)
X2
90.93 91.13
82.10
52.83 54.11
42.91
39.00
41.56
24.32
X3 X4
24
30
36
14
9
5
X3 X4
X2
VMAF
SR-ABR Net CARN-M Bilinear

SR-ABR Results
20
3098
1818
2670
1748 1738
BBA-0 EP SQUAD WISH WISH-SR
Average Bitrate (kbps)
3.54
4.05
3.35
4.06
4.09
QoE Score (ITU.T P.1203)
90.87
81.75
86.55
81.29
84.91
VMAF
22
1.85
1
0.3
24
1.8
0 0
Stall Duration (s)
# of Stalls
0.029
0.045
0.032
0.046
0.049
VMAF / Bitrate (1 kbps)

Conclusion
21
SR-ABR Net
WISH-SR
Lightweight SR DNN that considers the limitations of the mobile environment
Significant improvement (up to 60%) over bilinear interpolation (default in Android)
On-par performance with SoTA SR DNNs while running in real time on mobile GPU
ABR algorithm that leverages SR networks to improve quality
Weighted sum model of throughput cost, buffer cost, and quality cost
SR-ABR
SR-ABR Net integrated into WISH-SR and deployed on ExoPlayer
Significant data reduction (up to 43%) while providing high QoE

Thank you!
ekrem.cetinkaya@aau.at
minh.nguyen@aau.at
@ekremcetinkaya_
@minhkstn
linkedin.com/in/ekrcet
linkedin.com/in/minhkstn

MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices

More Related Content

What's hot (19)

Similar to MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices (20)

More from Minh Nguyen (8)

Recently uploaded (20)

MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices