1
BSQ-rate: a new approach for video-codec performance comparison and
drawbacks of current solutions
𝐴.𝑍𝑣𝑒𝑧𝑑𝑎𝑘𝑜𝑣𝑎
1
,𝐷.𝐾𝑢𝑙𝑖𝑘𝑜𝑣
1,2
,𝑆.𝑍𝑣𝑒𝑧𝑑𝑎𝑘𝑜𝑣
1
,𝐷.𝑉𝑎𝑡𝑜𝑙𝑖𝑛
1
^1: Lomonosov Moscow State University, Moscow, Russia; ^2: Dubna State University, Dubna, Russia
Keywords: video quality, no-reference metric, quality measuring, video-codec comparison, comparison methodology, bsq-rate
Abstract. This paper is dedicated to the analysis of the existing approaches to video codecs comparisons. It includes the
revealed drawbacks of popular comparison methods and proposes new techniques. The performed analysis of user-generated
videos collection showed that two of the most popular open video collections from media.xiph.org which are widely used for
video-codecs analysis and development do not cover real-life videos complexity distribution. A method for creating
representative video sets covering all segments of user videos the spatial and temporal complexity is also proposed. One of the
sections discusses video quality estimation algorithms used for video codec comparisons and shows the disadvantages of
popular methods VMAF and NIQE. Also, the paper describes the drawbacks of the BD-rate — generally used method for
video codecs final ranking during comparisons. A new ranking method called BSQ-rate which considers the identified issues
is proposed. The results of this investigation were obtained during the series of research conducted as part of the annual
video-codecs comparisons organized by video group of computer graphics and multimedia laboratory at Moscow State
University.
1. Introduction
Today video content takes more than 70% of world Internet traffic. According to Cisco forecasts [1], the video traffic rate will
grow up to 82% by 2022. Reduction of the amount of this kind of traffic and video transmission costs is becoming one of the
most important directions of research and development in companies. Apart from open-source video compression solutions,
which license taxes are rising, a number of big companies started the development of their own video encoders and cooperate
to contribute to new coding standards. As the existing standards are being improved and new standards are being developed in
many open-source and commercial solutions, trends and technologies in this area evolving very fast. In order to help end-user
a choose a codec for their particular tasks (cloud transcoding, online streaming, CCTV or any other practical application),
independent laboratories and specialists conduct comparisons of different video compression solutions. The organization of
such comparisons require resolving several fundamental goals which influence the objectivity and reasonability of comparison
results.
This article overviews common methodologies of video-codecs comparison including conclusions on the advantages and
disadvantages of different approaches and the description of our suggested methods for video selection, application of quality
metrics application and calculation of competitors overall ranking scores. The paper is organized as follows: Section 2
describes the general steps of video-codecs comparison; Section 3 describes in detail the features of video set construction for
comparison, also it contains the drawbacks of open video collections and description of the proposed approach for video
selection; Section 4 overviews the features of different quality metrics application for encoding performance comparison;
Section 5 describes the drawbacks of commonly used approach for quality rates calculation and offers a new method for this
purpose; Section 6 includes overall conclusions.
2. Stages and methods of video-codec comparisons
There are many approaches for performing video-codec comparisons which differ in a chosen quality metric, video or codec
format, or even in the way of making a decision about winners. However, they all have the following common stages: video
selection, encoding presets selection, compression, measurement of encoded videos quality, choosing the winners depending
on the task. Preparation for codecs contests requires the methodology development, reserving the computing resources and
videos to be used for codec testing, searching for the participants and conversations with them. Getting the encoder binary
from the participant does not mean the definite ability to test it, as usually encoders have many parameters need to be set for
each test as they influence encoding performance unpredictably. The simplest way is to test default encoding configurations,
but not all encoders have them. Even if the encoder has standard presets, these presets may be too different and incomparable
to other competitors. The more honest way is to ask the developers of each encoder to provide configurations for tests which
satisfy the given limitations. Finally, if there is no possibility to obtain encoding parameters from the developers, it is possible
to find optimal configurations using different optimization algorithms. For a fair comparison, encoders should be launched on
one or several equal servers to eliminate differences in hardware performance. For eliminating the influence of other external
factors, such as overheating or system processes, it is better to perform several launches and choose the average or the best
encoding time. For big contests, codecs launches take more time than expected, as sometimes codecs crash during encoding
and the developers need time to make the required fixes.
