Group Listening Tests of Various Formats at 128 kbit/s

Notes by ff123

• Rationale
• Choice of Encoder/Settings
• Exact Settings of Encoders/Decoders
• Test Method
• Bitrate Calculation and Comparison

• "Dogies" sample (test is completed, only reference file is available at my Audio Samples Page)
  • Results and Listener Comments
  • Analysis

Rationale:

With the release of Ogg Vorbis RC2, I was interested to find out how it would stack up against other competing formats at 128 kbit/s. I chose 128 kbit/s because it is a popular bitrate, as well as an interesting one to me: the bitrate is high enough to sound quite good to some people using certain formats, but low enough to sound quite bad to others. I decided to organize a series of blind tests using various different samples encoded in the following formats: MP3, AAC, MPC, Ogg Vorbis, and WMA8.

Choice of Encoder/Settings:

To keep the burden on the testers down, I decided to choose only one representative from each format (except for MP3, which I will explain shortly).

For MP3, I chose Lame using average bitrate encoding (ABR) and also using Naoki Shibata's psymodel (instead of the default gpsycho), a combination which produces relatively high-quality encodes for mp3 at an average bitrate of 128 kbit/s -- although many would argue that another encoder, e.g., Fraunhofer's FastEnc, would be a better choice. Also for MP3, I chose to include Xing, as implemented in AudioCatalyst 2.1. Xing's quality is generally agreed to leave much desired when compared to other MP3 encoders at 128 kbit/s. The purpose of its inclusion in these tests is to provide an "anchor" for the subjective quality ratings.

For AAC, the decision to make was between Liquifier Pro AAC and Psytel AAC. In general, most people seem to agree that the implementation within Liquifier Pro (designed by Fraunhofer) is superior at 128 kbit/s. I chose the "streaming 128" option from within Liquifier Pro 5.

For Ogg Vorbis, I chose the latest implementation of the codec, which in the case of the beginning of this series of tests, is Release Candidate 2 (RC2).

Microsoft's Windows Media Audio 8 (WMA8) is an interesting case. People with highly sensitive ears say that WMA7 is superior in quality to WMA8! In this instance however, I decided that I would pick WMA8 instead of WMA7, because it is the latest implementation, and people tend to think that the more recent release must be the best. Besides that, Microsoft doesn't generally make its older codecs available once the newer versions are out. I risk exposing myself to criticism that I am stacking the deck against WMA, but then again, it was Microsoft's own choice to optimize its codec for low bitrates (i.e., 64 kbit/s) at the apparent expense of higher bitrates. For WMA8, I chose the latest implementation available: the encoder/decoder combination within Windows Media Player 7.1.

MPC is not generally tested at such low bitrates, since it produces optimal quality at somewhat higher bitrates. Also, it is somewhat difficult to make direct comparisons against other codecs since it uses variable bitrate encoding (VBR) and fluctuates quite a lot in bitrate compared with other the other codecs. Nevertheless, by tweaking its settings some (with the help of some MPC enthusiasts), and by encoding several different types of albums with that setting, I convinced myself that at least averaged over a large sample of files, MPC could be comparable in bitrate to the other formats, depending though on the type of music chosen.

Exact Settings of Encoders/Decoders

Notes:

1. There have been reports of poor quality at the beginning of MP3 files (the first second or two). To circumvent such problems, I paste together two copies of the original together with some silence before, after, and in between. After encoding and decoding to WAV, I cut and discard the first copy.

2. Files are edited to make sure that they are aligned with each other and are the same length. I also checked to make sure that the volumes are the same for each format (one-time check only).

Test Method

The test method I chose is to involve as many listeners as possible via Internet in a blind test. Please note that this is not a strictly scientific test, otherwise it would involve lots of listeners (at least 30) under supervision and with uniform and very high quality listening conditions and audio equipment. Listeners are asked to rate the subjective quality from 1 to 5 of a selected short sample encoded with the various formats. The original unencoded sample is provided as a reference. Listeners are asked to grade the encoded samples on a 5-point scale.

The scale is identical to the one used by the MPEG group, as defined by the ITU-R:

5.0 = Imperceptible (not perceptible)
4.0 = Perceptible but not annoying
3.0 = Slightly annoying
2.0 = Annoying
1.0 = Very Annoying

1.0 is the lowest possible score and 5.0 is the highest (which should be assigned only if the listener cannot distinguish between the original and the encoded version). However, even though the grading scale is the same, it is not used in the same way that the MPEG group uses it. In the listening tests that they perform (for example the MPEG-2 AAC Stereo Verification Tests), the scale is absolute, in the sense that listeners are provided with reference samples which are supposed to have a specific grade. They can do this because the listeners as a group are highly sensitive listeners trained in hearing codec artifacts. For the tests I wish to perform, I don't have the opportunity to pick a large group of highly sensitive listeners -- every listener, sensitive or not, must count for something. It is quite possible, probable in fact, that a sample rated as Annoying to one listener should be rated Imperceptible to another. Both are valid results and should not be subject to an absolute grading system. The analysis I perform is of a type suited to the wide variability of listeners available for these tests.

Where appropriate, the use of an ABX listening tool is suggested (when differences are subtle and may not be repeatable).

Bitrate Calculation and Comparison

I needed to make sure that on average, the various encoders were making files at about the correct bitrate. I calculate bitrate as follows (size in bytes) * (8 bits/byte) / (1000 bits/kbit) / (duration in seconds). So for a 4 minute song that's 3,850,000 bytes large, the calculated bitrate is: 3850000 * 8 / 1000 / 240 = 128 kbit/s.

I encoded several different albums from with various musical styles to see how the various formats/settings fared. Xing and WMA8 are constant bitrate codecs, and Liquifier Pro 5 AAC was set for constant bitrate. I chose CBR mode for Liquifier because its VBR mode encodes at too high of an average bitrate unless the frequency bandwidth is set lower than desirable (a little less than 16 kHz). Ogg Vorbis and Lame (with the specified settings) are not constant bitrate, but come pretty close on average to 128 kbit/s. MPC bitrate varies quite a lot by comprison.

Return to ff123's Home Page