HttpRequest.get_preferred_type misorders types when more specific accepted types have lower q

[Anders Kaseorg]

Consider this example from ​RFC 9110 §12.5.1 (adjusted for ​erratum 7138):

The media type quality factor associated with a given type is determined by finding the media range with the highest precedence that matches the type. For example,

Accept: text/*;q=0.3, text/plain;q=0.7, text/plain;format=flowed,
       text/plain;format=fixed;q=0.4, */*;q=0.5

would cause the following values to be associated:

Media Type	Quality Value
text/plain;format=flowed	1
text/plain	0.7
text/html	0.3
image/jpeg	0.5
text/plain;format=fixed	0.4
text/html;level=3	0.3

Django’s HttpRequest.get_preferred_type fails to match this behavior.

>>> from django.conf import settings
>>> from django.http import HttpRequest
>>> settings.configure()
>>> request = HttpRequest()
>>> request.META["HTTP_ACCEPT"] = "text/*;q=0.3, text/plain;q=0.7, text/plain;format=flowed, text/plain;format=fixed;q=0.4, */*;q=0.5"
>>> request.get_preferred_type(['text/plain', 'text/plain;format=fixed']) # expected text/plain (0.7 > 0.4)
'text/plain;format=fixed'
>>> request.get_preferred_type(['text/html', 'image/jpeg']) # expected image/jpeg (0.3 < 0.5)
'text/html'
>>> request.get_preferred_type(['image/jpeg', 'text/html']) # expected image/jpeg (0.5 > 0.3)
'text/html'
>>> request.get_preferred_type(['image/jpeg', 'text/plain;format=fixed']) # expected image/jpeg (0.5 > 0.4)
'text/plain;format=fixed'
>>> request.get_preferred_type(['image/jpeg', 'text/html;level=3']) # expected image/jpeg (0.5 > 0.3)
'text/html;level=3'
>>> request.get_preferred_type(['text/plain;format=fixed', 'text/plain']) # expected text/plain (0.4 < 0.7)
'text/plain;format=fixed'
>>> request.get_preferred_type(['text/plain;format=fixed', 'image/jpeg']) # expected image/jpeg (0.4 < 0.5)
'text/plain;format=fixed'
>>> request.get_preferred_type(['text/html;level=3', 'image/jpeg']) # expected image/jpeg (0.3 < 0.5)
'text/html;level=3'

[Natalia Bidart]

Hello Anders, thanks for this ticket and for providing clear examples.

We'll investigate further to determine whether this qualifies as a release blocker. In any case, it won't be fixed in time for tomorrow’s special release. If it's deemed a release blocker, the fix will likely be included in an early July release.

[Jake Howard]

I've done some more digging, and I think the current implementation is working as expected. Notably, there's this sentence in the RFC:

Media ranges can be overridden by more specific media ranges or specific media types. If more than one media range applies to a given type, the most specific reference has precedence.

More specific definitions override less specific, even if the quality value is higher. This was found and fixed as part of #36411.

The same example from the previous version of the RFC (7231) has a similar example, which is copied verbatim into ​Django's codebase. If I plug the updated values into the same test case, they pass (​besides the final case). Notably, to find the specific entry in the Accept header which matches the given type, the .accepted_type (internal) method should be used, rather than is_preferred_type (which abstracts these considerations away).

Unfortunately, these RFCs are quite hard to interpret. I've opened a ​thread on the forum to collect some more input.

[Anders Kaseorg]

The most specific reference has precedence for the purpose of assigning the correct quality factor. This is clear from the part I quoted.

The media type quality factor associated with a given type is determined by finding the media range with the highest precedence that matches the type.

The precedence is used to compute the quality factor for each type, and then the computed quality factors are compared to find the most favorable type.

Your interpretation, where the precedence is used a second time to override the already-computed qualities during comparison, makes no sense: the RFC would not go to the effort of showing a detailed example where text/plain;format=flowed receives a lower quality factor than text/plain if there were no way for that lower quality to affect the final outcome.

[Anders Kaseorg]

Just to point out a real-world example—Firefox sends <video> requests with ​this Accept header:

Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5

which Django incorrectly interprets as a preference for application/ogg over video/*:

>>> request = HttpRequest()
>>> request.META["HTTP_ACCEPT"] = "video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5"
>>> request.get_preferred_type(["video/mp4", "application/ogg"])
'application/ogg'

[Natalia Bidart]

After a very useful conversation in the forum, we have concluded this is a valid report and also qualifies as release blocker. Aiming at releasing the fix during the next planned release (July 2nd).

[Jake Howard]

​PR

[Sarah Boyce <42296566+sarahboyce@…>]

In 12c15570:

Fixed #36447 -- Selected preferred media type based on quality.

When matching which entry in the Accept header should be used for
a given media type, the specificity matters. However once those are
resolved, only the quality matters when selecting preference.

Regression in c075508b4de8edf9db553b409f8a8ed2f26ecead.

Thank you to Anders Kaseorg for the report.

[Sarah Boyce <42296566+sarahboyce@…>]

In 4de4edf:

[5.2.x] Fixed #36447 -- Selected preferred media type based on quality.

When matching which entry in the Accept header should be used for
a given media type, the specificity matters. However once those are
resolved, only the quality matters when selecting preference.

Regression in c075508b4de8edf9db553b409f8a8ed2f26ecead.

Thank you to Anders Kaseorg for the report.

Backport of 12c1557060fc94fe5e1fbddc4578a4e29d38f77c from main.