Wide gamut support for Canvas/OffscreenCanvas/ImageBitmap

[fserb]

こんにちはTAG！

I'm requesting a TAG review of:

• Name: Wide gamut support for Canvas/OffscreenCanvas/ImageBitmap
• Specification URL: see below
• Explainer, Requirements Doc, or Example code: https://github.com/WICG/canvas-color-space/blob/master/CanvasColorSpaceProposal.md
• Tests: WPT 2dcontext/wide-gamut-canvas
• Primary contacts: fserb

Further details (optional):

• Relevant time constraints or deadlines: We are planning to ship this on Chrome by end of November.
• I have read and filled out the Self-Review Questionnare on Security and Privacy. The assessment is here.
• I have reviewed the TAG's API Design Principles

You should also know that...

We are not releasing the full WICG proposal, only the wide-gamut part of it. When in wide-gamut the color space defaults to extended sRGB. We are planing a separate release with more discussion over color spaces and maybe other storage possibilities.

The current WICG proposal is still under WhatWG review:
whatwg/html#3926
whatwg/html#3882
There are 2 other PR missing from this, that need to be sent after those two land.

We'd prefer the TAG provide feedback as (please select one):

• open issues in our Github repo for each point of feedback
• open a single issue in our Github repo for the entire review
• leave review feedback as a comment in this issue and @-notify fserb

[slightlyoff]

FYI for @plinss and @dbaron that this got another ping today. I know we discussed the media query events as a way to detect when this changes, but not sure if we resolved one way or the other if the API provides enough support to ensure that a developer can actually respond correctly to a window being dragged from one window w/o wide gamut to another tha does.

Thoughts?

[annevk]

It's rather hard to review this as the formal description isn't really there yet (as I also pointed out in the HTML PRs).

[fserb]

I summarized the surface changes of this spec here: whatwg/html#4167

[dbaron]

A few questions to start:

• how does this relate to the wide-gamut work happening in CSS (primarily in css-color-4, I think)? See in particular the section on Profiled, Device-dependent Colors (although I'm really not a fan of that section name). /cc @svgeesus @grorg @smfr
• what happens with color spaces here? It seems like that's an essential part of the problem space given that it's my understanding that sRGB isn't well-defined outside of [0, 1]. It seems to be discussed in the explainer but not in the proposed spec changes.
• how does this relate to capabilities of the OS or device? @slightlyoff alludes to that above, but I don't see the relationship explained in the explainer.
• how do the proposed solutions relate to the three use cases given at the top of the explainer? And what application use cases benefit from those (still mostly very technical) use cases? Are the proposed solutions sufficient for those applications?
• the explainer says that 12 bits is needed for rec2020, but then suggests float16, which has 11 bits of precision. Is that good enough?

[svgeesus]

We are not releasing the full WICG proposal, only the wide-gamut part of it. When in wide-gamut the color space defaults to extended sRGB.

The term extended sRGB is not defined, or even used, in the explainer. As @dbaron correctly notes, the sRGB transfer function is undefined outside [0, 1] and there are multiple plausible ways of extending it, so at a minimum that needs to be fully defined. Having done so, the capability of the extended space to represent WGB (and indeed, HDR) can be evaluated.

The explainer merely notes that 8bit " is below the capabilities of some monitors" which is true, but fails to connect that statement to why 10 or 12 bits is needed:

• to represent wider gamuts (or wider dynamic range) without increasing the size of steps between adjacent color values (which would give banding, etc)
• to allow correction of tone curves for calibration and linearization (this is why some monitors have, for example, 14bit lookup tables for gamma correction, even when the inputs are 8,10 or 12bit)

@dbaron makes the good point that Rec.2020 uses 12bits (in fact either 10 or 12, UHD uses 10 with 2020) while the proposed 16-bit half-float has 11 effective bits. That might well be okay, but it would be good to know that half-float is not painting us into a corner.

[svgeesus]

The Summary of the wide gamut support on Canvas proposal says

enum ImageDataStorageType {
  "uint8", // default
  "uint16",
  "float32",
};

The float16 option is missing there. Or is it being stored as a union inside the uint16?

[fserb]

@svgeesus, regarding the float16 option. The backing storage is float16, but the output to javascript (on ImageData) is either uint16 or float32.

[fserb]

I agree we need a better stable definition of extended sRGB. I'm going to look into that and report back.

[ccameron-chromium]

Here's a blub defining Extended-sRGB.

sRGB

Let's start with the familiar definition of sRGB (from Wikipedia), and then I'll introduce extended-sRGB.

And let's just look at the forward transformation (from CIE XYZ to sRGB). The reverse transformation is just the inverse.

We start with a triple of reals [X,Y,Z] in CIE XYZ space, and the output is a triple of reals [R,G,B] in sRGB space. There are two steps to the transformation. The output of the first step is a triple [Rlinear,Glinear,BLinear].

CIE XYZ to sRGB Step 1

The first step is defined as:

    [ Rlinear ]   [  3.2 -1.5 -0.5 ][ X ]
    [ Glinear ] = [ -1.0  1.9  0.0 ][ Y ]
    [ Blinear ]   [  0.1 -0.2  1.1 ][ Z ]

CIE XYZ to sRGB Step 2

For the second step, we define the function T (taking reals to reals) as follows:

           { 0               : x < 0
    T(x) = { 12.92*x         : 0 <= x <= 0.00313
           { 1.055*x^(1/2.4) : 0.00313 < x <= 1
           { 1               : x > 1

The second step and final step in the transformation is then

    [ R ]   [ T(Rlinear) ]
    [ G ] = [ T(Glinear) ]
    [ B ]   [ T(Blinear) ]

Remarks

That's it. Note that the function T only outputs values in the range [0,1].

Many applications encode the range [0,1] as 8-bit fixed-point.

Extended-sRGB

As before, we start with a triple of reals [X,Y,Z] in CIE XYZ space, and the output is a triple of reals [R,G,B] this time in Extended sRGB space. There are two steps to the transformation. The output of the first step is a triple [Rlinear,Glinear,BLinear].

CIE XYZ to Extended-sRGB Step 1

This is idential to CIE XYZ to sRGB.

CIE XYZ to Extended-sRGB Step 2

For the second step, we define the function U (taking reals to reals) as follows:

        { 1.055*(-x)^(1/2.4) : x < -0.00313
 U(x) = { 12.92*x            : -0.00313 <= x <= 0.00313
        { 1.055*x^(1/2.4)    : x > 0.00313

Note that in the range [-1,1], U is a point-symmetric extension of T, that is, U(x) == sign(x)*T(abs(x)).

The second step and final step in the transformation is then

 [ R ]   [ U(Rlinear) ]
 [ G ] = [ U(Glinear) ]
 [ B ]   [ U(Blinear) ]

Remarks

Note that it is entirely possible for R, G, and B to be negative or to be arbitrarily large (greater than 1). While these values may not correspond to physically realizable light spectra, it is valid to do math on them (just like in the CIE XYZ space), and once they are to be displayed in a particular gamut, the values may be clipped to that gamut.

Reverse transformation

The reverse transformation from Extended-sRGB to CIE XYZ is the inverse function, as with sRGB. Unlike with sRGB, the forward and reverse transformation are 1-to-1.

Remarks about compatibility

Note that for any triple [R,G,B] with R,G,B in the interval (0,1), the transformation of [R,G,B] from sRGB to CIE XYZ is exactly the same as the transformation from Extended-sRGB to CIE XYZ. This means that any program that was written in the sRGB color space can be changed to use the Extended-sRGB color space with no extra modifications.

Note that when using Extended-sRGB it is natural to store the color values as floating-point (or half-float) triples. It is not reasonable to encode them as 8-bit fixed-point values. If a canvas that is Extended-sRGB is to be converted to an image for export, then it would be natural to

• First convert from Extended-sRGB to the desired output image gamut (in the case of sRGB, this is the identity)
• Then clamp to the range [0,1] and encode as the range [0,1] as 8-bit fixed-point.

[svgeesus]

OK so of the three ways to do it:

a) extend sRGB function by not clamping to [0,1]
b) extend sRGB function by mirroring about 0
c) use a linear function, and more bits

this proposal is using b). This means it is well defined above 1 and below 0, can represent wider gamut colors that require a negative component value (typically red), and can represent a modest increase in maximum linear component value (the curve flattens off after 1). This is probably enough to deal with WCG and may be enough to deal with HDR.

Please put that description into the proposal.

This means that any program that was written in the sRGB color space can be changed to use the Extended-sRGB color space with no extra modifications.

With few modifications:

• remove any clamping of out of range colors
• use a higher bit depth