Add first-class support for differential script loading

[mathiasbynens]

The type=module/nomodule pattern gave developers a “clean break” to ship small, modern JavaScript bundles (with minimal polyfilling + transpilation) vs. legacy bundles (with lots of polyfills + transpilated code), which is great not just for module adoption but also for web performance. However, as more features are added to the JavaScript language, more polyfilling and transpilation becomes necessary even for these “modern” type=module script bundles.

@kristoferbaxter, @philipwalton, and I have been thinking about ways to address this problem in a future-facing way, and have explored several potential solutions. One way we could introduce a new “clean break” once a year is by adding a new attribute to <script type="module">, perhaps syntax or srcset:

<script type="module"
        srcset="2018.mjs 2018, 2019.mjs 2019"
        src="2017.mjs"></script>
<script nomodule src="legacy.js"></script>

(Note that this is just an example of what a solution could look like.) The 2018 and 2019 descriptors would then refer to feature sets that browsers recognize (in particular, they do NOT refer to ECMAScript version numbers or anything like that). For more details, read our exploration doc.

At this stage we’d like to get feedback on whether others agree this is a problem worth solving. Feedback on any particular solution (such as <script type="module" srcset> vs. <script type="module" syntax> vs. something else) is also welcome, but less important at this time.

[mathiasbynens]

cc @philipwalton @nomadtechie @watilde

[domenic]

I'm -1 on this idea for the following reasons:

• I think the user-agent string already gives the correct amount of information here. Any additional information given is a privacy leak, so this proposal must be strictly less powerful if it is not to violate privacy invariants. (For example, if a user changes their UA string, the browser would need to change what it reports for these values too, in order to not add more bits of entropy. The exploration doc seems to say this is not desired, in the "Differential JavaScript for user-agent Buckets" section, but I assume the intent was not to add more fingerprinting surface, so in fact there would be no difference.) As such it's best to stick with just one source of data.

• Agreement on "yearly feature sets" is not tractable. For example, it'd be ideal to ship BigInt or private field code to today's Chrome, but this proposal would not allow doing so, because "the majority of stable user-agents" do not contain those. (Or do they? See below.) Tests should be more granular than bundling together features in this way.

• Any definition of "majority of stable user agents" is not realistic. By some definitions, that would include exactly one user agent, the most-recently-stable Chrome version. By others, it would include Chrome and Safari, excluding Firefox. By others, it would include Chrome and Firefox, excluding Safari. (It's unclear how to count Edge given recent news.) In some geographic regions, it would include UC Browser or QQ browser. This isn't even mentioning the various Chromium-based browsers which are on less-than-latest-stable-Chrome versions. In the end, only app developers have a realistic idea of what features they want to use untranspiled, and how those features sit relative to the browsers they are targeting. They should make that determination on a per-feature/per-browser basis, not based on a committee agreement of what a year represents, or what the majority of stable user agents represent.

• Script loading is complicated and has many entry points. The exploration doc tries to thread this through <script> and <link>, but misses (in roughly descending order of importance) new Worker(), import statements, import() expressions, service workers, the varied-and-growing worklet loading entry points, importScripts(), and javascript: URLs. A unified solution would involve the server taking responsibility for the scripts based on user agent, as can already be done today, instead of speccing, implementing, and waiting for wide availability of browser-side mechanisms such as the OP, and burdening all current and future script-loading entry points with the need to support this.

• This attempts to bake in a division between the JavaScript platform and the web platform which I think we should discourage, not encourage.

As to whether this is a problem worth solving, it depends on what you mean. I agree it's a worthwhile thing to do for authors to serve browsers code based on the syntax and capabilities they support. I think that problem is already solvable with today's technology though.

[matthewp]

I like the general idea of differential loading but I don't think this solution is the right one. My main problem is surrounding how these yearly feature sets will be defined. I think it would be difficult to gain consensus on what is included.

I can also see a scenario where a Popular Website uses srcset and browsers feel pressure to lie about their support, knowing that Popular Website doesn't use feature Y (the thing they don't support) anyways.

---

I don't have a firm alternative, but I feel like some combination of import maps and top-level-await provide the primitives needed for differential loading. I could see a future feature of import maps that makes it a bit cleaner to do.

[mathiasbynens]

Some initial responses:

I think that problem is already solvable with today's technology though.

It may be “solvable” through UA sniffing and differential serving, but in practice this approach somehow hasn’t gotten much traction. We commonly see websites shipping megabytes of unnecessary JavaScript. To apply the technique you describe, currently developers have to implement and maintain:

• custom tooling configuration to output multiple separate JS bundles, and
• custom server-side UA sniffing that maps exactly to the tooling configuration

If instead, we could somehow standardize on some idea of “feature sets”, then browsers and tooling could align around that, and reduce this friction altogether. Developers could then perform a one-off change to their build configuration and reap the benefits.

This attempts to bake in a division between the JavaScript platform and the web platform which I think we should discourage, not encourage.

Which division are you seeing? There’s no reason npm and Node.js couldn’t adopt the same “feature sets” we standardize on.

Script loading is complicated and has many entry points.

Why do other entry points such as dynamic import() or javascript: URLs need to be supported? The tooling that generates the output bundles would know whether import() is supported or not based on the feature set (e.g. 2019, or whatever kind of identifier we come up with) that was used to generate it. As such, the tool could decide whether or not to transpile/polyfill import() for that particular bundle.

I think it would be difficult to gain consensus on what is included.

It would depend on the chosen process. We can make this as complicated or as simple as we want. It could be as simple as just picking a date. The date then maps to a list of latest versions of stable browsers at that point in time. That list of browsers then maps to a set of features that are fully supported (by some heuristic, e.g. 100% Test262 pass rate for ECMAScript-specific features). There’s no point in arguing about which features should be included if we can just look at browser reality and figure it out from there.

[domenic]

If instead, we could somehow standardize on some idea of “feature sets”, then browsers and tooling could align around that, and reduce this friction altogether. Developers could then perform a one-off change to their build configuration and reap the benefits.

I don't think this alignment necessitates new browser features.

Which division are you seeing?

The proposal includes language features, but not web platform features.

Why do other entry points such as dynamic import() or javascript: URLs need to be supported?

Because they are other ways of loading scripts, and if the problem statement is differential script loading, then you need to ensure those also allow differential script loading.

It would depend on the chosen process. We can make this as complicated or as simple as we want. It could be as simple as just picking a date.

As I tried to point out, it is not that simple. A concept such as "latest versions of stable browsers" is itself super-fraught.

[mathiasbynens]

Which division are you seeing?

The proposal includes language features, but not web platform features.

There’s no reason it cannot include web platform features.

[bkardell]

Given all of about 15 minutes worth of thought I am a little hesitant to share a anything like a 'real' opinion here, but my gut reaction was kind of similar to what @domenic said except that I fall way short of

I don't think this alignment necessitates new browser features.

That's not to say "it does" either, just that that I also fully accept @mathiasbynens general premise what "can" technically be done doesn't seem to have caught on and is probably more challenging than it should be - but I don't know how to fix that either.

[jkrems]

FYI: In the node modules working group, we're currently exploring extending the existing import map alternatives pattern to support this kind of environment matching: jkrems/proposal-pkg-exports#29

[kristoferbaxter]

I think the user-agent string already gives the correct amount of information here.

The User-Agent is usable for many scenarios to provide a varied document or script response, but not all scenarios. For instance, within a Signed HTTP exchange, how would an author vary the response for either a document or subresource script resource based on the user-agent header? When hosting a simple static document, how would the document author vary a script source based on user-agent?

Additionally, User-Agent requires document authors to correctly parse and leverage the information within. There are efforts to reduce the complexity of this burden, but it's still not clear if they will happen. Allowing the User-Agent to provide a clear signal (via Syntax request header) and use the exact same logic on static documents would open this functionality up to a much larger audience.

This proposal attempts to provide a similar mechanism as srcset does for images, which could arguably be mostly redundant if a document author uses Client Hints.

Any additional information given is a privacy leak, so this proposal must be strictly less powerful if it is not to violate privacy invariants. (For example, if a user changes their UA string, the browser would need to change what it reports for these values too, in order to not add more bits of entropy. The exploration doc seems to say this is not desired, in the "Differential JavaScript for user-agent Buckets" section, but I assume the intent was not to add more fingerprinting surface, so in fact there would be no difference.) As such it's best to stick with just one source of data.

This is an interesting point, the intention is the syntax version would remain stable between browser versions, until a new version passed the set of defined tests and could change the version to the next revision. Similar to the Accept header, this value would change relatively infrequently and fully align with the reported User-Agent string changing. There is no scenario where the Syntax value would change outside of a User-Agent change. I'm struggling to understand where this adds additional bits of entropy. Perhaps we could use Accept as a similar request header for comparison?

Agreement on "yearly feature sets" is not tractable. For example, it'd be ideal to ship BigInt or private field code to today's Chrome, but this proposal would not allow doing so, because "the majority of stable user-agents" do not contain those. (Or do they? See below.) Tests should be more granular than bundling together features in this way.

This proposal doesn't attempt to reduce transpilation to zero for specific User-Agents. If a document author wanted to specifically ship code that worked in Chrome alone, they would want to use User-Agent parsing. The "yearly feature set" is a stake in the ground, a compromise between shipping the absolute latest version of syntax and transpiling everything to ES5.

Any definition of "majority of stable user agents" is not realistic. By some definitions, that would include exactly one user agent, the most-recently-stable Chrome version. By others, it would include Chrome and Safari, excluding Firefox. By others, it would include Chrome and Firefox, excluding Safari. (It's unclear how to count Edge given recent news.) In some geographic regions, it would include UC Browser or QQ browser.

A goal of this proposal is to reduce the complexity in safely shipping differential JavaScript. This would require browser vendors working with one another to establish the items included in each yearly revision. However, I and other Web Developers would hope this is achievable... the goal is to make documents use more of the code they were authored with. If a User-Agent doesn't pass the defined set of tests for a yearly revision, they should not report that version in the Syntax request header, nor use a corresponding value in a HTMLScriptElement.syntax attribute.

Script loading is complicated and has many entry points. The exploration doc tries to thread this through <script> and , but misses (in roughly descending order of importance) new Worker(), import statements, import() expressions, service workers, the varied-and-growing worklet loading entry points, importScripts(), and javascript: URLs.

All of the above items are addressable with support added to HTMLScriptElement, HTMLLinkElement, and the Syntax request header. The expectation is once a HTMLScriptElement chooses a syntax version, the resource it chose is responsible for leveraging the correct references to its dependencies (Workers, import statement, import expressions, service workers, importScripts() and javascript: URLs).

Would specifying the behaviour for these items independently (as done with HTMLScriptElement and HTMLLinkElement) address these concerns?

This attempts to bake in a division between the JavaScript platform and the web platform which I think we should discourage, not encourage.

Not intentional. This proposal starts with a smaller target than the entire web platform, but no division is intended.

[littledan]

I agree it's a worthwhile thing to do for authors to serve browsers code based on the syntax and capabilities they support. I think that problem is already solvable with today's technology though.

There's more than one way to get at this sort of information; I wonder what you'd recommend. I like the idea of making the decision on the client side, as import maps does. I've heard it can be impractical to deploy UA testing in some scenarios.

If inefficient JavaScript is being served today, I'm wondering why. Is not efficient enough to do the tests? Are tools authors unaware of the technique? Is it impractical to deploy for some reason? I bet framework and bundler authors would have some relevant experience.

[daKmoR]

that seems too ambitious of a solution... (agreeing on which feature are in which "group" seems to be way too hard... and it can fastly different on which technology you are using)
imho you will always need to have some logic like

if (this and that feature is supported) { 
  load(this); 
} else if (other and more is supported) {
  load(other); 
} else if (...) {}

imho it's about having a way of getting these checks auto-generated into your index.html by bundlers.

[jridgewell]

I think the user-agent string already gives the correct amount of information here. Any additional information given is a privacy leak, so this proposal must be strictly less powerful if it is not to violate privacy invariants... As such it's best to stick with just one source of data.

The userAgent string does provide a ton of information, but it's inscrutable. Browsers add additional text to fool UA sniffers, and adding any additional significance (trying to determine JS support) based on it is going to cause errors.

Agreement on "yearly feature sets" is not tractable. For example, it'd be ideal to ship BigInt or private field code to today's Chrome, but this proposal would not allow doing so, because "the majority of stable user-agents" do not contain those.

It's impossible to solve this with a lowest-common-denominator approach. So you can either ship (and maintain!) multiple highly-specialized builds to each browser, or you can ship and maintain LCD builds.

Having just a yearly LCD build seems like an excellent middle ground compared to compile-everything-to-es5 or every-browser-version-gets-its-own-build.

Any definition of "majority of stable user agents" is not realistic... They should make that determination on a per-feature/per-browser basis, not based on a committee agreement of what a year represents, or what the majority of stable user agents represent.

I agree. This is the most hand-wavey part of the design, and will probably make it more difficult for devs to determine what needs to be done to generate an LCD build.

But what if we change it a bit? Instead of making the browsers vendors (or any standards body) determine what needs to be natively supported for Chrome to include "2018" support, we make it the build year. Every Chrome/Safari/Firefox/X built in 2018 advertises "2018". The community can then decide what 2018 means in an LCD build.

Eg, Chrome XX shipped in 2018 and advertises "2018". Firefox YY shipped in 2018 and advertises "2018". We know YY supports some feature (say, Private Fields) that XX doesn't. So, we know that if we want to ship a 2018 build that all 2018 browsers can understand, we need to transpile Private Fields. If Chrome adds support for Private Fields in 2018, the transpile is still necessary, because the 2018 LCD doesn't support it. By the time 2019 rolls around, everything supports Private Fields, and we know we no longer need to transpile it in the 2019 LCD.

Script loading is complicated and has many entry points. The exploration doc tries to thread this through <script> and , but misses (in roughly descending order of importance) new Worker(), import statements, import() expressions, service workers, the varied-and-growing worklet loading entry points, importScripts(), and javascript: URLs.

The 2018 build should be responsible for only loading 2018 build files. The 2017 build should be responsible for only loading 2017 build files. What's needed is the way to load the build's entry point, not the way for the build to load other files from the same build.