Skip to content

Investigate replacing qs #5723

Closed
Closed

Description

@TheDevMinerTV

Pros

Cons

  • fast-querystring requires at least Node 8:
    • it uses object destructuring and trailing commas
      • it might be possible to convince the author to remove the trailing commas, then it would be usable with Node 6
    • Node 8 has been out of support for more than 5 years now (support stopped on Jan 1st 2019, according to endoflife.date/nodejs)
    • Node 0.10 is even older, I can't even find an EOL date for it
  • fast-querystring doesn't support setting prototypes
    • this is unsafe either way, though it can probably get released in express v5

Notes

  • Slightly better performance:
    • this test was done using the benchmark folder in this repo and just replacing the require call to fast-querystring on a idling 7950X running Node v8.17.0
  • It might be useful to maybe port to using URLSearchParams which has been supported since Node 10

Activity

  1. ctcpip commented on Jun 24, 2024

    @ctcpip
    Member

    It can't be done in Express 4; as you correctly noted, it would require a higher minimum version of Node. Express 5 will require a minimum of Node 18, and if a dependency is no longer required, I would rather just drop it. However, a significant perf difference could be a compelling enough reason to use it over built-ins.

  2. TheDevMinerTV commented on Jun 24, 2024

    @TheDevMinerTV
    Author

    Sounds good, do you want me to make a PR to use the built-in URLSearchParams?

  3. ctcpip commented on Jun 24, 2024

    @ctcpip
    Member

    @TheDevMinerTV if you want to make a PR targeting 5.x, that would be great

  4. TheDevMinerTV commented on Jun 24, 2024

    @TheDevMinerTV
    Author

    Will do ^^
    Can you assign me to this issue?

  5. krzysdz commented on Jun 25, 2024

    @krzysdz
    Contributor

    In #4990 it was suggested to replace querystring (built-in Node module) with URLSearchParams.
    Now Express offers 2 "query parser" values:

    1. "simple" - using querystring, default since 5.0-beta.1
    2. "extended" - using qs, default in 4.x

    Does replacing both of them with URLSearchParams make sense? Doesn't qs have more features than URLSearchParams?

  6. TheDevMinerTV commented on Jun 25, 2024

    @TheDevMinerTV
    Author

    URLSearchParams is the standard way of doing query param parsing nowadays. We should probably stick to that, especially since it will work in all runtimes because it's in the ECMAscript URL spec. Though, using node:querystring would also be fine, I guess.

  7. ctcpip commented on Jun 25, 2024

    @ctcpip
    Member

    your point is valid, but note that URLSearchParams is part of the URL spec, not ECMAScript

    node:querystring is not sufficient because it doesn't handle deep nesting

  8. TheDevMinerTV commented on Jun 25, 2024

    @TheDevMinerTV
    Author

    URLSearchParams doesn't seem to parse /?foo[0][bar]=baz&foo[0][fizz]=buzz&foo[]=done! into {"foo":[{"bar":"baz","fizz":"buzz"},"done!"]}, but rather as {"foo[0][bar]":"baz","foo[0][fizz]":"buzz","foo[]":"done!"}. So it doesn't support automatic array parsing (which can be worked around, but that might be vulnerable). This was also described here: sindresorhus/got#1559 (comment)

    Edit: fast-querystring also doesn't implement this.

    Edit: This is very likely because the spec never mentions arrays. I'm not sure how to proceed here. Should we implement (or pull in a package for) the array parsing algorithm, if so, do we use the comma or the index syntax. Or should we drop the feature? IMO we should implement this ourselves with the index syntax.

  9. ctcpip commented on Jun 25, 2024

    @ctcpip
    Member

    sounds like we should continue to use qs

  10. TheDevMinerTV commented on Jun 25, 2024

    @TheDevMinerTV
    Author

    There seems to be picoquery which is ~40KB and seems to support the different modes but seems to fails with parsing on some stuff because it only supports on syntax at a time. I've opened an issue on their repository ^

  11. krzysdz commented on Jun 25, 2024

    @krzysdz
    Contributor

    I believe it would be great to provide a URLSearchParams based interface by default and offer pluggable support for qs or other parsers to allow users to decide for themselves without forcing qs on them.

    There is a query parser setting, which allows users to configure any query parser they want. Since 5.0 it defaults to "simple" (which may move from node:querystring to URLSearchParams - #4990 (comment); PRs welcome), with qs left as the "extended" option for compatibility.

    Completely removing qs and the "extended" option probably would require deprecating it first.

  12. 43081j commented on Jun 25, 2024

    @43081j

    it probably does make sense that the query parser setting can be simple or a function (in some future breaking version)

    since then you can use whatever query parser you like by passing it in, without having to bloat express itself with such a library

  13. changed the title [-]Investigate replacing `qs` with `fast-querystring`[/-] [+]Investigate replacing `qs`[/+] on Jun 25, 2024
  14. TheDevMinerTV commented on Jun 25, 2024

    @TheDevMinerTV
    Author

    Removing qs from express, assuming the 29,679,417 downloads / week from NPMJS.com is correct, would save over 75 TB / month (or over ~900 TB / year). body-parser also depends on qs and that's another ~30 million downloads / week.
    (correct me if my math is wrong though: https://www.wolframalpha.com/input?i=%2829679417+%2F+7++*+30+*+12%29+*+594+KB)

    @ctcpip will express v5 be ESM-only?

  15. ctcpip commented on Jun 25, 2024

    @ctcpip
    Member

    No, it will continue to be CJS-only

  16. TheDevMinerTV commented on Jun 25, 2024

    @TheDevMinerTV
    Author

    Alright, so it we'd have to use picoquery v1.*.*, though I'm willing to make a PR for the project to support both, ESM and CJS, at the same time. @43081j lmk if you want me to work on that.

  17. zmzlois commented on Jun 25, 2024

    @zmzlois
  18. 43081j commented on Jun 25, 2024

    @43081j

    Alright, so it we'd have to use picoquery v1.*.*, though I'm willing to make a PR for the project to support both, ESM and CJS, at the same time. @43081j lmk if you want me to work on that.

    It was an active choice to keep them separate (1.x cjs, 2.x esm - at the next breaking change, esm only)

    We will be backporting non breaking features and fixes to 1.x meanwhile

  19. anonrig commented on Jul 18, 2024

    @anonrig
  20. wesleytodd commented on Jul 18, 2024

    @wesleytodd
    Member

    I dont think we will be replacing this module, at least not any time soon. We are pushing to get v5 our the door and this is not on the list for that. Just want to temper folks expectations since the new group of maintainers, while no longer just one person, is still small and we have a lot of catch up we are doing.

    Additionally, with #5731 and v5 landing the default swap I would not be surprised if we went the path of removing all these as direct deps and go to them as 3rd party configuration additions instead for v6. Not promising anything, but also I don't want folks to waste a lot of time bikeshedding this issue for no reason.

  21. PuruVJ commented on Jul 23, 2024

    @PuruVJ

    I just made neoqs, fully backward compatible and forever going to follow qs(even in terms of code). It doesn't aim to be faster or drastically smaller. It just aims to do exact same as qs without adding additional deps.

    I'd suggest URLSearchParams as the endgame, but in the meanwhile, neoqs could fill in the gap without being disruptive 😄

    https://x.com/puruvjdev/status/1815658973946626209

  22. locked and limited conversation to collaborators on Jul 23, 2024
  23. converted this issue into a discussion #5783 on Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions