How do I preload the R packages in a node app?

[ColinFay]

Hey,

Given the following express app:

const app = require('express')()
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();
  console.log("webR is ready");
  await webR.installPackages(['attempt'])
  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })
})();


app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR('tools::toTitleCase("hello from r!")');
  let output = await result.toJs();
  res.send(output.values)
});

Every time I relaunch the app, a package installation is performed:

$ node index.js 
webR is ready
Downloading webR package: rlang

Downloading webR package: attempt

http://localhost:3000

$ node index.js
webR is ready
Downloading webR package: rlang

Downloading webR package: attempt

http://localhost:3000

It works but I feel like there should be a better way to do it, because :

• I feel like there is no good reason for downloading the packages every time the app is launched
• If I have a lot of packages, it would take a lot of time every time
• I can't deploy this on a service that has no access to internet

Wishlist :

I would love to have something that allows (in the spirit of node_modules) to pre-install the packages on my machine and then load them at runtime.

(async () => {
 webR = new WebR();
  await webR.init();
  await webR.loadPackage( path.join(__dirname, '/r-packages/attempt'))
})();

Random thoughts:

I could do it myself by downloading the binaries and toying the WebRFS interface but :

• I would have to do the dependency tree resolution myself, while webr::install() and install.packages() already have everything set up.
• It might be a bit complex to read the file/folder tree and replicate it on webr filesystem.

Happy to discuss this idea :)

Cheers

[ColinFay]

Some update on this: I've drafted (what seems to be) a working code for doing this.

I will release it in an npm package for now but I would be happy to discuss potential integration in the webR API if it sounds like a usefull tool :)

I'm currently writing a blogpost about this experiment, will keep you updated.

[georgestagg]

Hi Colin,

Sorry for the slow reply, I've been working on other things recently.

I think you're right in that at the moment the way to handle this would be to load the package binaries into the webR VFS -- I agree, this is not ideal.

Alternatively, it is also possible to build webR from source and provide a library directory to be written into the initial root filesystem. Though, again, this is not ideal as building webR from source can be a long process the first time (due to building a patched LLVM/flang).

I think the right way to handle this is to use a local CRAN-like repo containing webR binaries on disk, with a PACKAGES file acting as an index. This allows us to make use of R's built-in dependency resolution. This would involve dropping the package binaries into a directory and running write_PACKAGES().

This already works mostly OK in the current version of webR. The URL pointing to such a cran-like repo can be set at webR's startup using the repoUrl option when starting webR.

However, the repoUrl option does not currently work with a relative path for loading packages from disk when running under Node.js. I consider that a bug. When that is fixed in the future, and package repositories can be loaded from a local disk when running under node, I think that will be a good solution to the problem. One would simply set something like repoUrl: './pkgs' at startup, then run installPackages(), or webr::install() to install from the local disk and avoid hitting the net.

[ColinFay]

Thanks for the comment :)

My current approach is:

• before loading Node, I download the binaries and untar them in the local folder
• copy the dir tree (folder and files) at load time from ./webr_packages to webR /usr/lib/R/library with webR.FS

Will share the code this week after polishing the post :)

Another idea, but I'm not sure if this is doable with webR: could we "mount" a local folder to the webR filesystem, in the same way that we mount a docker volume? That way, I could pre-install on the local dir, and use this local dir as a library for webR

[ColinFay]

Reading the wasm doc, there seems to be an FS.mount https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.mount

[georgestagg]

could we "mount" a local folder to the webR filesystem

I believe using FS.mount() should indeed work in that way, but only on Node where one can use NODEFS. I guess on the web a library could be built with Emscripten's tools/file_packager and mounted in a similar way with FS.mount().

webR.FS does not currently expose mount() over the communication channel, but it should be possible. It might be nice to just directly mount a library rather than installing packages one by one from a repo. I'll think about it some more and look into it when I can.

[ColinFay]

Hey,

Some update about my current explorations: https://colinfay.me/preloading-your-r-packages-in-webr-in-an-express-js-api/

Happy to discuss this :)