From: "martinemde (Martin Emde) via ruby-core" Date: 2023-10-16T20:46:47+00:00 Subject: [ruby-core:115065] [Ruby master Feature#19744] Namespace on read Issue #19744 has been updated by martinemde (Martin Emde). Besides my position on integrating with RG&B, I wanted to add my thoughts about the feature itself separately and how it might end up being used in gems. ## Usage in gems I immediately imagine this feature being picked up by gems (again, please ignore whether or not bundler supports multiple versions of gems). ### Gem Namespaces Most gems want to isolate their own namespace and not pollute the namespaces into which they are required. Right now it is _customary_ but not required that a gem defines a top level module that matches the gem name, following a naming convention. These modules are "weak" namespaces compared to the proposed namespace because I can't contain anything I require within the namespace as well. If namespaces had always been available, modules and namespaces would be the same thing. We would just consider any require/load statement within a module to load nested into that module. `require` on Kernel is global, `require` within a module is namespaced. You might even expect places that currently require a file within a method or module to instead write `Kernel.require`. Gems may, with this feature, become more "strongly" namespaced, like the following: ``` # lib/my_gem.rb namespace MyGem require "oj" autoload :Feature, "my_gem/feature.rb" # maybe this feature uses json. end ``` Now this gem doesn't need to worry about which json library is already loaded or if this gem has polluted the global namespace by requiring a certain json library. Gems like this become self contained "cells", that provide an interface and functionality but don't alter any code outside of their own namespace, even when they require external libraries. ### Patching/plugins If my_gem wanted to add features to something outside of itself, if it was a plugin for another gem, then it needs to interact with an external namespace. In that case I might do something like this: ``` require "rack" # same code as above in lib/my_gem.rb Rack.include(MyGem::Feature) # `::Rack` should also work the same here ``` Just looks like normal code, but I bring it up because in means I should be able to include a module from one namespace into a module from another namespace. That module may even have access to a specific set of libraries that aren't in the destination module. Is support for that planned? ### Breaking patches out of a namespace If I require a namespaced library using a namespace, I could control what the top level name of the lib, as proposed, but then I can't still use the patch to Rack that I wrote above. Assuming MyGem is a name conflict for me, I could do the following. ``` namespace Libs # How do I add the Rack patch again? Like this? I don't have a good idea for how this would work. Rack == ::Rack # but how does the external require come into this namespace. require("my_gem") # Maybe end Rack = Libs::Rack # This feels more natural, but still weird and wouldn't work for multiple patches. ``` ### Top level ::MyGem Not being able to use `::MyGem` seems like a big problem. Without a solution for accessing "the top level of my namespace" and "the top level Kernel namespace", I expect there will be some unfixable module references. There should be a syntax for how this can be done. Within MyGem, I would expect ::MyGem to access the gem's namespace declared in the code above. If MyGem was required into another namespace by an external require, that should be invisible to this gem and ::MyGem should not change how it works. When you mix the patching of rack with a nested `outside_namespace.require("my_gem")`, then we must expect that rack will also be required into that namespace and patched inside the namespace. ## Conclusion Appreciate your time to read my exploration of this feature. Unless we do something to prevent it, I think gems will do this because it makes sense to do so. Is this going to work? Will it break everything? Thank you! ---------------------------------------- Feature #19744: Namespace on read https://bugs.ruby-lang.org/issues/19744#change-104940 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- # What is the "Namespace on read" This proposes a new feature to define virtual top-level namespaces in Ruby. Those namespaces can require/load libraries (either .rb or native extension) separately from the global namespace. Dependencies of required/loaded libraries are also required/loaded in the namespace. ### Motivation The "namespace on read" can solve the 2 problems below, and can make a path to solve another problem: The details of those motivations are described in the below section ("Motivation details"). #### Avoiding name conflicts between libraries Applications can require two different libraries safely which use the same module name. #### Avoiding unexpected globally shared modules/objects Applications can make an independent/unshared module instance. #### (In the future) Multiple versions of gems can be required Application developers will have fewer version conflicts between gem dependencies if rubygems/bundler will support the namespace on read. ### Example code with this feature ```ruby # your_module.rb module YourModule end # my_module.rb require 'your_module' module MyModule end # example.rb namespace1 = NameSpace.new namespace1.require('my_module') #=> true namespace1::MyModule #=> #::MyModule (or #::MyModule ?) namespace1::YourModule # similar to the above MyModule # NameError YourModule # NameError namespace2 = NameSpace.new # Any number of namespaces can be defined namespace2.require('my_module') # Different library "instance" from namespace1 require 'my_module' # require in the global namespace MyModule.object_id != namespace1::MyModule.object_id #=> true namespace1::MyModule.object_id != namespace2::MyModule.object_id ``` The required/loaded libraries will define different "instances" of modules/classes in those namespaces (just like the "wrapper" 2nd argument of `Kernel.load`). This doesn't introduce compatibility problems if all libraries use relative name resolution (without forced top-level reference like `::Name`). # "On read": optional, user-driven feature "On read" is a key thing of this feature. That means: * No changes are required in existing/new libraries (except for limited cases, described below) * No changes are required in applications if it doesn't need namespaces * Users can enable/use namespaces just for limited code in the whole library/application Users can start using this feature step by step (if they want it) without any big jumps. ## Motivation details This feature can solve multiple problems I have in writing/executing Ruby code. Those are from the 3 problems I mentioned above: name conflicts, globally shared modules, and library version conflicts between dependencies. I'll describe 4 scenarios about those problems. ### Running multiple applications on a Ruby process Modern computers have many CPU cores and large memory spaces. We sometimes want to have many separate applications (either micro-service architecture or modular monolith). Currently, running those applications require different processes. It requires additional computation costs (especially in developing those applications). If we have isolated namespaces and can load applications in those namespaces, we'll be able to run apps on a process, with less overhead. (I want to run many AWS Lambda applications on a process in isolated namespaces.) ### Running tests in isolated namespaces Tests that require external libraries need many hacks to: * require a library multiple times * require many different 3rd party libraries into isolated spaces (those may conflict with each other) Software with plugin systems (for example, Fluentd) will get benefit from namespaces. In addition to it, application tests can avoid unexpected side effects if tests are executed in isolated namespaces. ### Safely isolated library instances Libraries may have globally shared states. For example, [Oj](https://github.com/ohler55/oj) has a global `Obj.default_options` object to change the library behavior. Those options may be changed by any dependency libraries or applications, and it changes the behavior of `Oj` globally, unexpectedly. For such libraries, we'll be able to instantiate a safe library instance in an isolated namespace. ### Avoiding dependency hells Modern applications use many libraries, and those libraries require much more dependencies. Those dependencies will cause version conflicts very often. In such cases, application developers should resolve those by updating each libraries, or should just wait for the new release of libraries to conflict those libraries. Sometimes, library maintainers don't release updated versions, and application developers can do nothing. If namespaces can require/load a library multiple times, it also enables to require/load different versions of a library in a process. It requires the support of rubygems, but namespaces should be a good fundamental of it. ## Expected problems ### Use of top-level references In my expectation, `::Name` should refer the top-level `Name` in the global namespace. I expect that `::ENV` should contain the environment variables. But it may cause compatibility problems if library code uses `::MyLibrary` to refer themselves in their deeply nested library code. ### Additional memory consumption An extension library (dynamically linked library) may be loaded multiple times (by `dlopen` for temporarily copied dll files) to load isolated library "instances" if different namespaces require the same extension library. That consumes additional memory. In my opinion, additional memory consumption is a minimum cost to realize loading extension libraries multiple times without compatibility issues. This occurs only when programmers use namespaces. And it's only about libraries that are used in 2 or more namespaces. ### The change of `dlopen` flag about extension libraries To load an extension library multiple times without conflicting symbols, all extensions should stop sharing symbols globally. Libraries referring symbols from other extension libraries will have to change code & dependencies. (About the things about extension libraries, [Naruse also wrote an entry](https://naruse.hateblo.jp/entry/2023/05/22/193411).) # Misc The proof-of-concept branch is here: https://github.com/tagomoris/ruby/pull/1 It's still work-in-progress branch, especially for extension libraries. -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/