Hi Paolo,
For fixes and things needed for qemu I'm likely to release point updates per-fix, but it hasn't really been a necessity yet because of the code duplication. Even so, it's in a pretty stable state that doesn't appear to need much maintainer bandwidth at the moment.
I'd like to finally drop the in-tree mirror so I don't have to manually sync changes back and forth. The tough bit is that the in-tree python tooling would now have its first mandatory "external" dependency. i.e. qemu.git/python/setup.cfg would start requiring e.g. qemu.qmp == 0.0.3.
That'd mean that scripts and tooling in-tree that rely on import path hacks to utilize the in-tree packages could start breaking if the dependencies aren't met. To ensure they're met, we need to pip install the in-tree packages.
(To my knowledge: qom tools, iotests, and potentially other things. Maybe the VM tests? The Avocado tests already handle this installation via the Makefile, but this approach isn't suitable/scalable for everything in-tree that needs it.)
However, doing so unconditionally even when we have no intention to run tests/scripts means a 2-3 second delay on every last configure invocation - something we worked extremely hard to avoid.
Is it time to add a "testing" group to pythondeps.toml?
I imagine I need to do these things:
1. vendor the qemu.qmp wheel. simple enough.
2. add qemu.qmp as a requirement to python/setup.cfg. I like to do this to ensure that the python stuff in this directory is as "normal" as possible, even if we handle venv config/instantiation elsewhere; e.g. I've documented you can just "pip install" this directory when needed for years and years now.
3. add a testing group to pythondeps.toml; I'm not currently sure if I can add the in-tree source package by path or if I'll need to handle it "specially" like I did for avocado. I may or may not need to specify the qemu.qmp dependency explicitly here, based on how mkvenv tries to source dependencies. I'll test this and report back.
4. Figure out when the testing group should be ensured during initial configuration. My hunch is "almost always", but if it cannot be configured for some reason, it should be non-fatal to the build.
(But at the moment I don't know why, if I vendor the wheel, it should ever fail except for mkvenv toolchain failures - unforeseen problems with the way we use venv/pip/setuptools/distlib et al. Still, maybe future requirements may be less promise-able and we should perhaps be flexible here.)
Do we want --enable-tests and --disable-tests, with a default to "enable if possible", like we do for docs?
Or should I just hit the heavy hammer and force it always like for meson?
(If I do it always, I'm just concerned about the installation speed because unlike meson, we will *never* already have the package! It needs to be installed from the sourcs tree as a matter of course.)
5. Figure out how to perform a delayed initialization of the testing group to achieve just-when-needed testing support. I believe this is important for running iotests outside of the make system, which last I checked with Kevin and Hannah et al was a non-negotiable argument.
(IIRC it should be simple enough to have the iotests environment bootstrapper make a call to mkvenv ensure-group as long as it's fast in the do-nothing case. I'm less sure about other scripts that may require this env prepping. I'll audit that.)
Thoughts? Lets schedule a call to discuss if we can. (Anyone else interested can join if they want.)
~~js