Do not download all wheels of pip when the release history cannot be acquired

[dgelessus]

Issue

Occasionally when I create a virtual environment, virtualenv spawns three background Python processes that continue running even after virtualenv has exited. These background processes consume about 60-80 % CPU each and don't seem to exit on their own. Each process also has another Python subprocess that seems to be restarted every few seconds.

This doesn't happen on every virtualenv run - in particular, after it happens once (and I terminate all of the background Python processes), it doesn't happen again when I re-run virtualenv. But if I wait a while and then run virtualenv again, the problem reappears.

Here is a part of the command lines of these processes. The command lines are incomplete because I copied them from the output of htop and my terminal window wasn't wide enough. Because the issue is difficult to reproduce I haven't been able to get a better copy of the command lines yet.

eworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /
eworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /
eworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /
eworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('setuptools', '3.8', '/Li
eworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('wheel', '3.8', '/Library
eworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('pip', '3.8', '/Library/F

It looks like this has something to do with the new "periodic update" feature, which would explain why the problem only reappears after waiting for a while. Although according to virtualenv --help the periodic update should only happen every 14 days, and I'm having this problem much more frequently than that (it reappears after a day at most).

Environment

OS: macOS 10.14.6
Python: CPython 3.8.3 (macOS installer from Python.org)

`pip list` of the host python where `virtualenv` is installed

$ python3 -m pip list
Package            Version    Location
------------------ ---------- -------------------------------------------------------
appdirs            1.4.4
appnope            0.1.0
attrs              19.3.0
backcall           0.2.0
beautifulsoup4     4.9.1
bleach             3.1.5
certifi            2020.6.20
cffi               1.14.0
chardet            3.0.4
cloudpickle        1.3.0
colorama           0.4.3
colorclass         2.2.0
cryptography       2.9.2
cycler             0.10.0
decorator          4.4.2
defusedxml         0.6.0
distlib            0.3.1
docutils           0.16
easygui            0.98.1
entrypoints        0.3
filelock           3.0.12
future             0.18.2
gym                0.17.2
gym-minigrid       1.0.1
html5lib           1.1
idna               2.10
ipykernel          5.3.0
ipython            7.16.1
ipython-genutils   0.2.0
ipywidgets         7.5.1
jedi               0.17.1
Jinja2             2.11.2
jsonschema         3.2.0
jupyter-client     6.1.5
jupyter-core       4.6.3
keyring            21.2.1
kiwisolver         1.2.0
lxml               4.5.1
MarkupSafe         1.1.1
matplotlib         3.2.2
mistune            0.8.4
msoffcrypto-tool   4.10.2
nbconvert          5.6.1
nbformat           5.0.7
notebook           6.0.3
numpy              1.19.0
olefile            0.46
oletools           0.55.1
packaging          20.4
pandocfilters      1.4.2
parso              0.7.0
pcodedmp           1.2.6
pexpect            4.8.0
pickleshare        0.7.5
Pillow             7.2.0
pip                20.1.1
pkginfo            1.5.0.1
pluggy             0.13.1
prometheus-client  0.8.0
prompt-toolkit     3.0.5
ptyprocess         0.6.0
py                 1.9.0
pycparser          2.20
pyglet             1.5.0
Pygments           2.6.1
pyparsing          2.4.7
pyrsistent         0.16.0
python-dateutil    2.8.1
PyYAML             5.3.1
pyzmq              19.0.1
readme-renderer    26.0
regex              2020.6.8
requests           2.24.0
requests-toolbelt  0.9.1
rfc3986            1.4.0
rsrcfork           1.7.1.dev0 [...]/python-rsrcfork
scipy              1.5.0
Send2Trash         1.5.0
setuptools         47.3.1
six                1.15.0
soupsieve          2.0.1
terminado          0.8.3
testpath           0.4.4
toml               0.10.1
tornado            6.0.4
tox                3.16.1
tqdm               4.47.0
traitlets          4.3.3
twine              3.2.0
typedstream        0.0.1.dev0 [...]/python-typedstream
urllib3            1.25.9
virtualenv         20.0.25
wcwidth            0.2.5
webencodings       0.5.1
wheel              0.34.2
widgetsnbextension 3.5.1

Output of the virtual environment creation

$ python3 -m virtualenv -vvv --with-traceback testenv
138 setup logging to NOTSET [DEBUG report:42]
166 find interpreter for spec PythonSpec(path=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3) [INFO builtin:44]
167 proposed PythonInfo(spec=CPython3.8.3.final.0-64, exe=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3, platform=darwin, version='3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
167 accepted PythonInfo(spec=CPython3.8.3.final.0-64, exe=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3, platform=darwin, version='3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
173 filesystem is not case-sensitive [DEBUG info:28]
236 create virtual environment via CPython3Posix(dest=/Users/david/Downloads/work/testenv, clear=False, global=False) [INFO session:52]
237 write /Users/david/Downloads/work/testenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
237 	home = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
237 	implementation = CPython [DEBUG pyenv_cfg:38]
237 	version_info = 3.8.3.final.0 [DEBUG pyenv_cfg:38]
237 	virtualenv = 20.0.25 [DEBUG pyenv_cfg:38]
237 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
237 	base-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
237 	base-exec-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
237 	base-executable = /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 [DEBUG pyenv_cfg:38]
239 remove file /Users/david/Downloads/work/testenv/bin/python [DEBUG _sync:38]
239 symlink /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 to /Users/david/Downloads/work/testenv/bin/python [DEBUG _sync:44]
240 create virtualenv import hook file /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/_virtualenv.pth [DEBUG api:94]
241 create /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/_virtualenv.py [DEBUG api:97]
242 ============================== target debug ============================== [DEBUG session:54]
242 debug via /Users/david/Downloads/work/testenv/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:209]
242 {
  "sys": {
    "executable": "/Users/david/Downloads/work/testenv/bin/python",
    "_base_executable": "/Users/david/Downloads/work/testenv/bin/python",
    "prefix": "/Users/david/Downloads/work/testenv",
    "base_prefix": "/Library/Frameworks/Python.framework/Versions/3.8",
    "real_prefix": null,
    "exec_prefix": "/Users/david/Downloads/work/testenv",
    "base_exec_prefix": "/Library/Frameworks/Python.framework/Versions/3.8",
    "path": [
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python38.zip",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload",
      "/Users/david/Downloads/work/testenv/lib/python3.8/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]",
  "makefile_filename": "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/config-3.8-darwin/Makefile",
  "os": "<module 'os' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py'>",
  "site": "<module 'site' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site.py'>",
  "datetime": "<module 'datetime' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/datetime.py'>",
  "math": "<module 'math' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/math.cpython-38-darwin.so'>",
  "json": "<module 'json' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py'>"
} [DEBUG session:55]
308 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/david/Library/Application Support/virtualenv) [INFO session:59]
313 got embed update of distribution setuptools from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]
314 got embed update of distribution pip from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]
314 got embed update of distribution wheel from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]
319 wrote embed update of distribution pip at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:159]
319 wrote embed update of distribution setuptools at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:159]
319 wrote embed update of distribution wheel at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:159]
324 triggered periodic upgrade of pip==20.1.1 (for python 3.8) via background process having PID 57954 [INFO periodic_update:182]
326 triggered periodic upgrade of wheel==0.34.2 (for python 3.8) via background process having PID 57955 [INFO periodic_update:182]
332 got embed update of distribution wheel from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]
333 triggered periodic upgrade of setuptools==47.3.1 (for python 3.8) via background process having PID 57956 [INFO periodic_update:182]
335 got embed update of distribution setuptools from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]
336 got embed update of distribution pip from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]
338 install wheel from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
340 install setuptools from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-47.3.1-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
347 Attempting to acquire lock 4451582448 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [DEBUG filelock:270]
350 Lock 4451582448 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [INFO filelock:274]
341 Attempting to acquire lock 4451582256 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:270]
353 Lock 4451582256 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:274]
350 install pip from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.1.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
362 Attempting to acquire lock 4451582976 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:270]
352 Attempting to release lock 4451582448 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [DEBUG filelock:315]
365 Lock 4451582448 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [INFO filelock:318]
355 Attempting to release lock 4451582256 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:315]
365 Lock 4451582256 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:318]
365 Lock 4451582976 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:274]
366 Attempting to release lock 4451582976 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:315]
368 Lock 4451582976 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:318]
377 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/easy_install.py to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/easy_install.py [DEBUG _sync:52]
387 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:52]
410 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools-47.3.1.virtualenv to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/setuptools-47.3.1.virtualenv [DEBUG _sync:52]
437 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/wheel [DEBUG _sync:52]
454 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/setuptools [DEBUG _sync:52]
481 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:52]
499 generated console scripts wheel3 wheel wheel-3.8 [DEBUG base:53]
555 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/pip [DEBUG _sync:52]
601 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/pkg_resources to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/pkg_resources [DEBUG _sync:52]
630 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools-47.3.1.dist-info to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/setuptools-47.3.1.dist-info [DEBUG _sync:52]
648 generated console scripts easy_install3 easy_install easy_install-3.8 [DEBUG base:53]
1288 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.virtualenv to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/pip-20.1.1.virtualenv [DEBUG _sync:52]
1291 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.dist-info to /Users/david/Downloads/work/testenv/lib/python3.8/site-packages/pip-20.1.1.dist-info [DEBUG _sync:52]
1305 generated console scripts pip pip3.8 pip-3.8 pip3 [DEBUG base:53]
1306 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:64]
1314 write /Users/david/Downloads/work/testenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
1314 	home = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1314 	implementation = CPython [DEBUG pyenv_cfg:38]
1314 	version_info = 3.8.3.final.0 [DEBUG pyenv_cfg:38]
1314 	virtualenv = 20.0.25 [DEBUG pyenv_cfg:38]
1314 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
1314 	base-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1315 	base-exec-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1315 	base-executable = /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 [DEBUG pyenv_cfg:38]
1315 created virtual environment CPython3.8.3.final.0-64 in 1179ms
  creator CPython3Posix(dest=/Users/david/Downloads/work/testenv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/david/Library/Application Support/virtualenv)
    added seed packages: pip==20.1.1, setuptools==47.3.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:18]
-> exit 0

[gaborbernat]

This is expected, see https://virtualenv.pypa.io/en/latest/user_guide.html#wheels. though it shouldn't use much CPU, can you post their logs? 🤔

[dgelessus]

Where would I look for the relevant logs? I can't find anything under ~/Library/Logs or ~/Library/Application Support/virtualenv (or .../pip).

If it's of any use: it just happened to me again, and this time I let the processes run for a short while. Here are the relevant lines from ps, including the full command lines:

58469 ttys001    0:00.45 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('pip', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.1.1-py2.py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58470 ttys001    0:00.47 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('wheel', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.34.2-py2.py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58471 ttys001    0:00.42 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('setuptools', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-47.3.1-py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58569 ttys001    0:02.19 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house setuptools<47.0.0
58573 ttys001    0:01.63 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house pip<19.1
58576 ttys001    0:01.02 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house wheel<0.31.0

Interestingly, if I wait for a few seconds and run ps again, I can see that the command lines of the last three processes change:

58469 ttys001    0:00.64 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('pip', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.1.1-py2.py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58470 ttys001    0:00.68 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('wheel', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.34.2-py2.py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58471 ttys001    0:00.51 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('setuptools', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-47.3.1-py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58675 ttys001    0:05.43 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house setuptools<45.3.0
58679 ttys001    0:02.55 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house wheel<0.10.2
58681 ttys001    0:00.89 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house pip<8.0.0

And after a while, the processes even finish naturally:

58469 ttys001    0:00.78 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('pip', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.1.1-py2.py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58471 ttys001    0:00.60 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('setuptools', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-47.3.1-py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58738 ttys001    0:02.40 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house setuptools<42.0.1
58739 ttys001    0:01.41 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house pip<6.0.4

58471 ttys001    0:00.66 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -c from virtualenv.seed.wheels.periodic_update import do_update;do_update('setuptools', '3.8', '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-47.3.1-py3-none-any.whl', '/Users/david/Library/Application Support/virtualenv', [], True)
58776 ttys001    0:01.20 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -m pip download --disable-pip-version-check --only-binary=:all: --no-deps --python-version 3.8 -d /Users/david/Library/Application Support/virtualenv/wheel/house setuptools<41.2.0

It looks like for some reason virtualenv is downloading every available version of pip, setuptools and wheel, by calling pip download repeatedly, each time with a lower version constraint than the previous call. And if I look under ~/Library/Application Support/virtualenv/wheel/house, there are indeed wheels for every pip/setuptools/wheel version.

[gaborbernat]

That's the expected behaviour here. The only question remain to be seen is that why it keeps retrying all the time and not just once every two weeks. Can you post the content of /Users/bgabor8/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json? Also, can you access
https://pypi.org/pypi/pip/json?

For what it's worth you can disable this behaviour by using the https://virtualenv.pypa.io/en/latest/cli_interface.html#no-periodic-update flag, that can be also enabled globally by setting VIRTUALENV_NO_PERIODIC_UPDATE=true env var, or no_periodic_update = true in ~/Library/Application Support/virtualenv/virtualenv.ini

Thanks!

[dgelessus]

pip.json just contains this:

{
  "completed": null,
  "periodic": true,
  "started": "2020-07-03T16:10:39.723035Z",
  "versions": []
}

I am able to access https://pypi.org/pypi/pip/json, both in the browser and from Python using requests.get, and the returned JSON data contains information about all pip releases.

That's the expected behaviour here.

To clarify - is it expected that virtualenv downloads and keeps wheels for every version of pip/setuptools/wheel that was ever released? The total size of all of those wheels is about 250 MB, which is not huge, but I don't think virtualenv should use that much bandwidth/disk space without asking. I'm not sure what the old versions are useful for either - shouldn't you only need the latest version that supports the environment's Python version?

[gaborbernat]

That's a problem, seems we start acquiring the information but never finish. If we'd do the versions would not be empty and completed would have a value. Can you delete that pip.json file and then run:

env _VIRTUALENV_PERIODIC_UPDATE_INLINE=1 virtualenv --clear env --with-traceback -vvv

This should now put the update logs on the output, and do the update inline rather than background process.

To clarify - is it expected that virtualenv downloads and keeps wheels for every version of pip/setuptools/wheel that was ever released?

We go back in history until we find one that has been released 28 days ago, or previous minor and is compatible with the current Python version... So should not take all, just the last few versions.

[dgelessus]

I didn't even have to delete the pip.json, the periodic update ran again automatically for all three packages. Here's the output:

Output

$ env _VIRTUALENV_PERIODIC_UPDATE_INLINE=1 virtualenv --clear env --with-traceback -vvv
144 setup logging to NOTSET [DEBUG report:42]
177 find interpreter for spec PythonSpec(path=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3) [INFO builtin:44]
178 proposed PythonInfo(spec=CPython3.8.3.final.0-64, exe=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3, platform=darwin, version='3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
178 accepted PythonInfo(spec=CPython3.8.3.final.0-64, exe=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3, platform=darwin, version='3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
182 filesystem is not case-sensitive [DEBUG info:28]
333 create virtual environment via CPython3Posix(dest=/Users/david/Downloads/work/env, clear=True, global=False) [INFO session:52]
333 create folder /Users/david/Downloads/work/env/bin [DEBUG _sync:25]
334 create folder /Users/david/Downloads/work/env/lib/python3.8/site-packages [DEBUG _sync:25]
334 write /Users/david/Downloads/work/env/pyvenv.cfg [DEBUG pyenv_cfg:34]
334 	home = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
335 	implementation = CPython [DEBUG pyenv_cfg:38]
335 	version_info = 3.8.3.final.0 [DEBUG pyenv_cfg:38]
335 	virtualenv = 20.0.25 [DEBUG pyenv_cfg:38]
335 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
335 	base-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
335 	base-exec-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
335 	base-executable = /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 [DEBUG pyenv_cfg:38]
336 symlink /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 to /Users/david/Downloads/work/env/bin/python [DEBUG _sync:44]
337 create virtualenv import hook file /Users/david/Downloads/work/env/lib/python3.8/site-packages/_virtualenv.pth [DEBUG api:94]
337 create /Users/david/Downloads/work/env/lib/python3.8/site-packages/_virtualenv.py [DEBUG api:97]
338 ============================== target debug ============================== [DEBUG session:54]
338 debug via /Users/david/Downloads/work/env/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:209]
338 {
  "sys": {
    "executable": "/Users/david/Downloads/work/env/bin/python",
    "_base_executable": "/Users/david/Downloads/work/env/bin/python",
    "prefix": "/Users/david/Downloads/work/env",
    "base_prefix": "/Library/Frameworks/Python.framework/Versions/3.8",
    "real_prefix": null,
    "exec_prefix": "/Users/david/Downloads/work/env",
    "base_exec_prefix": "/Library/Frameworks/Python.framework/Versions/3.8",
    "path": [
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python38.zip",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload",
      "/Users/david/Downloads/work/env/lib/python3.8/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]",
  "makefile_filename": "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/config-3.8-darwin/Makefile",
  "os": "<module 'os' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py'>",
  "site": "<module 'site' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site.py'>",
  "datetime": "<module 'datetime' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/datetime.py'>",
  "math": "<module 'math' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/math.cpython-38-darwin.so'>",
  "json": "<module 'json' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py'>"
} [DEBUG session:55]
401 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/david/Library/Application Support/virtualenv) [INFO session:59]
406 got embed update of distribution setuptools from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]
407 got embed update of distribution wheel from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]
407 got embed update of distribution pip from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]
411 wrote embed update of distribution setuptools at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:159]
413 wrote embed update of distribution wheel at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:159]
414 wrote embed update of distribution pip at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:159]
414 triggered periodic upgrade of setuptools==47.3.1 (for python 3.8) via background process having PID 59590 [INFO periodic_update:182]
417 triggered periodic upgrade of wheel==0.34.2 (for python 3.8) via background process having PID 59591 [INFO periodic_update:182]
422 triggered periodic upgrade of pip==20.1.1 (for python 3.8) via background process having PID 59592 [INFO periodic_update:182]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/periodic_update.py", line 214, in do_update
    dest = acquire.download_wheel(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 67, in download_wheel
    raise subprocess.CalledProcessError(process.returncode, cmd, **kwargs)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.8/bin/python3', '-m', 'pip', 'download', '--disable-pip-version-check', '--only-binary=:all:', '--no-deps', '--python-version', '3.8', '-d', '/Users/david/Library/Application Support/virtualenv/wheel/house', 'wheel<0.9']' returned non-zero exit status 1.
155299 got embed update of distribution wheel from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/periodic_update.py", line 214, in do_update
    dest = acquire.download_wheel(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 67, in download_wheel
    raise subprocess.CalledProcessError(process.returncode, cmd, **kwargs)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.8/bin/python3', '-m', 'pip', 'download', '--disable-pip-version-check', '--only-binary=:all:', '--no-deps', '--python-version', '3.8', '-d', '/Users/david/Library/Application Support/virtualenv/wheel/house', 'pip<1.5.1']' returned non-zero exit status 1.
216355 got embed update of distribution pip from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/periodic_update.py", line 214, in do_update
    dest = acquire.download_wheel(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 67, in download_wheel
    raise subprocess.CalledProcessError(process.returncode, cmd, **kwargs)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.8/bin/python3', '-m', 'pip', 'download', '--disable-pip-version-check', '--only-binary=:all:', '--no-deps', '--python-version', '3.8', '-d', '/Users/david/Library/Application Support/virtualenv/wheel/house', 'setuptools<0.9.8']' returned non-zero exit status 1.
1184058 got embed update of distribution setuptools from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]
1184058 install wheel from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1184059 Attempting to acquire lock 4476672128 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:270]
1184059 install pip from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.1.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1184059 Lock 4476672128 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:274]
1184060 Attempting to acquire lock 4476671696 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:270]
1184060 install setuptools from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-47.3.1-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1184060 Lock 4476671696 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:274]
1184061 Attempting to acquire lock 4476671360 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [DEBUG filelock:270]
1184061 Attempting to release lock 4476672128 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:315]
1184061 Attempting to release lock 4476671696 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:315]
1184061 Lock 4476671360 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [INFO filelock:274]
1184061 Lock 4476672128 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:318]
1184061 Lock 4476671696 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:318]
1184062 Attempting to release lock 4476671360 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [DEBUG filelock:315]
1184062 Lock 4476671360 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [INFO filelock:318]
1184063 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/easy_install.py to /Users/david/Downloads/work/env/lib/python3.8/site-packages/easy_install.py [DEBUG _sync:52]
1184065 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /Users/david/Downloads/work/env/lib/python3.8/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:52]
1184066 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pip [DEBUG _sync:52]
1184067 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools-47.3.1.virtualenv to /Users/david/Downloads/work/env/lib/python3.8/site-packages/setuptools-47.3.1.virtualenv [DEBUG _sync:52]
1184068 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /Users/david/Downloads/work/env/lib/python3.8/site-packages/wheel [DEBUG _sync:52]
1184071 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools to /Users/david/Downloads/work/env/lib/python3.8/site-packages/setuptools [DEBUG _sync:52]
1184093 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /Users/david/Downloads/work/env/lib/python3.8/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:52]
1184137 generated console scripts wheel wheel3 wheel-3.8 [DEBUG base:53]
1184267 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/pkg_resources to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pkg_resources [DEBUG _sync:52]
1184301 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools-47.3.1.dist-info to /Users/david/Downloads/work/env/lib/python3.8/site-packages/setuptools-47.3.1.dist-info [DEBUG _sync:52]
1184326 generated console scripts easy_install3 easy_install easy_install-3.8 [DEBUG base:53]
1184772 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.virtualenv to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pip-20.1.1.virtualenv [DEBUG _sync:52]
1184773 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.dist-info to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pip-20.1.1.dist-info [DEBUG _sync:52]
1184791 generated console scripts pip3.8 pip-3.8 pip3 pip [DEBUG base:53]
1184791 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:64]
1184801 write /Users/david/Downloads/work/env/pyvenv.cfg [DEBUG pyenv_cfg:34]
1184801 	home = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1184801 	implementation = CPython [DEBUG pyenv_cfg:38]
1184801 	version_info = 3.8.3.final.0 [DEBUG pyenv_cfg:38]
1184801 	virtualenv = 20.0.25 [DEBUG pyenv_cfg:38]
1184801 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
1184801 	base-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1184802 	base-exec-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1184802 	base-executable = /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 [DEBUG pyenv_cfg:38]
1184802 created virtual environment CPython3.8.3.final.0-64 in 1184661ms
  creator CPython3Posix(dest=/Users/david/Downloads/work/env, clear=True, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/david/Library/Application Support/virtualenv)
    added seed packages: pip==20.1.1, setuptools==47.3.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:18]
-> exit 0

[gaborbernat]

Can you delete those three please, and run it again? For what I see now is that we ran out of wheels to download... but that would imply the content of this file should be something else than you posted.

[dgelessus]

Just to be sure, I renamed away the entire ~/Library/Application Support/virtualenv directory and re-ran the command from your previous comment (#1883 (comment)). The results are the same as before - every version of pip/setuptools/wheel is downloaded, but after everthing has finished running, .../virtualenv/wheel/3.8/embed/1/pip.json still looks like this:

{
  "completed": null,
  "periodic": true,
  "started": "2020-07-03T18:35:05.539654Z",
  "versions": []
}

setuptools.json and wheel.json are identical to pip.json, except for the "started" timestamp (which differs by a few milliseconds).

Full verbose output

$ env _VIRTUALENV_PERIODIC_UPDATE_INLINE=1 virtualenv --clear env --with-traceback -vvv
752 setup logging to NOTSET [DEBUG report:42]
753 created app data folder /Users/david/Library/Application Support/virtualenv [DEBUG __init__:30]
840 find interpreter for spec PythonSpec(path=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3) [INFO builtin:44]
840 proposed PythonInfo(spec=CPython3.8.3.final.0-64, exe=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3, platform=darwin, version='3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
841 accepted PythonInfo(spec=CPython3.8.3.final.0-64, exe=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3, platform=darwin, version='3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
845 filesystem is not case-sensitive [DEBUG info:28]
1021 create virtual environment via CPython3Posix(dest=/Users/david/Downloads/work/env, clear=True, global=False) [INFO session:52]
1021 create folder /Users/david/Downloads/work/env/bin [DEBUG _sync:25]
1022 create folder /Users/david/Downloads/work/env/lib/python3.8/site-packages [DEBUG _sync:25]
1022 write /Users/david/Downloads/work/env/pyvenv.cfg [DEBUG pyenv_cfg:34]
1022 	home = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1022 	implementation = CPython [DEBUG pyenv_cfg:38]
1022 	version_info = 3.8.3.final.0 [DEBUG pyenv_cfg:38]
1022 	virtualenv = 20.0.25 [DEBUG pyenv_cfg:38]
1022 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
1022 	base-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1022 	base-exec-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1023 	base-executable = /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 [DEBUG pyenv_cfg:38]
1023 symlink /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 to /Users/david/Downloads/work/env/bin/python [DEBUG _sync:44]
1024 create virtualenv import hook file /Users/david/Downloads/work/env/lib/python3.8/site-packages/_virtualenv.pth [DEBUG api:94]
1025 create /Users/david/Downloads/work/env/lib/python3.8/site-packages/_virtualenv.py [DEBUG api:97]
1026 ============================== target debug ============================== [DEBUG session:54]
1026 debug via /Users/david/Downloads/work/env/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:209]
1026 {
  "sys": {
    "executable": "/Users/david/Downloads/work/env/bin/python",
    "_base_executable": "/Users/david/Downloads/work/env/bin/python",
    "prefix": "/Users/david/Downloads/work/env",
    "base_prefix": "/Library/Frameworks/Python.framework/Versions/3.8",
    "real_prefix": null,
    "exec_prefix": "/Users/david/Downloads/work/env",
    "base_exec_prefix": "/Library/Frameworks/Python.framework/Versions/3.8",
    "path": [
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python38.zip",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload",
      "/Users/david/Downloads/work/env/lib/python3.8/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) \n[Clang 6.0 (clang-600.0.57)]",
  "makefile_filename": "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/config-3.8-darwin/Makefile",
  "os": "<module 'os' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py'>",
  "site": "<module 'site' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site.py'>",
  "datetime": "<module 'datetime' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/datetime.py'>",
  "math": "<module 'math' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/math.cpython-38-darwin.so'>",
  "json": "<module 'json' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py'>"
} [DEBUG session:55]
1095 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/david/Library/Application Support/virtualenv) [INFO session:59]
1099 wrote embed update of distribution pip at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:159]
1099 wrote embed update of distribution wheel at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:159]
1099 wrote embed update of distribution setuptools at /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:159]
1105 triggered periodic upgrade of pip==20.1.1 (for python 3.8) via background process having PID 61078 [INFO periodic_update:182]
1110 triggered periodic upgrade of wheel==0.34.2 (for python 3.8) via background process having PID 61079 [INFO periodic_update:182]
1111 triggered periodic upgrade of setuptools==47.3.1 (for python 3.8) via background process having PID 61080 [INFO periodic_update:182]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/periodic_update.py", line 214, in do_update
    dest = acquire.download_wheel(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 67, in download_wheel
    raise subprocess.CalledProcessError(process.returncode, cmd, **kwargs)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.8/bin/python3', '-m', 'pip', 'download', '--disable-pip-version-check', '--only-binary=:all:', '--no-deps', '--python-version', '3.8', '-d', '/Users/david/Library/Application Support/virtualenv/wheel/house', 'wheel<0.9']' returned non-zero exit status 1.
126431 got embed update of distribution wheel from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/periodic_update.py", line 214, in do_update
    dest = acquire.download_wheel(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 67, in download_wheel
    raise subprocess.CalledProcessError(process.returncode, cmd, **kwargs)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.8/bin/python3', '-m', 'pip', 'download', '--disable-pip-version-check', '--only-binary=:all:', '--no-deps', '--python-version', '3.8', '-d', '/Users/david/Library/Application Support/virtualenv/wheel/house', 'pip<1.5.1']' returned non-zero exit status 1.
183000 got embed update of distribution pip from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/periodic_update.py", line 214, in do_update
    dest = acquire.download_wheel(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 67, in download_wheel
    raise subprocess.CalledProcessError(process.returncode, cmd, **kwargs)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.8/bin/python3', '-m', 'pip', 'download', '--disable-pip-version-check', '--only-binary=:all:', '--no-deps', '--python-version', '3.8', '-d', '/Users/david/Library/Application Support/virtualenv/wheel/house', 'setuptools<0.9.8']' returned non-zero exit status 1.
1127421 got embed update of distribution setuptools from /Users/david/Library/Application Support/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]
1127422 install wheel from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1127422 install pip from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.1.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1127423 install setuptools from wheel /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-47.3.1-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1127423 Attempting to acquire lock 4511023696 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:270]
1127424 Attempting to acquire lock 4511023216 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [DEBUG filelock:270]
1127424 Attempting to acquire lock 4511023504 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:270]
1127424 Lock 4511023696 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:274]
1127424 Lock 4511023216 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [INFO filelock:274]
1127424 Lock 4511023504 acquired on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:274]
1127424 build install image for wheel-0.34.2-py2.py3-none-any.whl to /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any [DEBUG base:57]
1127424 build install image for setuptools-47.3.1-py3-none-any.whl to /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any [DEBUG base:57]
1127425 build install image for pip-20.1.1-py2.py3-none-any.whl to /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any [DEBUG base:57]
1127483 Attempting to release lock 4511023696 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:315]
1127483 Lock 4511023696 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:318]
1127485 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /Users/david/Downloads/work/env/lib/python3.8/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:52]
1127488 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /Users/david/Downloads/work/env/lib/python3.8/site-packages/wheel [DEBUG _sync:52]
1127547 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /Users/david/Downloads/work/env/lib/python3.8/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:52]
1127586 generated console scripts wheel-3.8 wheel3 wheel [DEBUG base:53]
1127742 Attempting to release lock 4511023216 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [DEBUG filelock:315]
1127743 Lock 4511023216 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any.lock [INFO filelock:318]
1127743 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/easy_install.py to /Users/david/Downloads/work/env/lib/python3.8/site-packages/easy_install.py [DEBUG _sync:52]
1127745 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools-47.3.1.virtualenv to /Users/david/Downloads/work/env/lib/python3.8/site-packages/setuptools-47.3.1.virtualenv [DEBUG _sync:52]
1127746 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools to /Users/david/Downloads/work/env/lib/python3.8/site-packages/setuptools [DEBUG _sync:52]
1127870 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/pkg_resources to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pkg_resources [DEBUG _sync:52]
1127891 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-47.3.1-py3-none-any/setuptools-47.3.1.dist-info to /Users/david/Downloads/work/env/lib/python3.8/site-packages/setuptools-47.3.1.dist-info [DEBUG _sync:52]
1127909 generated console scripts easy_install-3.8 easy_install3 easy_install [DEBUG base:53]
1127984 Attempting to release lock 4511023504 on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:315]
1127984 Lock 4511023504 released on /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:318]
1127985 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pip [DEBUG _sync:52]
1128249 copy /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.virtualenv to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pip-20.1.1.virtualenv [DEBUG _sync:52]
1128250 copy directory /Users/david/Library/Application Support/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.dist-info to /Users/david/Downloads/work/env/lib/python3.8/site-packages/pip-20.1.1.dist-info [DEBUG _sync:52]
1128258 generated console scripts pip-3.8 pip3.8 pip3 pip [DEBUG base:53]
1128258 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:64]
1128265 write /Users/david/Downloads/work/env/pyvenv.cfg [DEBUG pyenv_cfg:34]
1128265 	home = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1128265 	implementation = CPython [DEBUG pyenv_cfg:38]
1128265 	version_info = 3.8.3.final.0 [DEBUG pyenv_cfg:38]
1128266 	virtualenv = 20.0.25 [DEBUG pyenv_cfg:38]
1128266 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
1128266 	base-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1128266 	base-exec-prefix = /Library/Frameworks/Python.framework/Versions/3.8 [DEBUG pyenv_cfg:38]
1128266 	base-executable = /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 [DEBUG pyenv_cfg:38]
1128267 created virtual environment CPython3.8.3.final.0-64 in 1127517ms
  creator CPython3Posix(dest=/Users/david/Downloads/work/env, clear=True, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/david/Library/Application Support/virtualenv)
    added seed packages: pip==20.1.1, setuptools==47.3.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:18]
-> exit 0

[dgelessus]

ok, I think I found the underlying problem:

Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> resp = requests.get("https://pypi.org/pypi/pip/json")
>>> resp.status_code, resp.reason
(200, 'OK')
>>> from six.moves.urllib.request import urlopen
>>> stream = urlopen("https://pypi.org/pypi/pip/json")
Traceback (most recent call last):
  # ...
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

pip has no issues connecting to pypi.org, and neither does requests (which I think pip uses internally, so that makes sense), but urllib.request.urlopen (which virtualenv uses to get package release dates) can't connect because of certificate issues. The _get_release_date function silently returns None if the request fails:

virtualenv/src/virtualenv/seed/wheels/periodic_update.py

Lines 237 to 246 in 455afbc

	def _get_release_date(dest):
	wheel = Wheel(dest)
	# the most accurate is to ask PyPi - e.g. https://pypi.org/pypi/pip/json,
	# see https://warehouse.pypa.io/api-reference/json/ for more details
	try:
	with urlopen("https://pypi.org/pypi/{}/json".format(wheel.distribution)) as file_handler:
	content = json.load(file_handler)
	return datetime.strptime(content["releases"][wheel.version][0]["upload_time"], "%Y-%m-%dT%H:%M:%S")
	except Exception: # noqa
	return None

And if the actual release date is unknown, the release date checking code falls back to using the download time:

virtualenv/src/virtualenv/seed/wheels/periodic_update.py

Lines 99 to 101 in 455afbc

	def use(self, now):
	compare_from = self.release_date or self.found_date
	return now - compare_from >= timedelta(days=28)

And that will cause it to download every available version if none of the wheels have been downloaded before, because then every version's download time will be too new.

[gaborbernat]

To conclude the fix here would be to stop downloading wheels once we reach the embedded wheels version.