20.0.3: making a pypy2 virtualenv from pypy3 loses libpypy so

[asottile]

minimal reproduction:

rm -rf venv venvpp3 venvpp2

virtualenv venv
venv/bin/pip install virtualenv

venv/bin/virtualenv venvpp3 -ppypy3
venvpp3/bin/pip install virtualenv

venvpp3/bin/virtualenv venvpp2 -ppypy2 -vvv

venvpp2/bin/pypy --version

$ rm -rf venv venvpp3 venvpp2
$ virtualenv venv
Using real prefix '/usr'
Path not in prefix '/home/asottile/opt/venv/include/python3.6m' '/usr'
New python executable in /tmp/x/venv/bin/python3
Also creating executable in /tmp/x/venv/bin/python
Installing setuptools, pip, wheel...
done.
$ venv/bin/pip install virtualenv
Collecting virtualenv
  Using cached virtualenv-20.0.3-py2.py3-none-any.whl (4.6 MB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Processing /home/asottile/.cache/pip/wheels/33/d9/71/e4e3cac73529e1947df418af0f140cd7589d5d9ec0e17ecfc2/distlib-0.3.0-py3-none-any.whl
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting zipp>=0.5
  Using cached zipp-2.2.0-py36-none-any.whl (4.6 kB)
Installing collected packages: zipp, importlib-metadata, appdirs, distlib, filelock, six, importlib-resources, virtualenv
Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-20.0.3 zipp-2.2.0
$ venv/bin/virtualenv venvpp3 -ppypy3
created virtual environment in 263ms PyPy3Posix(dest=/tmp/x/venvpp3, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/home/asottile/.local/share/virtualenv/seed-v1 via=copy
$ venvpp3/bin/pip install virtualenv
Collecting virtualenv
  Using cached virtualenv-20.0.3-py2.py3-none-any.whl (4.6 MB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Processing /home/asottile/.cache/pip/wheels/1d/17/b9/58b26a9de4722a7fcfc624156dc630bb547a1a031625854214/distlib-0.3.0-py3-none-any.whl
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting zipp>=0.5
  Using cached zipp-2.2.0-py36-none-any.whl (4.6 kB)
Installing collected packages: appdirs, distlib, filelock, six, importlib-resources, zipp, importlib-metadata, virtualenv
Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-20.0.3 zipp-2.2.0
$ venvpp3/bin/virtualenv venvpp2 -ppypy2 -vvv
146 setup logging to NOTSET [DEBUG report:43]
160 find interpreter for spec PythonSpec(implementation=pypy, major=2) [INFO builtin:44]
161 discover system for PythonInfo(spec=PyPy3.6.9.final.0-64, exe=/tmp/x/venvpp3/bin/pypy3, platform=linux, version='3.6.9 (1608da62bfc7, Dec 23 2019, 10:50:04)\n[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]', encoding_fs_io=utf-8-UTF-8) in /home/asottile/opt/pypy3.6-v7.3.0-linux64 [DEBUG py_info:335]
162 filesystem is case-sensitive [DEBUG info:28]
165 Attempting to acquire lock 140317323200792 on /home/asottile/.local/share/virtualenv/py-info/20.0.3/d65722ccbd9d1a4506fa2c7808c0f53bc909b05d8994e87bd000bf498be75f27.lock [DEBUG filelock:270]
165 Lock 140317323200792 acquired on /home/asottile/.local/share/virtualenv/py-info/20.0.3/d65722ccbd9d1a4506fa2c7808c0f53bc909b05d8994e87bd000bf498be75f27.lock [INFO filelock:274]
166 get interpreter info via cmd: /home/asottile/opt/pypy3.6-v7.3.0-linux64/bin/pypy3 -s /tmp/x/venvpp3/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:107]
299 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/20.0.3/d65722ccbd9d1a4506fa2c7808c0f53bc909b05d8994e87bd000bf498be75f27.json for /home/asottile/opt/pypy3.6-v7.3.0-linux64/bin/pypy3 [DEBUG cached_py_info:88]
300 Attempting to release lock 140317323200792 on /home/asottile/.local/share/virtualenv/py-info/20.0.3/d65722ccbd9d1a4506fa2c7808c0f53bc909b05d8994e87bd000bf498be75f27.lock [DEBUG filelock:315]
300 Lock 140317323200792 released on /home/asottile/.local/share/virtualenv/py-info/20.0.3/d65722ccbd9d1a4506fa2c7808c0f53bc909b05d8994e87bd000bf498be75f27.lock [INFO filelock:318]
300 proposed PythonInfo(spec=PyPy3.6.9.final.0-64, system=/home/asottile/opt/pypy3.6-v7.3.0-linux64/bin/pypy3, exe=/tmp/x/venvpp3/bin/pypy3, platform=linux, version='3.6.9 (1608da62bfc7, Dec 23 2019, 10:50:04)\n[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:50]
301 discover PATH[0]=/home/asottile/bin [DEBUG builtin:77]
301 Attempting to acquire lock 140317323202304 on /home/asottile/.local/share/virtualenv/py-info/20.0.3/eaf3d0e7402d5363edd85a023ca0972de8e7f58fb4880eeea8be3a336a257777.lock [DEBUG filelock:270]
301 Lock 140317323202304 acquired on /home/asottile/.local/share/virtualenv/py-info/20.0.3/eaf3d0e7402d5363edd85a023ca0972de8e7f58fb4880eeea8be3a336a257777.lock [INFO filelock:274]
302 get interpreter info via cmd: /home/asottile/bin/pypy2 -s /tmp/x/venvpp3/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:107]
353 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/20.0.3/eaf3d0e7402d5363edd85a023ca0972de8e7f58fb4880eeea8be3a336a257777.json for /home/asottile/bin/pypy2 [DEBUG cached_py_info:88]
354 Attempting to release lock 140317323202304 on /home/asottile/.local/share/virtualenv/py-info/20.0.3/eaf3d0e7402d5363edd85a023ca0972de8e7f58fb4880eeea8be3a336a257777.lock [DEBUG filelock:315]
354 Lock 140317323202304 released on /home/asottile/.local/share/virtualenv/py-info/20.0.3/eaf3d0e7402d5363edd85a023ca0972de8e7f58fb4880eeea8be3a336a257777.lock [INFO filelock:318]
354 proposed PathPythonInfo(spec=PyPy2.7.13.final.42-64, exe=/home/asottile/bin/pypy2, platform=linux2, version='2.7.13 (724f1a7d62e8, Dec 23 2019, 15:36:24)\n[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]', encoding_fs_io=UTF-8-None) [INFO builtin:50]
355 accepted PathPythonInfo(spec=PyPy2.7.13.final.42-64, exe=/home/asottile/bin/pypy2, platform=linux2, version='2.7.13 (724f1a7d62e8, Dec 23 2019, 15:36:24)\n[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]', encoding_fs_io=UTF-8-None) [DEBUG builtin:52]
399 create virtual environment via PyPy2Posix(dest=/tmp/x/venvpp2, clear=False, global=False) [INFO session:51]
400 create folder /tmp/x/venvpp2/bin [DEBUG _sync:22]
400 create folder /tmp/x/venvpp2/lib-python/2.7 [DEBUG _sync:22]
400 create folder /tmp/x/venvpp2/lib_pypy [DEBUG _sync:22]
400 create folder /tmp/x/venvpp2/site-packages [DEBUG _sync:22]
400 write /tmp/x/venvpp2/pyvenv.cfg [DEBUG pyenv_cfg:34]
400 	home = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
400 	implementation = PyPy [DEBUG pyenv_cfg:38]
401 	version_info = 2.7.13.final.42 [DEBUG pyenv_cfg:38]
401 	virtualenv = 20.0.3 [DEBUG pyenv_cfg:38]
401 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
401 	base-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
401 	base-exec-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
401 	base-executable = /home/asottile/bin/pypy2 [DEBUG pyenv_cfg:38]
401 copy /home/asottile/bin/pypy2 to /tmp/x/venvpp2/bin/pypy [DEBUG _sync:49]
402 hard link /tmp/x/venvpp2/bin/pypy to python [DEBUG _sync:66]
402 hard link /tmp/x/venvpp2/bin/pypy to python2 [DEBUG _sync:66]
403 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/os.py to /tmp/x/venvpp2/lib-python/2.7/os.py [DEBUG _sync:41]
403 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/os.pyc to /tmp/x/venvpp2/lib-python/2.7/os.pyc [DEBUG _sync:41]
403 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/copy_reg.py to /tmp/x/venvpp2/lib-python/2.7/copy_reg.py [DEBUG _sync:41]
403 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/copy_reg.pyc to /tmp/x/venvpp2/lib-python/2.7/copy_reg.pyc [DEBUG _sync:41]
403 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/genericpath.py to /tmp/x/venvpp2/lib-python/2.7/genericpath.py [DEBUG _sync:41]
404 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/genericpath.pyc to /tmp/x/venvpp2/lib-python/2.7/genericpath.pyc [DEBUG _sync:41]
404 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/linecache.py to /tmp/x/venvpp2/lib-python/2.7/linecache.py [DEBUG _sync:41]
404 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/linecache.pyc to /tmp/x/venvpp2/lib-python/2.7/linecache.pyc [DEBUG _sync:41]
404 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/stat.py to /tmp/x/venvpp2/lib-python/2.7/stat.py [DEBUG _sync:41]
404 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/stat.pyc to /tmp/x/venvpp2/lib-python/2.7/stat.pyc [DEBUG _sync:41]
405 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/UserDict.py to /tmp/x/venvpp2/lib-python/2.7/UserDict.py [DEBUG _sync:41]
405 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/UserDict.pyc to /tmp/x/venvpp2/lib-python/2.7/UserDict.pyc [DEBUG _sync:41]
406 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/warnings.py to /tmp/x/venvpp2/lib-python/2.7/warnings.py [DEBUG _sync:41]
406 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/warnings.pyc to /tmp/x/venvpp2/lib-python/2.7/warnings.pyc [DEBUG _sync:41]
406 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/posixpath.py to /tmp/x/venvpp2/lib-python/2.7/posixpath.py [DEBUG _sync:41]
406 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/posixpath.pyc to /tmp/x/venvpp2/lib-python/2.7/posixpath.pyc [DEBUG _sync:41]
406 symlink directory /home/asottile/opt/pypy2.7-v7.3.0-linux64/include to /tmp/x/venvpp2/include [DEBUG _sync:41]
407 symlink directory /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib to /tmp/x/venvpp2/lib [DEBUG _sync:41]
407 ============================== target debug ============================== [DEBUG session:53]
408 debug via /tmp/x/venvpp2/bin/pypy /tmp/x/venvpp3/site-packages/virtualenv/create/debug.py [DEBUG creator:192]
407 {
  "out": "",
  "err": "/tmp/x/venvpp2/bin/pypy: error while loading shared libraries: libpypy-c.so: cannot open shared object file: No such file or directory\n",
  "returncode": 127,
  "exception": "SyntaxError('invalid syntax', ('<unknown>', 0, 1, '\\n', 0))"
} [DEBUG session:54]
411 add seed packages via FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/home/asottile/.local/share/virtualenv/seed-v1 via=copy [INFO session:58]
413 Attempting to acquire lock 140317322481352 on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:270]
413 Lock 140317322481352 acquired on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:274]
416 install setuptools from wheel /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels/setuptools-44.0.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:57]
416 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info to /tmp/x/venvpp2/site-packages/setuptools-44.0.0.dist-info [DEBUG _sync:49]
419 install wheel from wheel /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:57]
420 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /tmp/x/venvpp2/site-packages/wheel [DEBUG _sync:49]
424 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/pkg_resources to /tmp/x/venvpp2/site-packages/pkg_resources [DEBUG _sync:49]
415 install pip from wheel /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:57]
429 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /tmp/x/venvpp2/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:49]
430 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /tmp/x/venvpp2/site-packages/pip [DEBUG _sync:49]
437 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info.virtualenv to /tmp/x/venvpp2/site-packages/wheel-0.34.2.dist-info.virtualenv [DEBUG _sync:49]
441 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/easy_install.py to /tmp/x/venvpp2/site-packages/easy_install.py [DEBUG _sync:49]
448 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info.virtualenv to /tmp/x/venvpp2/site-packages/setuptools-44.0.0.dist-info.virtualenv [DEBUG _sync:49]
449 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools to /tmp/x/venvpp2/site-packages/setuptools [DEBUG _sync:49]
628 changing mode of /tmp/x/venvpp2/bin/wheel to 755 [INFO util:566]
629 changing mode of /tmp/x/venvpp2/bin/easy_install to 755 [INFO util:566]
630 changing mode of /tmp/x/venvpp2/bin/easy_install3 to 755 [INFO util:566]
630 changing mode of /tmp/x/venvpp2/bin/easy_install-3.6 to 755 [INFO util:566]
630 changing mode of /tmp/x/venvpp2/bin/wheel3 to 755 [INFO util:566]
631 changing mode of /tmp/x/venvpp2/bin/wheel-3.6 to 755 [INFO util:566]
632 generated console scripts easy_install easy_install3 easy_install-3.6 [DEBUG base:48]
631 generated console scripts wheel wheel3 wheel-3.6 [DEBUG base:48]
667 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info.virtualenv to /tmp/x/venvpp2/site-packages/pip-20.0.2.dist-info.virtualenv [DEBUG _sync:49]
667 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /tmp/x/venvpp2/site-packages/pip-20.0.2.dist-info [DEBUG _sync:49]
670 changing mode of /tmp/x/venvpp2/bin/pip to 755 [INFO util:566]
670 changing mode of /tmp/x/venvpp2/bin/pip3 to 755 [INFO util:566]
670 changing mode of /tmp/x/venvpp2/bin/pip-3.6 to 755 [INFO util:566]
670 generated console scripts pip pip3 pip-3.6 [DEBUG base:48]
671 Attempting to release lock 140317322481352 on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:315]
671 Lock 140317322481352 released on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:318]
671 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:64]
673 write /tmp/x/venvpp2/pyvenv.cfg [DEBUG pyenv_cfg:34]
673 	home = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
673 	implementation = PyPy [DEBUG pyenv_cfg:38]
673 	version_info = 2.7.13.final.42 [DEBUG pyenv_cfg:38]
673 	virtualenv = 20.0.3 [DEBUG pyenv_cfg:38]
674 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
674 	base-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
674 	base-exec-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
674 	base-executable = /home/asottile/bin/pypy2 [DEBUG pyenv_cfg:38]
675 created virtual environment in 531ms PyPy2Posix(dest=/tmp/x/venvpp2, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/home/asottile/.local/share/virtualenv/seed-v1 via=copy [WARNING __main__:24]
$ venvpp2/bin/pypy --version
venvpp2/bin/pypy: error while loading shared libraries: libpypy-c.so: cannot open shared object file: No such file or directory

[asottile]

I think (?) this is what I was hitting while trying to implement #1612

[asottile]

iiiiinteresting, this is what I'm seeing:

(Pdb) list
 37  	            yield src
 38  	        breakpoint()
 39  	        for host in cls._add_shared_libs(interpreter):
 40  	            yield PathRefToDest(host, dest=lambda self, s: self.bin_dir / s.name)
 41  	
 42  ->	    @classmethod
 43  	    def _add_shared_libs(cls, interpreter):
 44  	        # https://bitbucket.org/pypy/pypy/issue/1922/future-proofing-virtualenv
 45  	        python_dir = Path(interpreter.system_executable).parent
 46  	        for libname in cls._shared_libs():
 47  	            src = python_dir / libname
(Pdb) interpreter.system_executable
'/home/asottile/bin/pypy2'

I've got pypy set up so that pypy2 is a symlink -- I can probably sprinkle some readlink and fix this!

[gaborbernat]

Hello, a fix for this issue has been released via virtualenv 20.0.5; see https://pypi.org/project/virtualenv/20.0.5/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-5-2020-02-21) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.