Error when running debugging using pytest in IDE with geopandas

[arengel]

I ran into an error today while running a test in debug mode using PyCharm (I found the same issue on VS Code as well) when geopandas is imported. Outside of debug mode it seems to work fine. I've also found that downgrading pytest to 6.2.4 resolves the issue.

I was able to reproduce the problem by setting up a test environment using conda: A simple test script with a breakpoint set at the assert statement produces the error below

conda create --name test --channel=conda-forge python pytest geopandas

Today when I set it up that command resulted in the following version being installed (full list of packages installed below):
python=3.10.2 pytest=7.1.1 geopandas=0.10.2

I am using an ARM Mac running macOS 12.3, I've also gotten the same result on an Intel Mac macOS 11.6.4.

import geopandas


def test_pytest_and_geopandas():
    assert False

A simple test script with a breakpoint set at the assert statement like that above produces the following error if the test is run in debug mode in PyCharm:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 321, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 332, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 644, in perform_collect
INTERNALERROR>     self.ihook.pytest_collectreport(report=rep)
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/teamcity/pytest_plugin.py", line 328, in pytest_collectreport
INTERNALERROR>     self.report_test_failure(test_id, report)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/teamcity/pytest_plugin.py", line 223, in report_test_failure
INTERNALERROR>     self.ensure_test_start_reported(test_id)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/teamcity/pytest_plugin.py", line 186, in ensure_test_start_reported
INTERNALERROR>     self.teamcity.testStarted(test_id, flowId=test_id, captureStandardOutput=capture_standard_output, metainfo=metainfo)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_runner_tools.py", line 181, in testStarted
INTERNALERROR>     commands = _TREE_MANAGER_HOLDER.manager.level_opened(self._test_to_list(testName), _write_start_message)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_runner_tools.py", line 51, in manager
INTERNALERROR>     self._fill_manager()
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_runner_tools.py", line 59, in _fill_manager
INTERNALERROR>     from _jb_serial_tree_manager import SerialTreeManager
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 945, in _find_spec
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 92, in find_spec
INTERNALERROR>     if self._early_rewrite_bailout(name, state):
INTERNALERROR>   File "/Users/aengel/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 193, in _early_rewrite_bailout
INTERNALERROR>     path = PurePath(os.path.sep.join(parts) + ".py")
INTERNALERROR> AttributeError: module 'posixpath' has no attribute 'sep'

or VS Code

tests[/test_addl.py:10](): in <module>
    import geopandas
.[./../miniconda3/envs/test/lib/python3.10/site-packages/geopandas/__init__.py:1](): in <module>
    from geopandas._config import options  # noqa
.[./../miniconda3/envs/test/lib/python3.10/site-packages/geopandas/_config.py:109](): in <module>
    default_value=_default_use_pygeos(),
.[./../miniconda3/envs/test/lib/python3.10/site-packages/geopandas/_config.py:95](): in _default_use_pygeos
    import geopandas._compat as compat
.[./../miniconda3/envs/test/lib/python3.10/site-packages/geopandas/_compat.py:9](): in <module>
    import pyproj
.[./../miniconda3/envs/test/lib/python3.10/site-packages/pyproj/__init__.py:93](): in <module>
    pyproj.network.set_ca_bundle_path()
.[./../miniconda3/envs/test/lib/python3.10/site-packages/pyproj/network.py:51](): in set_ca_bundle_path
    ca_bundle_path = certifi.where()
.[./../miniconda3/envs/test/lib/python3.10/site-packages/certifi/core.py:36](): in where
    _CACERT_CTX = get_path("certifi", "cacert.pem")
.[./../miniconda3/envs/test/lib/python3.10/importlib/resources.py:119](): in path
    reader = _common.get_resource_reader(_common.get_package(package))
.[./../miniconda3/envs/test/lib/python3.10/importlib/_common.py:52](): in get_resource_reader
    return reader(spec.name)  # type: ignore
<frozen importlib._bootstrap_external>:548: in _check_name_wrapper
    ???
<frozen importlib._bootstrap_external>:1081: in get_resource_reader
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1002: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:945: in _find_spec
    ???
.[./../miniconda3/envs/test/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:92](): in find_spec
    if self._early_rewrite_bailout(name, state):
.[./../miniconda3/envs/test/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:193](): in _early_rewrite_bailout
    path = PurePath(os.path.sep.join(parts) + ".py")
E   AttributeError: module 'posixpath' has no attribute 'sep'

Package            Version
------------------ ------------
attrs              21.4.0
branca             0.4.2
brotlipy           0.7.0
certifi            2021.10.8
cffi               1.15.0
charset-normalizer 2.0.12
click              8.0.4
click-plugins      1.1.1
cligj              0.7.2
cryptography       36.0.1
cycler             0.11.0
Fiona              1.8.21
folium             0.12.1.post1
fonttools          4.31.1
GDAL               3.4.2
geopandas          0.10.2
idna               3.3
iniconfig          1.1.1
Jinja2             3.0.3
joblib             1.1.0
kiwisolver         1.4.0
mapclassify        2.4.3
MarkupSafe         2.1.1
matplotlib         3.5.1
munch              2.5.0
munkres            1.1.4
networkx           2.7.1
numpy              1.22.3
packaging          21.3
pandas             1.4.1
Pillow             9.0.1
pip                22.0.4
pluggy             1.0.0
py                 1.11.0
pycparser          2.21
pyOpenSSL          22.0.0
pyparsing          3.0.7
pyproj             3.3.0
PySocks            1.7.1
pytest             7.1.1
python-dateutil    2.8.2
pytz               2021.3
requests           2.27.1
Rtree              0.9.7
scikit-learn       1.0.2
scipy              1.8.0
setuptools         60.10.0
Shapely            1.8.0
six                1.16.0
threadpoolctl      3.1.0
tomli              2.0.1
unicodedata2       14.0.0
urllib3            1.26.9
wheel              0.37.1
xyzservices        2022.3.0

[Zac-HD]

Looks like a bug to me: we should also let PurePath do the joining, not hand-rolling string manipulation.

(and the error is because there's no such thing as os.path.sep, it's just os.sep... or os.pathsep, but confusingly that's an entirely different thing.)

[K0Te]

Hi,

I have found the same bug after updating project from Python3.9 to Python3.10.
Looks like proj==3.3.0 works with Python3.9 debugger and with Python3.10 without debugger, but breaks with 3.10 and debugger. This does not seem to be related to pytest, some code completely breaks os.path, pytest is just the first one to find that attribute is missing.
Minimal reproducible example:

from os import path

print(path.sep) # prints '/'
# First comment out _pyproj_global_context_initialize() call in
# pyproj/__init__.py, otherwise it will fail in import
from pyproj import _pyproj_global_context_initialize

print(path.sep) # prints '/'

_pyproj_global_context_initialize() # break os.path

print(path.sep) # fails

Stack-trace:

=========================================================================================== test session starts ============================================================================================
collected 0 items / 1 error                                                                                                                                                                                
##teamcity[testCount timestamp='2022-03-29T12:59:39.969' count='0']
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/_pytest/main.py", line 634, in perform_collect
INTERNALERROR>     self.items.extend(self.genitems(node))
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/_pytest/main.py", line 812, in genitems
INTERNALERROR>     node.ihook.pytest_collectreport(report=rep)
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/teamcity/pytest_plugin.py", line 328, in pytest_collectreport
INTERNALERROR>     self.report_test_failure(test_id, report)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/teamcity/pytest_plugin.py", line 223, in report_test_failure
INTERNALERROR>     self.ensure_test_start_reported(test_id)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/teamcity/pytest_plugin.py", line 186, in ensure_test_start_reported
INTERNALERROR>     self.teamcity.testStarted(test_id, flowId=test_id, captureStandardOutput=capture_standard_output, metainfo=metainfo)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_runner_tools.py", line 181, in testStarted
INTERNALERROR>     commands = _TREE_MANAGER_HOLDER.manager.level_opened(self._test_to_list(testName), _write_start_message)
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_runner_tools.py", line 51, in manager
INTERNALERROR>     self._fill_manager()
INTERNALERROR>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_runner_tools.py", line 59, in _fill_manager
INTERNALERROR>     from _jb_serial_tree_manager import SerialTreeManager
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 945, in _find_spec
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 91, in find_spec
INTERNALERROR>     if self._early_rewrite_bailout(name, state):
INTERNALERROR>   File "/Users/kote/portside/phenom/.env/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 195, in _early_rewrite_bailout
INTERNALERROR>     path = PurePath(os.path.sep.join(parts) + ".py")
INTERNALERROR> AttributeError: module 'posixpath' has no attribute 'sep'

[K0Te]

Re-building pyproj with non-released cython==3.0.0a10 fixes this issue, so I suspect that issue lies somewhere between pyproj and cython==3.0.0a10.

[RemiDesgrange]

@K0Te can you elaborate on the way you reproduce this ?

Here's what I did:

on a fresh .venv:

pip install --no-cache pyproj pytest
pytest test.py # the example you previsouly provide in the thread 
# works fine, should fail ?

I tried on another fresh venv:

pip install --no-cache cython==3.0.0a10
pip install --no-cache --no-binary ':all:' pyproj
pip install pytest
pytest test.py

Thx.

[songololo]

It looks like this may have been resolved in pyproj but that it will only be available in the next release per:
pyproj4/pyproj#1028