crash: free(): invalid pointer

[hazcod]

Very odd, but I have a Docker container on Apple Silicon (M1) which crashes when I import google.cloud.
It works when I remove the import line.

Environment details

• OS type and version: python:3.9.5-slim-buster on Docker
• google-cloud-pubsub version: 2.8.6

Steps to reproduce

1. from google.cloud import pubsub_v1

Code example

Stack trace

free(): invalid pointer

strace excerpt

openat(AT_FDCWD, "/lib/aarch64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\220\36\0\0\0\0\0\0"..., 832) = 832
fstat(6, {st_mode=S_IFREG|0644, st_size=31576, ...}) = 0
mmap(NULL, 95016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 6, 0) = 0xffffb039f000
mprotect(0xffffb03a6000, 61440, PROT_NONE) = 0
mmap(0xffffb03b5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 6, 0x6000) = 0xffffb03b5000
close(6)                                = 0
mprotect(0xffffb03b5000, 4096, PROT_READ) = 0
mprotect(0xffffb0acf000, 167936, PROT_READ) = 0
futex(0xffffb0b32070, FUTEX_WAKE_PRIVATE, 2147483647) = 0
writev(2, [{iov_base="free(): invalid pointer", iov_len=23}, {iov_base="\n", iov_len=1}], 2free(): invalid pointer
) = 24

[ellieayla]

I can reproduce this (and no k8s required) on a M1 MBP.

➜  ~ uname -a
Darwin Nemu.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 arm64

Using either of these starting images:

image python:3.9.5-slim-buster

apt install strace
python3 -m pip install google-cloud-pubsub

image debian:bullseye-slim

apt install python3 python3-pip libsodium-dev wget python3-setuptools libpq-dev git python3-dev build-essential strace
python3 -m pip install google-cloud-pubsub

@hazcod observed a fault when importing google.cloud.pubsub_v1 as above;

# strace python3 -c 'from google.cloud import pubsub_v1'
...
writev(2, [{iov_base="free(): invalid pointer", iov_len=23}, {iov_base="\n", iov_len=1}], 2free(): invalid pointer
) = 24
...
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=306, si_uid=0} ---
+++ killed by SIGABRT +++
Aborted

But importing google.cloud.pubsub_v1 isn't really required here; the fault can be reproduced with importing only these two dependencies...

python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted

Importing either by itself doesn't crash. Further, importing them in the other order doesn't crash. I do not understand why.

python3 -c 'import grpc, proto.message; print(grpc, proto.message)'
<module 'grpc' from '/usr/local/lib/python3.9/site-packages/grpc/__init__.py'> <module 'proto.message' from '/usr/local/lib/python3.9/site-packages/proto/message.py'>

[hazcod]

Opened up https://bugs.python.org/issue44161

[plamut]

This sounds like an external issue in one of the dependencies. It's the first time I'm seeing this, thanks for the report!

@lidizheng Is this issue familiar? Can you reproduce it as reported?
(basically trying to narrow it down to either proto-plus or grpcio)

@hazcod Can you please also share the versions of the proto-plus and grpcio dependencies used in the check? Or better yet, the entire pip freeze output? Thanks!

[lidizheng]

Crashing depending on the import order of grpcio and protobuf has happened before: grpc/grpc#24897. The quick fix is try to upgrade both grpcio and protobuf to in-sync, it's grpcio==1.37.1 grpcio-tools=1.37.1 protobuf==3.15.2.

CC @gnossen who found the ABI incompatibility in the first place.

---

EDIT: can not reproduce with macOS 11.3.1 + 3.7.9, here is the pip freeze: https://gist.github.com/lidizheng/391d0eb5f8be52ced2d259c9f806ee82

[gnossen]

The quick fix is try to upgrade both grpcio and protobuf to in-sync,

Actually, this issue should be fixed, regardless of protobuf version.

@alanjcastonguay Some questions for you:

• What version of grpcio do you have installed?
• Does this issue still occur if you update to the latest version of grpcio? (1.37.1)
• Do you have grpcio-tools installed as well? Or just grpcio? If both, please try upgrading grpcio-tools as well.
• If you do have grpcio-tools installed, try removing it. Does the issue persist?

If upgrading doesn't fix the issue for you, please help me generate a symbolized core dump. Install grpcio (and grpcio-tools) using our nightly builds, which do not have symbols stripped.

[gnossen]

I just noticed that this is in Docker on Mac M1, not x86 Mac OS, which the fix was for. Can you confirm which artifacts you're downloading? Is it grpcio-1.37.1-cp39-cp39-manylinux2014_aarch64.whl? These aarch64 artifacts are brand new. If this is the problematic artifact, then it's likely that a very similar issue to the one fixed by grpc/grpc#24992 is at fault here.

[ellieayla]

Starting from python:3.9.5-slim-buster, which has no packages.

➜  ~ docker run -it python:3.9.5-slim-buster bash
root@9bd98952a250:/# python3 -m pip freeze
root@9bd98952a250:/# python3 -m pip install google-cloud-pubsub
Collecting google-cloud-pubsub
  Downloading google_cloud_pubsub-2.4.2-py2.py3-none-any.whl (212 kB)
     |████████████████████████████████| 212 kB 3.9 MB/s
Collecting proto-plus>=1.7.1
  Downloading proto_plus-1.18.1-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 1.8 MB/s
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
Collecting libcst>=0.3.10
  Downloading libcst-0.3.19-py3-none-any.whl (513 kB)
     |████████████████████████████████| 513 kB 18.7 MB/s
Collecting google-api-core[grpc]<2.0.0dev,>=1.22.2
  Downloading google_api_core-1.26.3-py2.py3-none-any.whl (93 kB)
     |████████████████████████████████| 93 kB 3.4 MB/s
Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.9/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-pubsub) (56.2.0)
Collecting six>=1.13.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting packaging>=14.3
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 12.9 MB/s
Collecting google-auth<2.0dev,>=1.21.1
  Downloading google_auth-1.30.0-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 37.5 MB/s
Collecting pytz
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 22.2 MB/s
Collecting requests<3.0.0dev,>=2.18.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 10.0 MB/s
Collecting protobuf>=3.12.0
  Downloading protobuf-3.17.0-cp39-cp39-manylinux2014_aarch64.whl (925 kB)
     |████████████████████████████████| 925 kB 11.1 MB/s
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Downloading googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
     |████████████████████████████████| 198 kB 11.8 MB/s
Collecting grpcio<2.0dev,>=1.29.0
  Downloading grpcio-1.37.1-cp39-cp39-manylinux2014_aarch64.whl (37.1 MB)
     |████████████████████████████████| 37.1 MB 27.7 MB/s
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 68.1 MB/s
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting typing-inspect>=0.4.0
  Downloading typing_inspect-0.6.0-py3-none-any.whl (8.1 kB)
Collecting typing-extensions>=3.7.4.2
  Downloading typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Collecting pyyaml>=5.2
  Downloading PyYAML-5.4.1-cp39-cp39-manylinux2014_aarch64.whl (788 kB)
     |████████████████████████████████| 788 kB 18.9 MB/s
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 11.1 MB/s
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 10.2 MB/s
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
     |████████████████████████████████| 153 kB 29.2 MB/s
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 25.6 MB/s
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 17.5 MB/s
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 15.4 MB/s
Collecting mypy-extensions>=0.3.0
  Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Building wheels for collected packages: grpc-google-iam-v1
  Building wheel for grpc-google-iam-v1 (setup.py) ... done
  Created wheel for grpc-google-iam-v1: filename=grpc_google_iam_v1-0.12.3-py3-none-any.whl size=18499 sha256=63c626f89627ac51175131d58db3a10351b17b3028f9c0697b286fa919b8a012
  Stored in directory: /root/.cache/pip/wheels/a1/b4/ad/f2f4118c678bbc6496d5b7057d2508ba51f66009700b1afe8a
Successfully built grpc-google-iam-v1
Installing collected packages: six, pyasn1, urllib3, rsa, pyparsing, pyasn1-modules, protobuf, idna, chardet, certifi, cachetools, typing-extensions, requests, pytz, packaging, mypy-extensions, grpcio, googleapis-common-protos, google-auth, typing-inspect, pyyaml, google-api-core, proto-plus, libcst, grpc-google-iam-v1, google-cloud-pubsub
Successfully installed cachetools-4.2.2 certifi-2020.12.5 chardet-4.0.0 google-api-core-1.26.3 google-auth-1.30.0 google-cloud-pubsub-2.4.2 googleapis-common-protos-1.53.0 grpc-google-iam-v1-0.12.3 grpcio-1.37.1 idna-2.10 libcst-0.3.19 mypy-extensions-0.4.3 packaging-20.9 proto-plus-1.18.1 protobuf-3.17.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-2.4.7 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 rsa-4.7.2 six-1.16.0 typing-extensions-3.10.0.0 typing-inspect-0.6.0 urllib3-1.26.4
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

So installing google_cloud_pubsub==2.4.2 pulled in these packages:

root@9bd98952a250:/# python3 -m pip freeze
cachetools==4.2.2
certifi==2020.12.5
chardet==4.0.0
google-api-core==1.26.3
google-auth==1.30.0
google-cloud-pubsub==2.4.2
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3
grpcio==1.37.1
idna==2.10
libcst==0.3.19
mypy-extensions==0.4.3
packaging==20.9
proto-plus==1.18.1
protobuf==3.17.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
pytz==2021.1
PyYAML==5.4.1
requests==2.25.1
rsa==4.7.2
six==1.16.0
typing-extensions==3.10.0.0
typing-inspect==0.6.0
urllib3==1.26.4

The simple-case reproduction from above:

root@9bd98952a250:/# python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted

The questions posed:

• What version of grpcio do you have installed? grpcio==1.37.1
• Does this issue still occur if you update to the latest version of grpcio? (1.37.1) Yes
• Do you have grpcio-tools installed as well? Or just grpcio? If both, please try upgrading grpcio-tools as well. Only grpcio, no grpcio-tools.
• If you do have grpcio-tools installed, try removing it. Does the issue persist? Yes, because that's the initial case.
• Can you confirm which artifacts you're downloading? Is it grpcio-1.37.1-cp39-cp39-manylinux2014_aarch64.whl? Yes.

Questions not posed:

• If you do not have grpcio-tools installed, try installing it. Does the issue persist? Yes.

root@9bd98952a250:/# python3 -m pip install grpcio-tools
Collecting grpcio-tools
  Downloading grpcio_tools-1.37.1-cp39-cp39-manylinux2014_aarch64.whl (25.5 MB)
     |████████████████████████████████| 25.5 MB 47 kB/s
Requirement already satisfied: protobuf<4.0dev,>=3.5.0.post1 in /usr/local/lib/python3.9/site-packages (from grpcio-tools) (3.17.0)
Requirement already satisfied: grpcio>=1.37.1 in /usr/local/lib/python3.9/site-packages (from grpcio-tools) (1.37.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from grpcio-tools) (56.2.0)
Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.9/site-packages (from grpcio>=1.37.1->grpcio-tools) (1.16.0)
Installing collected packages: grpcio-tools
Successfully installed grpcio-tools-1.37.1

root@9bd98952a250:/# python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted

[gnossen]

CC @jtattermusch who built both the aarch64 grpcio wheels and the aarch64 protobuf wheels.

[ellieayla]

If upgrading doesn't fix the issue for you, please help me generate a symbolized core dump. Install grpcio (and grpcio-tools) using our nightly builds, which do not have symbols stripped.

Using only grpcio-1.39.0 + grpcio_tools-1.39.0 + proto_plus-1.18.1 (+ six-1.16.0 + protobuf-3.17.0, which are pulled in as dependencies).

➜  ~ docker run -it python:3.9.5-slim-buster bash
root@fa796b3a1f02:/# python3 -m pip install https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio/grpcio-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio-tools/grpcio_tools-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl proto_plus==1.18.1
Collecting grpcio==1.39.0.dev0
  Downloading https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio/grpcio-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl (39.0 MB)
     |████████████████████████████████| 39.0 MB 92.2 MB/s
Collecting grpcio-tools==1.39.0.dev0
  Downloading https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio-tools/grpcio_tools-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl (25.5 MB)
     |████████████████████████████████| 25.5 MB 41.0 MB/s
Collecting proto_plus==1.18.1
  Downloading proto_plus-1.18.1-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 862 kB/s
Collecting six>=1.5.2
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from grpcio-tools==1.39.0.dev0) (56.2.0)
Collecting protobuf<4.0dev,>=3.5.0.post1
  Downloading protobuf-3.17.0-cp39-cp39-manylinux2014_aarch64.whl (925 kB)
     |████████████████████████████████| 925 kB 6.2 MB/s
Installing collected packages: six, protobuf, grpcio, proto-plus, grpcio-tools
Successfully installed grpcio-1.39.0.dev0 grpcio-tools-1.39.0.dev0 proto-plus-1.18.1 protobuf-3.17.0 six-1.16.0
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

root@fa796b3a1f02:/# ulimit -c unlimited

root@fa796b3a1f02:/# python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted (core dumped)

root@fa796b3a1f02:/# python3 -c 'import grpc, proto.message'
# no crash

root@fa796b3a1f02:/# python3 -m pip freeze
grpcio @ https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio/grpcio-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl
grpcio-tools @ https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio-tools/grpcio_tools-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl
proto-plus==1.18.1
protobuf==3.17.0
six==1.16.0

Attached: py39-googleapis-python-pubsub-issue-414.core.gz decompressing to give

➜  ~ shasum py39-googleapis-python-pubsub-issue-414.core
26975b5d6e3a7f57e2a410ad51976b27d7b32851  py39-googleapis-python-pubsub-issue-414.core
➜  ~ file py39-googleapis-python-pubsub-issue-414.core
py39-googleapis-python-pubsub-issue-414.core: ELF 64-bit LSB core file, ARM aarch64, version 1 (SYSV), SVR4-style, from 'python3 -c import proto.message, grpc;'

[gnossen]

Either my gdb-multiarch isn't set up properly, or there's something very wrong in this process:

Core was generated by `python3 -c import proto.message, grpc;'.
Program terminated with signal SIGABRT, Aborted.
#0  0x0000ffff88ba6714 in ?? ()
(gdb) bt
#0  0x0000ffff88ba6714 in ?? ()
#1  0x0000ffff89217010 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) disassemble 0x0000ffff88ba6700,0x0000ffff88ba6730
Dump of assembler code from 0xffff88ba6700 to 0xffff88ba6730:
   0x0000ffff88ba6700:  udf     #0
   0x0000ffff88ba6704:  udf     #0
   0x0000ffff88ba6708:  udf     #0
   0x0000ffff88ba670c:  udf     #0
   0x0000ffff88ba6710:  udf     #0
=> 0x0000ffff88ba6714:  udf     #0
   0x0000ffff88ba6718:  udf     #0
   0x0000ffff88ba671c:  udf     #0
   0x0000ffff88ba6720:  udf     #0
   0x0000ffff88ba6724:  udf     #0
   0x0000ffff88ba6728:  udf     #0
   0x0000ffff88ba672c:  udf     #0
End of assembler dump.
(gdb) f 1
#1  0x0000ffff89217010 in ?? ()
(gdb) disassemble 0x0000ffff89217000,0x0000ffff89217020
Dump of assembler code from 0xffff89217000 to 0xffff89217020:
   0x0000ffff89217000:  udf     #0
   0x0000ffff89217004:  udf     #0
   0x0000ffff89217008:  .inst   0x89217000 ; undefined
   0x0000ffff8921700c:  udf     #65535
=> 0x0000ffff89217010:  udf     #0
   0x0000ffff89217014:  udf     #0
   0x0000ffff89217018:  udf     #0
   0x0000ffff8921701c:  udf     #0
End of assembler dump.