Python Hashlib & A True Story of One Bug

Python Hashlib &
A True Story of One Bug
Dmitry Alimov

$ whoami
Software engineer
Python Software Foundation (PSF) contributing member
Python meetups member & speaker
Co-organized Python meetups/drinkups
Played CTF with SiBears
2

Plan
A True Story of One Bug
3
hashlib

Test Framework
Cloud
Telephony
Services
Shared storage
IP Phone
Test Automation Framework
5

Authentication with remote machine "SERVER" for user "user" will be using NTLM v1 authentication (with extended
security)
Now switching over to SMB2 protocol communication
SMB2 dialect negotiation successful
Performing NTLMv2 authentication (on SMB2) with server challenge "b'da16c5c3e38ca2df'"
Performing NTLMv1 authentication (on SMB2) with server challenge "b'da16c5c3e38ca2df'"
Server supports SMB signing
SMB signing deactivated. SMB messages will NOT be signed.
Authentication (on SMB2) failed. Please check username and password.
Traceback (most recent call last):
File "/home/user/proj/smb_module.py", line 43, in <module>
downloader.list_dir()
File "/home/user/proj/
smb_module.py", line 36, in list_dir
all_builds = self._connection.listPath(self.service_name, path)
File "/home/user/proj/venv/lib/python3.9/site-packages/smb/SMBConnection.py", line 201, in listPath
self._listPath(service_name, path, cb, eb, search = search, pattern = pattern, timeout = timeout)
File "/home/user/proj/venv/lib/python3.9/site-packages/smb/base.py", line 601, in _listPath_SMB2
raise NotReadyError('SMB connection not authenticated')
smb.base.NotReadyError: SMB connection not authenticated
Problem
6

Code
import logging
from smb.SMBConnection import SMBConnection
logging.getLogger().setLevel(logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler())
class SMBDownloader:
def __init__(self, host: str, username: str, password: str, service_name: str = 'BIN'):
self.service_name = service_name
self._connection = SMBConnection(
remote_name=host, username=username, password=password, my_name='...')
self._connection.connect(remote_name)
def list_dir(self, path: str = '.'):
all_files = self._connection.listPath(
self.service_name, path)
print([item.filename for item in all_files])
downloader = SMBDownloader(
'server', 'user', 'password')
downloader.list_dir()
7

Python verbose mode
$ python -v ...
$ PYTHONVERBOSE=1 python ...
Source: [1]
16

Run in verbose mode, to analyse imported modules
$ python -v smb_module.py # on host where OK
...
import 'nmb.utils' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3199c4b370>
import 'nmb.base' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3199c42ca0>
# /usr/local/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc matches
/usr/local/lib/python3.9/site-packages/smb/ntlm.py
# code object from '/usr/local/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc'
# /usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc matches
/usr/local/lib/python3.9/site-packages/smb/utils/rc4.py
# code object from '/usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc'
import 'smb.utils.rc4' # <_frozen_importlib_external.SourceFileLoader object at 0x7f319bdd2a00>
# /usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc matches
/usr/local/lib/python3.9/site-packages/smb/utils/pyDes.py
# code object from '/usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc'
import 'smb.utils.pyDes' # <_frozen_importlib_external.SourceFileLoader object at 0x7f319bdd2970>
import 'smb.ntlm' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3199c4b220>
...
17

Run in verbose mode, to analyse imported modules
$ python -v smb_module.py # on host where FAILS
...
import 'nmb.utils' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea1a700>
import 'nmb.base' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea0ca90>
# /home/user/proj/venv/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/ntlm.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc'
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/rc4.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc'
import 'smb.utils.rc4' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea03640>
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/pyDes.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc'
import 'smb.utils.pyDes' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea03580>
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/md4.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/md4.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/md4.cpython-39.pyc'
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/U32.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/U32.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/U32.cpython-39.pyc'
import 'smb.utils.U32' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9e572a00>
import 'smb.utils.md4' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9e572550>
import 'smb.ntlm' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea03e80>
...
18

Where is it imported from?
Source: [2]
19
try:
import hashlib
hashlib.new('md4')
def MD4(): return hashlib.new('md4')
except ( ImportError, ValueError ):
from .utils.md4 import MD4

Try to use hash type – OK
(venv) $ python # on host where OK
Python 3.9.15 (main, Oct 12 2022, 19:14:37)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.new( 'md4')
<md4 _hashlib.HASH object @ 0x7ffbedfbbc70>
20

Try to use hash type – FAILS
21
(venv) $ python # on host where FAILS
Python 3.9.15 (main, Oct 12 2022, 19:14:37)
>>> import hashlib
File "/usr/lib/python3.9/hashlib.py", line 160, in __hash_new
return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported
During handling of the above exception, another exception occurred:
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/hashlib.py", line 166, in __hash_new
return __get_builtin_constructor(name)(data)
File "/usr/lib/python3.9/hashlib.py", line 123, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md4

pysmb-provided MD4 implementation for Python3 is broken
https:/
/github.com/miketeo/pysmb/issues/196
pysmb 1.2.8 ﬁxes this issue
Bug
22
Source: [3]

Fix
23
b'x01x00x02x00x03x00'
b'x01x02x03'
Fix embedded MD4 algorithm
https:/
/github.com/miketeo/pysmb/pull/198
Source: [4]

1. Update pysmb in project to 1.2.8
2. Enable legacy providers in openssl conﬁg
Solutions
24

$ openssl version -d
OPENSSLDIR: "/usr/lib/ssl"
Update /usr/lib/ssl/openssl.cnf
...
openssl_conf = openssl_init
[openssl_init]
providers = provider_sect
[provider_sect]
default = default_sect
legacy = legacy_sect
[default_sect]
activate = 1
[legacy_sect]
activate = 1
...
How to enable legacy providers in conﬁg
Source: [5]
25

Try to use hash type after enable legacy providers
(venv) $ python # on host where FAILED
Python 3.9.15 (main, Oct 12 2022, 19:14:37)
>>> import hashlib
<md4 _hashlib.HASH object @ 0x7fe885fade10>
26

Authentication with remote machine "SERVER" for user "user" will be using NTLM v1 authentication (with extended
security)
Now switching over to SMB2 protocol communication
SMB2 dialect negotiation successful
Performing NTLMv2 authentication (on SMB2) with server challenge "b'137550e8bf56b78d'"
Performing NTLMv1 authentication (on SMB2) with server challenge "b'137550e8bf56b78d'"
Server supports SMB signing
SMB signing deactivated. SMB messages will NOT be signed.
Authentication (on SMB2) successful!
...
Problem solved
27

How hashlib is related to OpenSSL
28
hashlib.py _hashlib.so libcrypto.so
python3.9
$ objdump -T python3.9 | egrep "blake|md4|md5|sha1|sha3|sha256|sha512"
003a90df g DF .text 0000008c Base PyInit__sha512
003a5998 g DF .text 00000058 Base PyInit__sha1
003a5e9d g DF .text 000001da Base PyInit__sha3
00304aca g DF .text 00000058 Base PyInit__md5
00388b47 g DF .text 000002fb Base PyInit__blake2
003a5b7f g DF .text 0000008c Base PyInit__sha256
$ objdump -T libcrypto.so
...
... g DF .text ...0c OPENSSL_1_1_1 EVP_sha3_384
... g DF .text ...0c OPENSSL_1_1_0 EVP_md4
... g DF .text ...0c OPENSSL_1_1_0 EVP_md5
... g DF .text ...0c OPENSSL_1_1_0
EVP_blake2s256
... g DF .text ...0c OPENSSL_1_1_0 EVP_sha1
... g DF .text ...0c OPENSSL_1_1_0 EVP_md5_sha1
...
$ objdump -T _hashlib.cpython-39-x86_64-linux-gnu.so
...
00000000 DF *UND* 00000000 OPENSSL_1_1_0
EVP_blake2s256
00000000 DF *UND* 00000000 OPENSSL_1_1_0 EVP_sha384
00000000 DF *UND* 00000000 OPENSSL_1_1_1 EVP_sha3_512
00000000 DF *UND* 00000000 OPENSSL_1_1_1 EVP_shake128
00000000 DF *UND* 00000000 OPENSSL_1_1_0 EVP_md5
...
OpenSSL consists of two libraries: libcrypto and libssl

Changes between 1.1.1 and 3.0.0 [7 sep 2021]
...
* The implementation of the EVP digests MD2, MD4, MDC2, WHIRLPOOL and
RIPEMD-160 have been moved to the legacy provider.
…
* The low-level MD2, MD4, MD5, MDC2, RIPEMD160 and Whirlpool digest
functions have been deprecated.
…
MD4 is deprecated and disabled by default in OpenSSL 3.0.0
29
Source: [6]

>>> import hashlib
>>> import ssl
>>> print(ssl.OPENSSL_VERSION)
>>> print(hashlib.algorithms_available)
OpenSSL 1.1.1g 21 Apr 2020
{'sha3_224', 'sha512', 'shake_128' , 'md4', 'sha3_256', 'sha3_384', 'shake_256' , 'ripemd160',
'sha384', 'blake2b', 'sha1', 'whirlpool', 'sha3_512', 'md5', 'sha224', 'sha512_224' , 'md5-sha1',
'blake2s', 'sha256', 'sha512_256' , 'sm3'}
OpenSSL 3.0.2 15 Mar 2022
{'sha3_224', 'sm3', 'sha3_384', 'sha512_256' , 'sha224', 'blake2b', 'sha512', 'sha3_512', 'sha384',
'blake2s', 'sha1', 'shake_128' , 'md5', 'sha256', 'sha512_224' , 'md5-sha1', 'sha3_256',
'shake_256' }
...
ValueError: unsupported hash type md4
Hashlib and OpenSSL version
30

Hashlib
and OpenSSL
version
algorithms_guaranteed
OpenSSL 1.1.1
algorithms_available
OpenSSL 3.0.2
BLAKE2B BLAKE2B BLAKE2B
BLAKE2S BLAKE2S BLAKE2S
MD4
MD5 MD5 MD5
MD5-SHA1 MD5-SHA1
RIPEMD160
SHA1 SHA1 SHA1
SHA224 SHA224 SHA224
SHA3_224 SHA3_224 SHA3_224
SHA3_256 SHA3_256 SHA3_256
SHA3_384 SHA3_384 SHA3_384
SHA3_512 SHA3_512 SHA3_512
SHA512_224 SHA512_224
SHA512_256 SHA512_256
SHAKE_128 SHAKE_128 SHAKE_128
SHAKE_256 SHAKE_256 SHAKE_256
SM3 SM3
WHIRLPOOL
31

Source: [7]
hashlib.py – algorithms constants
34

Source: [8]
hashlib.py – algorithms_guaranteed
35
# This tuple and __get_builtin_constructor() must be modified if a new
# always available algorithm is added.
__always_supported = ( 'md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512',
'blake2b', 'blake2s',
'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512',
'shake_128' , 'shake_256' )
algorithms_guaranteed = set(__always_supported)

algorithms_available = set(__always_supported)
...
try:
import _hashlib
new = __hash_new
__get_hash = __get_openssl_constructor
algorithms_available = algorithms_available.union(
_hashlib.openssl_md_meth_names)
except ImportError :
new = __py_new
__get_hash = __get_builtin_constructor
hashlib.py – algorithms_available
Source: [9]
36
hashlib.py _hashlib.so
$ objdump -T _hashlib.cpython-39-x86_64-linux-gnu.so
...
00000000 DF *UND* 00000000 OPENSSL_1_1_0
EVP_blake2s256
00000000 DF *UND* 00000000 OPENSSL_1_1_1 EVP_sha3_512
00000000 DF *UND* 00000000 OPENSSL_1_1_1 EVP_shake128
00000000 DF *UND* 00000000 OPENSSL_1_1_0 EVP_md5
...
libcrypto.so

/* Ask OpenSSL for a list of supported ciphers, filling in a Python set. */
static int
hashlib_md_meth_names(PyObject *module)
{
...
#if OPENSSL_VERSION_NUMBER >= 0x30000000L
// get algorithms from all activated providers in default context
EVP_MD_do_all_provided(NULL, &_openssl_hash_name_mapper, &state);
#else
EVP_MD_do_all(&_openssl_hash_name_mapper, &state);
#endif
...
if (PyModule_AddObject(module, "openssl_md_meth_names", state.set) < 0) {
Py_DECREF(state.set);
return -1;
}
return 0;
}
_hashopenssl.c
Source: [10]
OpenSSL API functions
>= OpenSSL 3.0.0
37

>>> import hashlib
>>> import _hashlib
>>> import ssl
>>> _hashlib.openssl_md_meth_names
frozenset({'sha3_512', 'sha3_384', 'md5', 'ripemd160' , 'sha512', 'sha256', 'whirlpool' , 'blake2b',
'sha3_224', 'blake2s', 'sha1', 'shake_128' , 'shake_256' , 'sha3_256', 'sm3', 'sha224', 'md4',
'sha512_256' , 'sha512_224' , 'sha384', 'md5-sha1'})
>>> print(ssl.OPENSSL_VERSION)
OpenSSL 1.1.1g 21 Apr 2020
>>> print(hashlib.algorithms_available)
{'sha3_512', 'sha3_384', 'md5', 'ripemd160' , 'sha512', 'sha256', 'whirlpool' , 'blake2b',
'sha3_224', 'sha1', 'blake2s', 'shake_128' , 'shake_256' , 'sha3_256', 'sm3', 'sha224', 'md4',
'sha512_256' , 'sha512_224' , 'sha384', 'md5-sha1'}
>>> hashlib.algorithms_available ^ hashlib.algorithms_guaranteed
{'whirlpool' , 'md4', 'sha512_256' , 'sha512_224' , 'ripemd160' , 'sm3', 'md5-sha1'}
hashlib.py – algorithms available
38

hashlib.py
and OpenSSL*
algorithms
available
39
algorithms_guaranteed
OpenSSL 1.1.1
BLAKE2B BLAKE2B
BLAKE2S BLAKE2S
MD4
MD5 MD5
MD5-SHA1
RIPEMD160
SHA1 SHA1
SHA224 SHA224
SHA256 SHA256
SHA384 SHA384
SHA3_224 SHA3_224
SHA3_256 SHA3_256
SHA3_384 SHA3_384
SHA3_512 SHA3_512
SHA512 SHA512
SHA512_224
SHA512_256
SHAKE_128 SHAKE_128
SHAKE_256 SHAKE_256
SM3
WHIRLPOOL
* OpenSSL 1.1.1

Sources: [7], [11]
hashlib.py – new
40
allows using insecure hashes
in OpenSSL FIPS mode

try:
import _hashlib
new = __hash_new
new = __py_new
hashlib.py – new
Source: [12]
41
__get_builtin_constructor

def __hash_new(name, data= b'', **kwargs):
"""new(name, data=b'') - Return a new hashing object using the named algorithm;
optionally initialized with data (which must be a bytes-like object).
"""
if name in __block_openssl_constructor:
# Prefer our builtin blake2 implementation.
return __get_builtin_constructor(name)(data, **kwargs)
try:
return _hashlib.new(name, data, **kwargs)
except ValueError:
# If the _hashlib module (OpenSSL) doesn't support the named
# hash, try using our builtin implementations.
# This allows for SHA224/256 and SHA384/512 support even though
# the OpenSSL library prior to 0.9.8 doesn't provide them.
return __get_builtin_constructor(name)(data)
hashlib.py – __hash_new
Source: [13]
'blake2b', 'blake2s'
42

def __get_builtin_constructor(name):
cache = __builtin_constructor_cache
constructor = cache.get(name)
if constructor is not None:
return constructor
try:
if name in {'SHA1', 'sha1'}:
import _sha1
cache['SHA1'] = cache['sha1'] = _sha1.sha1
elif name in {'MD5', 'md5'}:
import _md5
cache['MD5'] = cache['md5'] = _md5.md5
...
elif name in {'blake2b', 'blake2s'}:
import _blake2
cache['blake2b'] = _blake2.blake2b
cache['blake2s'] = _blake2.blake2s
...
except ImportError:
pass # no extension module, this hash is unsupported.
constructor = cache.get(name)
if constructor is not None:
return constructor
raise ValueError('unsupported hash type ' + name)
hashlib.py – __get_builtin_constructor
Source: [14]
43
hashlib.py
python3.9
$ objdump -T python3.9 | egrep "blake|md4|md5|sha1|sha3|sha256|sha512"
003a90df g DF .text 0000008c Base PyInit__sha512
003a5998 g DF .text 00000058 Base PyInit__sha1
003a5e9d g DF .text 000001da Base PyInit__sha3
00304aca g DF .text 00000058 Base PyInit__md5
00388b47 g DF .text 000002fb Base PyInit__blake2
003a5b7f g DF .text 0000008c Base PyInit__sha256

try:
import _hashlib
new = __hash_new
new = __py_new
hashlib.py – __get_hash
Source: [15]
44
to prepare named constructors

def __get_openssl_constructor (name):
if name in __block_openssl_constructor:
# Prefer our builtin blake2 implementation.
return __get_builtin_constructor(name)
try:
# MD5, SHA1, and SHA2 are in all supported OpenSSL versions
# SHA3/shake are available in OpenSSL 1.1.1+
f = getattr(_hashlib, 'openssl_' + name)
# Allow the C module to raise ValueError. The function will be
# defined but the hash not actually available. Don't fall back to
# builtin if the current security policy blocks a digest, bpo#40695.
f(usedforsecurity =False)
# Use the C function directly (very fast)
return f
except (AttributeError , ValueError):
return __get_builtin_constructor(name)
hashlib.py – __get_openssl_constructor
Source: [16]
'blake2b', 'blake2s'
45

for __func_name in __always_supported:
# try them all, some may not work due to the OpenSSL
# version not supporting that algorithm.
try:
globals()[__func_name] = __get_hash(__func_name)
except ValueError:
import logging
logging.exception( 'code for hash %s was not found.' , __func_name)
hashlib.py – prepare all named constructors for algorithms_guaranteed
>>> hashlib. __dict__
{..., 'md5': <built-in function openssl_md5>, 'sha1': <built-in function openssl_sha1>,
'sha224': <built-in function openssl_sha224>, 'sha256': <built-in function openssl_sha256>,
'sha384': <built-in function openssl_sha384>, 'sha512': <built-in function openssl_sha512>,
'blake2b': <class '_blake2.blake2b'>, 'blake2s': <class '_blake2.blake2s'>, 'sha3_224': <built-in
function openssl_sha3_224>, 'sha3_256': <built-in function openssl_sha3_256>, 'sha3_384':
<built-in function openssl_sha3_384>, 'sha3_512': <built-in function openssl_sha3_512>,
'shake_128' : <built-in function openssl_shake_128>, 'shake_256' : <built-in function
openssl_shake_256>}
Source: [17]
__get_openssl_constructor
or
__get_builtin_constructor
46

hashlib.py – key derivation
Source: [7]
48

hashlib.py – new feature in Python 3.11
Source: [7]
49

Sources: [18], [19]
PEP 644 – Require OpenSSL 1.1.1 or newer
Version 3.0 will be supported until 2026-09-07 (LTS).
Version 1.1.1 will be supported until 2023-09-11 (LTS).
Version 1.0.2 is no longer supported. Extended support for 1.0.2 to gain access to security ﬁxes
for that version is available.
Versions 1.1.0, 1.0.1, 1.0.0 and 0.9.8 are no longer supported.
50
OpenSSL 1.1.1 -> TLS 1.3 version with faster handshake and more secure.
OpenSSL 1.1.0c+ -> Fully fork and thread safe.
OpenSSL 1.1.0+ ships with SHA-3 and SHAKE. Python could rely on OpenSSL’s libcrypto.
Beneﬁts:

Summary
51
Python has a set of guaranteed hash algorithms.
Some extra algorithms Python loads from OpenSSL’s libcrypto.
The available OpenSSL algorithms may differ depending on the versions.
Remember about “python -v” verbose mode for analysis.

References
52
1. https:/
/docs.python.org/3/using/cmdline.html
2. https:/
/github.com/miketeo/pysmb/blob/pysmb-1.2.6/python3/smb/ntlm.py#L6-L12
3. https:/
/github.com/miketeo/pysmb/issues/196/
4. https:/
/github.com/miketeo/pysmb/pull/198/ﬁles/
5. https:/
/wiki.openssl.org/index.php/OpenSSL_3.0
6. https:/
/www.openssl.org/news/changelog.txt
7. https:/
/docs.python.org/3/library/hashlib.html
8. https:/
/github.com/python/cpython/blob/v3.9.15/Lib/hashlib.py#L56-L65
9. https:/
10. https:/
/github.com/python/cpython/blob/v3.9.15/Modules/_hashopenssl.c#L1891-L1921
11. https:/
/peps.python.org/pep-0452
12. https:/
13. https:/
14. https:/
15. https:/
16. https:/
17. https:/
18. https:/
/peps.python.org/pep-0644
19. https:/
/www.openssl.org/policies/releasestrat.html

Python Hashlib & A True Story of One Bug

More Related Content

Similar to Python Hashlib & A True Story of One Bug (20)

More from delimitry (19)

Recently uploaded (20)

Python Hashlib & A True Story of One Bug