Exposing cuDSS in the Python bindings

[PTNobel]

Here's my initial attempt at developing Python bindings for the new cuDSS solver. It compiles without error when I run

pip install --verbose -Csetup-args=-Dlink_cudss=true -Csetup-args=-Dint32=true .

but then python -m _scs_cudss fails, so I could use some help understanding what is wrong.

CC: @kalmarek

[kalmarek]

I've never done this, so I most probably won't be of much immediate help;
What is the error massage?

[PTNobel]

In [2]: import _scs_direct

In [2]: import _scs_cudss
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 1
----> 1 import _scs_cudss

ModuleNotFoundError: No module named '_scs_cudss'

Yet,

(scs-python) [ptnobel@PTNobel-P15 scs-python]$ ls .venv/lib/python3.12/site-packages/scs
__init__.py  _scs_cudss.cpython-312-x86_64-linux-gnu.so  _scs_direct.cpython-312-x86_64-linux-gnu.so  _scs_indirect.cpython-312-x86_64-linux-gnu.so

[PTNobel]

The legacy_setup.py script doesn't work with Python 3.12 since NumPy got rid of distutil

[PTNobel]

So we need to figure out the meson build

[PTNobel]

And it is all fixed and working locally for me!

[PTNobel]

FYI, the MKL tests are broken (i.e. never running), so I'm going to fix them in this PR too

============================= test session starts ==============================
platform linux -- Python 3.12.11, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/runner/work/scs-python/scs-python
configfile: pyproject.toml
testpaths: test
collected 26 items

test/test_scs_basic.py .....                                             [ 19%]
test/test_scs_object.py ......                                           [ 42%]
test/test_scs_rand.py .....                                              [ 61%]
test/test_scs_sdp.py .....                                               [ 80%]
test/test_solve_random_cone_prob.py .....                                [100%]

============================== 26 passed in 5.05s ==============================

[PTNobel]

Fixed!

============================= test session starts ==============================
platform linux -- Python 3.12.11, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/runner/work/scs-python/scs-python
configfile: pyproject.toml
testpaths: test
collected 29 items

test/test_scs_basic.py .....                                             [ 17%]
test/test_scs_object.py ......                                           [ 37%]
test/test_scs_rand.py .....                                              [ 55%]
test/test_scs_sdp.py .....                                               [ 72%]
test/test_solve_random_cone_prob.py .....                                [ 89%]
test/test_solve_random_cone_prob_mkl.py ...                              [100%]

============================== 29 passed in 5.14s ==============================

[kalmarek]

@PTNobel great to hear! did you also run tests with scs_cudss?

[kalmarek]

we might ask if link_cudss is better than use_cudss, but in general there's some inconsistency in the names, so hmm...

I'd say

option('use_cudss', type: 'boolean',
       value: false, description: 'enable direct solver on GPU using cuDSS library')
option('use_gpu`, type: 'boolean',
       value: false, description: 'enable indirect solver on GPU')

would be more descriptive.

(this is bikeshedding, feel free to ignore ;))

[PTNobel]

I personally think link for libraries and use for other things makes sense to me, but I'll leave it up to Brendan.

[PTNobel]

did you also run tests with scs_cudss?

Of course.

[kalmarek]

this is what I get from ldd libscscudss.so

	linux-vdso.so.1 (0x00007ffc69ba4000)
	libcudart.so.12 => /usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudart.so.12 (0x0000702b59000000)
	libcudss.so.0 => /opt/cudss/lib/./libcudss.so.0 (0x0000702b55000000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000702b59375000)
	libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x0000702b59308000)
	liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x0000702b54800000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000702b54400000)
	/lib64/ld-linux-x86-64.so.2 (0x0000702b59493000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x0000702b59301000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000702b592fc000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x0000702b592f7000)
	libcublas.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.12 (0x0000702b4dc00000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000702b592c7000)
	libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0 (0x0000702b4b820000)
	libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x0000702b4b400000)
	libcublasLt.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.12 (0x0000702b19200000)

Looking at the corresponding line for the gpu indirect build I guess we need to say

     libraries=list(libraries) + ["cudart", "cublas", "cublasLt", "cudss"]

I'm not sure what to do with include_dirs and library_dirs, the solution above is very rudimentary ;)
Following scs.mk I guess we should ask for os.environ["CUDA_PATH"] and os.environ["CUDSS_PATH"] and add $CUDA_PATH/lib and $CUDA_PATH/include where necessary (same for cudss).

maybe @enzbus who knows more about setup.py can suggest something

[PTNobel]

The setup.py isn't available on Python 3.12 and later because of the np.distutils deprecation. I'm not sure it's worth trying to add new features to it, but I guess that is Brendan's call.

[kalmarek]

@enzbus who is the author of the meson build also had some afterthoughts about it (see #120). Let's leave it as a comment for him then.

[enzbus]

Sorry guys, I did my best to get the meson build up and running back in '23 (there were no SCS wheels compatible with cPython 3.12 for quite some time) but know very little about it. Seemed like the easiest way to get it to work, we copied from Scipy and Numpy setups, I'm no expert. I guess you made it to work, nice!

[kalmarek]

@bodono This looks ready, but take into the account that I'm not a python specialist ;)

[PTNobel]

Quick note: This PR is currently blocking the CVXPY 1.7.0 release

[bodono]

Great, thanks @PTNobel and @kalmarek !