ImportError: numpy.core.multiarray failed to import with NumPy 2.0.0

[khaeru]

Describe the bug
Bottleneck 1.3.8 cannot be imported with NumPy 2.0.0:

>>> import bottleneck

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "/home/khaeru/.venv/3.12/bin/ipython", line 8, in <module>
    sys.exit(start_ipython())
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/__init__.py", line 129, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/traitlets/config/application.py", line 1053, in launch_instance
    app.start()
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/terminal/ipapp.py", line 317, in start
    self.shell.mainloop()
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/terminal/interactiveshell.py", line 888, in mainloop
    self.interact()
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/terminal/interactiveshell.py", line 881, in interact
    self.run_cell(code, store_history=True)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3046, in run_cell
    result = self._run_cell(
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3101, in _run_cell
    result = runner(coro)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
    coro.send(None)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3306, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3488, in run_ast_nodes
    if await self.run_code(code, result, async_=asy):
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3548, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-baae57b98b2a>", line 1, in <module>
    import bottleneck
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/bottleneck/__init__.py", line 7, in <module>
    from .move import (move_argmax, move_argmin, move_max, move_mean, move_median,
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
File ~/.venv/3.12/lib/python3.12/site-packages/numpy/core/_multiarray_umath.py:44, in __getattr__(attr_name)
     39     # Also print the message (with traceback).  This is because old versions
     40     # of NumPy unfortunately set up the import to replace (and hide) the
     41     # error.  The traceback shouldn't be needed, but e.g. pytest plugins
     42     # seem to swallow it and we should be failing anyway...
     43     sys.stderr.write(msg + tb_msg)
---> 44     raise ImportError(msg)
     46 ret = getattr(_multiarray_umath, attr_name, None)
     47 if ret is None:

ImportError: 
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.


---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[8], line 1
----> 1 import bottleneck

File ~/.venv/3.12/lib/python3.12/site-packages/bottleneck/__init__.py:7
      5 from . import slow
      6 from ._pytesttester import PytestTester
----> 7 from .move import (move_argmax, move_argmin, move_max, move_mean, move_median,
      8                    move_min, move_rank, move_std, move_sum, move_var)
      9 from .nonreduce import replace
     10 from .nonreduce_axis import (argpartition, nanrankdata, partition, push,
     11                              rankdata)

ImportError: numpy.core.multiarray failed to import

To Reproduce

1. Command/code being executed: as above.
2. Python version and OS: Python 3.12.3, Ubuntu 24.04.
3. pip version: pip 23.2.

4. Output of `pip list` or `conda list` (expand)

Package                       Version                                Editable project location
----------------------------- -------------------------------------- --------------------------------
accessible-pygments           0.0.4
alabaster                     0.7.15
alembic                       1.13.1
annotated-types               0.6.0
anyio                         4.3.0
appdirs                       1.4.4
argcomplete                   3.3.0
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
arrow                         1.3.0
asttokens                     2.4.1
async-lru                     2.0.4
asyncssh                      2.14.2
attrs                         23.2.0
Babel                         2.14.0
beautifulsoup4                4.12.2
bleach                        6.1.0
blinker                       1.7.0
bottle                        0.12.25
Bottleneck                    1.3.8
build                         1.2.1
cachelib                      0.9.0
cattrs                        23.2.3
certifi                       2024.2.2
cffi                          1.16.0
cfgv                          3.4.0
charset-normalizer            3.3.2
click                         8.1.7
cloudpickle                   3.0.0
colorama                      0.4.6
colorlog                      6.8.2
comm                          0.2.1
contourpy                     1.2.0
coverage                      7.4.0
cryptography                  41.0.7
cycler                        0.12.1
dask                          2024.5.1
dask-expr                     1.1.1
debugpy                       1.8.0
decorator                     5.1.1
defusedxml                    0.7.1
distlib                       0.3.8
docutils                      0.20.1
et-xmlfile                    1.1.0
execnet                       2.0.2
executing                     2.0.1
fastapi                       0.110.1
fastjsonschema                2.19.1
feedgenerator                 2.1.0
filelock                      3.13.1
Flask                         3.0.3
Flask-Caching                 2.1.0
flexcache                     0.3
flexparser                    0.3.1
fonttools                     4.47.0
fqdn                          1.5.1
fsspec                        2023.12.2
genno                         1.24.2.dev32+g6b95e83.d20240323        /home/khaeru/vc/genno
git-filter-repo               2.38.0
gitdb                         4.0.11
GitPython                     3.1.40
gprof2dot                     2022.7.29
graphviz                      0.20.1
greenlet                      3.0.3
h11                           0.14.0
h2                            4.1.0
hpack                         4.0.0
httpcore                      1.0.5
httpx                         0.27.0
hyperframe                    6.0.1
iam-units                     2023.9.12
icecream                      2.1.3
identify                      2.5.33
idna                          3.7
imagesize                     1.4.1
importlib-metadata            7.0.1
importlib_resources           6.4.0
iniconfig                     2.0.0
ipykernel                     6.28.0
ipython                       8.17.1
ipywidgets                    8.1.1
isodate                       0.6.1
isoduration                   20.11.0
itsdangerous                  2.1.2
ixmp                          3.8.1.dev13+g84c3817.d20240323         /home/khaeru/vc/iiasa/ixmp
ixmp4                         0.8.2
JayDeBeApi                    1.2.3
Jinja2                        3.1.2
joblib                        1.3.2
JPype1                        1.5.0
json5                         0.9.14
jsonpointer                   2.4
jsonschema                    4.20.0
jsonschema-specifications     2023.12.1
jupyter                       1.0.0
jupyter_client                8.6.0
jupyter-console               6.6.3
jupyter_core                  5.7.0
jupyter-events                0.9.0
jupyter-lsp                   2.2.1
jupyter_server                2.12.2
jupyter_server_terminals      0.5.1
jupyterlab                    4.0.10
jupyterlab_pygments           0.3.0
jupyterlab_server             2.25.2
jupyterlab-widgets            3.0.9
khaeru                        1.0.1.dev193+g2e4990c.d20240614        /home/khaeru/vc/khaeru
kiwisolver                    1.4.5
latexcodec                    2.0.1
llvmlite                      0.42.0
locket                        1.0.0
lxml                          5.1.0
lxml-stubs                    0.5.1
Mako                          1.3.3
markdown-it-py                3.0.0
MarkupSafe                    2.1.5
matplotlib                    3.8.2
matplotlib-inline             0.1.6
mdurl                         0.1.2
memory-profiler               0.61.0
message_data                  2020.6.21.dev3019+g184c70b00.d20240408 /home/khaeru/vc/iiasa/mdata
message_ix                    3.8.1.dev55+gd7c5f4f7.d20240323        /home/khaeru/vc/iiasa/message-ix
message-ix-models             2024.4.3.dev11+gf08a3447               /home/khaeru/vc/iiasa/models
mistune                       3.0.2
mizani                        0.9.3
multimethod                   1.10
mypy                          1.9.0
mypy-extensions               1.0.0
nbclient                      0.9.0
nbconvert                     7.14.0
nbformat                      5.9.2
nest-asyncio                  1.5.8
nodeenv                       1.8.0
nomenclature-iamc             0.2
notebook                      7.0.6
notebook_shim                 0.2.3
nox                           2024.4.15
numba                         0.59.0
numpy                         2.0.0
numpydoc                      1.6.0
openpyxl                      3.1.2
oracledb                      2.1.0
ordered-set                   4.1.0
overrides                     7.4.0
owlready2                     0.45
packaging                     24.0
pandas                        2.2.2
pandas-stubs                  2.2.1.240316
pandera                       0.18.3
pandocfilters                 1.5.0
parso                         0.8.3
partd                         1.4.1
patsy                         0.5.6
pelican                       4.9.1
pexpect                       4.8.0
pickleshare                   0.7.5
pillow                        10.2.0
Pint                          0.23
pip                           23.2
platformdirs                  4.1.0
plotnine                      0.12.4
pluggy                        1.3.0
pooch                         1.8.0
pprint36                      3.9.16.0
pre-commit                    3.6.0
pretenders                    1.4.5
prometheus-client             0.19.0
prompt-toolkit                3.0.39
psutil                        5.9.7
psycopg                       3.1.18
psycopg-binary                3.1.18
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
pyam-iamc                     2.2.1
pyarrow                       14.0.2
pybtex                        0.24.0
pybtex-docutils               1.0.3
pycountry                     23.12.11
pycparser                     2.21
pydantic                      2.7.0
pydantic_core                 2.18.1
pydantic-settings             2.2.1
pydata-sphinx-theme           0.15.1
Pygments                      2.17.2
PyJWT                         2.8.0
pyparsing                     3.1.1
pyproject_hooks               1.0.0
pytest                        8.0.0
pytest-benchmark              4.0.0
pytest-cov                    4.1.0
pytest-profiling              1.7.0
pytest-rerunfailures          13.0
pytest-xdist                  3.5.0
python-dateutil               2.9.0.post0
python-dotenv                 1.0.1
python-json-logger            2.0.7
pytz                          2024.1
pyxlsb                        1.0.10
PyYAML                        6.0.1
pyzmq                         25.1.2
qtconsole                     5.5.1
QtPy                          2.4.1
rdflib                        7.0.0
referencing                   0.32.1
requests                      2.31.0
requests-cache                1.2.0
requests-mock                 1.11.0
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rich                          13.7.1
rpds-py                       0.16.2
scikit-learn                  1.3.2
scipy                         1.11.4
sdmx1                         2.14.1.dev14                           /home/khaeru/vc/sdmx
seaborn                       0.13.1
Send2Trash                    1.8.2
setuptools                    69.0.3
setuptools-scm                8.0.4
shellingham                   1.5.4
six                           1.16.0
smmap                         5.0.1
sniffio                       1.3.1
snowballstemmer               2.2.0
soupsieve                     2.5
sparse                        0.15.1
Sphinx                        7.2.6
sphinx-book-theme             1.1.0
sphinx-rtd-theme              2.0.0
sphinxcontrib-applehelp       1.0.7
sphinxcontrib-bibtex          2.6.1
sphinxcontrib-devhelp         1.0.5
sphinxcontrib-htmlhelp        2.0.4
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.6
sphinxcontrib-serializinghtml 1.1.9
SQLAlchemy                    2.0.29
SQLAlchemy-Utils              0.41.2
stack-data                    0.6.3
starlette                     0.37.2
statsmodels                   0.14.1
tabulate                      0.9.0
terminado                     0.18.0
threadpoolctl                 3.2.0
tinycss2                      1.2.1
toml                          0.10.2
toolz                         0.12.0
tornado                       6.4
tqdm                          4.66.2
traitlets                     5.13.0
transport-energy              2021.5.5.dev45+ga7d092b                /home/khaeru/vc/item/db
typeguard                     4.2.1
typer                         0.12.3
types-docutils                0.20.0.20240317
types-python-dateutil         2.8.19.20240106
types-pytz                    2024.1.0.20240203
types-PyYAML                  6.0.12.20240311
types-requests                2.31.0.20240311
types-tqdm                    4.66.0.20240106
typing_extensions             4.11.0
typing-inspect                0.9.0
tzdata                        2024.1
Unidecode                     1.3.8
uri-template                  1.3.0
url-normalize                 1.4.3
urllib3                       2.1.0
uvicorn                       0.29.0
virtualenv                    20.25.0
watchfiles                    0.22.0
wcwidth                       0.2.9
webcolors                     1.13
webencodings                  0.5.1
websocket-client              1.7.0
Werkzeug                      3.0.2
widgetsnbextension            4.0.9
wquantiles                    0.6
wrapt                         1.16.0
xarray                        2023.12.0
xdg-base-dirs                 6.0.1
xlrd                          2.0.1
XlsxWriter                    3.2.0
zipp                          3.17.0

Expected behavior
bottleneck is importable and usable.

Additional context
Encountered via daily CI for the genno package, here.

[neutrinoceros]

@khaeru in case you didn't know: bottleneck 1.4.0rc5 is compatible, so you may be able to hotfix your CI by requiring the pre-release somehow.

[khaeru]

bottleneck 1.4.0rc5 is compatible, so you may be able to hotfix your CI by requiring the pre-release somehow.

Yes, thank you; I did see it at https://pypi.org/project/Bottleneck/#history and should have come back to mention for others in the same boat.

In our case:

• I'm not familiar with the maintenance practices of bottleneck.
• Thus, I don't know if the >3 week old RC means there might perhaps still be other release-blocking issues in 1.4.0, and don't have the bandwidth to discover this. (The release notes don't even mention 1.3.8.)
• We have other dependencies that also have NumPy 2.0 issues.

…so the simpler expedient is to temporarily limit to numpy < 2.

[seberg]

It would indeed be great to expedite a release, even if it might still have some issues. E.g. if users do a fresh install with pandas and bottleneck they otherwise run into issues (already at import time, although if not then it would happen later I guess).

[rdbisme]

Hello, I'm working on the release.

Thus, I don't know if the >3 week old RC means there might perhaps still be other release-blocking issues in 1.4.0, and don't have the bandwidth to discover this. (The release notes don't even mention 1.3.8.)

Unfortunately, I have no access to RTD. I tried to ask, but nobody gave me access. The RELEASE.rst file was up to date tho in the repo.

[seberg]

Thanks for getting the release on PyPI!

[khaeru]

Ditto, thanks!

Unfortunately, I have no access to RTD. I tried to ask, but nobody gave me access.

That's unfortunate indeed…my impression had been that Pydata were better organized than that 🙁

Perhaps until they resolve that access, the README could be updated to say “(up to v1.3.7 only)” right next to the RTD link.