CentOS 5 support ends soon

[lukeyeager]

CentOS 5 support ends in about a month and a half:

CentOS-5 updates until March 31, 2017
https://wiki.centos.org/FAQ/General#head-fe8a0be91ee3e7dea812e8694491e1dde5b75e6d

Is there a plan to migrate PEP-513 to a new minimum subset of libraries+versions eventually?

Please direct me to an existing discussion or a more appropriate forum if one exists.

[njsmith]

There is a plan in the sense that everyone agrees that this is a thing that should happen, but not in the sense of anyone having actually volunteered to do the work :-).

[ogrisel]

@lukeyeager would you like to volunteer to work on a draft of a new PEP for manylinux2 + CI infrastructure to actually generate manylinux2 compatible binaries (reusing but upgrading the existing scripts that we use to maintain the manylinux1 docker images)?

[lukeyeager]

Well that's a delightfully positive response! Glad to hear there is interest in pushing out an update. Unfortunately I don't think this falls under my purview at work, nor do I currently have the time to do it after hours.

[ogrisel]

Defining an updated whitelist for manylinux2 would also be a simple way to tackle the curses 5 -> 6 issue: #94.

[dvarrazzo]

A psycopg user reported that the manylinux psycopg2 2.7 package is failing connection because Centos 5.11 ships with libssl 0.9.8e (psycopg/psycopg2#518), while the missing features are available e.g. in 1.0.2k

How would you suggest to proceed?

• try to build openssl, libpq etc. on Centos 5?
• try to build wheels on CentOS 6?
• other?

[njsmith]

It's up to you really, but the"official" manylinux1 solution would be to build your own openssl on centos5 and then ship that, yeah. Technically this is pretty easy to do -- the manylinux build scripts actually do it because the centos 5 openssl is so useless that pip wouldn't work otherwise. It's just a curl / tar / configure / make. The hard part is that then you're responsible for releasing a new wheel whenever there's a new openssl security bug. (Which is why we don't make our openssl build available -- we want there to be a speedbump so people stop and think about this.) But IIRC you're already shipping openssl in all your wheels, so for you this responsibility isn't new...

[dvarrazzo]

Yes, needing to build a new package if a new bug in openssl is found is something I had already figured out, no problem with that.

The problem with psycopg is that both libssl and libpq - the postgres client library - must be built with all the dependencies needed to allow different form of authentications (pam, kerberos, etc.). But yesterday I already whipped together a package with several features in this script. I'll try building wheels with that, check if all the linked libraries are present (using system libpq vs. the one I build myself), rinse, repeat, ship.

[lukeyeager]

Update: EOL happened

[matham]

We were having issues getting manylinux to build on centos 5 because for kivy we need gl es2 and centos 5 doesn't seem to have it available. I started going down the rabit hole to compile mesa on centos 5, but there are just too many dependencies missing.

So, we were wondering, would we be terrible people if we build on centos 7 (even centos 6 is now in maintenance updates only and reaches eol at end of 2020) and distribute them as manylinux? We are aware it may only run on newer system, but it is probably ok for us.

[rmcgibbo]

So, we were wondering, would we be terrible people if we build on centos 7 (even centos 6 is now in maintenance updates only and reaches eol at end of 2020) and distribute them as manylinux? We are aware it may only run on newer system, but it is probably ok for us.

I would encourage you not to do this. It would be unfortunate, in my opinion, to distribute wheels on PyPI using the "manylinux1" tag which do not conform to PEP 513.

[njsmith]

Note though that we're actively seeking volunteers to help make manylinux2 happen officially.

[rmcgibbo]

Would this help? https://mail.python.org/pipermail/wheel-builders/2017-July/000282.html

[antocuni]

I mention this issue here because I think it is relevant to the manylinux2 roadmap; I tried to build wheels and run auditwheel repair on centos6, and got bitten by this bug of patchelf:
NixOS/patchelf#128

[njsmith]

@antocuni: not quite sure what you were hoping to accomplish there, but I can't think of any wheel-related situation where running patchelf on libm is a good idea :-).

If the problem also occurs for other centos6 binaries then that is more concerning...

[antocuni]

@njsmith I originally encountered the problem by developing this: https://github.com/antocuni/pypy-wheels
I have to use centos6 because pypy3 apparently does not build on centos5, and because the portable pypy binaries are built on centos6 as well: note that the antocuni/pypy-wheels image is basically the same as manylinux1, only based on centos6 instead of centos5..

However, it seems not to be related to pypy at all, as I can reproduce the problem even with cpython. I don't know why auditwheel repair chooses to patch libm: I assumed it was a normal behavior, but from what you write it seems it's not :):

$ docker run -it antocuni/pypy-wheels:3b01df8e /bin/bash
[root@e5eae337a5da /]# /opt/python/cp36-cp36m/bin/pip wheel numpy --no-binary :all:
Collecting numpy
  Downloading numpy-1.13.1.zip (5.0MB)
    100% |████████████████████████████████| 5.0MB 222kB/s 
Building wheels for collected packages: numpy
  Running setup.py bdist_wheel for numpy ... done
  Stored in directory: /
Successfully built numpy
[root@e5eae337a5da /]# auditwheel repair --plat linux_x86_64 numpy-1.13.1-cp36-cp36m-linux_x86_64.whl 
Repairing numpy-1.13.1-cp36-cp36m-linux_x86_64.whl
Grafting: /lib64/libm-2.12.so -> numpy/.libs/libm-2-e51d9a45.12.so
Grafting: /lib64/libpthread-2.12.so -> numpy/.libs/libpthread-2-fdb92d01.12.so
Grafting: /lib64/libc-2.12.so -> numpy/.libs/libc-2-8b5594ec.12.so
Setting RPATH: numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Grafting: /usr/lib64/atlas/libptf77blas.so.3.0 -> numpy/.libs/libptf77blas-842266ad.so.3.0
Grafting: /usr/lib64/libgfortran.so.3.0.0 -> numpy/.libs/libgfortran-6ca0b838.so.3.0.0
Grafting: /usr/lib64/atlas/libptcblas.so.3.0 -> numpy/.libs/libptcblas-f3eb6402.so.3.0
Grafting: /usr/lib64/atlas/libatlas.so.3.0 -> numpy/.libs/libatlas-b6fb69b0.so.3.0
Setting RPATH: numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/multiarray_tests.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/operand_flag_tests.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/struct_ufunc_test.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/test_rational.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/umath.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/umath_tests.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Grafting: /usr/lib64/atlas/libcblas.so.3.0 -> numpy/.libs/libcblas-70dd3697.so.3.0
Grafting: /usr/lib64/atlas/liblapack.so.3.0 -> numpy/.libs/liblapack-7aef6c42.so.3.0
Grafting: /usr/lib64/atlas/libf77blas.so.3.0 -> numpy/.libs/libf77blas-75a5c7e0.so.3.0
Grafting: /lib64/libgcc_s-4.4.7-20120601.so.1 -> numpy/.libs/libgcc_s-4-09117363.4.7-20120601.so.1
Setting RPATH: numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main_repair.py", line 81, in execute
    update_tags=args.UPDATE_TAGS)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/repair.py", line 91, in repair_wheel
    needed = elf_read_dt_needed(path)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/elfutils.py", line 16, in elf_read_dt_needed
    raise ValueError('Could not find soname in %s' % fn)
ValueError: Could not find soname in numpy/.libs/libm-2-e51d9a45.12.so

[adrientetar]

We were having issues getting manylinux to build on centos 5 because for kivy we need gl es2 and centos 5 doesn't seem to have it available.

wxPython had similar problems with centos 5 having only old gtk2 libs and no gtk3, see: https://github.com/RobinD42/Phoenix/tree/a8d9b0f144e8c12dcd0159e6999804bbff9b726d/vagrant/manylinux

[Mizux]

What's about PEP571 aka manylinux2010 ?

Bonus: It solve the issue with kernel vsyscall=emulate contrary to a Centos:6 based Dockerfile.
cf #141

[armudgal]

Hi,
should we close this issue since the discussion has entirely moved to #179

[trishankatdatadog]

We should, I'll let @mayeut make the call

[mayeut]

closing per previous comments. See manylinux2010 in #179