[WIP] Manylinux2010 (#279)

* Minimal usable merge of markrwilliams:manylinux2 into current pypa:master

- The dockerfiles build successfully.
- Compared to current master, there's a new intermediate image for
  glibc.  As the x86_64 build depends on it, this must have a predefined
  name.  In the PR the "naming prefix" is 'markrwilliams/manylinux2',
  therefore it's assumed for all images.
- There's a problem with libcurl which has been (temporarily) resolved
  with script which redirects yum.  See docker/build_scripts/build.sh for
  details.

The following script is proven to work:

  pushd docker/glibc/
  docker build -t markrwilliams/manylinux2:centos-6.9-no-vsyscall
  popd
  docker/build_scripts/prefetch.sh curl openssl
  docker build -t markrwilliams/manylinux2:x86_64 \
    -f docker/Dockerfile-x86_64 docker/
  docker build -t markrwilliams/manylinux2:i686 \
    -f docker/Dockerfile-i686 docker/

* Clean-up and adjustments for manylinux2010

- docker/build_scripts/build.sh:
  - removed libncurses.devel; not part of PEP 571
  - removed gpg as dependency, because yum already requires it
  - removed some /dev/null redirections, because they hinder debugging

- docker/build_scripts/manylinux1-check.py
  - renamed to manylinux-check.py
  - changed names/comments to new version
  - the check now targets glibc 2.12 instead of 2.5

- .travis.yml
  - only changed the image name
  - TODO: x86_64 still needs a script for the no-vsyscall image dependency

- docker/deploy.sh
  - changed image name

- note: .rst files have not been touched

* Fix for new release of pip (v10.0.0) breaking the build script

- The freshly released version of pip breaks build_utils.sh which tries
  to get the latest and greatest.  Added a condition to instead use the
  latest version below v10.

* Merge upstream changes (4a20e4b); remove get-pip hack

- Incorporated the lastest changes from the official repository.
  - That cleanup removed two packages from 'yum erase ...' which looks
    like an oversight. Readded.
- Removed the hack which fixed the broken build when the new pip was
  released.  This has been fixed upstream, see here:
  pypa/get-pip#19

* Re-add the two /dev/null redirections in build.sh

* Merge upstream changes (6f3bc0b)

* Merge upstream changes (3dd4551)

* Merge upstream changes (4132165)

* Adapt build.sh for the x86_64 two-stage build

- Add platform-dependent clause
- Assign the same prefix to the the no-vsyscall image (quay.io/pypa/...)
- The dependent docker/Dockerfile-x86_64 needs this prefix in FROM ...

* Getting Travis to play along

- Increase in `travis_wait` time
- Shunting build output off into a log file which can be inspected
  on failure.

* Drop 32-bit support and upgrade to devtoolset-7

* Update README.rst

manylinux2 -> manylinux2010

* Add comment regarding MANYLINUX2010_DEPS

* Add comment for Software collection & EPEL usage

* Move expat-devel to PYTHON_COMPILE_DEPS

* Use local gpgkey for RPM-GPG-KEY-CentOS-6

* Update to devtoolset-8

* Replace travis wait by custom command

Prevent stdout timeout. The global travis timeout is the only one to apply now.

* Use travis-ci build stages and cache glibc build

glibc patch is unlikely to change very often, use travis-ci cache mechanism in order not to rebuild it every time.

also, use build stages so that when it does change, glibc build is run as a pre-step, it will probably help not reaching the time limit, or at least, glibc build will be cached for job restart.

Author: mayeut

.travis.yml

@@ -10,25 +10,40 @@ branches:
   except:
     - /^pyup-/
 
+cache:
+  directories:
+    - $HOME/docker
+
 env:
   global:
     # QUAY_USERNAME and QUAY_PASSWORD for docker image upload
     - secure: "lKaTzEL6UNiEfp+BWLOUILG9BMtjwEMpwt6Yag0cQGHix7qJ/ElZ0t3oFw6ZwuDmA5qceAXIdxHLUK9HGVI2MloLk8czGhjvtfJ4XhOxtEJRQ0VkDGPsKN4cfhB4ZjGo6GAPtNqStMyNiY7BZuTrZa7coDLCoUeYcOmTpi6pmd1rrkk725B9QCTuhFHbPhkuL2yu/Jk6WxkHJBKjmuZek+iQa7lRItgMrG0/319PXLvwIGGl00nLFy+Ly5Ciwzux4wuHLTySZQKu0H9FX81A7smM0FW/42kg3ckGa2qLxRw/Pi8Nm/aIk8LD0QXzI5N7HhFfidOTgDS8Mt1HgfxmTk4wUXZ/KvCCshqjimzMc/s9i9wPZX9UqqcfrpZkmwz8dzhm1bndN45ZOCy6xAYT6dzf8T4mLMDjVWSW4+DUoW4sYHRLVujjcMk7ybcwGV43VruPTJnc8XVAhT+VIMQkoPjhQmTOn8h82LRNGYtLa5RReCh9OPKVYB2Quz18FXMWgFt7A6VWudL0c7/8CusLvuo+pLcxt9pnV40rvu1YEohpEj8qR/qTSaDUBZM0J9SVf5zrZR80pZUnXkDF8nm+mcLOTley3YWipU19lCR7dzVyCAiQdVAuNPdnyem3Yk8enGkAJbfLd6eaIDs+p73D0JXh1Nx1px1movVLQH3ohIw="
     - secure: "w1614pomHLltkBhqWM2bOvbymFWIWKqSqqIBDvaNn9tbQScioItJoELBT7g7+cD7nyU7OvpQ1U2fk0xVkCeNvYU0xS1vP4o/VnZRpup7f7Tkiq+2rf4fjwYr3HHnJjwak1l9bsw6FkgzKaVvSdiUJHMVxiIuLd3fVozR7qjBBhTDxSlWGOpSgd+ttpgMZwU5zQjdaVQr1D7E8M0979ZnWMrNRyLiAUeHaPILS815b+ijgqR+i5nmu0/FTCGM9Ik4KIzIfWq8AdfPdbRiq8c+LrrTPfyKcIQJaHmfduYRM4LycGWwzkXFBNtLrJ7uFLG9RDVemOHuHOWIJX8qCUIV4XuESXxH3fUQr6r+yxquTJbzXxNtoaLa6tBOTQWKDrRjT4z9Mf9Im14F2V59EUDoQowHx5bjunOH5wg3ruYNKYYBFRYra5kx0CkKrqFBzyl8fTUEQLyx1HWTVUC1WTXEeD/aFKOSIxW5DxZr5W4LLlW2+Raa52ZzY28Q6AdueFQCRzoJ70/GsJRlSsBdWNOHN4gSp1cZuToLWY15y64QhAMVDpikB+V4hmkbceLiTqeWzTStNL1sa32RHr6i/9zeFZw1pMD1+eOg9x6fgODfh2sqr/zPbu2oONsHnc4D2jwsEax4o+Dv5QHLvK7jdyWUmu47a9QReoexXK60jZXs3CA="
 
-
-matrix:
+jobs:
   include:
-    - env: PLATFORM="i686"
-    - env: PLATFORM="x86_64"
-
-script:
-  - PLATFORM=$PLATFORM TRAVIS_COMMIT=$TRAVIS_COMMIT ./build.sh
-
-
-deploy:
-  provider: script
-  script: docker/deploy.sh
-  on:
-    branch: master
-    repo: pypa/manylinux
+    - stage: "Patch glibc"
+      env:
+        - PLATFORM="x86_64"
+      before_install:
+        # Load cached docker images
+        - if [[ -d $HOME/docker ]]; then ls $HOME/docker/*.tar.gz | xargs -I {file} sh -c "zcat {file} | docker load"; fi
+      script:
+        - PLATFORM=$PLATFORM TRAVIS_COMMIT=$TRAVIS_COMMIT ./build.sh glibc_only
+      before_cache:
+        # Save tagged docker images
+        - mkdir -p $HOME/docker && docker images -a --filter='dangling=false' --format '{{.Repository}}:{{.Tag}} {{.ID}}' | grep 'centos-with-vsyscall:latest' | xargs -n 2 -t sh -c 'test -e $HOME/docker/$1.tar.gz || docker save $0 | gzip -2 > $HOME/docker/$1.tar.gz'
+    - stage: "Build manylinux images"
+      env:
+        - PLATFORM="x86_64"
+      before_install:
+        # Load cached docker images
+        - if [[ -d $HOME/docker ]]; then ls $HOME/docker/*.tar.gz | xargs -I {file} sh -c "zcat {file} | docker load"; fi
+      script:
+        - PLATFORM=$PLATFORM TRAVIS_COMMIT=$TRAVIS_COMMIT ./build.sh
+      deploy:
+        provider: script
+        script: docker/deploy.sh
+        on:
+          branch: master
+          repo: pypa/manylinux

build.sh

@@ -3,5 +3,18 @@
 # Stop at any error, show all commands
 set -ex
 
+
+if [ $PLATFORM == x86_64 ] || [ "$1" == "glibc_only" ]; then
+    # Output something every 10 minutes or Travis kills the job
+    while sleep 9m; do echo -n -e " \b"; done &
+    docker build --rm -t centos-with-vsyscall:latest --cache-from centos-with-vsyscall:latest --target centos-with-vsyscall -f docker/glibc/Dockerfile docker/glibc/
+    # Killing background sleep loop
+    kill %1
+    if [ "$1" == "glibc_only" ]; then
+        exit 0
+    fi
+    docker build --rm -t quay.io/pypa/manylinux2010_centos-6-no-vsyscall --cache-from quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest --cache-from centos-with-vsyscall:latest -f docker/glibc/Dockerfile docker/glibc/
+fi
+
 docker/build_scripts/prefetch.sh openssl curl
-docker build --rm -t quay.io/pypa/manylinux1_$PLATFORM:$TRAVIS_COMMIT -f docker/Dockerfile-$PLATFORM docker/
+docker build --rm -t quay.io/pypa/manylinux2010_$PLATFORM:$TRAVIS_COMMIT -f docker/Dockerfile-$PLATFORM docker/

docker/Dockerfile-i686

@@ -1,12 +1,13 @@
-FROM centos:5
-MAINTAINER The ManyLinux project
+# See docker/glibc/
+FROM quay.io/pypa/manylinux2010_centos-6-no-vsyscall
+LABEL maintainer="The ManyLinux project"
 
 ENV LC_ALL en_US.UTF-8
 ENV LANG en_US.UTF-8
 ENV LANGUAGE en_US.UTF-8
-ENV PATH /opt/rh/devtoolset-2/root/usr/bin:$PATH
-ENV LD_LIBRARY_PATH /opt/rh/devtoolset-2/root/usr/lib64:/opt/rh/devtoolset-2/root/usr/lib:/usr/local/lib64:/usr/local/lib
-ENV PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
+ENV PATH /opt/rh/devtoolset-8/root/usr/bin:$PATH
+ENV LD_LIBRARY_PATH /opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/usr/local/lib64:/usr/local/lib
+ENV PKG_CONFIG_PATH /usr/local/lib/pkgconfig
 
 COPY build_scripts /build_scripts
 COPY sources /

docker/Dockerfile-x86_64

@@ -10,20 +10,25 @@ MY_DIR=$(dirname "${BASH_SOURCE[0]}")
 
 # Dependencies for compiling Python that we want to remove from
 # the final image after compiling Python
-# GPG installed to verify signatures on Python source tarballs.
-PYTHON_COMPILE_DEPS="zlib-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel gpg libffi-devel"
-
-# Libraries that are allowed as part of the manylinux1 profile
-MANYLINUX1_DEPS="glibc-devel libstdc++-devel glib2-devel libX11-devel libXext-devel libXrender-devel  mesa-libGL-devel libICE-devel libSM-devel ncurses-devel"
-
-# Centos 5 is EOL and is no longer available from the usual mirrors, so switch
-# to http://vault.centos.org
-# From: https://github.com/rust-lang/rust/pull/41045
-# The location for version 5 was also removed, so now only the specific release
-# (5.11) can be referenced.
-sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf
-sed -i 's/mirrorlist/#mirrorlist/' /etc/yum.repos.d/*.repo
-sed -i 's/#\(baseurl.*\)mirror.centos.org\/centos\/$releasever/\1vault.centos.org\/5.11/' /etc/yum.repos.d/*.repo
+PYTHON_COMPILE_DEPS="zlib-devel bzip2-devel expat-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel"
+
+# Libraries that are allowed as part of the manylinux2010 profile
+# Extract from PEP: https://www.python.org/dev/peps/pep-0571/#the-manylinux2010-policy
+# On RPM-based systems, they are provided by these packages:
+# Package:    Libraries
+# glib2:      libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
+# glibc:      libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
+# libICE:     libICE.so.6
+# libX11:     libX11.so.6
+# libXext:    libXext.so.6
+# libXrender: libXrender.so.1
+# libgcc:     libgcc_s.so.1
+# libstdc++:  libstdc++.so.6
+# mesa:       libGL.so.1
+#
+# PEP is missing the package for libSM.so.6 for RPM based system
+# Install development packages (except for libgcc which is provided by gcc install)
+MANYLINUX2010_DEPS="glibc-devel libstdc++-devel glib2-devel libX11-devel libXext-devel libXrender-devel mesa-libGL-devel libICE-devel libSM-devel"
 
 # Get build utilities
 source $MY_DIR/build_utils.sh
@@ -41,39 +46,27 @@ echo "multilib_policy=best" >> /etc/yum.conf
 # Decided not to clean at this point: https://github.com/pypa/manylinux/pull/129
 yum -y update
 
-# EPEL support
-yum -y install wget
-# https://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
-cp $MY_DIR/epel-release-5-4.noarch.rpm .
-check_sha256sum epel-release-5-4.noarch.rpm $EPEL_RPM_HASH
-
-# Dev toolset (for LLVM and other projects requiring C++11 support)
-wget -q http://people.centos.org/tru/devtools-2/devtools-2.repo
-check_sha256sum devtools-2.repo $DEVTOOLS_HASH
-mv devtools-2.repo /etc/yum.repos.d/devtools-2.repo
-rpm -Uvh --replacepkgs epel-release-5*.rpm
-rm -f epel-release-5*.rpm
-
-# from now on, we shall only use curl to retrieve files
-yum -y erase wget
+# Software collection (for devtoolset-8) and EPEL support (for cmake28 & yasm)
+yum -y install centos-release-scl https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
 
 # Development tools and libraries
 yum -y install \
     automake \
     bison \
     bzip2 \
     cmake28 \
-    devtoolset-2-binutils \
-    devtoolset-2-gcc \
-    devtoolset-2-gcc-c++ \
-    devtoolset-2-gcc-gfortran \
+    devtoolset-8-binutils \
+    devtoolset-8-gcc \
+    devtoolset-8-gcc-c++ \
+    devtoolset-8-gcc-gfortran \
     diffutils \
-    expat-devel \
     gettext \
-    kernel-devel-`uname -r` \
     file \
+    kernel-devel-`uname -r` \
+    libffi-devel \
     make \
     patch \
+    perl-devel \
     unzip \
     which \
     yasm \
@@ -147,18 +140,39 @@ rm -rf patchelf.tar.gz patchelf-$PATCHELF_VERSION
 
 ln -s $PY36_BIN/auditwheel /usr/local/bin/auditwheel
 
+# HACK: The newly compiled and installed curl messes with the system's
+# py2.6 installation, on which yum depends.  Work around it by
+# rewiring libcurl.so specifically for yum.  /usr/local/bin/ has higher
+# priority on the PATH than /usr/bin/
+cat <<'EOF' > /usr/local/bin/yum && chmod +x /usr/local/bin/yum
+#!/bin/bash
+if [ "x$(arch)" != xi686 ]; then
+  LD_PRELOAD=/usr/lib64/libcurl.so.4
+else
+  LD_PRELOAD=/usr/lib/libcurl.so.4
+fi
+export LD_PRELOAD
+/usr/bin/yum "$@"
+EOF
+# the above might not shadow the real yum just yet, so call hash to be
+# sure:
+type yum
+hash yum
+
+
 # Clean up development headers and other unnecessary stuff for
 # final image
 yum -y erase \
     avahi \
     bitstream-vera-fonts \
     freetype \
+    gettext \
     gtk2 \
     hicolor-icon-theme \
     libX11 \
     wireless-tools \
-    ${PYTHON_COMPILE_DEPS}  > /dev/null 2>&1
-yum -y install ${MANYLINUX1_DEPS}
+    ${PYTHON_COMPILE_DEPS} > /dev/null 2>&1
+yum -y install ${MANYLINUX2010_DEPS}
 yum -y clean all > /dev/null 2>&1
 yum list installed
 
@@ -180,7 +194,7 @@ find /opt/_internal -depth \
 for PYTHON in /opt/python/*/bin/python; do
     # Smoke test to make sure that our Pythons work, and do indeed detect as
     # being manylinux compatible:
-    $PYTHON $MY_DIR/manylinux1-check.py
+    $PYTHON $MY_DIR/manylinux-check.py
     # Make sure that SSL cert checking works
     $PYTHON $MY_DIR/ssl-check.py
 done

docker/build_scripts/build.sh

@@ -37,5 +37,5 @@ GIT_HASH=ba2fed9d02e424b735e035c4f2b0bdb168ef0df7e35156b5051d900dc7247787
 GIT_DOWNLOAD_URL=https://github.com/git/git/archive
 
 GET_PIP_URL=https://bootstrap.pypa.io/get-pip.py
-EPEL_RPM_HASH=0dcc89f9bf67a2a515bad64569b7a9615edc5e018f676a578d5fd0f17d3c81d4
+EPEL_RPM_HASH=e5ed9ecf22d0c4279e92075a64c757ad2b38049bcf5c16c4f2b75d5f6860dc0d
 DEVTOOLS_HASH=a8ebeb4bed624700f727179e6ef771dafe47651131a00a78b342251415646acc

docker/build_scripts/build_env.sh

@@ -54,10 +54,7 @@ function do_cpython_build {
     if [ -e ${prefix}/bin/python3 ]; then
         ln -s python3 ${prefix}/bin/python
     fi
-    # --force-reinstall is to work around:
-    #   https://github.com/pypa/pip/issues/5220
-    #   https://github.com/pypa/get-pip/issues/19
-    ${prefix}/bin/python get-pip.py --force-reinstall
+    ${prefix}/bin/python get-pip.py
     if [ -e ${prefix}/bin/pip3 ] && [ ! -e ${prefix}/bin/pip ]; then
         ln -s pip3 ${prefix}/bin/pip
     fi

docker/build_scripts/build_utils.sh

@@ -1,6 +1,6 @@
 # Logic copied from PEP 513
 
-def is_manylinux1_compatible():
+def is_manylinux2010_compatible():
     # Only Linux, and only x86-64 / i686
     from distutils.util import get_platform
     if get_platform() not in ["linux-x86_64", "linux-i686"]:
@@ -14,8 +14,8 @@ def is_manylinux1_compatible():
         # Fall through to heuristic check below
         pass
 
-    # Check glibc version. CentOS 5 uses glibc 2.5.
-    return have_compatible_glibc(2, 5)
+    # Check glibc version. CentOS 6 uses glibc 2.12.
+    return have_compatible_glibc(2, 12)
 
 def have_compatible_glibc(major, minimum_minor):
     import ctypes
@@ -45,9 +45,9 @@ def have_compatible_glibc(major, minimum_minor):
     return True
 
 import sys
-if is_manylinux1_compatible():
-    print("%s is manylinux1 compatible" % (sys.executable,))
+if is_manylinux2010_compatible():
+    print("%s is manylinux2010 compatible" % (sys.executable,))
     sys.exit(0)
 else:
-    print("%s is NOT manylinux1 compatible" % (sys.executable,))
+    print("%s is NOT manylinux2010 compatible" % (sys.executable,))
     sys.exit(1)

docker/build_scripts/epel-release-5-4.noarch.rpm

@@ -1,5 +1,5 @@
 #!/bin/bash
 docker login -u $QUAY_USERNAME -p $QUAY_PASSWORD quay.io
-tag="quay.io/pypa/manylinux1_$PLATFORM"
+tag="quay.io/pypa/manylinux2010_$PLATFORM"
 docker tag ${tag}:${TRAVIS_COMMIT} ${tag}:latest
 docker push ${tag}:latest

docker/build_scripts/manylinux1-check.py renamed to docker/build_scripts/manylinux-check.py

@@ -0,0 +1,11 @@
+FROM centos:6 as centos-with-vsyscall
+
+COPY ./build_scripts /build_scripts
+RUN bash /build_scripts/rebuild-glibc-without-vsyscall.sh
+
+FROM centos:6
+LABEL maintainer="The Manylinux project"
+
+COPY --from=centos-with-vsyscall /rpms /rpms
+
+RUN yum -y install /rpms/* && rm -rf /rpms