PyPI "TLS<1.2 Brownout" is causing travis OS-X tests to fail #2913

New Issue

warner · 2018-03-28T23:10:39Z

warner commented

2018-03-28 23:10:39 +00:00

We're seeing some intermittent failures of the Travis-CI OS-X build, where the symptom is that tox is unable to install the "incremental" package (which is a dependency of Twisted):

https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/359145018

Collecting Twisted[tls]>=16.4.0 (from tahoe-lafs==0.0.0)
  Downloading Twisted-17.9.0.tar.bz2 (3.0MB)
    Complete output from command python setup.py egg_info:
    Couldn't find index page for 'incremental' (maybe misspelled?)
    No local packages or working download links found for incremental>=16.10.1
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1')

http://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html reports that PyPI is soon to be dropping support for TLS-1.0 and 1.1, requiring all clients to use TLS-1.2 or newer. To test this, they're conducting rolling brownouts: as of today, https://status.python.org/ reports that TLS-1.2 is being enforced for the first 15 minutes of each hour (clients speaking older versions get an HTTP 403 with an explanatory error message). Unfortunately many versions of pip/setuptools don't report these error messages very well, making them look like random network outages.

OS-X-10.12 (the not-most-recent-version, which is Travis's default) ships with an ancient OpenSSL which does not support TLS-1.2 . OS-X-10.13 (the current version) ships with a modern-ish version of libressl that can do TLS-1.2.

The system /usr/bin/python is linked against the system OpenSSL. Our Travis OS-X build appears to use that (although note that we don't have language: python turned on, for various reasons that need to be fixed, so we might get a different python if we told travis we wanted python).

pip-9.0.3 knows how to use OpenTransport on a mac, instead of OpenSSL, and OT is nicely modern and speaks TLS-1.2 just fine. But setuptools does not.

So when TLS<=1.2 is turned off, the only way to install things on OS-X are:

be using OS-X-10.13 (the system libressl works), OR
be using a Homebrew python (which doesn't link against system openssl), OR
be using pip-9.0.3 or higher (which uses OpenTransport), OR

In 4eac3ca, we modified our setup_requires= to need setuptools >= 28.8.0, since that's (roughly?) the oldest that understands the python_requires= syntax that we use in our setup.py. However upgrading setuptools from inside a setup_requires= is pretty explody (https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/OS-X%2010.13/builds/14/steps/tox/logs/stdio), so in 6f20dbc we changed travis to upgrade setuptools before running tox, and in 526b97c we changed tox to stop building sdists (which we didn't use, and which were built with the external python and it's old setuptools).

But the remaining problem is that when Twisted says setup_requires: ["incremental"], it's setuptools that attempts to do the install, not pip. Since Travis is using OS-X 10.12, and the system python, and it's not using pip, the setuptools attempt to install incremental uses an old version of TLS, which gets blocked by the PyPI brownout, and unhelpfully reported as a lookup failure.

To fix this, our workaround will be to pre-install/upgrade incremental, in our tox.ini. We're going to need to pre-install anything that is referenced by setup_requires in any dependency. We can probably remove this workaround when Travis moves their default to OS-X-10.13, or if setuptools acquires the same kind of workaround that pip has (not likely), or if we switch to telling travis language: python and that happens to use something like Homebrew python.

Travis is having other problems right now (OS-X builds are backed up pretty badly), but I think this TLS thing is what's biting us. It certainly makes it harder to experiment, though.

We're seeing some intermittent failures of the Travis-CI OS-X build, where the symptom is that tox is unable to install the "incremental" package (which is a dependency of Twisted): <https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/359145018> ``` Collecting Twisted[tls]>=16.4.0 (from tahoe-lafs==0.0.0) Downloading Twisted-17.9.0.tar.bz2 (3.0MB) Complete output from command python setup.py egg_info: Couldn't find index page for 'incremental' (maybe misspelled?) No local packages or working download links found for incremental>=16.10.1 raise DistutilsError(msg) distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1') ``` <http://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html> reports that PyPI is soon to be dropping support for TLS-1.0 and 1.1, requiring all clients to use TLS-1.2 or newer. To test this, they're conducting rolling brownouts: as of today, <https://status.python.org/> reports that TLS-1.2 is being enforced for the first 15 minutes of each hour (clients speaking older versions get an HTTP 403 with an explanatory error message). Unfortunately many versions of pip/setuptools don't report these error messages very well, making them look like random network outages. OS-X-10.12 (the not-most-recent-version, which is Travis's default) ships with an ancient OpenSSL which does not support TLS-1.2 . OS-X-10.13 (the current version) ships with a modern-ish version of libressl that *can* do TLS-1.2. The system `/usr/bin/python` is linked against the system OpenSSL. Our Travis OS-X build appears to use that (although note that we don't have `language: python` turned on, for various reasons that need to be fixed, so we might get a different python if we told travis we wanted python). pip-9.0.3 knows how to use OpenTransport on a mac, instead of OpenSSL, and OT is nicely modern and speaks TLS-1.2 just fine. But setuptools does not. So when TLS<=1.2 is turned off, the only way to install things on OS-X are: * be using OS-X-10.13 (the system libressl works), OR * be using a Homebrew python (which doesn't link against system openssl), OR * be using pip-9.0.3 or higher (which uses [OpenTransport](wiki/OpenTransport)), OR In 4eac3ca, we modified our `setup_requires=` to need `setuptools >= 28.8.0`, since that's (roughly?) the oldest that understands the `python_requires=` syntax that we use in our `setup.py`. However upgrading setuptools from inside a `setup_requires=` is pretty explody (<https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/OS-X%2010.13/builds/14/steps/tox/logs/stdio>), so in 6f20dbc we changed travis to upgrade setuptools before running tox, and in 526b97c we changed tox to stop building sdists (which we didn't use, and which were built with the external python and it's old setuptools). But the remaining problem is that when Twisted says `setup_requires: ["incremental"]`, it's setuptools that attempts to do the install, not pip. Since Travis is using OS-X 10.12, and the system python, and it's not using pip, the setuptools attempt to install `incremental` uses an old version of TLS, which gets blocked by the PyPI brownout, and unhelpfully reported as a lookup failure. To fix this, our workaround will be to pre-install/upgrade `incremental`, in our tox.ini. We're going to need to pre-install anything that is referenced by `setup_requires` in any dependency. We can probably remove this workaround when Travis moves their default to OS-X-10.13, or if setuptools acquires the same kind of workaround that pip has (not likely), or if we switch to telling travis `language: python` and that happens to use something like Homebrew python. Travis is having other problems right now (OS-X builds are backed up pretty badly), but I *think* this TLS thing is what's biting us. It certainly makes it harder to experiment, though.

warner added the

labels 2018-03-28 23:10:39 +00:00

warner added this to the 1.13.0 milestone 2018-03-28 23:10:39 +00:00

Brian Warner <warner@lothar.com> commented

2018-03-29 01:40:50 +00:00

In acc2b57/trunk:

tox: use newer (tox-2.4) settings, pre-install 'incremental'

* use 'extras' for our `[test]` additions instead of abusing 'deps'
* use 'deps' to pre-install 'incremental', which we couldn't do when we
  filled it with --editable to get `[test]`
* pre-install 'incremental' to work around a bug that strikes on Travis under
  OS-X-10.12 as PyPI gradually disables TLS<1.2. See ticket 2913 for details
* remove redundant configuration lines
* require tox-2.4 or newer, to get 'extras'

refs ticket:2913
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2913

In [acc2b57/trunk](/tahoe-lafs/trac-2024-07-25/commit/acc2b5744c481d64c3775ed7cbf8d08b24f4536c): ``` tox: use newer (tox-2.4) settings, pre-install 'incremental' * use 'extras' for our `[test]` additions instead of abusing 'deps' * use 'deps' to pre-install 'incremental', which we couldn't do when we filled it with --editable to get `[test]` * pre-install 'incremental' to work around a bug that strikes on Travis under OS-X-10.12 as PyPI gradually disables TLS<1.2. See ticket 2913 for details * remove redundant configuration lines * require tox-2.4 or newer, to get 'extras' refs ticket:2913 https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2913 ```

Brian Warner <warner@lothar.com> commented

2018-03-29 01:40:51 +00:00

In 479588d/trunk:

Merge branch '2913-travis-osx'

closes ticket:2913

In [479588d/trunk](/tahoe-lafs/trac-2024-07-25/commit/479588d427ab5aabba049a0ac4da134e938328b3): ``` Merge branch '2913-travis-osx' closes ticket:2913 ```

tahoe-lafs added the

fixed

label 2018-03-29 01:40:51 +00:00

tahoe-lafs closed this issue

2018-03-29 01:40:51 +00:00

Sign in to join this conversation.