add dependency on Twisted[tls] to overcome pip's non-resolver #2760

New Issue

warner · 2016-03-30T18:32:26Z

warner commented

2016-03-30 18:32:26 +00:00

Tahoe itself doesn't strictly depend on TLS support. It depends on Foolscap, which does, but in a perfect world Tahoe wouldn't have to know about that.

To ask Twisted to be TLS-capable, a package uses a square-bracketed "extra" in its dependencies, like Twistedtls >= 15.2.1 instead of just Twisted >= 15.2.1.

In that perfect world, we'd have:

Tahoe: Twisted >= 13.0.0, Foolscap
Foolscap: Twisted[tls] >= 16.0.0

But unless/until we bypass the issue by using a requirements.txt file (maybe for #2055), we're affected by a missing pip feature: it lacks a full resolver.

A full resolver is what lets Debian's "apt" try all possible combinations of packages and versions to find any (hopefully the "best") that will meet the given constraints. It takes a lot more work, and is scarily powerful (someone once proved that the constraint solver is Turing-complete).

Pip installs the highest allowable version of the first thing that it encounters, and then explores the dependencies. It doesn't go back to try something different if the subsequent dependencies don't fit. And in particular, if it installs a package without any "extras", it won't go back and re-install it (with the extras) if it sees a later dependency that wants them.

So in the above example, when we install tahoe, pip will first install the latest version of Twisted it can find (e.g. 16.0.0, with no extras), then it installs foolscap, then, it sees that Foolscap wants Twistedtls >= 16.0.0. It knows it can satisfy the >= 16.0.0 requirement, but it can't go back and re-install the tls extra. It emits a warning, but can't fix it.

To overcome this, for the 1.11.0 release, we made Tahoe aware of the need for TLS, by using:

Tahoe: Twisted[tls] >= 13.0.0, Foolscap

But this hits another problem, which is that Twisted didn't start offering the tls extra until 15.2.1 . So we're actually using:

Tahoe: Twisted[tls] >= 15.2.1, Foolscap

even though Tahoe, itself, doesn't need that recent of a Twisted version.

(incidentally, foolscap#249 is what added Twistedtls to Foolscap)

This version bump might be inconvenient for OS packagers who are backporting the new tahoe-1.11.0 to older distributions that don't have the more recent Twisted. As I mentioned on the mailing list just now, porters who find themselves in this situation (and can't upgrade their Twisted packages) should consider modifying Tahoe's src/allmydata/_auto_deps.py to reduce this constraint back to the previous version. As long as there are OS-package-level dependency constraints on everything that the older version of Twisted needs for TLS support, things should work.

(although note that the most recent version of Foolscap does, in fact, depend upon Twisted >= 16.0.0, so if you're upgrading that, you should probably go all-in and upgrade everything).

This ticket is just to record the reasons for this version bump.

Tahoe itself doesn't strictly depend on TLS support. It depends on Foolscap, which *does*, but in a perfect world Tahoe wouldn't have to know about that. To ask Twisted to be TLS-capable, a package uses a square-bracketed "extra" in its dependencies, like `Twistedtls >= 15.2.1` instead of just `Twisted >= 15.2.1`. In that perfect world, we'd have: ``` Tahoe: Twisted >= 13.0.0, Foolscap Foolscap: Twisted[tls] >= 16.0.0 ``` But unless/until we bypass the issue by using a requirements.txt file (maybe for #2055), we're affected by a missing pip feature: it lacks a full resolver. A full resolver is what lets Debian's "apt" try all possible combinations of packages and versions to find any (hopefully the "best") that will meet the given constraints. It takes a lot more work, and is scarily powerful (someone once proved that the constraint solver is Turing-complete). Pip installs the highest allowable version of the first thing that it encounters, and then explores the dependencies. It doesn't go back to try something different if the subsequent dependencies don't fit. And in particular, if it installs a package without any "extras", it won't go back and re-install it (with the extras) if it sees a later dependency that wants them. So in the above example, when we install tahoe, pip will first install the latest version of Twisted it can find (e.g. 16.0.0, with no extras), then it installs foolscap, then, it sees that Foolscap wants `Twistedtls >= 16.0.0`. It knows it can satisfy the `>= 16.0.0` requirement, but it can't go back and re-install the `tls` extra. It emits a warning, but can't fix it. To overcome this, for the 1.11.0 release, we made Tahoe aware of the need for TLS, by using: ``` Tahoe: Twisted[tls] >= 13.0.0, Foolscap ``` But this hits another problem, which is that Twisted didn't start offering the `tls` extra until 15.2.1 . So we're actually using: ``` Tahoe: Twisted[tls] >= 15.2.1, Foolscap ``` even though Tahoe, itself, doesn't need that recent of a Twisted version. (incidentally, [foolscap#249](https://foolscap.lothar.com/trac/ticket/249) is what added `Twistedtls` to Foolscap) This version bump might be inconvenient for OS packagers who are backporting the new tahoe-1.11.0 to older distributions that don't have the more recent Twisted. As I mentioned on the [mailing list](https://tahoe-lafs.org/pipermail/tahoe-dev/2016-March/009710.html) just now, porters who find themselves in this situation (and can't upgrade their Twisted packages) should consider modifying Tahoe's `src/allmydata/_auto_deps.py` to reduce this constraint back to the previous version. As long as there are OS-package-level dependency constraints on everything that the older version of Twisted needs for TLS support, things should work. (although note that the most recent version of Foolscap does, in fact, depend upon `Twisted >= 16.0.0`, so if you're upgrading that, you should probably go all-in and upgrade everything). This ticket is just to record the reasons for this version bump.

warner added the

labels 2016-03-30 18:32:26 +00:00

warner added this to the 1.11.0 milestone 2016-03-30 18:32:26 +00:00

warner self-assigned this 2016-03-30 18:32:26 +00:00

Brian Warner <warner@lothar.com> commented

2016-03-30 18:50:30 +00:00

In d57c8d5/trunk:

bump Twisted dependency (>=15.1.0) to get the [tls] extra

We only really need "Twisted >= 13.0.0", but we must add "[tls]" because
otherwise pip won't install it when Foolscap asks for it later, and we
need ">= 15.1.0" because that's the first version that provided "[tls]".

Fixes ticket:2760.

In [d57c8d5/trunk](/tahoe-lafs/trac-2024-07-25/commit/d57c8d5e39c8223fdf5b1a30770dcf18b499bb4f): ``` bump Twisted dependency (>=15.1.0) to get the [tls] extra We only really need "Twisted >= 13.0.0", but we must add "[tls]" because otherwise pip won't install it when Foolscap asks for it later, and we need ">= 15.1.0" because that's the first version that provided "[tls]". Fixes ticket:2760. ```

tahoe-lafs added the

fixed

label 2016-03-30 18:50:30 +00:00

tahoe-lafs closed this issue

2016-03-30 18:50:30 +00:00

Sign in to join this conversation.