allow automatic use of pyOpenSSL 0.14 #2221

New Issue

tahoe-lafs · 2014-04-14T23:02:16Z

daira commented

2014-04-14 23:02:16 +00:00

For Tahoe-LAFS 1.11 we decided to fix the pyOpenSSL version requirement to == 0.13 [quite; see comment:94877 below]not, in order to mitigate problems with building the cryptography library that is a dependency of pyOpenSSL >= 0.14 (see #2193 for details).

This ticket is for a longer-term solution to allow use of pyOpenSSL >= 0.14 without causing build/install regressions (such as #2217, or the requirement to manually install libffi on some platforms).

For Tahoe-LAFS 1.11 we decided to fix the pyOpenSSL version requirement to == 0.13 [quite; see [comment:94877](/tahoe-lafs/trac-2024-07-25/issues/2221#issuecomment-94877) below]not, in order to mitigate problems with building the `cryptography` library that is a dependency of pyOpenSSL >= 0.14 (see #2193 for details). This ticket is for a longer-term solution to allow use of pyOpenSSL >= 0.14 without causing build/install regressions (such as #2217, or the requirement to manually install `libffi` on some platforms).

tahoe-lafs added the

labels 2014-04-14 23:02:16 +00:00

tahoe-lafs added this to the soon milestone 2014-04-14 23:02:16 +00:00

glyph commented

2014-05-12 21:42:37 +00:00

For what it's worth, you don't need to manually install libffi on the mac, at least; it's bundled with the platform.

daira commented

2014-10-17 11:15:17 +00:00

The solution we eventually arrived at for 1.11 (for all platforms) was to attempt to import the OpenSSL module at build time and check its version; if 0.14 or above is already installed then we allow it, otherwise we use 0.13 or 0.13.1.

This is not entirely satisfactory; it's complicated and means that we don't get pyOpenSSL security fixes unless 0.14+ is installed manually.

The solution we eventually arrived at for 1.11 (for all platforms) was to attempt to import the OpenSSL module at build time and check its version; if 0.14 or above is already installed then we allow it, otherwise we use 0.13 or 0.13.1. This is not entirely satisfactory; it's complicated and means that we don't get pyOpenSSL security fixes unless 0.14+ is installed manually.

tahoe-lafs changed title from ~~allow use of pyOpenSSL 0.14~~ to allow automatic use of pyOpenSSL 0.14

2014-10-28 19:10:08 +00:00

daira commented

2015-05-01 19:31:31 +00:00

https://caremad.io/2014/11/distributing-a-cffi-project/ explains what is wrong with cffi from a packaging point of view. Until those design issues are fixed, we can't make much progress on this ticket.

<https://caremad.io/2014/11/distributing-a-cffi-project/> explains what is wrong with cffi from a packaging point of view. Until those design issues are fixed, we can't make much progress on this ticket.

glyph commented

2015-05-02 01:39:25 +00:00

That post includes workarounds for all of the issues described; it also says that the cryptography project has now already worked around all the issues described therein. Given that Tahoe doesn't make use of cffi directly, it's not clear to me why those particular issues would prevent it from being used indirectly in the project that all the workarounds come from.

That said, certainly the requirement to install libffi is still there.

That post includes workarounds for all of the issues described; it also says that the cryptography project has now already worked around all the issues described therein. Given that Tahoe doesn't make use of cffi directly, it's not clear to me why those particular issues would prevent it from being used indirectly in the project that all the workarounds come from. That said, certainly the requirement to install `libffi` is still there.

daira commented

2015-05-02 18:28:34 +00:00

Sorry, I missed the fact that they were already in cryptography (probably because I'd seen some of these problems when building previous versions of it). Which is the first version that has all of these fixes?

Sorry, I missed the fact that they were already in `cryptography` (probably because I'd seen some of these problems when building previous versions of it). Which is the first version that has all of these fixes?

daira commented

2015-05-02 18:34:19 +00:00

Also, is it possible to build a cryptography egg that allows building without a compiler or any manually installed dependencies on Windows?

(Note: we can't use wheels because we're stuck with zetuptoolz for the time being.)

Also, is it possible to build a cryptography egg that allows building without a compiler or any manually installed dependencies on Windows? (Note: we can't use wheels because we're stuck with zetuptoolz for the time being.)

glyph commented

2015-05-03 06:54:45 +00:00

I don't know what version the fixes appeared in; they have all trickled in over time. I'm not sure if there's even a complete release with all of them.

I assume it would be possible to build an egg, since the metadata provided is the same as for wheels, but eggs are pretty strongly deprecated at this point, and I'm not sure if you could convince the cryptography team to spend much time investigating it if it doesn't work :-. It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it?

I don't know what version the fixes appeared in; they have all trickled in over time. I'm not sure if there's even a complete release with all of them. I assume it would be possible to build an egg, since the metadata provided is the same as for wheels, but eggs are pretty strongly deprecated at this point, and I'm not sure if you could convince the cryptography team to spend much time investigating it if it doesn't work :-\. It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it?

zooko commented

2015-05-04 22:49:57 +00:00

Replying to glyph:

It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it?

Last time I considered it, which was one and a half years ago, we had made seven patches to our fork, zetuptoolz, and I wasn't sure if those issues had also been fixed in setuptools or not. Nowadays I guess I would be unsure if those issues would regress if we switched from zetuptoolz to wheel…

Replying to [glyph](/tahoe-lafs/trac-2024-07-25/issues/2221#issuecomment-94883): > It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it? Last time I considered it, which was [one and a half years ago](/tahoe-lafs/trac-2024-07-25/issues/2044#issuecomment-92869), we had made seven patches to our fork, zetuptoolz, and I wasn't sure if those issues had also been fixed in setuptools or not. Nowadays I guess I would be unsure if those issues would regress if we switched from zetuptoolz to `wheel`…

dstufft commented

2015-05-04 22:57:44 +00:00

For what it's worth, cffi is being worked on which fixes most (or all) of the issues in my post natively within cffi instead of relying on the hacks that we have in place in cryptography.

glyph commented

2015-05-04 23:13:05 +00:00

Replying to [zooko]comment:11:

Replying to glyph:

It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it?

Last time I considered it, which was one and a half years ago, we had made seven patches to our fork, zetuptoolz, and I wasn't sure if those issues had also been fixed in setuptools or not. Nowadays I guess I would be unsure if those issues would regress if we switched from zetuptoolz to wheel…

My understanding is that the context of those patches was a fairly unresponsive setuptools development team who wasn't fixing things. The Python Packaging Authority is substantially more responsive to bug reports, and users frequently get updates. Ironically zetuptoolz is now creating the exact situation it was created to avoid :-).

The only setuptools bug referenced by the linked comment is http://bugs.python.org/setuptools/issue54, which, if it hasn't been fixed (I think maybe it has? maybe dstufft can opine directly) has been mostly obviated by the ecosystem's consensus converging on virtualenv and related toolchains to produce isolated environments.

I want to say more things about this but I can't even find zetuptoolz's source code; all the old links are broken. Where does it live now?

Replying to [zooko]comment:11: > Replying to [glyph](/tahoe-lafs/trac-2024-07-25/issues/2221#issuecomment-94883): > > It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it? > > Last time I considered it, which was [one and a half years ago](/tahoe-lafs/trac-2024-07-25/issues/2044#issuecomment-92869), we had made seven patches to our fork, zetuptoolz, and I wasn't sure if those issues had also been fixed in setuptools or not. Nowadays I guess I would be unsure if those issues would regress if we switched from zetuptoolz to `wheel`… My understanding is that the context of those patches was a fairly unresponsive setuptools development team who wasn't fixing things. The Python Packaging Authority is *substantially* more responsive to bug reports, and users frequently get updates. Ironically `zetuptoolz` is now creating the exact situation it was created to avoid :-). The only setuptools bug referenced by the linked comment is <http://bugs.python.org/setuptools/issue54>, which, if it hasn't been fixed (I think maybe it has? maybe dstufft can opine directly) has been mostly obviated by the ecosystem's consensus converging on virtualenv and related toolchains to produce isolated environments. I want to say more things about this but I can't even find zetuptoolz's source code; all the old links are broken. Where does it live now?

zooko commented

2015-05-04 23:24:27 +00:00

The links in the comment mentioned above are not broken! They still work. The tahoe-lafs trac appears to be responding slowly though… give them a minute to load?

The links in [the comment mentioned above](/tahoe-lafs/trac-2024-07-25/issues/2044#issuecomment-92869) are not broken! They still work. The tahoe-lafs trac appears to be responding slowly though… give them a minute to load?

daira commented

2015-05-04 23:26:40 +00:00

glyph: it's in the Tahoe source tree: https://github.com/tahoe-lafs/tahoe-lafs/tree/master/setuptools-0.6c16dev6.egg

Last time I looked, the main change that upstream didn't have was the Windows script support that doesn't require an executable blob.

glyph: it's in the Tahoe source tree: <https://github.com/tahoe-lafs/tahoe-lafs/tree/master/setuptools-0.6c16dev6.egg> Last time I looked, the main change that upstream didn't have was the Windows script support that doesn't require an executable blob.

glyph commented

2015-05-04 23:29:29 +00:00

What do you mean by "require an executable blob"?

glyph commented

2015-05-04 23:31:11 +00:00

Also, I apologize for cluttering up this ticket's comments, we're getting pretty far off track. Should we make a new "eliminate zetuptoolz" ticket?

daira commented

2015-05-04 23:38:26 +00:00

There used to be (don't know whether there still is) a 32-bit Windows executable in the setuptools source, with no instructions on how to rebuild it. There was no corresponding 64-bit executable, and the 32-bit one was buggy; it failed to pass on Unicode arguments to the Python process correctly, for example.

There used to be (don't know whether there still is) a 32-bit Windows executable in the setuptools *source*, with no instructions on how to rebuild it. There was no corresponding 64-bit executable, and the 32-bit one was buggy; it failed to pass on Unicode arguments to the Python process correctly, for example.

zooko commented

2015-05-04 23:40:32 +00:00

Replying to glyph:

Also, I apologize for cluttering up this ticket's comments, we're getting pretty far off track. Should we make a new "eliminate zetuptoolz" ticket?

Here are [all of the related tickets]query:status=!closed&keywords=~setuptools&order=priority.

There are no fewer than six open tickets for wholesale replacement of zetuptoolz with something else. I must confess I no longer understand — if I ever did — which of the following tools are alternatives vs. complementary, which ones are currently maintained, etc.:

#1582 setuptools delenda est
#2044 investigate setuptools 0.7+
#2055 Building tahoe safely is non-trivial
#2020 write requirements doc for what we want from a package installer
#2077 pip packaging plan
[#2255 Experiment with virtualenv based packaging in the petmail style.]ticket:2255

Replying to [glyph](/tahoe-lafs/trac-2024-07-25/issues/2221#issuecomment-94890): > Also, I apologize for cluttering up this ticket's comments, we're getting pretty far off track. Should we make a new "eliminate zetuptoolz" ticket? Here are [all of the related tickets]query:status=!closed&keywords=~setuptools&order=priority. There are no fewer than six open tickets for wholesale replacement of `zetuptoolz` with something else. I must confess I no longer understand — if I ever did — which of the following tools are alternatives vs. complementary, which ones are currently maintained, etc.: * [#1582 setuptools delenda est](/tahoe-lafs/trac-2024-07-25/issues/6644) * [#2044 investigate setuptools 0.7+](/tahoe-lafs/trac-2024-07-25/issues/7074) * [#2055 Building tahoe safely is non-trivial](/tahoe-lafs/trac-2024-07-25/issues/7085) * [#2020 write requirements doc for what we want from a package installer](/tahoe-lafs/trac-2024-07-25/issues/7050) * [#2077 pip packaging plan](/tahoe-lafs/trac-2024-07-25/issues/7107) * [#2255 Experiment with virtualenv based packaging in the petmail style.]ticket:2255

zooko commented

2015-05-04 23:41:08 +00:00

P.S. Since I no longer understand these issues, I would welcome someone who does, such as dstufft, glyph, daira, warner, etc., taking the lead on proposing a path forward.

glyph commented

2015-05-04 23:44:48 +00:00

I think #1582 is probably the right place to continue discussion, so I will move there. Thanks for the overview.

rvs commented

2016-03-07 23:03:14 +00:00

FWIW, I'm having some problems with installing tahoe with --single-version-externally-managed while using pyOpenSSL 0.15.1: it still complains "pkg_resources.DistributionNotFound: The 'pyOpenSSL<=0.13.1,>=0.13' distribution was not found and is required by the application"

logs are here:

https://copr-be.cloud.fedoraproject.org/results/vrusinov/sundry/fedora-rawhide-x86_64/00166429-tahoe-lafs/build.log.gz
https://copr-be.cloud.fedoraproject.org/results/vrusinov/sundry/fedora-rawhide-x86_64/00166429-tahoe-lafs/root.log.gz (this one lists all installed packages)

Here's the spec file I try to build: https://github.com/vrusinov/copr-sundry/blob/tahoe-lafs/third_party/subtrees/tahoe-lafs/tahoe-lafs.spec

Note that I'm doing something weird before install:

rm -rf setuptools-*.egg
mkdir -p setuptools-0.0.0.egg

But it the only way I was able to build it avoiding weird setuptools version conflicts.

I can live without it now, but it may quickly become very annoying.

FWIW, I'm having some problems with installing tahoe with --single-version-externally-managed while using pyOpenSSL 0.15.1: it still complains "pkg_resources.DistributionNotFound: The 'pyOpenSSL<=0.13.1,>=0.13' distribution was not found and is required by the application" logs are here: * <https://copr-be.cloud.fedoraproject.org/results/vrusinov/sundry/fedora-rawhide-x86_64/00166429-tahoe-lafs/build.log.gz> * <https://copr-be.cloud.fedoraproject.org/results/vrusinov/sundry/fedora-rawhide-x86_64/00166429-tahoe-lafs/root.log.gz> (this one lists all installed packages) Here's the spec file I try to build: <https://github.com/vrusinov/copr-sundry/blob/tahoe-lafs/third_party/subtrees/tahoe-lafs/tahoe-lafs.spec> Note that I'm doing something weird before install: ``` rm -rf setuptools-*.egg mkdir -p setuptools-0.0.0.egg ``` But it the only way I was able to build it avoiding weird setuptools version conflicts. I can live without it now, but it may quickly become very annoying.

warner commented

2016-03-26 22:52:04 +00:00

We no longer pin the pyOpenSSL version as we used to do, nor do we use zetuptoolz. Tahoe now uses the current version of everything.

We currently have multiple pathways that depend on pyopenssl:

a direct dependency that requires >=0.14
one through Twistedtls that wants >=0.13
one through Foolscap that doesn't declare a version
one through Foolscap that wants Twistedtls
one through service-identity that wants pyopenssl >=0.12

After this release, we'll probably get rid of one on service-identity (which is no longer used by twisted; tahoe was depending on it just out of paranoia). We might also get rid of the direct dependency, unless there's something special about 0.14 that we care about more than Twisted or Foolscap.

But in any event, 0.14 is fine, and in fact modern builds will use the current release (which is "16.0.0"). So I'm closing this one out.

We no longer pin the pyOpenSSL version as we used to do, nor do we use zetuptoolz. Tahoe now uses the current version of everything. We currently have multiple pathways that depend on pyopenssl: * a direct dependency that requires `>=0.14` * one through `Twistedtls` that wants `>=0.13` * one through Foolscap that doesn't declare a version * one through Foolscap that wants `Twistedtls` * one through `service-identity` that wants pyopenssl `>=0.12` After this release, we'll probably get rid of one on service-identity (which is no longer used by twisted; tahoe was depending on it just out of paranoia). We might also get rid of the direct dependency, unless there's something special about 0.14 that we care about more than Twisted or Foolscap. But in any event, 0.14 is fine, and in fact modern builds will use the current release (which is "16.0.0"). So I'm closing this one out.

warner added the

fixed

label 2016-03-26 22:52:04 +00:00

warner modified the milestone from soon to 1.11.0

2016-03-26 22:52:04 +00:00

warner closed this issue

2016-03-26 22:52:04 +00:00

glyph commented

2016-03-28 01:48:04 +00:00

For what it's worth, service_identity is still totally imported directly by Twisted. It's part of twistedtls though, so the direct dependency shouldn't be necessary.

For what it's worth, `service_identity` is still totally imported directly by Twisted. It's part of `twistedtls` though, so the direct dependency shouldn't be necessary.

warner commented

2016-03-28 04:04:33 +00:00

Oops, it's characteristic that's no longer used by Twisted. But yeah, we'll also remove our dependency on service-identity and allow twistedtls to be responsible for it.

Oops, it's `characteristic` that's no longer used by Twisted. But yeah, we'll also remove our dependency on `service-identity` and allow `twistedtls` to be responsible for it.

Sign in to join this conversation.