zetuptoolz delenda est #1582
Labels
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
No Milestone
No Assignees
4 Participants
Notifications
Due Date
No due date set.
Reference: tahoe-lafs/trac-2024-07-25#1582
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
We need to stop using setuptools, for the following reasons:
Dealing with the effects of setuptools' problems on Tahoe-LAFS has inconvenienced users on many occasions and wasted a huge amount of core developer time. This ticket is to find, or to design and implement, an alternative.
See also /tahoe-lafs/trac-2024-07-25/issues/6513#comment:-1 .
I feel like this isn't as urgent as most bugs marked "Priority: Critical". If you disagree, then I apologize for overwriting the priority setting you left. Note that as the months and years have gone by, various other tools have been developed or improved; tools that we might be able to use, such as "Wheel" by Daniel Tolth, and "Paver". There are probably a few others that I haven't even heard of. Also pip and virtualenv have been ubiquitous and popular.
Major because of the sheer amount of pain setuptools has caused us. (pip doesn't solve most of those problems because it's just a frontend to setuptools, and the issues with setuptools are not
primarily inlimited to itseasy_installfrontend.)peep is another tool that intends to address this problem. I couldn't get it to work on Tahoe; it downloaded the required sdists and then blew up with an exception from pip, which it depends on:
If I understand correctly, pip by itself does not help you at all in verifying the integrity of dependencies; at most it will download the package you're directly installing over https.
Replying to daira:
More about what pip actually does: https://github.com/TheTorProject/ooni-backend/pull/1#discussion-diff-4084881.
I despair of Python package installation tools getting any better unless one is written COMPLETELY FROM SCRATCH by someone who knows what they're doing.
I don't believe the packaging tools need rewritten from scratch to enable secure install.
pip will verify the integrity of the downloads if it has that information available. Currently it is using md5 to do that (although that's not pip's doing, it supports md5, sha1, and sha2). At the urging of some members of ooni I am preparing to "once more into the breach" to fight for a better hash on PyPI.
The behavior described in the ooni pull request about scraping is mostly correct. However on July 1st I removed most link scraping from PyPI and the imminent pip 1.4 release makes the insecure external scraping opt-out on the installer side, and 1.5 will make it opt-in.
There is currently nothing like peep where you can bake a hash into a requirements.txt and verify against that, but it is something I want to enable. (As a side note, the above probably failed, because I don't believe a requires.txt is going to have the same format as requirements.txt).
Beyond that can you provide more information on:
The other issues I believe are either fixed, or are becoming fixed.
dstufft: thank you for your comment! In my opinion, an important thing going on here is just that Daira has strong negative emotions about setuptools. I don't want to push her to keep working on debugging and tweaking setuptools-based tools. On the other hand, I don't want to rip out all the setuptools stuff and replace it, either! So the ideal path forward in my opinion is either that we all pretend to ignore the problem and work on other things, or that someone other than Daira takes responsibility for diagnosing and fixing packaging-related issues.
Also Brian is pretty much in the same boat as Daira in hating and resenting a lot of setuptools-related issues. (In my opinion. Not to put words in their mouths!)
Actually I'm not too keen on spending my time debugging that stuff either. ;-)
So, if someone (like dstufft!) wants to help us with our packaging, that would be greatly appreciated.
zooko: I have no problem with helping to debug things, although I'm unlikely to have a ton of time to submit actual patches though D:
I'm more than willing to help debug things though if people have specific problems or tracebacks/logs that they can point me towards to see what's going on :) I joined the #tahoe-lafs channel and folks can ping me there for more immediate help if needed.
I know setuptools isn't the greatest thing in the world, and there is work being done to phase it out eventually :)
Replying to dstufft:
#1258 and #1287, for example.
Also #1450 and #1451.
And /tahoe-lafs/trac-2024-07-25/issues/6308#comment:86335 .
So those tickets are all variations on the same theme, and that theme is multi version installs are terrible and setuptools does them by default. That's why pip disabled them. You could probably override setup.py build|install|egg_info|develop etc to use pip instead of easy_install, or just tell everyone to use pip ;)
It would be very good for this ticket description to link to as many specific issues as possible in the setuptools, pip, and https://github.com/pypa/packaging-problems issue trackers, so that the problems can be run down in the place most obviously responsible for them. As it stands, most of this is accurate, but nebulous, griping about an old version of setuptools that Tahoe is stuck on, rather than an accurate framing of the state of the art.
When this ticket was filed, the setuptools team was leaving a period of being insular, unresponsive, and basically user-hostile. In the interim, setuptools has vastly accelerated its release cadence, increased its responsiveness to user bugs, and developed an ecosystem of tools and practices to obviate many of the issues discussed here.
The situation is still pretty bad in some corners, but it's vastly better than it once was, and most importantly, it is at least now worthwhile to go through the effort of filing specific issues because they do, in fact, get addressed eventually.
I would very much like to switch to using pip with a virtualenv rather than zetuptoolz with the hack to build dependencies under the
support/directory. However we're at completely the wrong stage in Tahoe's release cycle to work on that right now; it will have to wait until 1.11 at least.Sure; no rush. I am just trying to get my head around why exactly zetuptoolz exists, and which of these issues still exist vs. which don't. For example - would switching to a virtualenv fully address the things that motivated this ticket?
BTW there is a lot of very precise detail in the linked tickets about exactly what goes wrong whenever there are multiple packages installed under the same directory. As far as I know, none of that has been fixed.
Discussion of how to use pip and virtualenv should be on ticket #2077.
Review needed for https://github.com/tahoe-lafs/tahoe-lafs/pull/213!
I think we also want to enforce a minimum version of setuptools and/or use virtualenv automatically where available, but this branch can act as a base for that.
Glyph mentioned that Tahoe pins pyOpenSSL to 0.13 because of various setuptools reasons. Since I want to phase out support for non-Cryptography-using pyOpenSSL, I thought I'd post a comment here saying that using anything below 0.15.1 will be deprecated soon, and I don't want Tahoe to be left in the dark. :)
Tahoe doesn't actually pin pyOpenSSL to 0.13. It will happily use >= 0.14 if such a version is already installed (i.e. it reports its version as >= 0.14 when imported), but it will only request to build >= 0.13, <= 0.13.1 otherwise.
Also, this is not for setuptools reasons, at least not directly. It is because our experience was that the dependency on cffi was causing too many build failures.
Replying to daira:
I vaguely remember discussing this (it is kind of hard to find history in all these different tickets) but I don't see it mentioned here, and the situation has changed a lot since 2013, so, here goes:
Right now, Cryptography distributes wheels for OS X and Windows, which bundle OpenSSL and seem to install very reliably; I am fielding far fewer complaints about Twisted not installing due to build errors on those platforms, and it was a real headache when pyOpenSSL first moved.
This effort is also underway - https://github.com/manylinux/manylinux - to allow wheels to be provided that will robustly work across any possible reasonable linux distribution.
Does this change the situation at all for Tahoe?
Separately you may be interested in https://github.com/manylinux/auditwheel for your own purposes if Tahoe is producing its own build artifacts.
I'm quite happy to accept that the installation situation has improved. Does it still require manual installation of libffi, or is that now statically linked with the wheel? (Split the answer by platform and/or distribution if necessary.)
setuptools delenda estto zetuptoolz delenda estReplying to daira:
I believe the wheels statically link libffi on all platforms. It certainly must on Windows, because I've done
pip install cryptographythere, and it works, and I definitely never installed libffi on that platform. On OS X, there is guaranteed to be a system version of libffi, so it's less important there.On Linux I would presume static libffi would be a must.
On Windows 7 using the 1582.setuptools-delenda-est.1 branch:
It looks like the enthusiastically-long pycryptopp version string is causing a
shutil.copy()call to exceed the windows maximum pathname length. It's exacerbated by the string being used twice (in some sort of$PKGVER.tmp/../$PKGVERsyntax). It's possible that this could be fixed in setuptools, but it might also mean that zooko has to give up his beloved versioning scheme.We don't yet know what's going on with the pyasn1 error.
At today's meeting we decided to (at least provisionally) include this in the next release. It fixes #2728, which is probably a blocker, and should help reduce the need for a compiler, which has always been a problem on windows. It'd also close about half our outstanding "packaging" tickets.
There are other steps necessary if we really do include this:
pip install --editable .pip install allmydata-tahoeand get the version from PyPIallmydata-tahoetotahoe-lafs(#2011, but not #1950)I'll assemble a branch with proposed docs changes, and will look into the buildbot changes.
Packaging tickets that would probably be closed if we land this: #709 #717 #799 #1270 #1403 #1450 #1451 #1504 #1896 #2044 #2221 #2728 #1168 #1753 #2021 #2077 #2210 #2255 #2283 #2286 #2306 #2317 #2468 #2473 #2474 #657
Tickets that would not be closed, but would become a lot easier to resolve: #1220 #1244 #1343 #2055 #2475 #1342 #142
Do you have any 32-bit Windows users? I notice that Twisted is presently only shipping a wheel for Windows, and only for 64-bit. We could probably do a bit better here and ship 32-bit Windows wheels and OS X wheels as well.
Also: it's not causing any problems right now, but you'll probably be pleased to know that in Twisted trunk right now, we've removed PyCrypto as a dependency, and are now doing everything in
conchwithcryptography. We still need to get rid ofpyOpenSSLbut at least we only have one backend now. (Once we get rid of the dependency on OpenSSL's TLS specifics, which is a goal that's in sight now, we might even be able to have acryptographybackend that pulls from pycryptopp exclusively, modulo just a little dependency-mangling magic on their side...)The 1582.setuptools-delenda-est.1 branch is now looking in much better shape on Windows. In a virtualenv:
works for me and passes all tests.
As I suspected, Unicode command-line support on Windows is broken on this branch. It works fine if you use
python tahoe-script.py ...though; the problem is just thetahoe.exeexecutable generated by setuptools not passing through arguments correctly (see #565 and #1074).daira, I looked for a bug on the setuptools tracker ( https://bitbucket.org/pypa/setuptools/issues?status=new&status=open ) and could not find unicode command-line arguments as a bug there. Can you file one?
BTW, for those visiting this ticket in the future, the original intent was to stop using setuptools, i.e. go back to
distutils. This was borne out of frustration with the bad-old-days when setuptools was not being maintained, forks likedistributewere rampant, andvirtualenvwas not yet a thing. Later, setuptools improved for the better, andvirtualenvprovided a well-supported way of doing the funky custom thing we'd built in Tahoe. So some time around 2014 or 2015, we changed our minds, embraced the new setuptools, and this ticket was changed to be about stopping our use of zetuptoolz (Tahoe's embedded fork of setuptools, by this point horribly horribly ancient).Thanks for the update, warner. Posterity thanks you :). Perhaps we should update the ticket description as well, to clarify the present intention?
Replying to warner:
That's the wrong way to describe this problem. The facts are:
setuptools uses the short-paths Windows API instead of the long-paths Windows API. That's the root of the problem, and it is probably easy to fix (you just prepend
\\or something like that to the paths, and then Windows handles long paths).Because of this bug, the build will fail whenever the base directory that you're starting in, plus 2 times the name of your package, exceeds 256 bytes (I think).
Therefore, the build will always fail if the base directory that you're starting in has too long of a pathname, and it will always succeed (at least with all current package names), if the base directory you're in has sufficiently short of a pathname.
Therefore, reducing the size of pycryptopp's package name would be a kludge that would only prevent the error in some of the cases. Fixes that can fix the problem in all of the cases are:
Build in a directory with a short name, e.g. "C:/build". That is a workaround that anybody can do without changing the code.
Fix setuptools to use the correct Windows API, by prepending
\\or whatever it is to the pathnames before callingshutil.copy().I'm not going to change the pycryptopp versioning scheme.
Here's a way to test whether I'm right about what I said in comment:38:/tahoe-lafs/trac-2024-07-25/issues/6644:
Make a directory named 'C:/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'.
chdir into it.
Try to build any package at all.
If I'm right, the build will fail, regardless of which package you try to build, because of this bug in setuptools. This is not pycryptopp's problem.
If I'm right, then the workaround, whether you're building pycryptopp or any other package is "chdir into a directory with a shorter and shorter name until the build works".
Moving the setuptools-vs-pycryptopp issue to #2739 .
In d8549a2/trunk:
In 389efdf/trunk:
In aa8a5be/trunk:
In fe360ee4/trunk:
Reviewed https://github.com/warner/tahoe-lafs/commit/01d7f11e1ff88c9cd489287fc77e1fe64c5dcb5d
Zetuptoolz is no more!!11!1 \o/
Replying to daira:
🎉
👍
Thank you to everyone who participated in this issue, especially daira, who I know suffered greatly on the torturous path to this eventual, glorious resolution :-).
Milestone renamed