yet another failure of setuptools to set up a correct sys.path #1451

New Issue

tahoe-lafs · 2011-07-28T03:27:38Z

davidsarah commented

2011-07-28 03:27:38 +00:00

(http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/534/steps/install-to-egg/logs/stdio)

Even though the setup-time requirement for zope.interface in that build was "zope.interface == 3.3.1, == 3.5.3, == 3.6.1", the version that was on the sys.path when building Nevow was 3.6.4.

(http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/534/steps/install-to-egg/logs/stdio) Even though the setup-time requirement for zope.interface in that build was "zope.interface == 3.3.1, == 3.5.3, == 3.6.1", the version that was on the sys.path when building Nevow was 3.6.4.

tahoe-lafs added the

labels 2011-07-28 03:27:38 +00:00

tahoe-lafs added this to the undecided milestone 2011-07-28 03:27:38 +00:00

davidsarah commented

2011-07-28 03:29:25 +00:00

This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go.

davidsarah commented

2011-07-28 03:34:34 +00:00

Note that the Tahoe version here is r5073, not the 1.8.2 release.

zooko commented

2011-07-28 05:35:11 +00:00

Replying to davidsarah:

This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go.

Maybe we should open a ticket named "setuptools delenda est", or instead start a mailing list thread. My first question would be: is there reason to believe it would be worth switching to distribute instead? (My instinct: no. The distribute project changed a bunch of things without first writing unit tests, thus resulting in an even buggier codebase than the original setuptools. Then they abandoned (?) it.)

What about the new Distutils2 project? (The successor from the same group that did the aforementioned "distribute" project.) As far as I know it isn't yet supported by the makers of our deps, so for example as far as I understand the pyOpenSSL project doesn't -- or can't? -- produce a package which provides pyOpenSSL on Windows when the user doesn't have a compiler.

Now we could still make our build system build Tahoe-LAFS on all supported operating systems including Windows when the user does have a compiler, and we could still provide a complete pre-built bundle of binaries of pyOpenSSL with Tahoe-LAFS for certain specific operating systems, but this would mean users no longer have the ability to acquire binaries of deps from upstream (e.g. pyOpenSSL maintainers) and source code from Tahoe-LAFS and use the two together.

Also, even to get that far would be a great deal of effort on our part, wouldn't it? Unless Distutils2, or pip, or something is a lot more functional than I thought.

Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/1451#issuecomment-84494): > This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go. Maybe we should open a ticket named "setuptools delenda est", or instead start a mailing list thread. My first question would be: is there reason to believe it would be worth switching to `distribute` instead? (My instinct: no. The distribute project changed a bunch of things without first writing unit tests, thus resulting in an even buggier codebase than the original setuptools. Then they abandoned (?) it.) What about the new Distutils2 project? (The successor from the same group that did the aforementioned "distribute" project.) As far as I know it isn't yet supported by the makers of our deps, so for example as far as I understand the pyOpenSSL project doesn't -- or can't? -- produce a package which provides pyOpenSSL on Windows when the user doesn't have a compiler. Now we could still make our build system build Tahoe-LAFS on all supported operating systems including Windows when the user does have a compiler, and we could still provide a complete pre-built bundle of binaries of pyOpenSSL with Tahoe-LAFS for certain specific operating systems, but this would mean users no longer have the ability to acquire binaries of deps from upstream (e.g. pyOpenSSL maintainers) and source code from Tahoe-LAFS and use the two together. Also, even to get that far would be a great deal of effort on our part, wouldn't it? Unless Distutils2, or pip, or something is a lot more functional than I thought.

davidsarah commented

2011-07-28 16:49:09 +00:00

Replying to [zooko]comment:3:

Replying to davidsarah:

This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go.

Maybe we should open a ticket named "setuptools delenda est", or instead start a mailing list thread. My first question would be: is there reason to believe it would be worth switching to distribute instead?

No. distribute doesn't seem to have fixed any of the things in setuptools that I consider most broken, and has introduced its own bugs.

What about the new Distutils2 project? (The successor from the same group that did the aforementioned "distribute" project.) As far as I know it isn't yet supported by the makers of our deps, so for example as far as I understand the pyOpenSSL project doesn't -- or can't? -- produce a package which provides pyOpenSSL on Windows when the user doesn't have a compiler.

If setuptools is used to produce a given pyOpenSSL egg, that doesn't mean you need setuptools to run an application that uses it. That's just a matter of putting the egg in the right position on the sys.path (which setuptools can't do reliably), and then the Python ZipImport machinery will load it, including loading native libraries.

So run-time should be easy, and we shouldn't need the script that setuptools generates in the support directory.

At build-time, we do need to run the setup.py files of dependencies. If we build them in subprocesses, set up the sys.path for each subprocess correctly (i.e. with that package's build-time dependencies), and prevent setuptools from downloading packages, then the scope that setuptools has to mess up each build is much reduced. Then we can incrementally change (or persuade its maintainers to change) each package to not depend on setuptools at build time.

Now we could still make our build system build Tahoe-LAFS on all supported operating systems including Windows when the user does have a compiler, and we could still provide a complete pre-built bundle of binaries of pyOpenSSL with Tahoe-LAFS for certain specific operating systems, but this would mean users no longer have the ability to acquire binaries of deps from upstream (e.g. pyOpenSSL maintainers) and source code from Tahoe-LAFS and use the two together.

No, because if the pyOpenSSL maintainers want to continue to use setuptools to generate their eggs, we're not stopping them. It's our own use of setuptools that we want to replace first, because that's where most of the problems lie.

Replying to [zooko]comment:3: > Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/1451#issuecomment-84494): > > This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go. > > Maybe we should open a ticket named "setuptools delenda est", or instead start a mailing list thread. My first question would be: is there reason to believe it would be worth switching to `distribute` instead? No. `distribute` doesn't seem to have fixed any of the things in setuptools that I consider most broken, and has introduced its own bugs. > What about the new Distutils2 project? (The successor from the same group that did the aforementioned "distribute" project.) As far as I know it isn't yet supported by the makers of our deps, so for example as far as I understand the pyOpenSSL project doesn't -- or can't? -- produce a package which provides pyOpenSSL on Windows when the user doesn't have a compiler. If setuptools is used to produce a given pyOpenSSL egg, that doesn't mean you need setuptools to run an application that uses it. That's just a matter of putting the egg in the *right* position on the `sys.path` (which setuptools can't do reliably), and then the Python ZipImport machinery will load it, including loading native libraries. So run-time should be easy, and we shouldn't need the script that setuptools generates in the support directory. At build-time, we do need to run the `setup.py` files of dependencies. If we build them in subprocesses, set up the `sys.path` for each subprocess correctly (i.e. with that package's build-time dependencies), and prevent setuptools from downloading packages, then the scope that setuptools has to mess up each build is much reduced. Then we can incrementally change (or persuade its maintainers to change) each package to not depend on setuptools at build time. > Now we could still make our build system build Tahoe-LAFS on all supported operating systems including Windows when the user does have a compiler, and we could still provide a complete pre-built bundle of binaries of pyOpenSSL with Tahoe-LAFS for certain specific operating systems, but this would mean users no longer have the ability to acquire binaries of deps from upstream (e.g. pyOpenSSL maintainers) and source code from Tahoe-LAFS and use the two together. No, because *if* the pyOpenSSL maintainers want to continue to use setuptools to generate their eggs, we're not stopping them. It's our own use of setuptools that we want to replace first, because that's where most of the problems lie.

davidsarah commented

2011-07-28 16:54:56 +00:00

Attachment Freestorm-WinXP-x86-py26-build-534-install-to-egg.txt (8382 bytes) added

Copy of http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/534/steps/install-to-egg/logs/stdio/text

**Attachment** Freestorm-WinXP-x86-py26-build-534-install-to-egg.txt (8382 bytes) added Copy of <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/534/steps/install-to-egg/logs/stdio/text>

Freestorm-WinXP-x86-py26-build-534-install-to-egg.txt

8.2 KiB

zooko commented

2011-07-28 17:40:23 +00:00

Let's move this discussion to its own ticket or the mailing list.

davidsarah commented

2012-03-29 19:55:03 +00:00

The "setuptools delenda est" ticket is #1582. (It doesn't replace this ticket which is documenting a specific problem with setuptools.)

warner commented

2016-03-26 22:43:21 +00:00

We've stopped using eggs, virtualenvs are protecting us against system-installed packages, and modern setuptools is working pretty well for us. I think we can close this now.

warner added the

fixed

label 2016-03-26 22:43:21 +00:00

warner modified the milestone from undecided to 1.11.0

2016-03-26 22:43:21 +00:00

warner closed this issue

2016-03-26 22:43:21 +00:00

Sign in to join this conversation.