setuptools downloads and builds a correct version of a dependency in the install-to-egg step, but then adds a different version not satisfying the requirement to easy_install.pth #1450

New Issue

tahoe-lafs · 2011-07-28T01:30:36Z

davidsarah commented

2011-07-28 01:30:36 +00:00

In the attached install-to-egg build on the 'Eugen lenny' buildslave, the requirement for Twisted is >= 10.1.0.

setuptools downloads and builds Twisted 10.2:

Searching for Twisted>=10.1.0
Best match: Twisted 10.2.0
Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/Twisted-10.2.0.tar.bz2
Processing Twisted-10.2.0.tar.bz2
Running Twisted-10.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QTimkw/Twisted-10.2.0/egg-dist-tmp-yBXrHn
[...]

but then it adds Twisted 8.1 to easy-install.pth, resulting in a failed Tahoe build:

Adding Twisted 8.1.0 to easy-install.pth file
[...]
Installed /home/tahoebuildslave/tahoebuildslave/Eugen lenny-amd64/build/egginstalldir/Twisted-8.1.0-py2.5-linux-x86_64.egg
error: Could not find required distribution Twisted>=10.1.0

Since we're installing to an egg rather than globally, there's no good reason why that shouldn't work even if Twisted 8.1 is installed single-version-externally-managed. But if that's not supported for some reason, it's still incorrect to build 10.2 and then fail with a misleading error message.

In the attached install-to-egg build on the 'Eugen lenny' buildslave, the requirement for Twisted is >= 10.1.0. setuptools downloads and builds Twisted 10.2: ``` Searching for Twisted>=10.1.0 Best match: Twisted 10.2.0 Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/Twisted-10.2.0.tar.bz2 Processing Twisted-10.2.0.tar.bz2 Running Twisted-10.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QTimkw/Twisted-10.2.0/egg-dist-tmp-yBXrHn [...] ``` but then it adds Twisted 8.1 to easy-install.pth, resulting in a failed Tahoe build: ``` Adding Twisted 8.1.0 to easy-install.pth file [...] Installed /home/tahoebuildslave/tahoebuildslave/Eugen lenny-amd64/build/egginstalldir/Twisted-8.1.0-py2.5-linux-x86_64.egg error: Could not find required distribution Twisted>=10.1.0 ``` Since we're installing to an egg rather than globally, there's no good reason why that shouldn't work even if Twisted 8.1 is installed single-version-externally-managed. But if that's not supported for some reason, it's still incorrect to build 10.2 and then fail with a misleading error message.

tahoe-lafs added the

labels 2011-07-28 01:30:36 +00:00

tahoe-lafs added this to the undecided milestone 2011-07-28 01:30:36 +00:00

davidsarah commented

2011-07-28 01:32:27 +00:00

Attachment Eugen-lenny-amd64-build-850-install-to-egg.txt (12931 bytes) added

Copy of http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/850/steps/install-to-egg/logs/stdio/text

**Attachment** Eugen-lenny-amd64-build-850-install-to-egg.txt (12931 bytes) added Copy of <http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/850/steps/install-to-egg/logs/stdio/text>

Eugen-lenny-amd64-build-850-install-to-egg.txt

13 KiB

zooko commented

2011-08-01 18:22:20 +00:00

According to http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/860/steps/show-tool-versions/logs/stdio , that is setuptools-0.6c8 doing that. That is a very old version. Perhaps this is one of the bugs that has been fixed between setuptools-0.6c8 and the current setuptools-0.6c11 or distribute-0.6.19. We don't see the same bug on other builders, that have newer version of setuptools installed.

I'll ask Eugen in email if he could upgrade the setuptools on there. If not, we could try to reproduce this by installing setuptools-0.6c8, reproducing this failure, then upgrading setuptools to 0.6c11 or distribute-0.6.19 and seeing if the failure goes away.

According to <http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/860/steps/show-tool-versions/logs/stdio> , that is setuptools-0.6c8 doing that. That is a very old version. Perhaps this is one of the bugs that has been fixed between setuptools-0.6c8 and the current setuptools-0.6c11 or distribute-0.6.19. We don't see the same bug on other builders, that have newer version of setuptools installed. I'll ask Eugen in email if he could upgrade the setuptools on there. If not, we could try to reproduce this by installing setuptools-0.6c8, reproducing this failure, then upgrading setuptools to 0.6c11 or distribute-0.6.19 and seeing if the failure goes away.

zooko commented

2011-08-04 16:30:07 +00:00

Eugen upgraded to setuptools-0.6c12dev-!r88846 (show-tool-versions), and this problem went away: test-from-egg stdio.

Now there is another problem in the subsequent test-from-egg step. The sys.path has /usr/lib/python2.5/site-packages earlier than /home/tahoebuildslave/tahoebuildslave/Eugen lenny-amd64/build/egginstalldir/Twisted-11.0.0-py2.5-linux-x86_64.egg, so Twisted 8.1 (which is in /usr/lib/python2.5/site-packages gets imported and then our [check_all_requirements()]source:trunk/src/allmydata/init.py?annotate=blame&rev=5079#L346 function correctly objects that we imported a disallowed version.

I don't know why /usr/lib/python2.5/site-packages is earlier than the Twisted egg. Setuptools normally puts all .egg's earlier on the sys.path earlier than all non-egg entries (which is the cause of the most widely hated setuptools issue: setuptools issue #53 -- respect the PYTHONPATH).

Eugen upgraded to setuptools-0.6c12dev-!r88846 ([show-tool-versions](http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/878/steps/show-tool-versions/logs/stdio)), and this problem went away: [test-from-egg stdio](http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/878/steps/install-to-egg/logs/stdio). Now there is another problem in the subsequent [test-from-egg](http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/878/steps/test-from-egg/logs/stdio) step. The `sys.path` has `/usr/lib/python2.5/site-packages` earlier than `/home/tahoebuildslave/tahoebuildslave/Eugen lenny-amd64/build/egginstalldir/Twisted-11.0.0-py2.5-linux-x86_64.egg`, so Twisted 8.1 (which is in `/usr/lib/python2.5/site-packages` gets imported and then our [check_all_requirements()]source:trunk/src/allmydata/*init*.py?annotate=blame&rev=5079#L346 function correctly objects that we imported a disallowed version. I don't know why `/usr/lib/python2.5/site-packages` is earlier than the Twisted egg. Setuptools normally puts all .egg's earlier on the `sys.path` earlier than all non-egg entries (which is the cause of the most widely hated setuptools issue: [setuptools issue #53 -- respect the PYTHONPATH](http://bugs.python.org/setuptools/issue53)).

daira commented

2014-03-17 18:42:18 +00:00

Replying to zooko:

I don't know why /usr/lib/python2.5/site-packages is earlier than the Twisted egg. Setuptools normally puts all .egg's earlier on the sys.path earlier than all non-egg entries (which is the cause of the most widely hated setuptools issue: setuptools issue #53 -- respect the PYTHONPATH).

setuptools does not always put .egg entries earlier than non-egg entries. If a non-egg is needed to satisfy a dependency and it is in a shared package directory (such as a site-packages directory, either the global one or one in a virtualenv), then other packages from that directory can end up being found before the ones required by Tahoe, even when setuptools thinks theose requirements have been correctly satisfied (see /tahoe-lafs/trac-2024-07-25/issues/6308#comment:6). This behaviour is nondeterministic.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1450#issuecomment-84482): > I don't know why `/usr/lib/python2.5/site-packages` is earlier than the Twisted egg. Setuptools normally puts all .egg's earlier on the `sys.path` earlier than all non-egg entries (which is the cause of the most widely hated setuptools issue: [setuptools issue #53 -- respect the PYTHONPATH](http://bugs.python.org/setuptools/issue53)). setuptools does not always put .egg entries earlier than non-egg entries. If a non-egg is needed to satisfy a dependency and it is in a shared package directory (such as a `site-packages` directory, either the global one or one in a virtualenv), then other packages from that directory can end up being found before the ones required by Tahoe, even when setuptools thinks theose requirements have been correctly satisfied (see [/tahoe-lafs/trac-2024-07-25/issues/6308](/tahoe-lafs/trac-2024-07-25/issues/6308)#comment:6). This behaviour is nondeterministic.

warner commented

2016-03-26 22:41:59 +00:00

We no longer use eggs, and pip installs our dependencies in an unpacked form. I think we can close this now.

warner added the

fixed

label 2016-03-26 22:41:59 +00:00

warner modified the milestone from undecided to 1.11.0

2016-03-26 22:41:59 +00:00

warner closed this issue

2016-03-26 22:41:59 +00:00

Sign in to join this conversation.