setuptools fails to be satisfied with already installed Nevow and tries to install a new one #229

New Issue

zooko · 2007-12-12T01:08:09Z

zooko commented

2007-12-12 01:08:09 +00:00

It is supposed to be the case that you can install the "MANUAL DEPENDENCIES", download the tahoe source distribution, move to a desert island with no Internet, follow the README, and get a working version of Tahoe.

To that end, we bundle source tarballs of the "AUTOMATIC DEPENDENCIES", and we configure our setup process to use those locally available tarballs instead of fetching things from the Internet.

However, this feature is not working. Our setup process keeps fetching things from the Internet even when it doesn't have to, and worse builds fail (sometimes? On some platforms?) if it is unable to download packages.

See also #212 "easy_install of a fat binary .egg which was built on Mac OS 10.4 fails on Mac OS 10.5", and #164 "bundled dep tarballs not used if... if some of the .egg's are already found on the system?".

It is supposed to be the case that you can install the "MANUAL DEPENDENCIES", download the tahoe source distribution, move to a desert island with no Internet, follow the README, and get a working version of Tahoe. To that end, we bundle source tarballs of the "AUTOMATIC DEPENDENCIES", and we configure our setup process to use those locally available tarballs instead of fetching things from the Internet. However, this feature is not working. Our setup process keeps fetching things from the Internet even when it doesn't have to, and worse builds fail (sometimes? On some platforms?) if it is unable to download packages. See also #212 "easy_install of a fat binary .egg which was built on Mac OS 10.4 fails on Mac OS 10.5", and #164 "bundled dep tarballs not used if... if some of the .egg's are already found on the system?".

zooko added the

labels 2007-12-12 01:08:09 +00:00

zooko commented

2007-12-12 01:12:47 +00:00

The setuptools_darcs plugin is often a problem. We need to fix this, by either bundling that package and making the use-bundled-packages feature work, or by no longer requiring setuptools_darcs. This plugin is actually needed only for producing source packages from source -- not for producing binary packages nor for installing locally. The reason that it is required by default is that if someone does get a copy of our source tree and then tries to use it to produce a new source tree (i.e. by running ./setup.py sdist, then without the setuptools_darcs plugin they will get a bad (incomplete) source package with no warning or error message. In order to minimize the chance that somebody stubs their toe on that corner case, I have made the setuptools_darcs plugin mandatory.

The `setuptools_darcs` plugin is often a problem. We need to fix this, by either bundling that package and making the use-bundled-packages feature work, or by no longer requiring `setuptools_darcs`. This plugin is actually needed only for producing source packages from source -- not for producing binary packages nor for installing locally. The reason that it is required by default is that if someone does get a copy of our source tree and then tries to use it to produce a new source tree (i.e. by running `./setup.py sdist`, then without the `setuptools_darcs` plugin they will get a bad (incomplete) source package with no warning or error message. In order to minimize the chance that somebody stubs their toe on that corner case, I have made the `setuptools_darcs` plugin mandatory.

zooko commented

2007-12-25 22:46:02 +00:00

fixed by some subset of the patches between changeset:3028defb7e1f00f3 and changeset:bfc69178b2f07477 (inclusive on both ends).

zooko added the

fixed

label 2007-12-25 22:46:02 +00:00

zooko added this to the 0.7.0 milestone 2007-12-25 22:46:02 +00:00

zooko closed this issue

2007-12-25 22:46:02 +00:00

warner commented

2008-03-04 23:20:52 +00:00

I'm seeing this problem occur again, on my home machine running
debian/unstable. After I upgraded to the debian packaging of 0.6c8 (from
0.6c7), the tahoe build process started downloading and building a Nevow
tarball, even though there's a local copy already installed (with a
valid-looking .egg-info file) on sys.path .

http://allmydata.org/buildbot/builders/speed-DSL/builds/315/steps/compile/logs/stdio
is an example of the build process with setuptools-0.6c8, in which Nevow is
downloaded from divmod.org and compiled.

http://allmydata.org/buildbot/builders/speed-DSL/builds/312/steps/compile/logs/stdio
is an example of a build which failed because the local debian package of
nevow-0.9.29 was detected, which locked the target version at 0.9.29, and
setuptools then tried to download an 0.9.29 tarball (which failed because by
that point divmod.org was hosting a newer version, and so I presume they had
removed the old tarball).

http://allmydata.org/buildbot/builders/speed-DSL/builds/311/steps/compile/logs/stdio
is the first build with the new setuptools-0.6c8, doing the extra
download-and-build step.

http://allmydata.org/buildbot/builders/speed-DSL/builds/310/steps/compile/logs/stdio
is the last build with the old setuptools-0.6c7, in which it correctly
refrains from downloading or building Nevow.

I'm seeing this problem occur again, on my home machine running debian/unstable. After I upgraded to the debian packaging of 0.6c8 (from 0.6c7), the tahoe build process started downloading and building a Nevow tarball, even though there's a local copy already installed (with a valid-looking .egg-info file) on sys.path . <http://allmydata.org/buildbot/builders/speed-DSL/builds/315/steps/compile/logs/stdio> is an example of the build process with setuptools-0.6c8, in which Nevow is downloaded from divmod.org and compiled. <http://allmydata.org/buildbot/builders/speed-DSL/builds/312/steps/compile/logs/stdio> is an example of a build which failed because the local debian package of nevow-0.9.29 was detected, which locked the target version at 0.9.29, and setuptools then tried to download an 0.9.29 tarball (which failed because by that point divmod.org was hosting a newer version, and so I presume they had removed the old tarball). <http://allmydata.org/buildbot/builders/speed-DSL/builds/311/steps/compile/logs/stdio> is the first build with the new setuptools-0.6c8, doing the extra download-and-build step. <http://allmydata.org/buildbot/builders/speed-DSL/builds/310/steps/compile/logs/stdio> is the last build with the old setuptools-0.6c7, in which it correctly refrains from downloading or building Nevow.

warner removed the

fixed

label 2008-03-04 23:20:52 +00:00

warner reopened this issue

2008-03-04 23:20:52 +00:00

warner modified the milestone from 0.7.0 to undecided

2008-03-08 00:14:48 +00:00

zooko commented

2008-03-11 02:17:20 +00:00

Here's a test which could show whether setuptools/easy_install is working on your system:

Find a Python package which is already installed, such as Nevow.

Now run sudo easy_install nevow. If it downloads anything, then your easy_install is broken.

Alternately, you could get minproj-2.0.0.tar.gz.
This is a project that has install_requires=["nevow"] in it. If running sudo ./setup.py install downloads anything, then your setuptools is broken.

You can do this without sudo, by setting the PYTHONPATH and using --prefix...

Here's a test which could show whether setuptools/easy_install is working on your system: Find a Python package which is already installed, such as Nevow. Now run `sudo easy_install nevow`. If it downloads anything, then your `easy_install` is broken. Alternately, you could get minproj-2.0.0.tar.gz. This is a project that has `install_requires=["nevow"]` in it. If running `sudo ./setup.py install` downloads anything, then your `setuptools` is broken. You can do this without sudo, by setting the PYTHONPATH and using --prefix...

warner commented

2008-03-11 02:37:32 +00:00

On my sid workstation ('fluxx'), I did the following commands, and observed
setuptools download nevow from divmod.org, despite nevow-0.9.31 already being
installed such that 'import nevow' worked inside a python shell.

HERE = ~/trees/tahoe-mutable-205-312/t

go:
	mkdir -p $(HERE)/tmp/lib/python2.4/site-packages
	PYTHONPATH=$(HERE)/tmp/lib/python2.4/site-packages easy_install --prefix=$(HERE)/tmp nevow

I will see if I can find a machine on which this same test does not download anything, and then compare the setuptools/easy_install installations.

On my sid workstation ('fluxx'), I did the following commands, and observed setuptools download nevow from divmod.org, despite nevow-0.9.31 already being installed such that 'import nevow' worked inside a python shell. ``` HERE = ~/trees/tahoe-mutable-205-312/t go: mkdir -p $(HERE)/tmp/lib/python2.4/site-packages PYTHONPATH=$(HERE)/tmp/lib/python2.4/site-packages easy_install --prefix=$(HERE)/tmp nevow ``` I will see if I can find a machine on which this same test does *not* download anything, and then compare the setuptools/easy_install installations.

zooko commented

2008-03-11 04:18:44 +00:00

Here's a related question: is Nevow installed on fluxx in such a way that pkg_resources recognizes it and its version number? Try this:

python -c 'import pkg_resources;print pkg_resources.require("nevow")'

and

python -c 'import pkg_resources;print pkg_resources.require("nevow == 0.9.31")'

Here's a related question: is Nevow installed on fluxx in such a way that pkg_resources recognizes it and its version number? Try this: ``` python -c 'import pkg_resources;print pkg_resources.require("nevow")' ``` and ``` python -c 'import pkg_resources;print pkg_resources.require("nevow == 0.9.31")' ```

warner commented

2008-03-11 05:18:48 +00:00

yes, I believe so:

1293:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow")'
[Nevow 0.9.31 (/var/lib/python-support/python2.4)]
1294:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow == 0.9.31")'
[Nevow 0.9.31 (/var/lib/python-support/python2.4)]

yes, I believe so: ``` 1293:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow")' [Nevow 0.9.31 (/var/lib/python-support/python2.4)] 1294:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow == 0.9.31")' [Nevow 0.9.31 (/var/lib/python-support/python2.4)] ```

zooko commented

2008-05-31 01:18:40 +00:00

This has been ticketed on the setuptool s issue tracker issue17 and debian bug report 475440.

This has been ticketed on [the setuptool s issue tracker issue17](http://bugs.python.org/setuptools/issue17) and [debian bug report 475440](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475440).

zooko modified the milestone from undecided to 1.1.0

2008-05-31 01:18:40 +00:00

zooko changed title from ~~"Desert Island" build~~ to debian sid packaging of setuptools v0.6c8 broke the "Desert Island" build

2008-05-31 01:20:36 +00:00

warner commented

2008-06-03 07:23:19 +00:00

For reference, this is still broken on my debian/sid system: Nevow-0.9.31 is downloaded and built even though pkg_resources.require("nevow") indicates that 0.9.31 is available in /var/lib/python-support/python2.5 .

warner commented

2008-06-26 22:36:39 +00:00

we're also seeing this on a hardy box

zooko commented

2008-07-02 20:04:10 +00:00

Confirmed that this problem happens on our new Hardy server, and also I confirmed that it stops happening when you uninstall the python-setuptools package from Ubuntu and instead install setuptools from upstream directly. I'll update the Debian bug report and open a sibling bug report for Ubuntu...

zooko commented

2008-08-01 18:28:44 +00:00

Added a bug report for Ubuntu:

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035

Added a bug report for Ubuntu: <https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035>

zooko commented

2008-08-01 21:32:08 +00:00

Ah, it turns out that this is not caused by the Debian/Ubuntu packaging! It happens also with the upstream setuptools. I'm not sure why I thought it didn't (as earlier posted in this ticket).

Ah, it turns out that this is *not* caused by the Debian/Ubuntu packaging! It happens also with the upstream setuptools. I'm not sure why I thought it didn't (as earlier posted in this ticket).

warner commented

2008-08-20 23:23:36 +00:00

still happening on a hardy box (deharo2). My workaround was to uninstall tahoe+nevow, thus allowing the buildslave to download+compile nevow on each build.

zooko commented

2008-08-21 04:01:59 +00:00

By the way, I suspect that #455 (setuptools sandbox isn't tight enough to hold Twisted's sand in so that it doesn't get in Nevow's eyes) might be an underlying cause of this, or at least might be related.

zooko changed title from ~~debian sid packaging of setuptools v0.6c8 broke the "Desert Island" build~~ to setuptools fails to be satisfied with already installed Nevow and tries to install a new one

2008-08-21 04:02:37 +00:00

warner commented

2008-09-10 23:38:25 +00:00

The same problem was causing build failures via python-openssl, since 0.6 is not being distributed anymore (or it's been removed from certain sf.net mirrors, or sf.net is having connectivity problems). In changeset:ae6cc14a2da9d7f1 I added a workaround: just add --site-dirs=/var/lib/python-support/python2.5/ to the 'setup.py develop' command line. This should fix the problem on debian-like systems and not cause any ill effects on other platforms.

I expect that this ought to cause our buildslaves to stop trying to build nevow and openssl when they are already installed.

The same problem was causing build failures via python-openssl, since 0.6 is not being distributed anymore (or it's been removed from certain sf.net mirrors, or sf.net is having connectivity problems). In changeset:ae6cc14a2da9d7f1 I added a workaround: just add --site-dirs=/var/lib/python-support/python2.5/ to the 'setup.py develop' command line. This should fix the problem on debian-like systems and not cause any ill effects on other platforms. I expect that this ought to cause our buildslaves to stop trying to build nevow and openssl when they are already installed.

robk commented

2008-09-24 17:42:24 +00:00

so I had a very similar problem today, on the mac

it turned out that easy_install had at some point installed twisted into /Library, and somehow tahoe (specifcally make test) was pulling in twisted from /System/Library (the apple supplied version) which wound up barfing on an error it got whilst cross-importing bits of twisted from out of /Library. I removed the twisted egg etc from /Library, but also needed to add

"--site-dirs /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python"

to the build args in order to get tahoe's build to not try and install a different version of twisted and instead use the extant system one. I'll submit a patch to that end shortly.

so I had a very similar problem today, on the mac it turned out that easy_install had at some point installed twisted into /Library, and somehow tahoe (specifcally make test) was pulling in twisted from /System/Library (the apple supplied version) which wound up barfing on an error it got whilst cross-importing bits of twisted from out of /Library. I removed the twisted egg etc from /Library, but also needed to add "--site-dirs /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python" to the build args in order to get tahoe's build to not try and install a different version of twisted and instead use the extant system one. I'll submit a patch to that end shortly.

warner commented

2008-09-24 18:38:46 +00:00

Note that we had to add a --site-dirs argument to get setuptools to recognize the python modules that get installed (on debian) into a /var/lib directory that is on sys.path but isn't one of the traditional places.

Take a look at setup.py for the place to add it. It should be harmless to add another one.

Note that we had to add a `--site-dirs` argument to get setuptools to recognize the python modules that get installed (on debian) into a /var/lib directory that is on sys.path but isn't one of the traditional places. Take a look at setup.py for the place to add it. It should be harmless to add another one.

zooko commented

2008-10-22 00:50:28 +00:00

Well, I guess this is fixed by our --site-dirs hack. Not very satisfying (I would really prefer it if setuptools would search the sys.path for requirements per http://bugs.python.org/setuptools/issue17 ), but the allmydata-tahoe-relevant portion of this issue is fixed. Right?

Well, I guess this is fixed by our `--site-dirs` hack. Not very satisfying (I would really prefer it if setuptools would search the `sys.path` for requirements per <http://bugs.python.org/setuptools/issue17> ), but the allmydata-tahoe-relevant portion of this issue is fixed. Right?

zooko added the

fixed

label 2008-10-22 00:50:28 +00:00

zooko closed this issue

2008-10-22 00:50:28 +00:00

launchpad commented

2008-10-31 15:33:28 +00:00

Updating Launchpad bug reference

zooko commented

2009-03-09 16:36:04 +00:00

This was marked as fixed long ago, possibly in 1.2.0 but definitely in 1.3.0. Note that there is currently an observed bug on some platforms which smells like this one, so perhaps this should be reopened. (On Zandr's NAS device and on Brian's Debian system, pycryptopp gets built both during python ./setup.py build and during python ./setup.py test. Okay, that's actually a different underlying issue so I'll open a new ticket for that.

This was marked as fixed long ago, possibly in 1.2.0 but definitely in 1.3.0. Note that there is currently an observed bug on some platforms which smells like this one, so perhaps this should be reopened. (On Zandr's NAS device and on Brian's Debian system, pycryptopp gets built both during `python ./setup.py build` and during `python ./setup.py test`. Okay, that's actually a different underlying issue so I'll open a new ticket for that.

zooko added this to the 1.3.0 milestone 2009-03-09 16:36:04 +00:00

Sign in to join this conversation.