unnecessary rebuild of dependencies when tahoe-deps/ is present #717

New Issue

warner · 2009-05-29T01:15:10Z

warner commented

2009-05-29 01:15:10 +00:00

Using a trunk changeset:9a02f4a1043d6288 tree, on my debian/sid box, performing "python setup.py build" with the tahoe-deps tarball unpacked in the parent directory causes several dependency libraries to be built and installed in support/lib/python2.5/site-packages even though those libraries are already present, usable, and visible to pkg_resources.requires in the standard PYTHONPATH /usr/lib/python2.5/site-packages directory.

The libraries that are unnecessarily rebuilt are:

foolscap-0.4.1
twisted-8.2.0
simplejson-2.0.9

Removing ../tahoe-deps, erasing support/lib, and re-running "python setup.py build" correctly skips the build of these libraries, resulting in a support/lib that only contains setuptools-0.6c12dev and the allmydata-tahoe.egg-link file (well, and the usual setuptools noise: setup.py, setuptools.pth, and easy-install.pth).

Something in setuptools is buggy: having ../tahoe-deps available should not influence the decision to install a library or not.

This needs to be added to the setuptools bugtracker at http://bugs.python.org/setuptools/ .

Using a trunk changeset:9a02f4a1043d6288 tree, on my debian/sid box, performing "python setup.py build" with the tahoe-deps tarball unpacked in the parent directory causes several dependency libraries to be built and installed in support/lib/python2.5/site-packages even though those libraries are already present, usable, and visible to `pkg_resources.requires` in the standard PYTHONPATH /usr/lib/python2.5/site-packages directory. The libraries that are unnecessarily rebuilt are: * foolscap-0.4.1 * twisted-8.2.0 * simplejson-2.0.9 Removing ../tahoe-deps, erasing support/lib, and re-running "python setup.py build" correctly skips the build of these libraries, resulting in a support/lib that only contains setuptools-0.6c12dev and the allmydata-tahoe.egg-link file (well, and the usual setuptools noise: setup.py, setuptools.pth, and easy-install.pth). Something in setuptools is buggy: having ../tahoe-deps available should not influence the decision to install a library or not. This needs to be added to the setuptools bugtracker at <http://bugs.python.org/setuptools/> .

warner added the

labels 2009-05-29 01:15:10 +00:00

warner added this to the undecided milestone 2009-05-29 01:15:10 +00:00

zooko commented

2009-05-29 01:40:26 +00:00

Thank you for the bug report. As I'm sure you know, a good way to report this bug to setuptools devs would be to create a minimal test case. Here's a crack at one:

Make a setup.py file in a new empty directory with the following contents:

from setuptools import setup
setup(name="minbug", install_requires="simplejson>=1.4", find_links="../tahoe-deps")

Then run python ./setup.py build, with and without a simplejson-2.0.9.tar.gz located in ../tahoe-deps.

Thank you for the bug report. As I'm sure you know, a good way to report this bug to setuptools devs would be to create a minimal test case. Here's a crack at one: Make a `setup.py` file in a new empty directory with the following contents: ``` from setuptools import setup setup(name="minbug", install_requires="simplejson>=1.4", find_links="../tahoe-deps") ``` Then run `python ./setup.py build`, with and without a `simplejson-2.0.9.tar.gz` located in `../tahoe-deps`.

zooko commented

2009-06-04 16:23:08 +00:00

This may be related to http://bugs.python.org/setuptools/issue17 (easy_install will install a package that is already there). There is a work-around documented in that ticket.

This may be related to <http://bugs.python.org/setuptools/issue17> (easy_install will install a package that is already there). There is a work-around documented in that ticket.

warner commented

2009-06-06 01:07:01 +00:00

I'm not sure it is related.. as I understand it, setuptools-17 is about setuptools building packages that are present in an unusual directory that the platform has added to sys.path (like the /var/lib/python-support/python2.5 on debian), probably because it isn't honoring the platform's changes to sys.path when it scans for those packages.

In contrast, what I observed was that all of the unecessarily-rebuilt libraries were present in /usr/lib/python2.5/site-packages . None were in /var/lib/python-support/python2.5 .

I'm not sure it is related.. as I understand it, setuptools-17 is about setuptools building packages that are present in an unusual directory that the platform has added to sys.path (like the /var/lib/python-support/python2.5 on debian), probably because it isn't honoring the platform's changes to sys.path when it scans for those packages. In contrast, what I observed was that all of the unecessarily-rebuilt libraries were present in /usr/lib/python2.5/site-packages . None were in /var/lib/python-support/python2.5 .

zooko commented

2009-08-06 18:30:42 +00:00

This may be related to #657 ("python ./setup.py test" rebuilds packages).

cgalvan commented

2009-09-24 19:59:14 +00:00

Note that once #668 is resolved, switching to Distribute, this issue may or may not be resolved as a result.

zooko commented

2009-10-14 15:08:20 +00:00

It looks like this may be related to http://bugs.python.org/setuptools/issue65 (setuptools bug,inconsistent replacement of eggs when installing from file:// urls.) which is fixed in setuptools-0.6c10 prerelease. setuptools 65 is related to http://bugs.python.org/setuptools/issue17 (easy_install will install a package that is already there) and http://bugs.python.org/setuptools/issue20 (package required at build time seems to be not fully present at install time?).

It looks like this may be related to <http://bugs.python.org/setuptools/issue65> (setuptools bug,inconsistent replacement of eggs when installing from file:// urls.) which is fixed in setuptools-0.6c10 prerelease. setuptools 65 is related to <http://bugs.python.org/setuptools/issue17> (easy_install will install a package that is already there) and <http://bugs.python.org/setuptools/issue20> (package required at build time seems to be not fully present at install time?).

zooko commented

2009-10-14 15:09:51 +00:00

Maybe someone (cgalvan?) could test this issue with the setuptools v0.6c10.

zooko commented

2010-09-18 17:40:10 +00:00

There was a different bug which caused a similar behavior in #229.

zooko commented

2010-09-18 18:21:50 +00:00

In comment:7:ticket:1168 and comment:10:ticket:1168 we commented on this issue. David-Sarah wrote:

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

and I wrote:

Looks like it might be #717. Isn't the buildbot already testing this by running the "Desert Island build"? e.g: http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:-1)?

[Makefile]source:trunk/Makefile@4650#L266

In comment:7:ticket:1168 and comment:10:ticket:1168 we commented on this issue. David-Sarah wrote: I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully. and I wrote: Looks like it might be #717. Isn't the buildbot already testing this by running the "Desert Island build"? e.g: <http://tahoe-lafs.org/buildbot/builders/clean/builds/2522> In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario ([comment:-1](/tahoe-lafs/trac-2024-07-25/issues/717#issuecomment--1))? [Makefile]source:trunk/Makefile@4650#L266

davidsarah commented

2010-09-18 20:35:42 +00:00

Replying to zooko:

[...] Isn't the buildbot already testing this by running the "Desert Island build"? e.g: http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:-1)?

In my scenario, I download a non-SUMO tarball, run ~~python setup.py trial~~ python setup.py build followed by python setup.py trial (expecting the dependencies to be downloaded and built before the test), and then run python setup.py trial again (expecting the previously downloaded dependencies to be used and not rebuilt). Are these expectations correct?

In any case, this is not the same thing as downloading tahoe-deps.tar.gz, untarring it in the distribution directory and then running python setup.py build, which is what test-desert-island does.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/717#issuecomment-71341): > [...] Isn't the buildbot already testing this by running the "Desert Island build"? e.g: <http://tahoe-lafs.org/buildbot/builders/clean/builds/2522> > > In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario ([comment:-1](/tahoe-lafs/trac-2024-07-25/issues/717#issuecomment--1))? In my scenario, I download a non-SUMO tarball, run ~~`python setup.py trial`~~ `python setup.py build` followed by `python setup.py trial` (expecting the dependencies to be downloaded and built before the test), and then run `python setup.py trial` again (expecting the previously downloaded dependencies to be used and not rebuilt). Are these expectations correct? In any case, this is not the same thing as downloading `tahoe-deps.tar.gz`, untarring it in the distribution directory and then running `python setup.py build`, which is what test-desert-island does.

warner commented

2016-03-26 21:22:37 +00:00

We no longer provide or support the tahoe-deps tarball/directory, and the pip http+wheel cache works correctly to avoid rebuilds. Closing this out.

We no longer provide or support the `tahoe-deps` tarball/directory, and the pip http+wheel cache works correctly to avoid rebuilds. Closing this out.

warner added the

fixed

label 2016-03-26 21:22:37 +00:00

warner modified the milestone from undecided to 1.11.0

2016-03-26 21:22:37 +00:00

warner closed this issue

2016-03-26 21:22:37 +00:00

Sign in to join this conversation.