Nevow doesn't declare its dependency on Twisted in a machine-readable way. #440

New Issue

zooko · 2008-06-03T00:12:00Z

zooko commented

2008-06-03 00:12:00 +00:00

So Adam Langley's problem building Tahoe on Ubuntu turned out to be completely reproducible. Any time you run make in a Tahoe source tree on a system where neither Twisted nor Nevow is installed, setuptools will first install Twisted into a directory named Twisted-$VERSIONNUMBER.egg in the current working directory, because our source:setup.py told setuptools (in source:_auto_deps.py) that we need to have Twisted installed at build time.

We told it that for two reasons: 1. We require twisted to run tests, which is a build-time activity, and 2. We were attempting to work-around the fact that Nevow doesn't declare its dependency on Twisted in a machine-readable way, so that setuptools doesn't know whether to install Twisted first or Nevow first, and if it picks Nevow then the Nevow installation fails at install time when it can't import twisted modules.

However, it turns out that on at least Ubuntu Hardy (Adam's machine) and on my Macbook Pro (Mac OS 10.4), having Twisted listed as setup_requires and having Nevow being listed as install_requires leads to this bizarre error message which takes the name of PIL in vain:

 File "/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py",
line 27, in run_setup
 File "/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py",
line 63, in run
 File "/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py",
line 29, in <lambda>
 File "setup.py", line 8, in <module>
   #
 File "/tmp/easy_install-fqJkNC/Nevow-0.9.18/setupcommon.py", line 2,
in <module>
 File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 10, in <module>
   #
ImportError: No module named components

Here is a minimal Python project that you can use to test this yourself. Simply put the following two lines into a file named setup.py:

import setuptools
setuptools.setup(name="minproj_that_needs_nevow", setup_requires=["Twisted"], install_requires=["Nevow"]

And then execute python ./setup.py install. If your platform is like Adam's and mine, then you'll get this bizarre traceback that takes the name of PIL in vain.

Here is another minimal project's setup.py file:

import setuptools
setuptools.setup(name="minproj_that_needs_nevow", install_requires=["Nevow"]

If you try to install this project, you will get a nice ImportError from Nevow attempting to import twisted while it (Nevow) is being installed.

Now, one way that this could all be made to work is if the Nevow setup.py declared that it required Twisted. Twisted itself declares that it requires zope.interface, like this:

http://twistedmatrix.com/trac/browser/trunk/setup.py?rev=23010#L75

The Nevow (and Twisted) maintainers have an admirable policy of not applying patches to code that isn't automatically tested. Therefore in order to get this changed in Nevow's setup.py it is first necessary to have automated testing of the process of installing Nevow. That is the topic of

http://divmod.org/trac/ticket/2630 # installation of Nevow doesn't have automated tests

which I intend to help dash with as much as I can as soon as possible.

So Adam Langley's problem building Tahoe on Ubuntu turned out to be completely reproducible. Any time you run `make` in a Tahoe source tree on a system where neither Twisted nor Nevow is installed, `setuptools` will first install Twisted into a directory named `Twisted-$VERSIONNUMBER.egg` in the current working directory, because our source:setup.py told setuptools (in source:_auto_deps.py) that we need to have Twisted installed at build time. We told it that for two reasons: 1. We require twisted to run tests, which is a build-time activity, and 2. We were attempting to work-around the fact that Nevow doesn't declare its dependency on Twisted in a machine-readable way, so that setuptools doesn't know whether to install Twisted first or Nevow first, and if it picks Nevow then the Nevow installation fails at install time when it can't import twisted modules. However, it turns out that on at least Ubuntu Hardy (Adam's machine) and on my Macbook Pro (Mac OS 10.4), having Twisted listed as `setup_requires` and having Nevow being listed as `install_requires` leads to this bizarre error message which takes the name of PIL in vain: ``` File "/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py", line 27, in run_setup File "/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py", line 63, in run File "/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py", line 29, in <lambda> File "setup.py", line 8, in <module> # File "/tmp/easy_install-fqJkNC/Nevow-0.9.18/setupcommon.py", line 2, in <module> File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 10, in <module> # ImportError: No module named components ``` Here is a minimal Python project that you can use to test this yourself. Simply put the following two lines into a file named `setup.py`: ``` import setuptools setuptools.setup(name="minproj_that_needs_nevow", setup_requires=["Twisted"], install_requires=["Nevow"] ``` And then execute `python ./setup.py install`. If your platform is like Adam's and mine, then you'll get this bizarre traceback that takes the name of PIL in vain. Here is another minimal project's setup.py file: ``` import setuptools setuptools.setup(name="minproj_that_needs_nevow", install_requires=["Nevow"] ``` If you try to install this project, you will get a nice `ImportError` from Nevow attempting to import twisted while it (Nevow) is being installed. Now, one way that this could all be made to work is if the Nevow `setup.py` declared that it required Twisted. Twisted itself declares that it requires `zope.interface`, like this: <http://twistedmatrix.com/trac/browser/trunk/setup.py?rev=23010#L75> The Nevow (and Twisted) maintainers have an admirable policy of not applying patches to code that isn't automatically tested. Therefore in order to get this changed in Nevow's `setup.py` it is first necessary to have automated testing of the process of installing Nevow. That is the topic of <http://divmod.org/trac/ticket/2630> # installation of Nevow doesn't have automated tests which I intend to help dash with as much as I can as soon as possible.

zooko added the

labels 2008-06-03 00:12:00 +00:00

zooko added this to the 1.1.0 milestone 2008-06-03 00:12:00 +00:00

zooko self-assigned this 2008-06-03 00:12:00 +00:00

zooko commented

2008-06-09 23:01:12 +00:00

Okay, there is now an automated test of Nevow installation using setuptools (http://divmod.org/trac/ticket/2630 # installation of Nevow doesn't have automated tests ), so hopefully this opens the way for improvements to the setuptools build/packaging/metadata...

Okay, there is now an automated test of Nevow installation using setuptools (<http://divmod.org/trac/ticket/2630> # installation of Nevow doesn't have automated tests ), so hopefully this opens the way for improvements to the setuptools build/packaging/metadata...

cgalvan commented

2008-08-26 04:43:32 +00:00

Hey zooko, since http://bugs.python.org/setuptools/issue20 has been resolved, should this be closed as well? It seems like this was caused by that problem as well as #455.

Hey zooko, since <http://bugs.python.org/setuptools/issue20> has been resolved, should this be closed as well? It seems like this was caused by that problem as well as #455.

zooko commented

2008-10-22 01:44:18 +00:00

Yes, the fix in http://bugs.python.org/setuptools/issue20 does make this problem stop happening for Tahoe. That's because setuptools gets lucky and accidentally installs Twisted before installing Nevow. So while I still intend to get http://divmod.org/trac/ticket/2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way) fixes, we can in the meantime close this ticket as the issue is no longer negatively effecting Tahoe. In other words, the problem Tahoe had was really caused by #455 (setuptools sandbox isn't tight enough to hold Twisted's sand in so that it doesn't get in Nevow's eyes) and not by Nevow not declaring its dependency on Twisted in a machine-readable way. So I'm going to close this ticket as "Somebody Else's Problem".

Yes, the fix in <http://bugs.python.org/setuptools/issue20> does make this problem stop happening for Tahoe. That's because setuptools gets lucky and accidentally installs Twisted before installing Nevow. So while I still intend to get <http://divmod.org/trac/ticket/2629> (Nevow doesn't declare its dependency on Twisted in a machine-parseable way) fixes, we can in the meantime close this ticket as the issue is no longer negatively effecting Tahoe. In other words, the problem Tahoe had was really caused by #455 (setuptools sandbox isn't tight enough to hold Twisted's sand in so that it doesn't get in Nevow's eyes) and not by Nevow not declaring its dependency on Twisted in a machine-readable way. So I'm going to close this ticket as "Somebody Else's Problem".

zooko added the

wontfix

label 2008-10-22 01:44:18 +00:00

zooko closed this issue

2008-10-22 01:44:18 +00:00

zooko commented

2009-03-09 16:56:51 +00:00

This was wontfixed for tahoe-1.3.0.

zooko added this to the 1.3.0 milestone 2009-03-09 16:56:51 +00:00

Sign in to join this conversation.