hard to run against alternate dependencies, e.g. trunk version of Foolscap #709

New Issue

warner · 2009-05-18T21:23:23Z

warner commented

2009-05-18 21:23:23 +00:00

For the #653 work I'm doing now (which requires a change to foolscap:
foolscap#105), I need to run a
Tahoe trunk tree against a Foolscap trunk tree (to test changes I'm
considering putting into the upcoming foolscap release). The normal way to do
this (which used to work before we switched to setuptools) was:

PYTHONPATH=~/foolscap/trunk make test

I tried to do this yesterday, and here are the problems I ran into:

the Foolscap .egg that was built in support/lib overrides anything else on
PYTHONPATH, so I must delete the .egg first
on OS-X, there's also an .egg put in the root of the tahoe tree (the #657
problem), so I must delete that one too
"make test" takes forever (#591) and rebuilds a Foolscap .egg before
running any tests, so I use "make quicktest" instead
A Foolscap source tree (in my ~/foolscap/trunk directory) does not
appear to qualify as fulfilling the setuptools dependency for
foolscapsecure_connections>=0.3.1.
src/allmydata/scripts/runner.py has a line which does
pkg_resources.require('allmydata-tahoe'), and that throws a
DistributionNotFound exception. I commented that line out.
the pkg_resources.load_entry_point-based support/bin/tahoe script
does an implicit pkg_resources.require, also failing because it
thinks it doesn't have Foolscap available.

I think the only way I can work around this is to use "setup.py develop
TARGETDIR" on my Foolscap tree, and then set PYTHONPATH to point at TARGETDIR
instead of ~/foolscap/trunk.

This is annoying. I really wish that PYTHONPATH could be used as it was
originally intended, and that these eggs weren't getting in the way.

For the #653 work I'm doing now (which requires a change to foolscap: [foolscap#105](http://foolscap.lothar.com/trac/ticket/105)), I need to run a Tahoe trunk tree against a Foolscap trunk tree (to test changes I'm considering putting into the upcoming foolscap release). The normal way to do this (which used to work before we switched to setuptools) was: ``` PYTHONPATH=~/foolscap/trunk make test ``` I tried to do this yesterday, and here are the problems I ran into: * the Foolscap .egg that was built in support/lib overrides anything else on PYTHONPATH, so I must delete the .egg first * on OS-X, there's also an .egg put in the root of the tahoe tree (the #657 problem), so I must delete that one too * "make test" takes forever (#591) and rebuilds a Foolscap .egg before running any tests, so I use "make quicktest" instead * A Foolscap source tree (in my `~/foolscap/trunk` directory) does not appear to qualify as fulfilling the setuptools dependency for `foolscapsecure_connections>=0.3.1`. `src/allmydata/scripts/runner.py` has a line which does `pkg_resources.require('allmydata-tahoe')`, and that throws a DistributionNotFound exception. I commented that line out. * the `pkg_resources.load_entry_point`-based support/bin/tahoe script does an implicit `pkg_resources.require`, also failing because it thinks it doesn't have Foolscap available. I think the only way I can work around this is to use "setup.py develop TARGETDIR" on my Foolscap tree, and then set PYTHONPATH to point at TARGETDIR instead of `~/foolscap/trunk`. This is annoying. I really wish that PYTHONPATH could be used as it was originally intended, and that these eggs weren't getting in the way.

warner added the

labels 2009-05-18 21:23:23 +00:00

warner added this to the undecided milestone 2009-05-18 21:23:23 +00:00

warner commented

2009-05-18 21:24:10 +00:00

fix formatting of Foolscap ticket link

zooko commented

2009-05-18 21:38:24 +00:00

The first two problems in your list of five problems are http://bugs.python.org/setuptools/issue53 (respect the PYTHONPATH). There have been two different patches posted to that ticket (the first was by me), and I don't currently understand the objection of the setuptools maintainer (PJE) to the most recent patch. Perhaps you could help by reading that ticket and figuring out what needs to change to get PJE to accept Mark Sienkiwicz's patch.

Alternately, we could continue our toothpick of setuptools (which I call zetuptoolz) by applying my patch or Mark Sienkiwicz's patch to zetuptoolz.

For what it is worth, this is going to be fixed one way or another in the long run, as either distutils core or various setuptools competitors move ahead and we will be able to switch to that in order to gain "respect the PYTHONPATH" without losing other functionality that we currently use.

Now, as for workarounds, if you build a .egg-info file, I think it would eliminate the last 3 of your 5 complaints. I'm not sure if that will do it. .egg-info files are created by default by modern Python distutils when building.

The first two problems in your list of five problems are <http://bugs.python.org/setuptools/issue53> (respect the PYTHONPATH). There have been two different patches posted to that ticket (the first was by me), and I don't currently understand the objection of the setuptools maintainer (PJE) to the most recent patch. Perhaps you could help by reading that ticket and figuring out what needs to change to get PJE to accept Mark Sienkiwicz's patch. Alternately, we could continue our toothpick of setuptools (which I call `zetuptoolz`) by applying my patch or Mark Sienkiwicz's patch to `zetuptoolz`. For what it is worth, this is going to be fixed one way or another in the long run, as either distutils core or various setuptools competitors move ahead and we will be able to switch to that in order to gain "respect the PYTHONPATH" without losing other functionality that we currently use. Now, as for workarounds, if you build a `.egg-info` file, I *think* it would eliminate the last 3 of your 5 complaints. I'm not sure if that will do it. `.egg-info` files are created by default by modern Python distutils when building.

warner commented

2009-05-18 22:35:23 +00:00

Yup, by doing python setup.py build in the Foolscap tree, it created a foolscap.egg-info directory there, after which PYTHONPATH=~/foolscap/trunk make quicktest started working again (without commenting out the pkg_resources.require calls).

Also, PYTHONPATH=~/tahoe/trunk/support/lib/python2.5/site-packages python setup.py develop --prefix ~/tahoe/trunk/support put a foolscap.egg-link file in the tahoe lib directory, from which tahoe was able to find the foolscap code.

(I'm not used to doing python setup.py build with foolscap, because Foolscap doesn't need it for anything).

This works well enough for now. I look forward to seeing the "respect PYTHONPATH for .eggs" fix get into whatever packaging system we use. I guess I'll leave this ticket open until that happens. Thanks!

Yup, by doing `python setup.py build` in the Foolscap tree, it created a `foolscap.egg-info` directory there, after which `PYTHONPATH=~/foolscap/trunk make quicktest` started working again (without commenting out the `pkg_resources.require` calls). Also, `PYTHONPATH=~/tahoe/trunk/support/lib/python2.5/site-packages python setup.py develop --prefix ~/tahoe/trunk/support` put a `foolscap.egg-link` file in the tahoe lib directory, from which tahoe was able to find the foolscap code. (I'm not used to doing `python setup.py build` with foolscap, because Foolscap doesn't need it for anything). This works well enough for now. I look forward to seeing the "respect PYTHONPATH for .eggs" fix get into whatever packaging system we use. I guess I'll leave this ticket open until that happens. Thanks!

warner commented

2009-05-20 01:20:13 +00:00

I was mistaken. python setup.py build does not create a foolscap.egg-info directory. You must do a setup.py egg_info to get it to create the .egg-info . The setup.py develop merely creates it as a side-effect.

I was mistaken. `python setup.py build` does not create a `foolscap.egg-info` directory. You must do a `setup.py egg_info` to get it to create the .egg-info . The `setup.py develop` merely creates it as a side-effect.

cgalvan commented

2009-09-24 19:58:58 +00:00

Note that once #668 is resolved, switching to Distribute, this issue may or may not be resolved as a result.

tahoe-lafs changed title from ~~hard to run against alternate dependencies, i.e. trunk version of Foolscap~~ to hard to run against alternate dependencies, e.g. trunk version of Foolscap

2010-04-04 16:51:21 +00:00

zooko commented

2010-04-04 17:08:06 +00:00

I opened a ticket with the Python Distribute project: http://bitbucket.org/tarek/distribute/issue/136/respect-the-pythonpath

I opened a ticket with the Python Distribute project: <http://bitbucket.org/tarek/distribute/issue/136/respect-the-pythonpath>

zooko commented

2010-11-30 15:27:00 +00:00

zetuptoolz has a patch which is intended to fix this: http://tahoe-lafs.org/trac/zetuptoolz/changeset/20081115185932-92b7f-d451b10759897b0fa5d038f2c42cdcd3901289a8/trunk

If you wanted to try it out by installing zetuptoolz instead of your normal setuptools-or-distribute and try PYTHONPATH=~/foolscap/trunk make test, I would appreciate it.

zetuptoolz has a patch which is intended to fix this: <http://tahoe-lafs.org/trac/zetuptoolz/changeset/20081115185932-92b7f-d451b10759897b0fa5d038f2c42cdcd3901289a8/trunk> If you wanted to try it out by installing zetuptoolz instead of your normal setuptools-or-distribute and try `PYTHONPATH=~/foolscap/trunk make test`, I would appreciate it.

zooko commented

2010-11-30 17:26:46 +00:00

Oh, it isn't just installing zetuptoolz -- this code gets written into your easy-install.pth. So to test this, read the patch and make the appropriate adjustment to the code in your easy-install.pth, and then see if PYTHONPATH=~/foolscap/trunk make test works. If so, then maybe we can extend zetuptoolz so that it hunts down all easy-install.pth's that it has write access to and fixes them. ;-)

Here is the code in question: zetuptoolz's easy_install.py

Oh, it isn't just installing zetuptoolz -- this code gets written into your easy-install.pth. So to test this, read the patch and make the appropriate adjustment to the code in your easy-install.pth, and then see if `PYTHONPATH=~/foolscap/trunk make test` works. If so, then maybe we can extend zetuptoolz so that it hunts down all easy-install.pth's that it has write access to and fixes them. ;-) Here is the code in question: [zetuptoolz's easy_install.py](http://tahoe-lafs.org/trac/zetuptoolz/browser/trunk/setuptools/command/easy_install.py?annotate=blame&rev=592#L1371)

zooko commented

2010-11-30 17:27:09 +00:00

warner: are you willing to try this?

zooko commented

2011-08-04 16:28:41 +00:00

I've opened Launchpad bug #821000 to track the way this affects setuptools, distribute, Ubuntu, and Tahoe-LAFS (via our local fork of setuptools named "zetuptoolz").

I've opened [Launchpad bug #821000](https://bugs.launchpad.net/tahoe-lafs/+bug/821000) to track the way this affects setuptools, distribute, Ubuntu, and Tahoe-LAFS (via our local fork of setuptools named "zetuptoolz").

warner commented

2016-03-26 21:20:54 +00:00

This is now resolved: by using a virtualenv, I can pip install --editable ~/my/foolscap to use a locally-modified Foolscap along with my locally-modified Tahoe. It's not $PYTHONPATH manipulation, but it works just as well.

This is now resolved: by using a virtualenv, I can `pip install --editable ~/my/foolscap` to use a locally-modified Foolscap along with my locally-modified Tahoe. It's not $PYTHONPATH manipulation, but it works just as well.

warner added the

fixed

label 2016-03-26 21:20:54 +00:00

warner modified the milestone from undecided to 1.11.0

2016-03-26 21:20:54 +00:00

warner closed this issue

2016-03-26 21:20:54 +00:00

Sign in to join this conversation.