make setup.py more easily patchable by OS packagers #1168

New Issue

tahoe-lafs · 2010-08-11T01:38:13Z

davidsarah commented

2010-08-11 01:38:13 +00:00

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs or by deciding to use a different version of the dependency than would be provided by an OS package.

The attached diff adds an external_dependency_management switch to setup.py to cause this change in behaviour. It doesn't remove the need for a patch, but makes such patches much more likely to apply cleanly.

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs or by deciding to use a different version of the dependency than would be provided by an OS package. The attached diff adds an `external_dependency_management` switch to `setup.py` to cause this change in behaviour. It doesn't remove the need for a patch, but makes such patches much more likely to apply cleanly.

tahoe-lafs added the

labels 2010-08-11 01:38:13 +00:00

tahoe-lafs added this to the undecided milestone 2010-08-11 01:38:13 +00:00

davidsarah commented

2010-08-11 01:39:45 +00:00

Attachment setup.py.diff (8581 bytes) added

add 'external_dependency_management' flag to setup.py (context diff)

**Attachment** setup.py.diff (8581 bytes) added add 'external_dependency_management' flag to setup.py (context diff)

setup.py.diff

8.4 KiB

zooko commented

2010-08-11 06:36:25 +00:00

For install requirements you can turn off dependency management by passing --single-version-externally-managed or --root=targetdir options to python setup.py install. For build-time dependencies, they are bundled with the Tahoe-LAFS source tree so you probably don't need to disable the dependency on them.

For install requirements you can turn off dependency management by passing `--single-version-externally-managed` or `--root=targetdir` options to `python setup.py install`. For build-time dependencies, they are bundled with the Tahoe-LAFS source tree so you probably don't need to disable the dependency on them.

zooko commented

2010-08-11 06:38:07 +00:00

Replying to davidsarah:

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

or by deciding to use a different version of the dependency than would be provided by an OS package.

Does this mean the packager wants to supply a version of a dependent library older than the minimum version that we specify in source:_auto_deps.py? If so, I would like to know about this!

Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/6230): > Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present. > or by deciding to use a different version of the dependency than would be provided by an OS package. Does this mean the packager wants to supply a version of a dependent library older than the minimum version that we specify in source:_auto_deps.py? If so, I would like to know about this!

zooko commented

2010-08-11 07:10:13 +00:00

I chatted with rockstar on IRC about his packaging of Tahoe-LAFS for Ubuntu. He said that python setup.py test failed due to it trying to execute darcsver and darcsver not being found, which he solved by removing the aliases section of source:setup.cfg which makes the darcsver command be invoked when running the test command.

I don't understand how darcsver could not been found and caused the test to fail, but perhaps I don't understand something about the build environment (which is managed by debhelper). Here is an example of python setup.py test working on an Ubuntu buildslave: http://tahoe-lafs.org/buildbot/builders/Shawn%20jaunty%20amd64/builds/532/steps/test/logs/stdio .

I think the next step is to set up automated execution of a build as managed by debhelper on a buildslave.

Also I would very much appreciate it if someone who knows how to use debhelper would give me a paste of what the error message is when running python setup.py test.

I chatted with rockstar on IRC about his packaging of Tahoe-LAFS for Ubuntu. He said that `python setup.py test` failed due to it trying to execute darcsver and darcsver not being found, which he solved by removing the `aliases` section of source:setup.cfg which makes the darcsver command be invoked when running the test command. I don't understand how darcsver could not been found and caused the test to fail, but perhaps I don't understand something about the build environment (which is managed by `debhelper`). Here is an example of `python setup.py test` working on an Ubuntu buildslave: <http://tahoe-lafs.org/buildbot/builders/Shawn%20jaunty%20amd64/builds/532/steps/test/logs/stdio> . I think the next step is to set up automated execution of a build as managed by `debhelper` on a buildslave. Also I would very much appreciate it if someone who knows how to use `debhelper` would give me a paste of what the error message is when running `python setup.py test`.

zooko commented

2010-08-11 21:36:47 +00:00

(http://tahoe-lafs.org/pipermail/tahoe-dev/2010-August/004964.html)

davidsarah commented

2010-08-12 01:08:39 +00:00

Replying to [zooko]comment:3:

Replying to davidsarah:

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

I've seen it attempt to do so (for foolscap-0.5.1, for example). It's possible that the dependencies were not present in a way recognized by setuptools, but that seems to be a situation that is very easy to end up in. Also see #717 (which is about rebuilding rather than downloading, but that's still a problem).

or by deciding to use a different version of the dependency than would be provided by an OS package.

Does this mean the packager wants to supply a version of a dependent library older than the minimum version that we specify in source:_auto_deps.py? If so, I would like to know about this!

setuptools might want to download, build, or install a version that is newer than the one supplied by the OS package manager, even though the latter satisfies our declared requirement. (I'm skeptical that http://bugs.python.org/setuptools/issue17 and http://bugs.python.org/setuptools/issue20 have actually been fixed, or if they have, then there must be other bugs causing this problem, because I've definitely seen it happen.)

Replying to [zooko]comment:3: > Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/6230): > > Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs > > Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present. I've seen it attempt to do so (for foolscap-0.5.1, for example). It's possible that the dependencies were not present in a way recognized by setuptools, but that seems to be a situation that is very easy to end up in. Also see #717 (which is about rebuilding rather than downloading, but that's still a problem). > > or by deciding to use a different version of the dependency than would be provided by an OS package. > > Does this mean the packager wants to supply a version of a dependent library older than the minimum version that we specify in source:_auto_deps.py? If so, I would like to know about this! setuptools might want to download, build, or install a version that is newer than the one supplied by the OS package manager, even though the latter satisfies our declared requirement. (I'm skeptical that <http://bugs.python.org/setuptools/issue17> and <http://bugs.python.org/setuptools/issue20> have actually been fixed, or if they have, then there must be other bugs causing this problem, because I've definitely seen it happen.)

davidsarah commented

2010-08-12 01:13:35 +00:00

Replying to [davidsarah]comment:6:

Replying to [zooko]comment:3:

Replying to davidsarah:

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

I've seen it attempt to do so (for foolscap-0.5.1, for example).

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

Replying to [davidsarah]comment:6: > Replying to [zooko]comment:3: > > Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/6230): > > > Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that source:setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs > > > > Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present. > > I've seen it attempt to do so (for foolscap-0.5.1, for example). I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running `python setup.py trial`. foolscap 0.5.1 is downloaded again on every run of trial, even though `python setup.py build` copied it into `support/site-packages` and the build completed successfully.

davidsarah commented

2010-08-12 01:23:00 +00:00

Replying to [davidsarah]comment:7:

Replying to [davidsarah]comment:6:

Replying to [zooko]comment:3:

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

I've seen it attempt to do so (for foolscap-0.5.1, for example).

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

This may be the same problem as #657 (modulo trial vs test, but trial should not be doing any more than test, I think).

Replying to [davidsarah]comment:7: > Replying to [davidsarah]comment:6: > > Replying to [zooko]comment:3: > > > > > Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present. > > > > I've seen it attempt to do so (for foolscap-0.5.1, for example). > > I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running `python setup.py trial`. foolscap 0.5.1 is downloaded again on every run of trial, even though `python setup.py build` copied it into `support/site-packages` and the build completed successfully. This may be the same problem as #657 (modulo `trial` vs `test`, but `trial` should not be doing any more than `test`, I think).

zooko commented

2010-09-18 17:38:05 +00:00

Replying to [davidsarah]comment:7:

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

Okay let's open a ticket and write a deterministic unit test which shows this misbehavior.

Replying to [davidsarah]comment:7: > I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running `python setup.py trial`. foolscap 0.5.1 is downloaded again on every run of trial, even though `python setup.py build` copied it into `support/site-packages` and the build completed successfully. Okay let's open a ticket and write a deterministic unit test which shows this misbehavior.

zooko commented

2010-09-18 18:09:23 +00:00

Looks like it might be #717. Isn't the buildbot already testing this by running the "Desert Island build"? e.g: http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:8)?

[Makefile]source:trunk/Makefile@4650#L266

Looks like it might be #717. Isn't the buildbot already testing this by running the "Desert Island build"? e.g: <http://tahoe-lafs.org/buildbot/builders/clean/builds/2522> In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:8)? [Makefile]source:trunk/Makefile@4650#L266

zooko commented

2010-09-18 18:10:41 +00:00

Moving the discussion about re-downloading foolscap to #717.

zooko commented

2010-11-30 15:18:28 +00:00

I think this is a duplicate of #1220 (build/install should be able to refrain from getting dependencies). Please re-open this ticket if I'm missing something.

zooko added the

duplicate

label 2010-11-30 15:18:28 +00:00

zooko closed this issue

2010-11-30 15:18:28 +00:00

davidsarah commented

2011-07-31 21:08:13 +00:00

Replying to zooko:

I think this is a duplicate of #1220 (build/install should be able to refrain from getting dependencies). Please re-open this ticket if I'm missing something.

This ticket has a patch specifically to reduce the need for patching setup.py, and that patch got forgotten when it was closed. My interpretation of #1220 is that it's about longer-term fixes.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1168#issuecomment-79374): > I think this is a duplicate of #1220 (build/install should be able to refrain from getting dependencies). Please re-open this ticket if I'm missing something. This ticket has a patch specifically to reduce the need for patching `setup.py`, and that patch got forgotten when it was closed. My interpretation of #1220 is that it's about longer-term fixes.

tahoe-lafs removed the

duplicate

label 2011-07-31 21:08:13 +00:00

davidsarah reopened this issue

2011-07-31 21:08:13 +00:00

tahoe-lafs added

normal

and removed

major

labels 2012-04-01 05:13:38 +00:00

warner commented

2016-03-27 00:08:03 +00:00

I think our new setup.py now meets this requirement. At least, it behaves like all other setup.pys out there. The lack of setup_requires= is a big one, and using tox for tests means we no longer have custom setup.py test steps (with dependencies).

Closing this one out, but feel free to re-open if I've misinterpreted it.

I think our new setup.py now meets this requirement. At least, it behaves like all other setup.pys out there. The lack of `setup_requires=` is a big one, and using `tox` for tests means we no longer have custom `setup.py test` steps (with dependencies). Closing this one out, but feel free to re-open if I've misinterpreted it.

warner added the

fixed

label 2016-03-27 00:08:03 +00:00

warner modified the milestone from undecided to 1.11.0

2016-03-27 00:08:03 +00:00

warner closed this issue

2016-03-27 00:08:03 +00:00

Sign in to join this conversation.