remove the "build" step in the "edit, build, run" cycle #82

New Issue

zooko · 2007-07-10T01:01:15Z

zooko commented

2007-07-10 01:01:15 +00:00

setup.py build -i ought to work

`setup.py build -i` ought to work

zooko added the

labels 2007-07-10 01:01:15 +00:00

zooko added this to the undecided milestone 2007-07-10 01:01:15 +00:00

warner commented

2007-08-15 07:56:20 +00:00

Here's my plan:

~~copy AES/SHA256 into src/allmydata, get rid of PyCrypto~~: done
make sure there are .debs for all our dependent libraries
- this means zfec for all platforms, simplejson for the older ones that
  don't already have it. We know that Twisted, Nevow, and Foolscap already
  have .debs out there. We'll put copies of the .debs on the tahoe
  apt-repository for all platforms that don't already have them.
Stop putting anything but tahoe code in the tahoe .deb: remove zfec
and simplejson.

Once this is in place, it will be possible to install and use a tahoe .deb
that does not contain anything other than tahoe code. The debian dependencies
will be fulfilled by installing those other .debs .

make sure that PyPI knows about all our dependent libraries
- this means that 'easy_install zfec' and 'easy_install foolscap' should
  work. (this is already the case with foolscap).
- Twisted doesn't work this way, I think it is too large and in too many
  pieces to be installed with easy_install. That's ok.
change our setup.py to use setuptools, and declare a dependency on all our
dependent libraries (except Twisted).

At this point, 'python setup.py install' should handle dependencies
automatically. Our README can be changed to instruct users to make sure that
python, setuptools, and Twisted are installed, then just type 'python
setup.py install'. The 'make install' target should be a simple alias for
this. I'd like to create a small script that adds the necessary
--single-version-externally-managed and --prefix and PYTHONPATH=prefixything
to make it possible to install tahoe code to somewhere other than /usr/lib
(misc/install-to-prefix.py comes to mind), and then have the 'make install
PREFIX=x' target be an alias for that.

In addition, the 'python setup.py build_ext -i' command will work: it will
compile the crypto pieces to .so files in place. The 'make' target should be
a simple alias for this.

Next we should remove all non-tahoe source from our tree.
Then we modify the bin/allmydata-tahoe script to add "src" to PYTHONPATH
when it detects that we're running it from the source directory (at the
moment it adds instdir/lib in this case).

With that, users who have all the necessary dependent libraries installed
(perhaps because they ran 'easy_install zfec', etc) can run tahoe from source
by just typing 'make; ./bin/tahoe COMMAND'.

Next, we create a utility (misc/build-deps.py) that looks to see which of
our dependent libraries are needed ('try: import zfec; except ImportError:
needed=True'), and for each one, run easy_install.py and tell it to put
the results in ./support . The 'make build-deps' target should be an
alias for this utility.
Then we modify bin/allmydata-tahoe to add src/ to the beginning of
PYTHONPATH and support/ to the end of PYTHONPATH.

With this, users who do not have the necessary dependent libraries can still
run from source, by typing 'make; make build-deps; ./bin/tahoe COMMAND'.

For convenience, we could put a copy of the source tarballs on the tahoe
website, and change setup.py to add that as a possible download path: this
would allow a tahoe tree to be built on a system that does not already
have the dependent libraries installed even if the main
zfec/simplejson/foolscap site were down. If we put a copy inside the tahoe
source tree itself, this enables these sorts of builds without network
access.
Finally, we register tahoe on PyPI.

This last step would enable 'easy_install tahoe', again assuming that Twisted
is already installed.

Here's my plan: * ~~copy AES/SHA256 into src/allmydata, get rid of [PyCrypto](wiki/PyCrypto)~~: done * make sure there are .debs for all our dependent libraries * this means zfec for all platforms, simplejson for the older ones that don't already have it. We know that Twisted, Nevow, and Foolscap already have .debs out there. We'll put copies of the .debs on the tahoe apt-repository for all platforms that don't already have them. * Stop putting anything but tahoe code in the tahoe .deb: remove zfec and simplejson. Once this is in place, it will be possible to install and use a tahoe .deb that does not contain anything other than tahoe code. The debian dependencies will be fulfilled by installing those other .debs . * make sure that PyPI knows about all our dependent libraries * this means that 'easy_install zfec' and 'easy_install foolscap' should work. (this is already the case with foolscap). * Twisted doesn't work this way, I think it is too large and in too many pieces to be installed with easy_install. That's ok. * change our setup.py to use setuptools, and declare a dependency on all our dependent libraries (except Twisted). At this point, 'python setup.py install' should handle dependencies automatically. Our README can be changed to instruct users to make sure that python, setuptools, and Twisted are installed, then just type 'python setup.py install'. The 'make install' target should be a simple alias for this. I'd like to create a small script that adds the necessary --single-version-externally-managed and --prefix and PYTHONPATH=prefixything to make it possible to install tahoe code to somewhere other than /usr/lib (misc/install-to-prefix.py comes to mind), and then have the 'make install PREFIX=x' target be an alias for *that*. In addition, the 'python setup.py build_ext -i' command will work: it will compile the crypto pieces to .so files in place. The 'make' target should be a simple alias for this. * Next we should remove all non-tahoe source from our tree. * Then we modify the bin/allmydata-tahoe script to add "src" to PYTHONPATH when it detects that we're running it from the source directory (at the moment it adds instdir/lib in this case). With that, users who have all the necessary dependent libraries installed (perhaps because they ran 'easy_install zfec', etc) can run tahoe from source by just typing 'make; ./bin/tahoe COMMAND'. * Next, we create a utility (misc/build-deps.py) that looks to see which of our dependent libraries are needed ('try: import zfec; except [ImportError](wiki/ImportError): needed=True'), and for each one, run easy_install.py and tell it to put the results in ./support . The 'make build-deps' target should be an alias for this utility. * Then we modify bin/allmydata-tahoe to add src/ to the beginning of PYTHONPATH and support/ to the end of PYTHONPATH. With this, users who do not have the necessary dependent libraries can still run from source, by typing 'make; make build-deps; ./bin/tahoe COMMAND'. * For convenience, we could put a copy of the source tarballs on the tahoe website, and change setup.py to add that as a possible download path: this would allow a tahoe tree to be built on a system that does not already have the dependent libraries installed even if the main zfec/simplejson/foolscap site were down. If we put a copy inside the tahoe source tree itself, this enables these sorts of builds without network access. * Finally, we register tahoe on PyPI. This last step would enable 'easy_install tahoe', again assuming that Twisted is already installed.

zooko commented

2007-08-20 18:09:07 +00:00

Excellent plan, Brian!

I'm making this ticket the successor to #15. See also the Packaging page.

This is the biggest component of the "packaging/build system" task which I hope will go into the v0.6 release.

Excellent plan, Brian! I'm making this ticket the successor to #15. See also the [Packaging](wiki/Packaging) page. This is the biggest component of the "packaging/build system" task which I hope will go into the v0.6 release.

zooko modified the milestone from undecided to 0.6.0

2007-08-20 18:09:07 +00:00

warner commented

2007-08-20 18:44:45 +00:00

For reference, http://wiki.debian.org/DebianPythonFAQ suggests that
--single-version-externally-managed is the Right Way to build debian packages
out of setuptool-using python projects, and that the process is actually
fairly easy.

For reference, <http://wiki.debian.org/DebianPythonFAQ> suggests that --single-version-externally-managed is the Right Way to build debian packages out of setuptool-using python projects, and that the process is actually fairly easy.

warner commented

2007-08-21 20:47:46 +00:00

progress thus far:

we have i386 zfec .debs for sid, edgy, and feisty. These packages are good
enough to do 'import zfec', which is all we need for tahoe. They do
not support the zfec/zunfec command-line tools (they do not even
include them). Zandr is setting up an etch buildslave now (#24); once
that's in place we can make a .deb for etch too.
simplejson .debs already exist in feisty, edgy, etch, and sid.
we have foolscap-0.1.4 .debs for sid, edgy, and feisty. This is enough for
tahoe-0.5.0 . (foolscap-0.1.5 is the current release, but there are bugs
in its debian packaging, and tahoe-0.5.0 doesn't need any of the new
features, and foolscap-0.1.4 is wire-compatible with foolscap-0.1.5)

The next step will be to change the tahoe .deb generation to declare
dependencies upon python-zfec, python-simplejson, and python-foolscap, and
stop including zfec in the tahoe .deb .

progress thus far: * we have i386 zfec .debs for sid, edgy, and feisty. These packages are good enough to do 'import zfec', which is all we need for tahoe. They do **not** support the zfec/zunfec command-line tools (they do not even include them). Zandr is setting up an etch buildslave now (#24); once that's in place we can make a .deb for etch too. * simplejson .debs already exist in feisty, edgy, etch, and sid. * we have foolscap-0.1.4 .debs for sid, edgy, and feisty. This is enough for tahoe-0.5.0 . (foolscap-0.1.5 is the current release, but there are bugs in its debian packaging, and tahoe-0.5.0 doesn't need any of the new features, and foolscap-0.1.4 is wire-compatible with foolscap-0.1.5) The next step will be to change the tahoe .deb generation to declare dependencies upon python-zfec, python-simplejson, and python-foolscap, and stop including zfec in the tahoe .deb .

warner commented

2007-08-21 23:37:38 +00:00

I've modified .deb generation. 'apt-get install allmydata-tahoe' now works, pulling in foolscap/zfec/simplejson as necessary.

The next step requires that 'easy_install zfec' works, which will require zfec-1.0.2 to be published (to fix the MANIFEST problems that caused ez_setup.py and src/fec.h to be left out of the source tarball, as well as to remove the setup.py-declared dependency on "pyutil").

Assigning to zooko for zfec-1.0.2, please assign back to me when the release is complete.

I've modified .deb generation. 'apt-get install allmydata-tahoe' now works, pulling in foolscap/zfec/simplejson as necessary. The next step requires that 'easy_install zfec' works, which will require zfec-1.0.2 to be published (to fix the MANIFEST problems that caused ez_setup.py and src/fec.h to be left out of the source tarball, as well as to remove the setup.py-declared dependency on "pyutil"). Assigning to zooko for zfec-1.0.2, please assign back to me when the release is complete.

warner removed their assignment 2007-08-21 23:37:38 +00:00

zooko was assigned by warner

2007-08-21 23:37:38 +00:00

zooko commented

2007-09-12 20:21:16 +00:00

Okay, zfec-1.0.3 is easy_installable.

zooko removed their assignment 2007-09-12 20:21:16 +00:00

warner was assigned by zooko

2007-09-12 20:21:16 +00:00

warner commented

2007-09-14 05:15:45 +00:00

we just implemented most of the rest of this today.

The pieces that remain:

register the next release of tahoe with the cheeseshop
add 'nevow' to the list of dependencies in setup.py
identify the platforms on which setuptools is too old, preventing ez_setup from working
- convince the author of simplejson to use our ez_setup patch to accomodate old setuptools
- add back the simplejson source tarball to misc/dependencies/ , to enable desert-island
  building
find a way to not build tahoe in the 'make build-deps' step, rather just build the dependencies
- (this doesn't currently hurt anything, since we use the version from src/ before any
  others, but it's slow and confusing to look at)

we just implemented most of the rest of this today. The pieces that remain: * register the next release of tahoe with the cheeseshop * add 'nevow' to the list of dependencies in setup.py * identify the platforms on which setuptools is too old, preventing ez_setup from working * convince the author of simplejson to use our ez_setup patch to accomodate old setuptools * add back the simplejson source tarball to misc/dependencies/ , to enable desert-island building * find a way to not build tahoe in the 'make build-deps' step, rather just build the dependencies * (this doesn't currently hurt anything, since we use the version from src/ before any others, but it's slow and confusing to look at)

warner commented

2007-09-16 09:22:46 +00:00

added nevow
- there was a problem on dapper, where the nevow that is available in the distribution
  is old (0.6.0) but works, but has no .egg-info, so our build-deps target didn't see it,
  and tried to build the new one (0.9.18), but that requires twisted-2.4.0 or newer, and
  the platform has twisted-2.2.0 . I hacked it to remove nevow from the dependencies if it
  sees nevow-0.6.0 available to import, which at least makes dapper work.
stopped building tahoe in the 'build-deps' step, thanks to zooko's crazy/brilliant suggestion

of using a dummy package named "tahoe-deps"

Now that the tahoe-specific things are done, I'm going to close out this ticket. I'll add a new one for registering tahoe with the cheeseshop.

* added nevow * there was a problem on dapper, where the nevow that is available in the distribution is old (0.6.0) but works, but has no .egg-info, so our build-deps target didn't see it, and tried to build the new one (0.9.18), but that requires twisted-2.4.0 or newer, and the platform has twisted-2.2.0 . I hacked it to remove nevow from the dependencies if it sees nevow-0.6.0 available to import, which at least makes dapper work. * stopped building tahoe in the 'build-deps' step, thanks to zooko's crazy/brilliant suggestion > of using a dummy package named "tahoe-deps" Now that the tahoe-specific things are done, I'm going to close out this ticket. I'll add a new one for registering tahoe with the cheeseshop.

warner added the

fixed

label 2007-09-16 09:22:46 +00:00

warner closed this issue

2007-09-16 09:22:46 +00:00

Sign in to join this conversation.