automated tests of debian compatibility: run-from-source and run-from-.deb-package #630

New Issue

warner · 2009-02-18T02:11:28Z

warner commented

2009-02-18 02:11:28 +00:00

We should have a wiki page (perhaps InstallDetails) with a table. One one
axis we'll have a list of debian/ubuntu releases: gutsy, intrepid, etch, sid.
The other axis will have two values: "run-from-source" and "run-from-.deb".
In each cell we'll either say "not compatible" (maybe with some hints for
someone who wants to do the work to fix that), or "yes compatible" and a link
to the buildbot that proves it.

The "run-from-source" form of compatibility means that you can download a
tahoe tarball (or grab a darcs checkout) and run "python setup.py build" and
then run "./bin/tahoe --version" successfully. This kind of build is allowed
to download a variety of dependent libraries (or use ones from the tahoe-deps
tarball). It has some minimal set of dependencies that cannot be installed
via easy_install, like python itself, and either some glue libraries
(python-openssl) or the binary and header packages for the C libraries that
they link against (like openssl-dev).

The "run-from-package" form of compatibility means that you can download a
.deb package from the APT repository on allmydata.org and then run
"/usr/bin/tahoe --version" successfully. It also means that you can create
.deb packages from a source tree with the "make deb-$ARCH-head" target
(perhaps after installing a few extra packages, like build-essential and
debhelper). This form is more strict, because it requires that tahoe work
with the library versions that are present in the host environment, rather
than replacing them with newer versions via easy_install. For example, Ubuntu
"gutsy" includes simplejson-1.7.1, but the current version is 2.0.8 . If
Tahoe required 2.0.x, then it would work in the "run-from-source" mode on
gutsy, but not in the "run-from-package" mode.

To confirm run-from-package compatibility, we'd like to have a buildslave
which tests installation and execution of a tahoe debian package, like so:

generate a Tahoe .deb package
use 'debootstrap' or 'dchroot' or 'schroot' to create a new chroot jail
with a basic debian system installed. (this should probably cache the
necessary packages, to run faster and reduce network traffic, since it
will need the entire basic system on each build)
modify the chroot environment to point /etc/apt/sources.list at a local
web server, put the .deb in there, tell the chroot to 'apt-get install
allmydata-tahoe'
run 'tahoe --version' inside the chroot and make sure it succeeds
upload the .deb to the allmydata.org APT repository

We should have a wiki page (perhaps [InstallDetails](wiki/InstallDetails)) with a table. One one axis we'll have a list of debian/ubuntu releases: gutsy, intrepid, etch, sid. The other axis will have two values: "run-from-source" and "run-from-.deb". In each cell we'll either say "not compatible" (maybe with some hints for someone who wants to do the work to fix that), or "yes compatible" and a link to the buildbot that proves it. The "run-from-source" form of compatibility means that you can download a tahoe tarball (or grab a darcs checkout) and run "python setup.py build" and then run "./bin/tahoe --version" successfully. This kind of build is allowed to download a variety of dependent libraries (or use ones from the tahoe-deps tarball). It has some minimal set of dependencies that cannot be installed via easy_install, like python itself, and either some glue libraries (python-openssl) or the binary and header packages for the C libraries that they link against (like openssl-dev). The "run-from-package" form of compatibility means that you can download a .deb package from the APT repository on allmydata.org and then run "/usr/bin/tahoe --version" successfully. It also means that you can create .deb packages from a source tree with the "make deb-$ARCH-head" target (perhaps after installing a few extra packages, like build-essential and debhelper). This form is more strict, because it requires that tahoe work with the library versions that are present in the host environment, rather than replacing them with newer versions via easy_install. For example, Ubuntu "gutsy" includes simplejson-1.7.1, but the current version is 2.0.8 . If Tahoe required 2.0.x, then it would work in the "run-from-source" mode on gutsy, but not in the "run-from-package" mode. To confirm run-from-package compatibility, we'd like to have a buildslave which tests installation and execution of a tahoe debian package, like so: 1. generate a Tahoe .deb package 2. use 'debootstrap' or 'dchroot' or 'schroot' to create a new chroot jail with a basic debian system installed. (this should probably cache the necessary packages, to run faster and reduce network traffic, since it will need the entire basic system on each build) 3. modify the chroot environment to point /etc/apt/sources.list at a local web server, put the .deb in there, tell the chroot to 'apt-get install allmydata-tahoe' 4. run 'tahoe --version' inside the chroot and make sure it succeeds 5. upload the .deb to the allmydata.org APT repository

warner added the

labels 2009-02-18 02:11:28 +00:00

warner added this to the undecided milestone 2009-02-18 02:11:28 +00:00

zooko commented

2009-02-18 02:27:12 +00:00

Thank you for writing this up! This is a good plan.

Now, as a matter of policy, which Debian/Ubuntu versions should we target to meet the second kind of "supported" -- the apt-get-centric kind? I suggest Ubuntu Hardy, Ubuntu Intrepid, and Debian Lenny. Zandr the allmydata.com Ops Guy says that he is happy requiring Ubuntu Hardy on things instead of having tahoe maintain apt-get-style compatibility with Ubuntu Gutsy, and also the Tahoe-1.3.0 release doesn't have the apt-get-centric kind of compatibility with Gutsy, since the Nevow installation on Gutsy doesn't have a .egg-info file, and Tahoe-1.3.0 generates a bin/tahoe executable using setuptools. (This may not have been entirely intended -- I didn't mean to make the bin/tahoe executable which worked on Gutsy in Tahoe-1.2.0 stop working in Tahoe-1.3.0. Once we have the automated testing described in this ticket then such unexpected changes will be less common.)

Thank you for writing this up! This is a good plan. Now, as a matter of policy, which Debian/Ubuntu versions should we target to meet the second kind of "supported" -- the apt-get-centric kind? I suggest Ubuntu Hardy, Ubuntu Intrepid, and Debian Lenny. Zandr the allmydata.com Ops Guy says that he is happy requiring Ubuntu Hardy on things instead of having tahoe maintain apt-get-style compatibility with Ubuntu Gutsy, and also the Tahoe-1.3.0 release doesn't have the apt-get-centric kind of compatibility with Gutsy, since the Nevow installation on Gutsy doesn't have a .egg-info file, and Tahoe-1.3.0 generates a bin/tahoe executable using setuptools. (This may not have been entirely intended -- I didn't mean to make the bin/tahoe executable which worked on Gutsy in Tahoe-1.2.0 stop working in Tahoe-1.3.0. Once we have the automated testing described in this ticket then such unexpected changes will be less common.)

warner commented

2009-02-18 02:47:07 +00:00

Yeah, hardy/intrepid/lenny works for me. I use sid myself, but if it's compatible with lenny it should be compatible with sid. (we could test this explicitly, but it would be slightly more expensive network-wise, since sid is by definition a moving target). As long as Zandr is ok with giving up on gutsy, I'm ok with it.

warner commented

2009-02-18 03:15:46 +00:00

I've updated InstallDetails and DownloadDebianPackages with the beginnings of the compatibility table.

I've updated [InstallDetails](wiki/InstallDetails) and [DownloadDebianPackages](wiki/DownloadDebianPackages) with the beginnings of the compatibility table.

zandr commented

2009-02-18 03:34:54 +00:00

For Allmydata Ops, it's just Hardy that matters.

Having said that, I'd like various Intrepid machines to work as well, and I guess now that stable=Lenny, I suppose we should support that too.

For Allmydata Ops, it's just Hardy that matters. Having said that, I'd like various Intrepid machines to work as well, and I guess now that stable=Lenny, I suppose we should support that too.

warner commented

2009-02-18 05:57:28 +00:00

some notes: sudo debootstrap gutsy ./gutsy-chroot <http://us.archive.ubuntu.com/ubuntu> creates a 218MB gutsy chroot environment in ./gutsy-chroot, takes about 4 minutes from our colo. There are some more options to include extra packages in the base install. I think the next step is to start an sshd inside the chroot, then use schroot to run additional commands in there (starting with adding more directories to /etc/apt/sources.list, then running apt-get update and install).

some notes: `sudo debootstrap gutsy ./gutsy-chroot <http://us.archive.ubuntu.com/ubuntu>` creates a 218MB gutsy chroot environment in ./gutsy-chroot, takes about 4 minutes from our colo. There are some more options to include extra packages in the base install. I think the next step is to start an sshd inside the chroot, then use schroot to run additional commands in there (starting with adding more directories to /etc/apt/sources.list, then running apt-get update and install).

warner commented

2009-02-18 06:28:19 +00:00

ooh, it looks like 'schroot' is what I want: it can be configured to create a chroot environment from a tarball (i.e. a pre-generated gutsy base system) and then run some number of commands inside it. It also looks like, once configured, it can be invoked by (controlled) non-root users, which would be ideal for the buildslave.

warner commented

2009-02-18 17:32:54 +00:00

Ok, here's my plan:

install LVM and schroot on the host
create ~buildslave/public_html/EXPORT
write a script that takes a .deb file and replaces EXPORT with a
functional single-package APT repository
create a baseline LVM volume
use debootstrap to install a basic e.g. hardy system
create a user account with sudo-powers to run apt-get
configure /etc/apt/sources.list to point at both the 'main' section of
the allmydata.org repo (for dependency packages) and the local EXPORT
repo
declare this baseline to be readonly, don't modify it again
teach schroot.conf how to create a session that uses a read-write snapshot
of that baseline volume. Each time the session is started, schroot will
create an LVM snapshot initially populated with the baseline contents.
When the session is finished, the snapshot will be destroyed.
the buildslave process is then:
checkout
create .deb (I want this to happen before tests, in case the _trial_temp/
directory gets dragged into the .deb)
run tests (halt-on-failure)
run create-EXPORT-repo script with the previously-generated .deb
schroot start session
schroot apt-get update, apt-get install allmydata-tahoe
schroot tahoe --version, confirm output looks right
for extra credit, run 'trial allmydata' and check the output
for extra credit, create a no-storage client node, attach to testgrid or
prodnet, wait for connections, download a file, check its contents
schroot destroy session

Having one of these on each debian platform we care about should get us
tested installable packages.

Ok, here's my plan: * install LVM and schroot on the host * create ~buildslave/public_html/EXPORT * write a script that takes a .deb file and replaces EXPORT with a functional single-package APT repository * create a baseline LVM volume * use debootstrap to install a basic e.g. hardy system * create a user account with sudo-powers to run apt-get * configure /etc/apt/sources.list to point at both the 'main' section of the allmydata.org repo (for dependency packages) and the local EXPORT repo * declare this baseline to be readonly, don't modify it again * teach schroot.conf how to create a session that uses a read-write snapshot of that baseline volume. Each time the session is started, schroot will create an LVM snapshot initially populated with the baseline contents. When the session is finished, the snapshot will be destroyed. * the buildslave process is then: * checkout * create .deb (I want this to happen before tests, in case the _trial_temp/ directory gets dragged into the .deb) * run tests (halt-on-failure) * run create-EXPORT-repo script with the previously-generated .deb * schroot start session * schroot apt-get update, apt-get install allmydata-tahoe * schroot tahoe --version, confirm output looks right * for extra credit, run 'trial allmydata' and check the output * for extra credit, create a no-storage client node, attach to testgrid or prodnet, wait for connections, download a file, check its contents * schroot destroy session Having one of these on each debian platform we care about should get us tested installable packages.

warner commented

2009-02-18 17:34:15 +00:00

Oh, and http://www.pseudorandom.co.uk/2007/sbuild/ is a useful resource.

Oh, and <http://www.pseudorandom.co.uk/2007/sbuild/> is a useful resource.

zooko commented

2009-03-04 16:56:59 +00:00

It looks like the task of "explain debian compatibility" has been nicely completed by Brian. Shall we close this ticket, or change its name to "automated tests of debian compatibility"? :-)

warner commented

2009-03-04 18:48:02 +00:00

yes! updating title..

warner changed title from ~~explain debian compatibility: run-from-source vs run-from-.deb-package~~ to automated tests of debian compatibility: run-from-source and run-from-.deb-package

2009-03-04 18:48:02 +00:00

zooko commented

2009-03-09 16:54:45 +00:00

Is this now a duplicate of #593 (test installation of .deb's)?