the "update-apt" step for pycryptopp buildbots always says that a task is already queued #1257

New Issue

zooko · 2010-11-12T14:10:23Z

zooko commented

2010-11-12 14:10:23 +00:00

Here is a builder whose job is to update the apt repository:

http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/

It seems like every time the update-apt step is run, it says:

update already queued, deferring to them

stderr_dk on IRC says:

<stderr_dk> I wonder where
	    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/7
	    were uploaded...
<stderr_dk> It says "python-pycryptopp_0.5.25-r755-1_i386.deb:
	    uploaded"
<stderr_dk> But the latest version in
	    http://tahoe-lafs.org/debian/dists/lenny/main/binary-i386/ is
	    0.5.17 from 01-Jun-2010.
<stderr_dk> Maybe the .deb is uploaded to the wrong place? Maybe
	    that's part of the problem?

I think the reason it isn't served up by http://tahoe-lafs.org is that the script which always thinks it is already queued is supposed to scp it to tahoe-lafs.org.

Here is a builder whose job is to update the apt repository: <http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/> It seems like every time the update-apt step is run, it says: ``` update already queued, deferring to them ``` stderr_dk on IRC says: ``` <stderr_dk> I wonder where http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/7 were uploaded... <stderr_dk> It says "python-pycryptopp_0.5.25-r755-1_i386.deb: uploaded" <stderr_dk> But the latest version in http://tahoe-lafs.org/debian/dists/lenny/main/binary-i386/ is 0.5.17 from 01-Jun-2010. <stderr_dk> Maybe the .deb is uploaded to the wrong place? Maybe that's part of the problem? ``` I think the reason it isn't served up by <http://tahoe-lafs.org> is that the script which always thinks it is already queued is supposed to scp it to tahoe-lafs.org.

zooko added the

labels 2010-11-12 14:10:23 +00:00

zooko added this to the undecided milestone 2010-11-12 14:10:23 +00:00

zooko commented

2010-11-12 14:17:11 +00:00

I found the script on dev.allmydata.org and ran it manually:

buildslave@dev:~/tahoe-debs$ make
update already queued, deferring to them

inspecting the Makefile I see that it is looking for a file named queue.stamp:

-rw-------  1 buildslave buildslave        0 2010-06-09 01:39 queue.stamp

I found the script on dev.allmydata.org and ran it manually: ``` buildslave@dev:~/tahoe-debs$ make update already queued, deferring to them ``` inspecting the Makefile I see that it is looking for a file named `queue.stamp`: ``` -rw------- 1 buildslave buildslave 0 2010-06-09 01:39 queue.stamp ```

zooko commented

2010-11-12 14:22:11 +00:00

Reading the Makefile I don't see any flaw in its handling of queue.stamp. I guess what happened was that the system rebooted or in some other way the make process exited without finishing its clean-up of those .stamp files. I'm going to manually remove them.

Reading the `Makefile` I don't see any flaw in its handling of `queue.stamp`. I guess what happened was that the system rebooted or in some other way the make process exited without finishing its clean-up of those .stamp files. I'm going to manually remove them.

zooko commented

2010-11-12 14:34:59 +00:00

That fixed it! But there is a new failure:
http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/10/steps/update%20apt%20repo/logs/stdio

make[3]: Leaving directory `/home/buildslave/tahoe-debs'
~/bin/mirror-debs-to-org.sh
pushing .debs to tahoe-lafs.org
Host key verification failed.
make[2]: Leaving directory `/home/buildslave/tahoe-debs'
make[1]: Leaving directory `/home/buildslave/tahoe-debs'
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
make[2]: *** [run-update-and-mirror] Error 255
make[1]: *** [run-update-and-check-for-queued] Error 2
make: *** [update] Error 2
program finished with exit code 2

That fixed it! But there is a new failure: <http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/10/steps/update%20apt%20repo/logs/stdio> ``` make[3]: Leaving directory `/home/buildslave/tahoe-debs' ~/bin/mirror-debs-to-org.sh pushing .debs to tahoe-lafs.org Host key verification failed. make[2]: Leaving directory `/home/buildslave/tahoe-debs' make[1]: Leaving directory `/home/buildslave/tahoe-debs' rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7] make[2]: *** [run-update-and-mirror] Error 255 make[1]: *** [run-update-and-check-for-queued] Error 2 make: *** [update] Error 2 program finished with exit code 2 ```

zooko commented

2010-11-12 14:49:37 +00:00

I suspect it might have just been waiting for someone to type "yes" to ssh. I did so manually. I guess this means we should be configuring our ssh to not prompt that. There is a way to do that.

zooko commented

2010-11-12 15:14:29 +00:00

Aha, and that led to
http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/11/steps/update%20apt%20repo/logs/stdio

update in progress, queueing a new one

So I guess that timeout from scp waiting for someone to type "yes" is what led to the leftover stale queue.stamp file...

Aha, and that led to <http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/11/steps/update%20apt%20repo/logs/stdio> ``` update in progress, queueing a new one ``` So I guess that timeout from scp waiting for someone to type "yes" is what led to the leftover stale `queue.stamp` file...

zooko commented

2010-11-12 17:16:09 +00:00

Okay that worked:
http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/12/steps/update%20apt%20repo/logs/stdio

Now http://tahoe-lafs.org/debian/dists/lenny/main/binary-i386/?C=M;O=D has the latest files.

To fix this ticket: make it so failures of the upload code don't leave behind an immortal stale lock file (probably by using the debian/ubuntu package lockfile-progs and perhaps the bash trap builtin or else a subprocess).

Alternate fix for this ticket: decide that we can live with that script doing that whenever it dies and close this ticket as "fixed".

Okay that worked: <http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/12/steps/update%20apt%20repo/logs/stdio> Now <http://tahoe-lafs.org/debian/dists/lenny/main/binary-i386/?C=M;O=D> has the latest files. To fix this ticket: make it so failures of the upload code don't leave behind an immortal stale lock file (probably by using the debian/ubuntu package `lockfile-progs` and perhaps the bash `trap` builtin or else a subprocess). Alternate fix for this ticket: decide that we can live with that script doing that whenever it dies and close this ticket as "fixed".

davidsarah commented

2011-08-26 22:20:20 +00:00

The deb builders have been decommissioned.

tahoe-lafs added the

invalid

label 2011-08-26 22:20:20 +00:00

davidsarah closed this issue

2011-08-26 22:20:20 +00:00

Sign in to join this conversation.