Debian package installation is unreliable and spuriously fails CI jobs #2979

New Issue

exarkun · 2019-02-26T20:15:19Z

exarkun commented

2019-02-26 20:15:19 +00:00

For example, seen on <https://circleci.com/gh/tahoe-lafs/tahoe-lafs/2864>, apt-get --quiet --yes install git eventually fails like:

W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/main/binary-amd64/Packages  Hash Sum mismatch

For example, seen on <<https://circleci.com/gh/tahoe-lafs/tahoe-lafs/2864>>, `apt-get --quiet --yes install git` eventually fails like: ``` W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/main/binary-amd64/Packages Hash Sum mismatch ```

exarkun added the

labels 2019-02-26 20:15:19 +00:00

exarkun added this to the undecided milestone 2019-02-26 20:15:19 +00:00

exarkun commented

2020-01-17 17:11:36 +00:00

CI got refactored to build Debian (and other) Docker images first and only once they have been built successfully to use them to run tests.

It's still possible for Docker image builds to fail... But image builds aren't in the critical path for development. I guess we could add some kind of retry logic in the Docker image building code so that failures like the above don't cause an image build to fail ... but that strikes me as quite low priority since these failures don't get in the way of any development now (unless they happen repeatedly so many times that we eventually end up testing against ancient versions of Debian/whatever that no longer reflect what users will really be using)...

I'm going to call this "good enough" and say we won't do the extra work to try to avoid these failures ever failing an image build job.

CI got refactored to build Debian (and other) Docker images *first* and only once they have been built successfully to use them to run tests. It's still possible for Docker image builds to fail... But image builds aren't in the critical path for development. I guess we could add some kind of retry logic in the Docker image building code so that failures like the above don't cause an image build to fail ... but that strikes me as quite low priority since these failures don't get in the way of any development now (unless they happen repeatedly so many times that we eventually end up testing against ancient versions of Debian/whatever that no longer reflect what users will really be using)... I'm going to call this "good enough" and say we won't do the extra work to try to avoid these failures *ever* failing an image build job.