Integration test step goes green even if actual tests fail #2922

New Issue

meejah · 2018-05-08T01:42:26Z

meejah commented

2018-05-08 01:42:26 +00:00

Is maybe the "tox" command not propagating its exit-code .. "or something"? See e.g. https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/375959463

Is maybe the "tox" command not propagating its exit-code .. "or something"? See e.g. <https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/375959463>

meejah added the

labels 2018-05-08 01:42:26 +00:00

meejah added this to the undecided milestone 2018-05-08 01:42:26 +00:00

meejah commented

2018-05-08 02:28:54 +00:00

Locally at least (Debian stretch) tox -e integration does correctly return 1 (if e.g. the "Chutney" tests fail, in the case I was looking at).

Locally at least (Debian stretch) `tox -e integration` does correctly return `1` (if e.g. the "Chutney" tests fail, in the case I was looking at).

exarkun commented

2020-01-17 17:04:57 +00:00

Hm. I think someone clicked the rebuild button on that travis build. Travis re-uses build numbers! So when you rebuild a failed job you lose all of the state from the failed run and it is replaced by the results of the new run. Fun stuff.

This sounds like a terrible bug sadly all of the information associated with the failure is gone.

We should re-open this if we observe the problem again and include all relevant information on the ticket instead of linking out to travis to avoid losing it again.

FWIW, fortunately or unfortunately, I've seen failed integration test runs on CI... so this doesn't always happen. :/

Hm. I think someone clicked the rebuild button on that travis build. Travis re-uses build numbers! So when you rebuild a failed job you lose all of the state from the failed run and it is replaced by the results of the new run. Fun stuff. This sounds like a terrible bug sadly all of the information associated with the failure is gone. We should re-open this if we observe the problem again and include all relevant information on the ticket instead of linking out to travis to avoid losing it again. FWIW, fortunately or unfortunately, I've seen failed integration test runs on CI... so this doesn't *always* happen. :/

exarkun added the

cannot reproduce

label 2020-01-17 17:04:57 +00:00

exarkun closed this issue

2020-01-17 17:04:57 +00:00

Sign in to join this conversation.