uploader should keep trying other servers if its initially-chosen servers fail during the "scan" phase #2108

New Issue

zooko · 2013-11-15T07:24:32Z

zooko commented

2013-11-15 07:24:32 +00:00

In [v1.10 upload.py]source:trunk/src/allmydata/immutable/upload.py?annotate=blame&rev=196bd583b6c4959c60d3f73cdcefc9edda6a38ae#L390, during the uploader's "scan" phase (asking storage servers if they already have, or would be willing to accept upload of, shares of this file), if the uploader's first chosen servers answer "no can do" or fail, then it will keep asking more and more servers, until it either succeeds at uploading or runs out of candidates.

In 1382-rewrite-2 upload.py (which is hopefully going to be merged into trunk soon, and released in the upcoming Tahoe-LAFS v1.11), it instead chooses a few servers that it is going to ask, and if all of them fail then it gives up on the upload.

(I have a vague memory of discussing this on a conference call with the other Google Summer of Code Mentors and Mark Berger, and telling him to go ahead and do it this way, as it is simpler to implement. That might be a false memory.)

Anyway, I'd like to revisit this issue. For some situations, this would be a regression from 1.10 to 1.11, i.e. 1.10 would successfully upload and 1.11 would say that the upload failed. Therefore I'm adding the keywords "regression" and "blocks-release" to this ticket.

The reason to do it this way with a finite "scan" phase is that by first establishing which servers either already-have or are-willing-to-accept shares, we can then use our upload-strategy-of-happiness computation to plan which servers we want to upload to. Mixing planning with action is confusing, and the old 1.10 algorithm was hard to understand and has some undesirable behaviors. I suspect this is why we instructed Mark to go ahead with the simpler "phased" approach in #1382.

However, now that I've seen the 1382-rewrite-2 branch up close, I think I'm starting to see how a variant of it wouldn't be too complicated, would have the property of "always achieves Happiness if it is possible to do so" and would relieve it of this regression.

The idea would be that instead of:

Pick a few servers (e.g. 2*N).
Query them all about their state (whether they already have any shares and whether they would be willing to hold a share).
Wait til all queries resolve (by a response, a failure or network disconnect, or a timeout).
Run the "calculate-a-plan-for-happiness" algorithm.
If it is possible to achieve happiness, go on to the next phase of attempting to implement that plan (i.e. the uploading-shares phase).

We would instead have a state machine which does something like this:

Let R be the set of servers who have responded to our queries by indicating that they either already have shares or would be willing to hold a share. At the beginning of the state machine, R is ∅ (the empty set). Let A be the set of all servers that we have heard about.
Run the "calculate-a-plan-for-happiness" algorithm on R.
If it is possible to achieve happiness, go on to the next phase of attempting to implement the plan (i.e. the uploading-shares phase).
If it is not possible, then pop the next server out of A and send it a query. When the query comes back, go to step 2.

In [v1.10 upload.py]source:trunk/src/allmydata/immutable/upload.py?annotate=blame&rev=196bd583b6c4959c60d3f73cdcefc9edda6a38ae#L390, during the uploader's "scan" phase (asking storage servers if they already have, or would be willing to accept upload of, shares of this file), if the uploader's first chosen servers answer "no can do" or fail, then it will keep asking more and more servers, until it either succeeds at uploading or runs out of candidates. In [1382-rewrite-2 upload.py](https://github.com/markberger/tahoe-lafs/blob/fa6f3cfb1e258e4427f35a7aada05c7ad2c9dd13/src/allmydata/immutable/upload.py#L284) (which is hopefully going to be merged into trunk soon, and released in the upcoming Tahoe-LAFS v1.11), it instead chooses a few servers that it is going to ask, and if all of them fail then it gives up on the upload. (I have a vague memory of discussing this on a conference call with the other Google Summer of Code Mentors and Mark Berger, and telling him to go ahead and do it this way, as it is simpler to implement. That might be a false memory.) Anyway, I'd like to revisit this issue. For some situations, this would be a regression from 1.10 to 1.11, i.e. 1.10 would successfully upload and 1.11 would say that the upload failed. Therefore I'm adding the keywords "regression" and "blocks-release" to this ticket. The reason to do it this way with a finite "scan" phase is that by first establishing which servers either already-have or are-willing-to-accept shares, we can then use our upload-strategy-of-happiness computation to plan which servers we want to upload to. Mixing planning with action is confusing, and the old 1.10 algorithm was hard to understand and has some undesirable behaviors. I suspect this is why we instructed Mark to go ahead with the simpler "phased" approach in #1382. However, now that I've seen the 1382-rewrite-2 branch up close, I think I'm starting to see how a variant of it wouldn't be *too* complicated, would have the property of "always achieves Happiness if it is possible to do so" and would relieve it of this regression. The idea would be that instead of: 1. Pick a few servers (e.g. 2*N). 2. Query them all about their state (whether they already have any shares and whether they would be willing to hold a share). 3. Wait til all queries resolve (by a response, a failure or network disconnect, or a timeout). 4. Run the "calculate-a-plan-for-happiness" algorithm. 1. If it is possible to achieve happiness, go on to the next phase of attempting to implement that plan (i.e. the uploading-shares phase). We would instead have a state machine which does something like this: 1. Let `R` be the set of servers who have responded to our queries by indicating that they either already have shares or would be willing to hold a share. At the beginning of the state machine, `R` is `∅` (the empty set). Let `A` be the set of all servers that we have heard about. 2. Run the "calculate-a-plan-for-happiness" algorithm on `R`. 1. If it is possible to achieve happiness, go on to the next phase of attempting to implement the plan (i.e. the uploading-shares phase). 2. If it is not possible, then pop the next server out of `A` and send it a query. When the query comes back, go to step 2.

zooko added the

labels 2013-11-15 07:24:32 +00:00

zooko added this to the 1.11.0 milestone 2013-11-15 07:24:32 +00:00

zooko commented

2013-12-12 04:58:47 +00:00

Daira said, at some point, I think, that this is not an important regression for Tahoe-LAFS v1.11, because the only cases where this would occur are when the grid has a high rate of churn (servers coming and going), and in those cases, Tahoe-LAFS v1.10 immutable upload probably has other problems. I think that's what she said. Anyway, it sounded right to me at the time and I agreed with it, but apparently we forgot to write it down on this ticket. Assigning to Daira to confirm and moving this ticket out of v1.11.