tolerate simultaneous uploads better #2409

New Issue

warner · 2015-04-22T00:14:03Z

warner commented

2015-04-22 00:14:03 +00:00

In the Nuts+Bolts meeting this morning, we discussed what would happen if an application (in particular the "magic folder / drop-upload" feature) were to upload two copies of the same file at the same time. We thought about this a long time ago, but I can't seem to find a ticket on the particular issue.

I believe there's a race condition on the storage servers which would make the upload go less smoothly than we'd like. The first upload will see no shares for each storage index, so it will allocate a BucketWriter and start writing the share. The second upload will compute the storage-index, ask the server about pre-existing shares, and then.. probably get a yes?

The answer is uncertain, and depends upon the server implementation. The server's read-side might look on disk for the partially-written files, or the server's write-side might be using the write-to-tempfile atomic-swap technique, or the read-side might be looking in a leasedb for evidence of the share. Some of these will result in a "no" answer to the DYHB, in which case the second upload will try to allocate new BucketWriters to fill the shares (which might fail because of the existing writers, or might succeed with hilarious results as the two writers attempt to write the same file with hopefully the same data). It might get a "yes", in which case I think the uploader will ignore the shares and assume that they'll be present in the future.

We should probably:

nail down exactly what the server does in this situation
change the Uploader to be more cautious about pre-existing shares

The Uploader could read the pre-existing shares as it goes, comparing them against locally-generated ones. If they match, great, those shares can count against the servers-of-happiness criteria. If they don't, or if they aren't complete, then oops. The simplest way to deal with such problems is to treat them like a share write that failed (as if the server disconnected before the upload was complete), which may flunk the shares-of-happiness test and mark the upload as failing. A more sophisticated approach (which hopefully is ticketed elsewhere) is to have a second pass which writes out a new copy of any share that wasn't successfully placed during the first pass.

If we implement that verify-during-upload thing, we'll need to think carefully about how simultaneous uploads ought to work. I think we'll need a way to mark shares as "in-progress", which tells the second uploader that they can't verify the share now, but maybe they shouldn't upload it anyways.

This will get better when we make the storage-index be a hash of the share (or the root of a merkle tree with the shares in the leaves), because then the storage-index won't even be defined until the upload is complete, and the intermediate in-progress state will disappear. Simultaneous uploads will then turn into two uploads of the exact same share, detected at close(), which is inefficient but sound, I think.

Related tickets:

#610 "upload should take better advantage of existing shares"
#643 make the storage index be the verifier cap
#873 upload: tolerate lost of unacceptably slow servers
#1288 support streaming uploads in uploader
#1508 shortcut encryption and erasure coding when a file has already been uploaded

In the Nuts+Bolts meeting this morning, we discussed what would happen if an application (in particular the "magic folder / drop-upload" feature) were to upload two copies of the same file at the same time. We thought about this a long time ago, but I can't seem to find a ticket on the particular issue. I believe there's a race condition on the storage servers which would make the upload go less smoothly than we'd like. The first upload will see no shares for each storage index, so it will allocate a BucketWriter and start writing the share. The second upload will compute the storage-index, ask the server about pre-existing shares, and then.. probably get a yes? The answer is uncertain, and depends upon the server implementation. The server's read-side might look on disk for the partially-written files, or the server's write-side might be using the write-to-tempfile atomic-swap technique, or the read-side might be looking in a leasedb for evidence of the share. Some of these will result in a "no" answer to the DYHB, in which case the second upload will try to allocate new BucketWriters to fill the shares (which might fail because of the existing writers, or might succeed with hilarious results as the two writers attempt to write the same file with hopefully the same data). It might get a "yes", in which case I think the uploader will ignore the shares and assume that they'll be present in the future. We should probably: * nail down exactly what the server does in this situation * change the Uploader to be more cautious about pre-existing shares The Uploader could read the pre-existing shares as it goes, comparing them against locally-generated ones. If they match, great, those shares can count against the servers-of-happiness criteria. If they don't, or if they aren't complete, then oops. The simplest way to deal with such problems is to treat them like a share write that failed (as if the server disconnected before the upload was complete), which may flunk the shares-of-happiness test and mark the upload as failing. A more sophisticated approach (which hopefully is ticketed elsewhere) is to have a second pass which writes out a new copy of any share that wasn't successfully placed during the first pass. If we implement that verify-during-upload thing, we'll need to think carefully about how simultaneous uploads ought to work. I think we'll need a way to mark shares as "in-progress", which tells the second uploader that they can't verify the share now, but maybe they shouldn't upload it anyways. This will get better when we make the storage-index be a hash of the share (or the root of a merkle tree with the shares in the leaves), because then the storage-index won't even be defined until the upload is complete, and the intermediate in-progress state will disappear. Simultaneous uploads will then turn into two uploads of the exact same share, detected at `close()`, which is inefficient but sound, I think. Related tickets: * #610 "upload should take better advantage of existing shares" * #643 make the storage index be the verifier cap * #873 upload: tolerate lost of unacceptably slow servers * #1288 support streaming uploads in uploader * #1508 shortcut encryption and erasure coding when a file has already been uploaded

warner added the

labels 2015-04-22 00:14:03 +00:00

warner added this to the undecided milestone 2015-04-22 00:14:03 +00:00

daira commented

2015-04-27 19:19:17 +00:00

zooko wrote in #952:

As described in #950 (upload doesn't appear in the Recent Uploads Downloads page) and #951 (uploads aren't cancelled by closing the web page) I started an upload of a large (350 MB) file, then started a second upload of the same file. Later I discovered that both uploads were running:

# 2010-02-14 18:38:58.761541Z [152794]: <Encoder for 5ossh> uploaded 266352528 / 353388984 bytes (75%) of your file.
# 2010-02-14 18:39:06.215643Z [152863]: storage: allocate_buckets 5osshw7oonef6qlpvdvtmuskt4
# 2010-02-14 18:39:17.599753Z [152865]: <Encoder for 5ossh> uploaded 266483607 / 353388984 bytes (75%) of your file.
# 2010-02-14 18:39:17.608450Z [152867]: peer selection successful for <Tahoe2PeerSelector for upload 5ossh>: placed all 26 shares, sent 23 queries to 13 peers, 20 queries placed some shares, 2 placed none, got 1 errors
# 2010-02-14 18:39:17.610185Z [152868]: _send_shares, used_peers is set([<PeerTracker for peer we543qo4 and SI 5ossh>, <PeerTracker for peer hfhjhv7s and SI 5ossh>, <PeerTracker for peer 2bu4lxwa and SI 5ossh>, <PeerTracker for peer xjy2clbq and SI 5ossh>, <PeerTracker for peer 5ouv6kib and SI 5ossh>, <PeerTracker for peer t6d76eso and SI 5ossh>, <PeerTracker for peer rhug4hsl and SI 5ossh>, <PeerTracker for peer wbxexhfu and SI 5ossh>, <PeerTracker for peer 62u7qrne and SI 5ossh>, <PeerTracker for peer 5xzrcb27 and SI 5ossh>])
# 2010-02-14 18:39:17.887869Z [152869]: <Encoder for 5ossh> starting
# 2010-02-14 18:39:18.554996Z [152947]: <Encoder for 5ossh> uploaded 266614686 / 353388984 bytes (75%) of your file.
# 2010-02-14 18:39:18.597406Z [153004]: <Encoder for 5ossh> uploaded 131079 / 353388984 bytes (0%) of your file.
# 2010-02-14 18:39:21.569559Z [153081]: <Encoder for 5ossh> uploaded 266745765 / 353388984 bytes (75%) of your file.
# 2010-02-14 18:39:21.610486Z [153138]: <Encoder for 5ossh> uploaded 262158 / 353388984 bytes (0%) of your file.

To close this ticket, make so a second upload of the same file (when using the same added convergence secret of course) just waits for the first upload instead of starting a second task (which will compete with the first one for resources).

zooko wrote in #952: > As described in #950 (upload doesn't appear in the Recent Uploads Downloads page) and #951 (uploads aren't cancelled by closing the web page) I started an upload of a large (350 MB) file, then started a second upload of the same file. Later I discovered that both uploads were running: > > ``` > # 2010-02-14 18:38:58.761541Z [152794]: <Encoder for 5ossh> uploaded 266352528 / 353388984 bytes (75%) of your file. > # 2010-02-14 18:39:06.215643Z [152863]: storage: allocate_buckets 5osshw7oonef6qlpvdvtmuskt4 > # 2010-02-14 18:39:17.599753Z [152865]: <Encoder for 5ossh> uploaded 266483607 / 353388984 bytes (75%) of your file. > # 2010-02-14 18:39:17.608450Z [152867]: peer selection successful for <Tahoe2PeerSelector for upload 5ossh>: placed all 26 shares, sent 23 queries to 13 peers, 20 queries placed some shares, 2 placed none, got 1 errors > # 2010-02-14 18:39:17.610185Z [152868]: _send_shares, used_peers is set([<PeerTracker for peer we543qo4 and SI 5ossh>, <PeerTracker for peer hfhjhv7s and SI 5ossh>, <PeerTracker for peer 2bu4lxwa and SI 5ossh>, <PeerTracker for peer xjy2clbq and SI 5ossh>, <PeerTracker for peer 5ouv6kib and SI 5ossh>, <PeerTracker for peer t6d76eso and SI 5ossh>, <PeerTracker for peer rhug4hsl and SI 5ossh>, <PeerTracker for peer wbxexhfu and SI 5ossh>, <PeerTracker for peer 62u7qrne and SI 5ossh>, <PeerTracker for peer 5xzrcb27 and SI 5ossh>]) > # 2010-02-14 18:39:17.887869Z [152869]: <Encoder for 5ossh> starting > # 2010-02-14 18:39:18.554996Z [152947]: <Encoder for 5ossh> uploaded 266614686 / 353388984 bytes (75%) of your file. > # 2010-02-14 18:39:18.597406Z [153004]: <Encoder for 5ossh> uploaded 131079 / 353388984 bytes (0%) of your file. > # 2010-02-14 18:39:21.569559Z [153081]: <Encoder for 5ossh> uploaded 266745765 / 353388984 bytes (75%) of your file. > # 2010-02-14 18:39:21.610486Z [153138]: <Encoder for 5ossh> uploaded 262158 / 353388984 bytes (0%) of your file. > ``` > > To close this ticket, make so a second upload of the same file (when using the same added convergence secret of course) just waits for the first upload instead of starting a second task (which will compete with the first one for resources).

tahoe-lafs added

major

and removed

normal

labels 2015-04-27 19:19:17 +00:00

daira commented

2015-04-27 19:19:50 +00:00

zooko also wrote in #952:

This is related to #1032 (Display active HTTP upload operations on the status page), #951 (uploads aren't cancelled by closing the web page) and #320 (add streaming (on-line) upload to HTTP interface).

zooko also wrote in #952: > This is related to #1032 (Display active HTTP upload operations on the status page), #951 (uploads aren't cancelled by closing the web page) and #320 (add streaming (on-line) upload to HTTP interface).

Sign in to join this conversation.