sometimes a helper hurts instead of helping (if you need to upload less than K shares) #613

New Issue

zooko · 2009-02-10T06:11:30Z

zooko commented

2009-02-10 06:11:30 +00:00

If #610 were fixed so that uploader (e.g. on repair) were smart about which shares to upload, then sometimes it would decide to upload only a single share, for example if the file were encoded with M=10 and 9 shares were already being served. If it were doing that, then uploading through a helper would ironically multiply the upstream bandwidth used instead of dividing it. So in that case it might want to be clever and skip using the helper.

zooko added the

labels 2009-02-10 06:11:30 +00:00

zooko added this to the undecided milestone 2009-02-10 06:11:30 +00:00

zooko changed title from ~~sometimes a helper doesn't help (if you need to upload less than K shares)~~ to sometimes a helper hurts instead of helping (if you need to upload less than K shares)

2010-07-21 16:07:12 +00:00

lpirl commented

2015-02-09 11:25:03 +00:00

I thought about this too. Without knowing too much about the internals, here is the idea:

Let's assume the helper knows (better) which shares to upload and which to reuse.
Couldn't an uploader

ask the helper for it's encoding parameters,
do the encryption and the erasure coding,
then send the hashes of the resulting shares
and finally upload those shares, the helper reports as missing?

Additionally, if a share is unhappy, the helper could fetch it from a faster sever and re-distribute it (although it is probably a difficult decision if this is beneficial or not).

I thought about this too. Without knowing too much about the internals, here is the idea: Let's assume the helper knows (better) which shares to upload and which to reuse. Couldn't an uploader 1. ask the helper for it's encoding parameters, 1. do the encryption and the erasure coding, 1. then send the hashes of the resulting shares 1. and finally upload those shares, the helper reports as missing? Additionally, if a share is unhappy, the helper could fetch it from a faster sever and re-distribute it (although it is probably a difficult decision if this is beneficial or not).

zooko commented

2015-02-10 17:45:07 +00:00

lpirl: what would be the point of the helper if the uploader was still doing erasure coding and uploading shares?

lpirl commented

2015-02-10 18:15:04 +00:00

Sorry for not pointing that out clearly.
The idea is, that the uploader only uploads shares to the helper that are actually missing on the storage servers - or is it actually implemented this way?
In other words, the uploader would ask the helper if it needs a specific share prior uploading.
For shares that are available but unhappy, the helper could download and re-distribute them without requiring the client to upload them again.

Sorry for not pointing that out clearly. The idea is, that the uploader only uploads shares to the helper that are actually *missing* on the storage servers - or is it actually implemented this way? In other words, the uploader would ask the helper if it needs a specific share prior uploading. For shares that are available but unhappy, the helper could download and re-distribute them without requiring the client to upload them again.

zooko commented

2015-02-10 18:52:41 +00:00

lpirl: Happiness is currently a property of files, not of shares. (See https://github.com/zooko/tahoe-lafs/blob/1382-rewrite-5/docs/upload-happiness.rst for a new document about that.)

The point of the helper is to let the uploader avoid uploading all the shares, and instead upload just a copy of the ciphertext.

If there are only 1 or a few shares that need to be uploaded, then it would probably make more sense for the uploader to upload those shares directly to the servers, and not use the helper at all.

lpirl: Happiness is currently a property of files, not of shares. (See <https://github.com/zooko/tahoe-lafs/blob/1382-rewrite-5/docs/upload-happiness.rst> for a new document about that.) The point of the helper is to let the uploader avoid uploading all the shares, and instead upload just a copy of the ciphertext. If there are only 1 or a few shares that need to be uploaded, then it would probably make more sense for the uploader to upload those shares directly to the servers, and not use the helper at all.

Sign in to join this conversation.