maybe have separate code paths for upload and repair #1126

New Issue

tahoe-lafs · 2010-07-19T02:43:11Z

davidsarah commented

2010-07-19 02:43:11 +00:00

IRC discussion:

[20:43] as far as overall design goes, when we get around to rewriting the uploader, I think it should have a separate code path that it follows as soon as it sees any evidence of shares already being present

[20:44] we want the new-upload case to work quickly, but the moment we see an alreadygot= share, we should switch into a mode where we search fairly thoroughly for all the existing shares (without allocating anything), come up with a sharemap, then replace the earlier (speculative) allocations with the ideal ones

[20:44] (dealing with allocation failures along the way)

[20:45] it's probably worth losing some of the pipelining along this path to simplify the code

[20:45] neat idea

[20:45] I'm not sure I agree.

[20:45] Okay, afk...

[20:46] of course, second-upload (when we detect existing shares) is really a form of repair, better than normal repair because we have the full plaintext available locally

[20:48] so maybe what ought to happen is that we rewrite the repairer, and make the uploader's "hey there are already shares there" path do: abandon the current allocations, hand the alreadygot info and the plaintext/ciphertext filehandle to the repairer, start the repairer, wait for its results

[20:50] second-upload is a common case if you're storing a lot of files in tahoe and locally and 'moving'/copying back and forth

[20:50] hm, true

[20:51] and given immutable directories...

[20:51] we've got two sorts of frontend-duplicate detectors that might move some pressure off the uploader: the helper, and the backupdb

[20:51] some might want to have efficient upload-the-same-thing-again, as for backups

[20:51] it's kind of an open question as to where the responsibilities ought to lie

[20:51] i.e. tahoe backup without the builtin cleverness

[20:52] I think it's reasonable to add backupdb support to 'tahoe cp', and to add a contents-to-filecap table to backupdb, which would avoid network IO in the case of moving files around

[20:53] (currently if you move a file, the backupdb declares a miss, and we proceed to the uploader, which will either get a Helper hit or exercise the second-upload code. In my experience, second-upload results in duplicate shares, which is a drag, so I'd prefer to avoid it)

[20:55] (OTOH, having the backupdb track file-contents means that we'd do an extra hash of the file for each new upload, in addition to the subsequent CHK-computation hash. OT3H, unless the files are really large, the filesystem cache should save us from actually doing the disk IO an extra time)

[20:55] warner: 1.6.0 was changed to search for all extant shares before immutable upload, IIRC

[20:56] http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/NEWS?rev=d329759bb83ad6a0#L66

[20:56] I think it sends requests to everyone, yeah (which I still consider to be a scaling problem), but I don't believe that it waits to hear from everyone before starting the upload

[20:56] there's always a tradeoff between stallability and thoroughness there

IRC discussion: > [20:43] <warner> as far as overall design goes, when we get around to rewriting the uploader, I think it should have a separate code path that it follows as soon as it sees any evidence of shares already being present > [20:44] <warner> we want the new-upload case to work quickly, but the moment we see an alreadygot= share, we should switch into a mode where we search fairly thoroughly for all the existing shares (without allocating anything), come up with a sharemap, then replace the earlier (speculative) allocations with the ideal ones > [20:44] <warner> (dealing with allocation failures along the way) > [20:45] <warner> it's probably worth losing some of the pipelining along this path to simplify the code > [20:45] <zooko> neat idea > [20:45] <zooko> I'm not sure I agree. > [20:45] <zooko> Okay, afk... > [20:46] <warner> of course, second-upload (when we detect existing shares) is really a form of repair, better than normal repair because we have the full plaintext available locally > [20:48] <warner> so maybe what ought to happen is that we rewrite the repairer, and make the uploader's "hey there are already shares there" path do: abandon the current allocations, hand the alreadygot info and the plaintext/ciphertext filehandle to the repairer, start the repairer, wait for its results > [20:50] <kpreid> second-upload is a common case if you're storing a lot of files in tahoe and locally and 'moving'/copying back and forth > [20:50] <warner> hm, true > [20:51] <kpreid> and given immutable directories... > [20:51] <warner> we've got two sorts of frontend-duplicate detectors that might move some pressure off the uploader: the helper, and the backupdb > [20:51] <kpreid> some might want to have efficient upload-the-same-thing-again, as for backups > [20:51] <warner> it's kind of an open question as to where the responsibilities ought to lie > [20:51] <kpreid> i.e. tahoe backup without the builtin cleverness > [20:52] <warner> I think it's reasonable to add backupdb support to 'tahoe cp', and to add a contents-to-filecap table to backupdb, which would avoid network IO in the case of moving files around > [20:53] <warner> (currently if you move a file, the backupdb declares a miss, and we proceed to the uploader, which will either get a Helper hit or exercise the second-upload code. In my experience, second-upload results in duplicate shares, which is a drag, so I'd prefer to avoid it) > [20:55] <warner> (OTOH, having the backupdb track file-contents means that we'd do an extra hash of the file for each new upload, in addition to the subsequent CHK-computation hash. OT3H, unless the files are really large, the filesystem cache should save us from actually doing the disk IO an extra time) > [20:55] <zooko> warner: 1.6.0 was changed to search for all extant shares before immutable upload, IIRC > [20:56] <zooko> <http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/NEWS?rev=d329759bb83ad6a0#L66> > [20:56] <warner> I think it sends requests to everyone, yeah (which I still consider to be a scaling problem), but I don't believe that it waits to hear from everyone before starting the upload > [20:56] <warner> there's always a tradeoff between stallability and thoroughness there

tahoe-lafs added the

labels 2010-07-19 02:43:11 +00:00

tahoe-lafs added this to the undecided milestone 2010-07-19 02:43:11 +00:00

Sign in to join this conversation.