implement mutable-file upload/download helpers #283

New Issue

warner · 2008-01-24T01:44:31Z

warner commented

2008-01-24 01:44:31 +00:00

In addition to the CHK-upload helper (#116) and the CHK-download helper
(#268), we could use a helper for mutable files. This helper would allow
clients to read and write directories without talking to all of the storage
servers.

If all four helpers are implemented, then we can change the introducer
configuration to isolate the clients from the storage servers. If any of
these helpers are not implemented, the clients must continue to talk to all
the storage servers.

This ticket is specifically about the mutable-file upload/download helper.

In addition to the CHK-upload helper (#116) and the CHK-download helper (#268), we could use a helper for mutable files. This helper would allow clients to read and write directories without talking to all of the storage servers. If all four helpers are implemented, then we can change the introducer configuration to isolate the clients from the storage servers. If any of these helpers are not implemented, the clients must continue to talk to all the storage servers. This ticket is specifically about the mutable-file upload/download helper.

warner added the

labels 2008-01-24 01:44:31 +00:00

warner added this to the eventually milestone 2008-01-24 01:44:31 +00:00

warner added

code-mutable

and removed

code-encoding

labels 2008-04-24 23:46:43 +00:00

warner modified the milestone from eventually to undecided

2008-06-01 20:43:21 +00:00

zooko commented

2009-12-20 23:14:57 +00:00

Perhaps this could be part of the solution to #868 (use helper to bypass NAT/firewall).

zooko commented

2010-07-21 16:35:12 +00:00

But despite its potential use for #868, I'm -1 on this ticket. I think of the existence of the current immutable file upload erasure-coding helper as doubling the engineering costs of improving immutable upload. Inventing helpers for mutable files and for download would multiply this problem. On the other hand, I'm pretty enthusiastic about tickets like #993 (refactor download interfaces to treat immutable files and mutable versions more uniformly), which offer to reduce the number of codebases that we're maintaining and improving.

I recognize that immutable upload helper offers a performance benefit that the current immutable file upload code doesn't. I'm not proposing to kill it, which would be a regression for people who use that performance benefit. But I hope that we'll improve the normal upload, repair, and rebalancing features to such a point that it becomes better for users to rely on those features and they stop needing the upload helper.

Ways that we can improve those things: better and more customizable share placement, #610 (upload should take better advantage of existing shares), #809 (Measure how segment size affects upload/download speed.), #873 (upload: tolerate lost or missing servers), #1130 (Failure to achieve happiness in upload).

repair-and-rebalance behavior: #232 (Peer selection doesn't rebalance shares on overwrite of mutable file.), #483 (repairer service), #450 (checker / repair agent), #543 ('rebalancing manager'), #643 (Automatically schedule repair process), #699 (rebalance during repair or upload), #711 (repair to different levels of M), #1004 (how to fix 'multiple versions are recoverable'?).

See also the discussion on tahoe-dev that prompted me to write this comment.

But despite its potential use for #868, I'm -1 on this ticket. I think of the existence of the current immutable file upload erasure-coding helper as doubling the engineering costs of improving immutable upload. Inventing helpers for mutable files and for download would multiply this problem. On the other hand, I'm pretty enthusiastic about tickets like #993 (refactor download interfaces to treat immutable files and mutable versions more uniformly), which offer to reduce the number of codebases that we're maintaining and improving. I recognize that immutable upload helper offers a performance benefit that the current immutable file upload code doesn't. I'm not proposing to kill it, which would be a regression for people who use that performance benefit. But I hope that we'll improve the normal upload, repair, and rebalancing features to such a point that it becomes better for users to rely on those features and they stop needing the upload helper. Ways that we can improve those things: better and more customizable [share placement](wiki/ServerSelection), #610 (upload should take better advantage of existing shares), #809 (Measure how segment size affects upload/download speed.), #873 (upload: tolerate lost or missing servers), #1130 (Failure to achieve happiness in upload). repair-and-rebalance behavior: #232 (Peer selection doesn't rebalance shares on overwrite of mutable file.), #483 (repairer service), #450 (checker / repair agent), #543 ('rebalancing manager'), #643 (Automatically schedule repair process), #699 (rebalance during repair or upload), #711 (repair to different levels of M), #1004 (how to fix 'multiple versions are recoverable'?). See also [the discussion on tahoe-dev](http://tahoe-lafs.org/pipermail/tahoe-dev/2010-July/004718.html) that prompted me to write this comment.

lpirl commented

2015-10-07 23:29:37 +00:00

An implied limitation seems to be #2403 (daira pointed that out): when the helper is the only node that sees all other nodes (star topology), clients of that helper fail on a lot of commands since they can neither read nor write (enough shares of) directories.

Sign in to join this conversation.