implement URI:LIT encoding for very small files #81

New Issue

warner · 2007-07-08T15:48:34Z

warner commented

2007-07-08 15:48:34 +00:00

Uploading a zero-length file causes a divide-by-zero error, because segment_size ends up being zero.

It would probably be sufficient to simply make segment_size always be at least one.

Uploading a zero-length file causes a divide-by-zero error, because segment_size ends up being zero. It would probably be sufficient to simply make segment_size always be at least one.

warner added the

labels 2007-07-08 15:48:34 +00:00

warner added this to the 0.5.0 milestone 2007-07-08 15:48:34 +00:00

warner commented

2007-07-11 22:31:43 +00:00

upgrading priority because I really want this one fixed before the 0.5.0 release. Once the new webapi is in place and it is easy to upload a lot of files at once, this bug becomes a lot more annoying.

warner added

critical

and removed

major

labels 2007-07-11 22:31:43 +00:00

warner commented

2007-07-12 17:58:53 +00:00

we're going to fix this one by implementing URI:LIT uris, which are used for very small files (less than about 40 bytes), in which the entire file contents are contained verbatim inside the URI itself.

warner changed title from ~~uploading an empty file causes an exception~~ to implement URI:LIT encoding for very small files

2007-07-12 17:58:53 +00:00

warner commented

2007-07-12 19:31:46 +00:00

Our target for CHK URIs is URI:CHK:(base32(16byte AES key)):(base32(32byte URI Extension Hash)):needed_shares:total_shares:size

so that's 3+1+3+1+26+1+52+1+2+1+3+1+2 = 97 bytes (at least for <100B files).

Our plan for LIT uris is: URI:LIT:(base32(data))

So the breakeven point is 55 bytes. A 55 byte file will fit into a 3+1+3+1+88 = 96 byte LIT uri.

So if the file is 55 bytes or smaller, store it in a LIT, otherwise store it in the grid.

Our target for CHK URIs is URI:CHK:(base32(16byte AES key)):(base32(32byte URI Extension Hash)):needed_shares:total_shares:size so that's 3+1+3+1+26+1+52+1+2+1+3+1+2 = 97 bytes (at least for <100B files). Our plan for LIT uris is: URI:LIT:(base32(data)) So the breakeven point is 55 bytes. A 55 byte file will fit into a 3+1+3+1+88 = 96 byte LIT uri. So if the file is 55 bytes or smaller, store it in a LIT, otherwise store it in the grid.

warner commented

2007-07-12 19:32:49 +00:00

Oh, and of course this also includes 0-byte files, which neatly fixes the problem of the encoding algorithm exploding when faced with 0-byte files.