store shares in single files, instead of 7 files and a directory #85

New Issue

warner · 2007-07-12T18:33:59Z

warner commented

2007-07-12 18:33:59 +00:00

At the moment we have significant filesystem overhead in our share storage, mostly because common filesystems like ext3 consume a whole disk block (4096 bytes) even for 1-byte files. Since we use 7 files in a separate directory for each share, that means 8*4096=32768 bytes consumed per share, even for 1-byte shares. As a result, the lower bound on storage consumed occurs at 102400 bytes (i.e. all files 102400 bytes or smaller consume the same amount of storage), at which point the storage consumed is 3276800 (3.3MB).

Our storage format was chosen for simplicity and ease of implementation, but this represents a huge overhead. So the plan is to combine all 7 files into a single one, and to not put it in its own directory. That will reduce the minimum share size to one disk block (4096) instead of 8 (32768), and will bring the lower bound on storage to a filesize of 81250, at which point the storage consumed will be 409600 (410kB), an 8x improvement.

Reducing this filesystem-blocksize overhead below that would involve packing multiple shares (for different URIs) into a single file, which complicates the deletion and indexing of them. It might be useful, but hopefully we can avoid this step.

Also, we need to figure out a good place to put leases, once we implement them, but they can probably live in a separate database with different packing and access requirements.

At the moment we have significant filesystem overhead in our share storage, mostly because common filesystems like ext3 consume a whole disk block (4096 bytes) even for 1-byte files. Since we use 7 files in a separate directory for each share, that means 8*4096=32768 bytes consumed per share, even for 1-byte shares. As a result, the lower bound on storage consumed occurs at 102400 bytes (i.e. all files 102400 bytes or smaller consume the same amount of storage), at which point the storage consumed is 3276800 (3.3MB). Our storage format was chosen for simplicity and ease of implementation, but this represents a huge overhead. So the plan is to combine all 7 files into a single one, and to not put it in its own directory. That will reduce the minimum share size to one disk block (4096) instead of 8 (32768), and will bring the lower bound on storage to a filesize of 81250, at which point the storage consumed will be 409600 (410kB), an 8x improvement. Reducing this filesystem-blocksize overhead below that would involve packing multiple shares (for different URIs) into a single file, which complicates the deletion and indexing of them. It might be useful, but hopefully we can avoid this step. Also, we need to figure out a good place to put leases, once we implement them, but they can probably live in a separate database with different packing and access requirements.

warner added the

labels 2007-07-12 18:33:59 +00:00

warner added this to the 0.5.0 milestone 2007-07-12 18:33:59 +00:00

warner commented

2007-07-13 03:08:52 +00:00

I'm actively working on this one right now. The basics are in place, but the interfaces between the new bucket-proxies and the rest of the system are not working yet. I'm hoping to finish it tomorrow.

warner commented

2007-07-14 00:20:39 +00:00

Done, in changesets changeset:cd8648d39b897684, changeset:1f8e407d9cda19ed, changeset:7589a8ee82eb6531, changeset:35117d77a0bb2177, and changeset:4d868e6649c2c5d8. The new format increases the actual overhead slightly, the layout described in storageserver.py:WriteBucketProxy shows that we add about 36 bytes to allow the share to be self-describing. But the overhead caused by 4kB disk blocks is reduced by 8x (for small files).

Done, in changesets changeset:cd8648d39b897684, changeset:1f8e407d9cda19ed, changeset:7589a8ee82eb6531, changeset:35117d77a0bb2177, and changeset:4d868e6649c2c5d8. The new format increases the actual overhead slightly, the layout described in [storageserver.py:WriteBucketProxy](http://allmydata.org/trac/tahoe/browser/src/allmydata/storageserver.py?rev=35117d77a0bb2177#L148) shows that we add about 36 bytes to allow the share to be self-describing. But the overhead caused by 4kB disk blocks is reduced by 8x (for small files).

warner added the

fixed

label 2007-07-14 00:20:39 +00:00

warner closed this issue

2007-07-14 00:20:39 +00:00

Sign in to join this conversation.