Support running a storage node per drive easily #913

New Issue

tahoe-lafs · 2010-01-18T03:08:10Z

davidsarah commented

2010-01-18 03:08:10 +00:00

Suggested by badon on IRC:

Tahoe needs a slick method to aggregate drives on a single machine. Call it Tahoe drive spanning, or something like that. Then I can use all my obsolete drives again. They're otherwise too small to be useful, but together with Tahoe's fault tolerance, it could make RAID storage look restrictive. RAID has to have all the same size drives, or it'll waste the capacity of all but the smallest drive.

Suggested by badon on IRC: > Tahoe needs a slick method to aggregate drives on a single machine. Call it Tahoe drive spanning, or something like that. Then I can use all my obsolete drives again. They're otherwise too small to be useful, but together with Tahoe's fault tolerance, it could make RAID storage look restrictive. RAID has to have all the same size drives, or it'll waste the capacity of all but the smallest drive.

tahoe-lafs added the

labels 2010-01-18 03:08:10 +00:00

tahoe-lafs added this to the undecided milestone 2010-01-18 03:08:10 +00:00

qwasty commented

2010-01-18 03:17:49 +00:00

I see people all the time complaining about how their RAID system can't take new drives of random sizes. Tahoe could solve that problem. It'd be the killer app of distributed filesystems.

From what I gather, Tahoe is ready to do that NOW, with some minor technical work. If you could make it "just work", then cut the bloggers loose on it, because it's going to get some attention, much like Sun's ZFS has. I run http://www.livebusinesschat.com/ and I can see the potential from a business perspective. If a business could just keep adding disks without worrying about when they're going to fail, companies could save huge bucks. Most of them don't need blazing speed, they just need low cost reliable storage. Tahoe is perfect.

Actually, why not borrow Sun ZFS terminology and call it a "Tahoe storage pool"? Tahoe is a lake too, and I know how zooko loves wordplay.

It's basically some user friendliness that I'm asking for, since it appears all the core technology is already in place. You'd have an entirely new area that Tahoe would serve with that comparatively simple addition.

I see people all the time complaining about how their RAID system can't take new drives of random sizes. Tahoe could solve that problem. It'd be the killer app of distributed filesystems. From what I gather, Tahoe is ready to do that NOW, with some minor technical work. If you could make it "just work", then cut the bloggers loose on it, because it's going to get some attention, much like Sun's ZFS has. I run <http://www.livebusinesschat.com/> and I can see the potential from a business perspective. If a business could just keep adding disks without worrying about when they're going to fail, companies could save huge bucks. Most of them don't need blazing speed, they just need low cost reliable storage. Tahoe is perfect. Actually, why not borrow Sun ZFS terminology and call it a "Tahoe storage pool"? Tahoe is a lake too, and I know how zooko loves wordplay. It's basically some user friendliness that I'm asking for, since it appears all the core technology is already in place. You'd have an entirely new area that Tahoe would serve with that comparatively simple addition.

zooko commented

2010-01-18 04:27:48 +00:00

I think Tahoe-LAFS is already technically well suited for this. Maybe it needs more documentation or another layer of automation? Why don't you try it and post ot this ticket how it works for you.

qwasty commented

2010-01-18 04:56:24 +00:00

Yes, that's why I mention it. Since Tahoe is already technically capable of this, I think it's the application that will put Tahoe on the map. I would do it myself, but I'm on windows, and these things are a pain on windows. A layer of automation is exactly what I need, as well as out-of-the-box functionality.

I've been following Tahoe development for quite some time now, but I've had difficulty making it work. It's partly laziness, since I have the ability to figure it out if I really want to, but since what I'd really like to do is tell the world that they can use Tahoe, I don't want to put too much more effort into getting it to work than "they" are likely to invest.

Then again, I'd put the effort in for personal reasons so I can have my own system running, but I simply haven't had the time...So, if this is going to become a reality - and the blockbuster application Tahoe needs to get noticed - it's going to have to be easier to use on windows.

I suspect that it may not require the effort I've put into it so far, if only the documentation were better, but I'm not sure about that, since I wasn't successful in the few days of effort I put into it. I used the allmydata.com client on windows, which differs in sparsely documented ways from the "main" *nix version.

This final bit of polish needed to pull this off would also make it possible to swap my drive space for remote drive space in either the volunteer pool, or the allmydata.com pool. I'm not sure how the economics of that will work out, but it's still worth mentioning as an example of emergent applications of this technology once it's more accessible to people who may want to do novel things with it, but who don't want to fuss over the details of Tahoe technology.

Yes, that's why I mention it. Since Tahoe is already technically capable of this, I think it's the application that will put Tahoe on the map. I would do it myself, but I'm on windows, and these things are a pain on windows. A layer of automation is exactly what I need, as well as out-of-the-box functionality. I've been following Tahoe development for quite some time now, but I've had difficulty making it work. It's partly laziness, since I have the ability to figure it out if I really want to, but since what I'd really like to do is tell the world that they can use Tahoe, I don't want to put too much more effort into getting it to work than "they" are likely to invest. Then again, I'd put the effort in for personal reasons so I can have my own system running, but I simply haven't had the time...So, if this is going to become a reality - and the blockbuster application Tahoe needs to get noticed - it's going to have to be easier to use on windows. I suspect that it may not require the effort I've put into it so far, if only the documentation were better, but I'm not sure about that, since I wasn't successful in the few days of effort I put into it. I used the allmydata.com client on windows, which differs in sparsely documented ways from the "main" *nix version. This final bit of polish needed to pull this off would also make it possible to swap my drive space for remote drive space in either the volunteer pool, or the allmydata.com pool. I'm not sure how the economics of that will work out, but it's still worth mentioning as an example of emergent applications of this technology once it's more accessible to people who may want to do novel things with it, but who don't want to fuss over the details of Tahoe technology.

qwasty commented

2010-01-18 18:52:42 +00:00

Dugg! http://digg.com/software/Tahoe_LAFS_RAID_lets_you_use_drives_of_any_type_or_size

Dugg! <http://digg.com/software/Tahoe_LAFS_RAID_lets_you_use_drives_of_any_type_or_size>

warner commented

2010-01-18 22:37:59 +00:00

hm. Grids with heterogeneous-capacity storage nodes, whether comprised of a variety of sizes of local drives, or a variety of sizes of remote storage servers, will not be able to offer uniform high reliability for all files. Sure, we're more flexible than a simple RAID scheme (which will refuse to use the leftover space on the larger drives), but in our basic server-selection algorithm, we'll fill the small drives quickly and then eventually wind up putting all the shares on the largest disk, lowering reliability for those files down to the reliability of a single disk.

OTOH, it might still be useful to offer "reliable" storage up to the size of your smallest (or 2nd-largest) drive, and then unreliable non-redundant storage with the remaining space.

hm. Grids with heterogeneous-capacity storage nodes, whether comprised of a variety of sizes of local drives, or a variety of sizes of remote storage servers, will not be able to offer uniform high reliability for all files. Sure, we're more flexible than a simple RAID scheme (which will refuse to use the leftover space on the larger drives), but in our basic server-selection algorithm, we'll fill the small drives quickly and then eventually wind up putting all the shares on the largest disk, lowering reliability for those files down to the reliability of a single disk. OTOH, it might still be useful to offer "reliable" storage up to the size of your smallest (or 2nd-largest) drive, and then unreliable non-redundant storage with the remaining space.

imhavoc commented

2010-01-18 23:29:43 +00:00

I think this may be a possible design flaw. Not in the sense that Tahoe was originally designed for, but in the sense of how Tahoe is going to be used in the real world.

Production grids will fill up. New drives/servers will be added. The new drives in each generation will be larger than the previous generations. This may potentially cause the grid to have a cascading performance decrease. The other usage is for heterogeneous grids like those that have been discussed lately (including my own).

I have had some thoughts on this, but they are just thoughts.

Give a node the ability to flag itself "full," "read only," "storage not available," or some other flag. This will keep the grid from trying to store shares on full volumes and eliminate that performance penalty. (Let the requesting node know you're full before it asks to store shares on you.)
Have nodes report available storage. The storage algorithms could sort the nodes before polling for share storage by only looking at nodes that last reported >X bytes available to grid. (Each node would keep a table of available storage on each attached peer.)
With "servers of happiness" in place rather than "shares of happiness," a flag could be raised to indicate problems.
A daemon could be run to email grid administrators grid status updates related to grid balance/imbalance/alarms.

By hoping for a homogeneous grid, Tahoe is limited to expansive, relatively sterile use cases.

I think this may be a possible design flaw. Not in the sense that Tahoe was originally designed for, but in the sense of how Tahoe is going to be used in the real world. Production grids will fill up. New drives/servers will be added. The new drives in each generation will be larger than the previous generations. This may potentially cause the grid to have a cascading performance decrease. The other usage is for heterogeneous grids like those that have been discussed lately (including my own). I have had some thoughts on this, but they are just thoughts. - Give a node the ability to flag itself "full," "read only," "storage not available," or some other flag. This will keep the grid from trying to store shares on full volumes and eliminate that performance penalty. (Let the requesting node know you're full before it asks to store shares on you.) - Have nodes report available storage. The storage algorithms could sort the nodes before polling for share storage by only looking at nodes that last reported >X bytes available to grid. (Each node would keep a table of available storage on each attached peer.) - With "servers of happiness" in place rather than "shares of happiness," a flag could be raised to indicate problems. - A daemon could be run to email grid administrators grid status updates related to grid balance/imbalance/alarms. By hoping for a homogeneous grid, Tahoe is limited to expansive, relatively sterile use cases.

zooko commented

2010-01-19 01:08:18 +00:00

I don't agree with what Brian said "Grids with heterogeneous-capacity storage nodes ... will not be able to offer uniform high reliability for all files." I think whether it will work in practice depends on a lot of operational/sysadmin/tools/monitoring questions, such as the ones imhavoc mentioned. And in fact, several of the features that imhavoc asked for are already there:

mark a storage node as read-only: check -- see source:docs/configuration.txt
report available storage: you could set up munin and use the [munin plugins]source:misc/munin (or implement #648)
servers of happiness: coming soon in v1.6! (#778)
daemon to mail status updates: possibly triggered by munin?

I don't agree with what Brian said "Grids with heterogeneous-capacity storage nodes ... will not be able to offer uniform high reliability for all files." I think whether it will work in practice depends on a lot of operational/sysadmin/tools/monitoring questions, such as the ones imhavoc mentioned. And in fact, several of the features that imhavoc asked for are already there: * mark a storage node as read-only: check -- see source:docs/configuration.txt * report available storage: you could set up munin and use the [munin plugins]source:misc/munin (or implement #648) * servers of happiness: coming soon in v1.6! (#778) * daemon to mail status updates: possibly triggered by munin?

qwasty commented

2010-01-19 02:54:54 +00:00

Zooko: Don't forget to automate all that, or it will be inaccessible to people like me who are not sysadmins.

Also, would it be better to fill all nodes to the same percentage of capacity? For example, smaller drives are probably older and less reliable. If you have enough data to spread out evenly amongst all the drives by filling them all to exactly 90% capacity, that would mean that comparatively less data would be stored on smaller, presumably older and less reliable drives.

It would also have the effect of nearly guaranteeing that the larger, and presumably newer and more reliable drives would have enough excess capacity to host the data regenerated from a failure of one of the smaller drives.

So, instead of reporting absolute capacities, which can become meaningless relative to the massive new drives in the pool, maybe it would be better to report percent remaining capacity? For example, does it really matter if there's 2 MB left on an old 16 MB flash drive, if there's 2 TB left on a modern drive? That 2 MB is a lot for that flash drive, but not for the modern drive. If they report percentages instead, you can tell which drives are pulling their own weight.

From there, I assume Tahoe already makes sure that significant losses can be regenerated with the erasure coding methods that Tahoe uses.

Zooko: Don't forget to automate all that, or it will be inaccessible to people like me who are not sysadmins. Also, would it be better to fill all nodes to the same percentage of capacity? For example, smaller drives are probably older and less reliable. If you have enough data to spread out evenly amongst all the drives by filling them all to exactly 90% capacity, that would mean that comparatively less data would be stored on smaller, presumably older and less reliable drives. It would also have the effect of nearly guaranteeing that the larger, and presumably newer and more reliable drives would have enough excess capacity to host the data regenerated from a failure of one of the smaller drives. So, instead of reporting absolute capacities, which can become meaningless relative to the massive new drives in the pool, maybe it would be better to report percent remaining capacity? For example, does it really matter if there's 2 MB left on an old 16 MB flash drive, if there's 2 TB left on a modern drive? That 2 MB is a lot for that flash drive, but not for the modern drive. If they report percentages instead, you can tell which drives are pulling their own weight. From there, I assume Tahoe already makes sure that significant losses can be regenerated with the erasure coding methods that Tahoe uses.

imhavoc commented

2010-01-19 03:45:39 +00:00

qwasty: percentages are not useful. If a node is attempting to store a 6MB file, it needs to know what servers can store 2MB shares.

Also, if you base the distribution on percentages, then a new drive will get all of the shares until it reaches the percentage of the rest of the grid.

Zooko: A node should be able to either report it's remaining capacity to other nodes, or/and set a flag that says, "I'm full. Don't try to store shares on me. I'm read-only," automatically when it's full. The node administrator shouldn't have to reconfigure and restart for this behavior. The node administrator should even have to monitor storage space for this behavior.

When I'm talking about reporting available storage, I'm thinking "to other nodes," so that node 'n' knows how much available space (as last reported) is available on all other connected nodes before it starts asking to store shares. If it's trying to store 20MB shares, it can skip over all the nodes with <20MB available (as last reported). This is going to end up being a huge problem in large ad hoc grids, and Tahoe is a natural candidate for ad hoc grids of all shapes and sizes.

Servers of happiness: Woohoo! (I knew it was coming, but I had to put that in my comments for completeness.)

Daemon to mail status updates: Should be part of the default install. If it's additional work to add on, many (most?) users of ad hoc systems might not be capable (for lack of a better term) of installing and configuring additional components. If they are capable, the chances of them not "having time" to install additional components will lead to the grid performance flat-lining before diagnostics lead them to discovering that they should have added this earlier. I realize that delivering email is not as simple as most users assume it is -- especially when you get off of *nix systems, but this is something that needs to be addressed.

qwasty: percentages are not useful. If a node is attempting to store a 6MB file, it needs to know what servers can store 2MB shares. Also, if you base the distribution on percentages, then a new drive will get all of the shares until it reaches the percentage of the rest of the grid. Zooko: A node should be able to either report it's remaining capacity to other nodes, or/and set a flag that says, "I'm full. Don't try to store shares on me. I'm read-only," automatically when it's full. The node administrator shouldn't have to reconfigure and restart for this behavior. The node administrator should even have to monitor storage space for this behavior. When I'm talking about reporting available storage, I'm thinking "to other nodes," so that node 'n' knows how much available space (as last reported) is available on all other connected nodes before it starts asking to store shares. If it's trying to store 20MB shares, it can skip over all the nodes with <20MB available (as last reported). This is going to end up being a huge problem in large ad hoc grids, and Tahoe is a natural candidate for ad hoc grids of all shapes and sizes. Servers of happiness: Woohoo! (I knew it was coming, but I had to put that in my comments for completeness.) Daemon to mail status updates: Should be part of the default install. If it's additional work to add on, many (most?) users of ad hoc systems might not be capable (for lack of a better term) of installing and configuring additional components. If they are capable, the chances of them not "having time" to install additional components will lead to the grid performance flat-lining before diagnostics lead them to discovering that they should have added this earlier. I realize that delivering email is not as simple as most users assume it is -- especially when you get off of *nix systems, but this is something that needs to be addressed.

imhavoc commented

2010-01-19 03:47:33 +00:00

Defining "fullness": "Fullness" (if the "full" flag is chosen, would have to be defined by some threshold. That's why I keep leaning toward the reported available capacity idea.

kevan commented

2010-01-19 04:10:23 +00:00

Replying to imhavoc:

When I'm talking about reporting available storage, I'm thinking "to other nodes," so that node 'n' knows how much available space (as last reported) is available on all other connected nodes before it starts asking to store shares. If it's trying to store 20MB shares, it can skip over all the nodes with <20MB available (as last reported). This is going to end up being a huge problem in large ad hoc grids, and Tahoe is a natural candidate for ad hoc grids of all shapes and sizes.

I think the upload logic does that already. See [this logic in Tahoe2PeerSelector]source:src/allmydata/immutable/upload.py#L191, and [this corresponding logic from the storage server]source:src/allmydata/storage/server.py#L267; combined, they mean that a file upload won't attempt to store shares on peers that can't hold those shares. Maybe I'm not understanding your suggestion?

Replying to [imhavoc](/tahoe-lafs/trac-2024-07-25/issues/913#issuecomment-74810): > When I'm talking about reporting available storage, I'm thinking "to other nodes," so that node 'n' knows how much available space (as last reported) is available on all other connected nodes before it starts asking to store shares. If it's trying to store 20MB shares, it can skip over all the nodes with <20MB available (as last reported). This is going to end up being a huge problem in large ad hoc grids, and Tahoe is a natural candidate for ad hoc grids of all shapes and sizes. I think the upload logic does that already. See [this logic in Tahoe2PeerSelector]source:src/allmydata/immutable/upload.py#L191, and [this corresponding logic from the storage server]source:src/allmydata/storage/server.py#L267; combined, they mean that a file upload won't attempt to store shares on peers that can't hold those shares. Maybe I'm not understanding your suggestion?

zooko commented

2010-01-19 04:29:22 +00:00

Replying to qwasty:

Zooko: Don't forget to automate all that, or it will be inaccessible to people like me who are not sysadmins.

Well, much as I personally want to encourage this sort of use case, I have limited time and I don't think I will have time to implement these layers of automation in the forseeable future. I think the thing that is most needed at this stage is experience reports from people who actually deployed Tahoe-LAFS grids and observed the things that worked well and the things that didn't.

Now actually when wearing one of my other hats I am doing exactly that -- maintaining the allmydata.com production grid and posting bug reports or asking allmydata.com users to post bug reports to this trac.

However, my individual use case will not cover all of the layers of documentation and automation that would be necessary to achieve wide acceptance.

Therefore, if you see what I am getting at, it is up to you! You must deploy a Tahoe-LAFS grid! And you must open trac tickets explaining what didn't work, and posting to the tahoe-dev list summarizing what was good and bad about the experience.

One nice thing about layers of automation is that they can be written completely outside of the Tahoe-LAFS core, using the WAPI, cmdline, or perhaps the (new, unstable) FUSE interface. That means it is easier for them to be written by people who don't have the expertise to make changes to the Tahoe-LAFS core. See [the RelatedProjects page](wiki/RelatedProjects) (http://allmydata.org/trac/tahoe/wiki/RelatedProjects ) for examples of this sort of auxiliary tool such as Kevin Reid's cron script, Andrej Falout's obackup, and François Deppierraz's puppet-tahoe. These are all tools that can be used to automate and control your Tahoe-LAFS nodes.

Maybe people who are interested in this sort of use case should explore those tools and consider contributing to those tools to extend them to cover your use case.

Also, would it be better to fill all nodes to the same percentage of capacity?

David-Sarah recently had a similar idea and opened #872 (Adjust the probability of selecting a node according to its storage capacity (or other fitness measure)). It could be interesting! Again, we may need more help in terms of bug reports, testing, documentation, code review, and writing patches if we're going to accomplish that goal along with many other useful and widely-demanded features.

Replying to [qwasty](/tahoe-lafs/trac-2024-07-25/issues/913#issuecomment-74809): > Zooko: Don't forget to automate all that, or it will be inaccessible to people like me who are not sysadmins. Well, much as I personally want to encourage this sort of use case, I have limited time and I don't think I will have time to implement these layers of automation in the forseeable future. I think the thing that is most needed at this stage is experience reports from people who actually deployed Tahoe-LAFS grids and observed the things that worked well and the things that didn't. Now actually when wearing one of my other hats I am doing exactly that -- maintaining the allmydata.com production grid and posting bug reports or asking allmydata.com users to post bug reports to this trac. However, my individual use case will not cover all of the layers of documentation and automation that would be necessary to achieve wide acceptance. Therefore, if you see what I am getting at, it is up to you! You must deploy a Tahoe-LAFS grid! And you must open trac tickets explaining what didn't work, and posting to the tahoe-dev list summarizing what was good and bad about the experience. One nice thing about layers of automation is that they can be written completely outside of the Tahoe-LAFS core, using the WAPI, cmdline, or perhaps the (new, unstable) FUSE interface. That means it is easier for them to be written by people who don't have the expertise to make changes to the Tahoe-LAFS core. See [the [RelatedProjects](wiki/RelatedProjects) page](wiki/RelatedProjects) (<http://allmydata.org/trac/tahoe/wiki/RelatedProjects> ) for examples of this sort of auxiliary tool such as Kevin Reid's cron script, Andrej Falout's obackup, and François Deppierraz's puppet-tahoe. These are all tools that can be used to automate and control your Tahoe-LAFS nodes. Maybe people who are interested in this sort of use case should explore those tools and consider contributing to those tools to extend them to cover your use case. > Also, would it be better to fill all nodes to the same percentage of capacity? David-Sarah recently had a similar idea and opened #872 (Adjust the probability of selecting a node according to its storage capacity (or other fitness measure)). It could be interesting! Again, we may need more help in terms of bug reports, testing, documentation, code review, and writing patches if we're going to accomplish that goal along with many other useful and widely-demanded features.

warner commented

2010-01-19 07:32:14 +00:00

The discussion in #872 is good stuff (as well as #302): if you think you want to take advantage of both MB-sized nodes and TB-sized nodes in the same grid, you should read it. I don't believe it's possible to do that, but I'm eager to hear more people's thoughts about it.

zooko commented

2010-01-19 13:38:58 +00:00

I personally have read those tickets, and I still think it is possible to take advantage of GB-sized nodes and TB-sized nodes in the same grid. I don't know that it will turn out to be practical but I also don't know that it will turn out to be impractical. It remains to be seen.

Those tickets (correct me if I'm wrong -- they're complicated!) are about further optimizing the way Tahoe-LAFS uses heterogeneous-sized storage nodes in order to (a) let you the grid last as long as possible even if you fill it entirely up, and (b) make it so that if a disaster happens such as M servers all dying that this kills only a fraction of the files instead of having a worse pattern of destruction such as killing lots of files and killing more-recently-uploaded files. Those are worthy goals, but the absence of this improvement shouldn't deter people from experimenting with grids of heterogeneous-sized storage nodes.

For what its worth, the allmydata.com grid has dozens of nodes with 500 GB, dozens with 1 TB, and a dozen with 2 TB. I would assume that the volunteergrid currently has nodes ranging from the 100's of GBs to the 1's of TBs, but nobody knows. We need more experiment reports.

I personally have read those tickets, and I still think it is possible to take advantage of GB-sized nodes and TB-sized nodes in the same grid. I don't *know* that it will turn out to be practical but I also don't know that it will turn out to be impractical. It remains to be seen. Those tickets (correct me if I'm wrong -- they're complicated!) are about further optimizing the way Tahoe-LAFS uses heterogeneous-sized storage nodes in order to (a) let you the grid last as long as possible even if you fill it entirely up, and (b) make it so that if a disaster happens such as `M` servers all dying that this kills only a fraction of the files instead of having a worse pattern of destruction such as killing lots of files and killing more-recently-uploaded files. Those are worthy goals, but the absence of this improvement shouldn't deter people from experimenting with grids of heterogeneous-sized storage nodes. For what its worth, the allmydata.com grid has dozens of nodes with 500 GB, dozens with 1 TB, and a dozen with 2 TB. I would assume that the volunteergrid currently has nodes ranging from the 100's of GBs to the 1's of TBs, but nobody knows. We need more experiment reports.

zooko commented

2010-01-19 14:03:39 +00:00

Oh, and the tickets are also about maintaining good upload and download performance and about scaling up to thousands of nodes in a grid. I doubt that upload and download performance would be noticeably worse in a grid with heterogeneous-sized storage nodes than it is now on the testgrid. (Unfortunately.)