build some share-migration tools #481

New Issue

warner · 2008-07-01T23:37:32Z

warner commented

2008-07-01 23:37:32 +00:00

Zandr and I were talking about what sorts of tools we'd like to have
available when it comes time to move shares from one disk to another.

The Repairer is of course the first priority, and should be able to handle
share loss, but there are some techniques we might use to make things more
efficient: using shares that already exist instead of generating new ones.

If we have a large disk full of shares that has some problems (bad blocks,
etc), we should be able to dd or scp off the shares to another system. This
wants a tool that will try to read a file (skipping it if we get io errors),
verify as much of it as we can (seeing if the UEB hash matches), then sending
it over the network to somewhere else.

If a disk is starting to fail (we've seen SMART statistics, or we're starting
to see hash failures in the shares we return, etc), then we might want to
kick the disk into "abandon ship" mode: get all shares off the disk (and onto
better ones) as quickly as possible. The server could do the peer-selection
work and ask around and find the "right" server for each share (i.e. the
first one in the permuted order that doesn't already have a share), or it
could just fling them to a "lifeboat" node and leave the peer-selection work
until later.

Repair nodes should have a directory where we can dump shares that came from
other servers: the repair node should treat that directory as a work queue,
and it should find a home for each one (or discard it as a duplicate). The
repair node needs to be careful to not treat abandon-ship nodes as suitable
targets, so we can avoid putting shares back on the server that was trying to
get rid of them.

It might also be useful to split up a storage server, or to take a functional
server and export half of its shares in a kind of rebalancing step.

Zandr and I were talking about what sorts of tools we'd like to have available when it comes time to move shares from one disk to another. The Repairer is of course the first priority, and should be able to handle share loss, but there are some techniques we might use to make things more efficient: using shares that already exist instead of generating new ones. If we have a large disk full of shares that has some problems (bad blocks, etc), we should be able to dd or scp off the shares to another system. This wants a tool that will try to read a file (skipping it if we get io errors), verify as much of it as we can (seeing if the UEB hash matches), then sending it over the network to somewhere else. If a disk is starting to fail (we've seen SMART statistics, or we're starting to see hash failures in the shares we return, etc), then we might want to kick the disk into "abandon ship" mode: get all shares off the disk (and onto better ones) as quickly as possible. The server could do the peer-selection work and ask around and find the "right" server for each share (i.e. the first one in the permuted order that doesn't already have a share), or it could just fling them to a "lifeboat" node and leave the peer-selection work until later. Repair nodes should have a directory where we can dump shares that came from other servers: the repair node should treat that directory as a work queue, and it should find a home for each one (or discard it as a duplicate). The repair node needs to be careful to not treat abandon-ship nodes as suitable targets, so we can avoid putting shares back on the server that was trying to get rid of them. It might also be useful to split up a storage server, or to take a functional server and export half of its shares in a kind of rebalancing step.