document known scaling issues #1199

New Issue

zooko · 2010-09-18T01:00:49Z

zooko commented

2010-09-18 01:00:49 +00:00

There are some known issues with scaling Tahoe-LAFS up in terms of servers, users, files, aggregate data. Most of these issues were learned from the allmydata.com experience. Most of them are "soft limits" -- things start taking an inconveniently long time -- rather than hard limits where something that used to work fails.

Examples:

it took a long time to do deep-add-lease and deep-repair
user data was spread over all the servers and some files and directories were reachable only through other directories, so if a few servers were (even temporarily) off-line then some users accounts couldn't be deep-add-leased, deep-repaired, or recursively downloaded. There was no good support for a deep- operation to stop cleanly when it encounters an unrecoverable file or directory and show its partial results to the user or preserve its state for later retry.

There are some known issues with scaling Tahoe-LAFS up in terms of servers, users, files, aggregate data. Most of these issues were learned from the allmydata.com experience. Most of them are "soft limits" -- things start taking an inconveniently long time -- rather than hard limits where something that used to work fails. Examples: * it took a long time to do deep-add-lease and deep-repair * user data was spread over all the servers and some files and directories were reachable only through other directories, so if a few servers were (even temporarily) off-line then some users accounts couldn't be deep-add-leased, deep-repaired, or recursively downloaded. There was no good support for a deep- operation to stop cleanly when it encounters an unrecoverable file or directory and show its partial results to the user or preserve its state for later retry.