How does tahoe filesystem layer (lafs) handle lots of file-upload tasks? #173

New Issue

zooko · 2007-10-11T03:36:54Z

zooko commented

2007-10-11 03:36:54 +00:00

Rob, Peter, Justin and I just fixed up the commercial, proprietary cousin of Tahoe -- the http://allmydata.com product -- to behave better when the user asks it to upload 100,000 files ASAP. How would Tahoe handle a similar situation? To close this ticket, implement an automated measurement (akin to the memory and upload/download rate measurements) which measures how much RAM, wall-clock time, CPU load, etc. a Tahoe node requires to simultaneously upload 1000, 10,000, or 100,000 files.

Rob, Peter, Justin and I just fixed up the commercial, proprietary cousin of Tahoe -- the <http://allmydata.com> product -- to behave better when the user asks it to upload 100,000 files ASAP. How would Tahoe handle a similar situation? To close this ticket, implement an automated measurement (akin to the [memory and upload/download rate](wiki/Performance) measurements) which measures how much RAM, wall-clock time, CPU load, etc. a Tahoe node requires to simultaneously upload 1000, 10,000, or 100,000 files.

zooko added the

labels 2007-10-11 03:36:54 +00:00

zooko added this to the undecided milestone 2007-10-11 03:36:54 +00:00

zooko added

0.6.1

and removed

0.6.0

labels 2007-10-19 23:11:38 +00:00

zooko commented

2008-02-28 01:58:33 +00:00

Fabrice, the entrepeneur behind Allmydata, says that our reference test should be 25,000 files and 100 GB -- "VERY WELL and RELIABLY!". He says this is necessary to be competitive with Carbonite, Mozy, etc.. He promises to raise the bar to 100,000 files and 500 GB as soon as we reach 25,000 files and 100 GB. ;-)

Putting this into the 0.9.0 (Allmydata 3.0 final) Milestone.

Fabrice, the entrepeneur behind Allmydata, says that our reference test should be 25,000 files and 100 GB -- "VERY WELL and RELIABLY!". He says this is necessary to be competitive with Carbonite, Mozy, etc.. He promises to raise the bar to 100,000 files and 500 GB as soon as we reach 25,000 files and 100 GB. ;-) Putting this into the 0.9.0 (Allmydata 3.0 final) Milestone.

zooko modified the milestone from undecided to 0.9.0 (Allmydata 3.0 final)

2008-02-28 01:58:33 +00:00

zooko commented

2008-03-10 19:57:45 +00:00

This ticket is just for the tahoe filesystem layer ("laughfs"). See also #337 (how does the whole system including the Windows native client handle lots of file-upload tasks?).

This ticket is hereby vagueified to be unclear about whether we're talking about lots of parallel upload tasks or a lot of sequential upload tasks or some combination thereof.

This ticket is just for the tahoe filesystem layer ("laughfs"). See also #337 (how does the whole system including the Windows native client handle lots of file-upload tasks?). This ticket is hereby vagueified to be unclear about whether we're talking about lots of parallel upload tasks or a lot of sequential upload tasks or some combination thereof.

zooko changed title from ~~How does tahoe handle lots of simultaneous file-upload tasks?~~ to How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks?

2008-03-10 19:57:45 +00:00

warner commented

2008-03-10 20:04:18 +00:00

incidentally, I'm -1 on the "laughfs" term. I have no idea what it refers to.

zooko changed title from ~~How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks?~~ to How does tahoe filesystem layer (lafs) handle lots of file-upload tasks?

2008-03-21 22:31:09 +00:00

zooko added this to the undecided milestone 2008-03-21 22:31:29 +00:00

Sign in to join this conversation.