disk space resource limits on storage servers #34

New Issue

zooko · 2007-05-04T05:54:55Z

zooko commented

2007-05-04 05:54:55 +00:00

Currently if you run a storage server, then clients can ask you to store data, and you always say yes. Make it possible to configure a storage server to have a limit on how much disk space it will use, and when it is full it says no.

zooko added the

code

major

defect

labels 2007-05-04 05:54:55 +00:00

zooko self-assigned this 2007-05-04 05:54:55 +00:00

zooko commented

2007-05-04 05:59:02 +00:00

also mentioned in source:roadmap.txt

warner commented

2007-06-06 21:56:24 +00:00

Implementation notes: I'm thinking of having the node do a 'du -s' of the storage directory on startup and keep that value around as its estimate of total storage used. Each time a lease request arrives, subtract the estimate from the max-space target to decide whether we'll accept the lease or not. Keep a list of pending incoming shares and count them against the storage space (so that four billion lease requests that arrive at the same time won't all be honored). When the bucket is closed, do a 'du -s' of the bucket and add it to the estimate. Likewise, when a bucket is deleted, do a 'du -s' first and remove that from the estimate.

Sound reasonable? I'll look to amdlib.utils.fileutil for 'du -s' code, so we have something that'll work on windows too.

Implementation notes: I'm thinking of having the node do a 'du -s' of the storage directory on startup and keep that value around as its estimate of total storage used. Each time a lease request arrives, subtract the estimate from the max-space target to decide whether we'll accept the lease or not. Keep a list of pending incoming shares and count them against the storage space (so that four billion lease requests that arrive at the same time won't all be honored). When the bucket is closed, do a 'du -s' of the bucket and add it to the estimate. Likewise, when a bucket is deleted, do a 'du -s' first and remove that from the estimate. Sound reasonable? I'll look to amdlib.utils.fileutil for 'du -s' code, so we have something that'll work on windows too.

zooko commented

2007-06-07 03:00:46 +00:00

Sounds reasonable!

warner commented

2007-07-04 00:43:55 +00:00

I just implemented this, in changeset:9ddb92965161a067, changeset:c80ea7d69399b153, and changeset:94e6e6160b24959c. Write a number of bytes (perhaps with a suffix like kB or Mb or G) to a file named 'sizelimit' in the client's base directory. It will be read at node startup and used to constrain the StorageServer to that size.

It's not 100% accurate right now: buckets which are being filled count against the limit, but the metadata that goes into them is only counted once the bucket is closed. The whole storage pool is sized for real at startup, and the amount consumed is modified after that as buckets come and go. The sizelimit does not take into account space consumed by directories, but rather only pays attention to files.

I just implemented this, in changeset:9ddb92965161a067, changeset:c80ea7d69399b153, and changeset:94e6e6160b24959c. Write a number of bytes (perhaps with a suffix like kB or Mb or G) to a file named 'sizelimit' in the client's base directory. It will be read at node startup and used to constrain the [StorageServer](wiki/StorageServer) to that size. It's not 100% accurate right now: buckets which are being filled count against the limit, but the metadata that goes into them is only counted once the bucket is closed. The whole storage pool is sized for real at startup, and the amount consumed is modified after that as buckets come and go. The sizelimit does not take into account space consumed by directories, but rather only pays attention to files.

warner added the

fixed

label 2007-07-04 00:43:55 +00:00

warner closed this issue

2007-07-04 00:43:55 +00:00

warner added this to the 0.5.0 milestone 2007-07-04 00:44:08 +00:00

Sign in to join this conversation.