refuse to upload/download a mutable file if it cannot be done in the available memory #1796

New Issue

tahoe-lafs · 2012-08-06T23:00:47Z

davidsarah commented

2012-08-06 23:00:47 +00:00

If an operation on a mutable file requires significantly more memory than the amount of physical memory available, then it will just thrash uselessly. Depending on how well the operating system handles memory pressure, this may make the machine unusable or the gateway process difficult to kill. The gateway should simply refuse the operation in this case.

The method of getting the amount of usable memory is likely to be different on Unix and Windows.

If an operation on a mutable file requires significantly more memory than the amount of physical memory available, then it will just thrash uselessly. Depending on how well the operating system handles memory pressure, this may make the machine unusable or the gateway process difficult to kill. The gateway should simply refuse the operation in this case. The method of getting the amount of usable memory is likely to be different on Unix and Windows.

tahoe-lafs added the

labels 2012-08-06 23:00:47 +00:00

tahoe-lafs added this to the undecided milestone 2012-08-06 23:00:47 +00:00

davidsarah commented

2012-08-06 23:15:32 +00:00

(00:12:56) davidsarah: how to determine usable memory: free + buffers + cached as reported by the 'free' command?
(00:13:21) davidsarah: (free physical + buffers + cached that is)
(00:13:38) zooko: I don't think it is that simple.
(00:13:41) zooko: I mean,
(00:13:46) zooko: suppose that number comes out to 2 GB
(00:13:51) davidsarah: ok
(00:14:00) zooko: but if you then try to allocate 2 GB of RAM, you push something else into swap thrash.
(00:14:02) zooko: For example.
(00:14:17) davidsarah: right. but some swapping is fine, no?
(00:14:26) davidsarah: we're trying to stop pathological swapping
(00:14:29) zooko: Hm.
(00:14:36) zooko: I don't know.
(00:14:50) davidsarah: because the OS doesn't do a good job of stopping pathological swapping
(00:14:51) zooko: I'm inclined to leave the handling of too-large SDMFs simple and dumb and bad...

``` (00:12:56) davidsarah: how to determine usable memory: free + buffers + cached as reported by the 'free' command? (00:13:21) davidsarah: (free physical + buffers + cached that is) (00:13:38) zooko: I don't think it is that simple. (00:13:41) zooko: I mean, (00:13:46) zooko: suppose that number comes out to 2 GB (00:13:51) davidsarah: ok (00:14:00) zooko: but if you then try to allocate 2 GB of RAM, you push something else into swap thrash. (00:14:02) zooko: For example. (00:14:17) davidsarah: right. but some swapping is fine, no? (00:14:26) davidsarah: we're trying to stop pathological swapping (00:14:29) zooko: Hm. (00:14:36) zooko: I don't know. (00:14:50) davidsarah: because the OS doesn't do a good job of stopping pathological swapping (00:14:51) zooko: I'm inclined to leave the handling of too-large SDMFs simple and dumb and bad... ```