memory leak on 'tahoe get' #1910

New Issue

tahoe-lafs · 2013-01-31T08:09:34Z

T_X commented

2013-01-31 08:09:34 +00:00

When trying to retrieve a file either via sshfs or via 'tahoe get' I seem to have a memory leak. Every 'tahoe get' on a 350MB large file seems to grow the memory usage of the tahoe daemon by about 20MB:

$ for i in `seq 1 5`; do ps -o %mem,cmd 2303; tahoe get "root:path/to/myfile" /dev/null; done; ps -o %mem,cmd 2303
%MEM CMD
23.3 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
26.8 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
30.9 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
34.7 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
38.8 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
39.1 /usr/bin/python /usr/bin/tahoe start
$ free -m
             total       used       free     shared    buffers     cached
Mem:           497        267        229          0          7         37
-/+ buffers/cache:        222        274
Swap:          258          0        258

'myfile' is an immutable file according to the web interface. sshfs is unmounted for tahoe, so the 'tahoe get's above should be the only tahoe commands invoked during that time.

Versions (as provided by Debian wheezy):

allmydata-tahoe: 1.9.2, foolscap: 0.6.4, pycryptopp: 0.5.29, zfec: 1.4.5, Twisted: 12.0.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.7.3rc2, platform: Linux-debian_wheezy/sid-x86_64-64bit_ELF, pyOpenSSL: 0.13, simplejson: 2.5.2, pycrypto: 2.6, pyasn1: unknown, mock: 0.8.0, sqlite3: 2.6.0 [sqlite 3.7.13], setuptools: 0.6 [distribute]

Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.32-1 x86_64 unknown unknown GNU/Linux

Encoding parameters: shares.needed = 1, shares.happy = 2.

When trying to retrieve a file either via sshfs or via 'tahoe get' I seem to have a memory leak. Every 'tahoe get' on a 350MB large file seems to grow the memory usage of the tahoe daemon by about 20MB: ``` $ for i in `seq 1 5`; do ps -o %mem,cmd 2303; tahoe get "root:path/to/myfile" /dev/null; done; ps -o %mem,cmd 2303 %MEM CMD 23.3 /usr/bin/python /usr/bin/tahoe start root:path/to/myfile retrieved and written to /dev/null %MEM CMD 26.8 /usr/bin/python /usr/bin/tahoe start root:path/to/myfile retrieved and written to /dev/null %MEM CMD 30.9 /usr/bin/python /usr/bin/tahoe start root:path/to/myfile retrieved and written to /dev/null %MEM CMD 34.7 /usr/bin/python /usr/bin/tahoe start root:path/to/myfile retrieved and written to /dev/null %MEM CMD 38.8 /usr/bin/python /usr/bin/tahoe start root:path/to/myfile retrieved and written to /dev/null %MEM CMD 39.1 /usr/bin/python /usr/bin/tahoe start $ free -m total used free shared buffers cached Mem: 497 267 229 0 7 37 -/+ buffers/cache: 222 274 Swap: 258 0 258 ``` 'myfile' is an immutable file according to the web interface. sshfs is unmounted for tahoe, so the 'tahoe get's above should be the only tahoe commands invoked during that time. Versions (as provided by Debian wheezy): ``` allmydata-tahoe: 1.9.2, foolscap: 0.6.4, pycryptopp: 0.5.29, zfec: 1.4.5, Twisted: 12.0.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.7.3rc2, platform: Linux-debian_wheezy/sid-x86_64-64bit_ELF, pyOpenSSL: 0.13, simplejson: 2.5.2, pycrypto: 2.6, pyasn1: unknown, mock: 0.8.0, sqlite3: 2.6.0 [sqlite 3.7.13], setuptools: 0.6 [distribute] ``` Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.32-1 x86_64 unknown unknown GNU/Linux Encoding parameters: shares.needed = 1, shares.happy = 2.

tahoe-lafs added the

labels 2013-01-31 08:09:34 +00:00

tahoe-lafs added this to the undecided milestone 2013-01-31 08:09:34 +00:00

davidsarah commented

2013-02-01 02:59:29 +00:00

Does this pattern continue for more than, say, 200 MB without freeing any memory? Garbage collection doesn't necessarily happen immediately, so we need to exclude the possibility that it's just "floating garbage" rather than a leak.

tahoe-lafs added

code-frontend

and removed

unknown

labels 2013-02-01 03:01:23 +00:00

zooko commented

2013-02-01 18:13:50 +00:00

T_X: could you please do this experiment? Keep iterating the tahoe get's and see if the memory usage keeps rising indefinitely or if it levels off. I'm guessing it is the latter, which means that this isn't a "memory leak" per se, but it might still be a problem if the level at which it levels off is too high. Please report back! Also, what sort of machine is that? Consider pasting in the output (or part of the output) of python misc/build_helpers/show-tool-versions.py.

Re-assigning this ticket to T_X.

T_X: could you please do this experiment? Keep iterating the `tahoe get`'s and see if the memory usage keeps rising indefinitely or if it levels off. I'm guessing it is the latter, which means that this isn't a "memory leak" per se, but it might still be a problem if the level at which it levels off is too high. Please report back! Also, what sort of machine is that? Consider pasting in the output (or part of the output) of `python misc/build_helpers/show-tool-versions.py`. Re-assigning this ticket to T_X.

T_X commented

2013-02-02 03:08:54 +00:00

Attachment show-tools-versions.log (2979 bytes) added

**Attachment** show-tools-versions.log (2979 bytes) added

show-tools-versions.log

2.9 KiB

T_X commented

2013-02-02 03:09:08 +00:00

Attachment output (6217 bytes) added

**Attachment** output (6217 bytes) added

output

6.1 KiB

T_X commented

2013-02-02 03:18:16 +00:00

Ok, I temporarily increased the RAM from 256MB to 1.5GB in this VM instance and redid the tests with 50 'tahoe get's. You are right, pretty much exactly after 10 rounds (~200MB) the memory consumption does not increase as quickly as before anymore.

However it looks like it still, linearly increases by about 0.6MB per 'tahoe get', with one 'tahoe get' needing 2 minutes. At least during these 50 rounds needing 92 minutes in total - not sure whether this curve will stop increasing after the next n rounds.

I also verified that these 0.6MB per 2 minutes are due to the 'tahoe get' invocation: Four hours after this test run the the memory consumption still is at 18.1%.

Also see attached files for details.

Ok, I temporarily increased the RAM from 256MB to 1.5GB in this VM instance and redid the tests with 50 'tahoe get's. You are right, pretty much exactly after 10 rounds (~200MB) the memory consumption does not increase as quickly as before anymore. However it looks like it still, linearly increases by about 0.6MB per 'tahoe get', with one 'tahoe get' needing 2 minutes. At least during these 50 rounds needing 92 minutes in total - not sure whether this curve will stop increasing after the next n rounds. I also verified that these 0.6MB per 2 minutes are due to the 'tahoe get' invocation: Four hours after this test run the the memory consumption still is at 18.1%. Also see attached files for details. ![](http://metameute.de/~tux/tahoe-lafs/output.png)

davidsarah commented

2013-02-04 01:15:34 +00:00

The section of the graph after 11 invocations does indeed appear to show a 0.6 MB-per-invocation memory leak. ps -o %mem is based on virtual size, though, and we prefer to measure RSS. Can you repeat the test using ps -o rss?

The section of the graph after 11 invocations does indeed appear to show a 0.6 MB-per-invocation memory leak. `ps -o %mem` is based on virtual size, though, and we prefer to measure RSS. Can you repeat the test using `ps -o rss`?

Sign in to join this conversation.