MDMF upload via web-API uses much more memory in the gateway process than expected #1523
Labels
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
No Milestone
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Reference: tahoe-lafs/trac-2024-07-25#1523
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Split from #113:
The web-API interface does not support streaming (#113, #320), so it is expected for the gateway to need to hold the whole file in memory in order to upload it. However, when using
tahoe put
to upload an MDMF file, the increase in memory usage of the gateway process seems to be much larger than the file size. For example, when uploading a 191 MiB MDMF file in 1.9alpha usingtahoe put --mutable --mutable-type=mdmf
, the peak RSS of the gateway (which was also a storage server in this test) was over 1300 MiB. There is also a huge memory leak of more than 700 MiB after the upload has finished.I originally thought that the memory usage was larger when using the web-API than when updating the same file using SFTP, but apparently that was wrong (I may have been misled by at first doing the SFTP experiment without restarting the nodes).
I can imagine two problems here.. which are you thinking of?
Replying to warner:
SFTP will update existing mutable files that are either SDMF or MDMF. In this case we're using SFTP to update an MDMF file as a comparison for the memory that would be used if streaming were supported.
No, I'm comparing SFTP-to-MDMF against webapi-to-MDMF. We expect the memory usage for SDMF to be bad because that would use a whole-file segment, and there would be more than one segment-sized buffer in memory. For webapi-to-MDMF, on the other hand, we should be able to have only one file-sized buffer in memory even without supporting streaming, and so the memory usage should only be worse than for SFTP-to-MDMF by approximately the file size.
I started an introducer, 4 storage servers and a gateway. This time the gateway had storage disabled. The encoding parameters of the gateway were k=3, happy=1, N=10. Initially the memory usage as measured by
ps -O rss,vsize -C tahoe
(command paths snipped for readability) was:I ran
bin/tahoe put --mutable --mutable-type=mdmf zeros
, wherezeros
is a file containing 200000000 zero bytes (190.7 MiB). The memory usage of the gateway initially climbed to 1384.5 MiB RSS:and then the memory usage of the storage servers climbed uniformly to about 117 MiB RSS each:
and then more irregularly to a different amount for each server at the end of the command, while the gateway usage dropped to about 746 MiB RSS:
There seems to be quite a severe memory leak, since these figures hadn't decreased 20 minutes later.
I restarted all the nodes, and repeated the experiment using
bin/tahoe put --mutable --mutable-type=mdmf zeros <URI from previous run>
, i.e. an update rather than an initial upload. The results were similar except that the peak RSS of the gateway process was 1322.5 MiB, and the final usages were:(i.e. 684.5 MiB RSS for the gateway). Again these hadn't decreased several minutes later.
I restarted all the nodes again, and logged in using
sftp -P 8022 127.0.0.1
. The memory usage at that point was:I issued the SFTP client command
put zeros /uri/<URI from first run>
, i.e. updating the same MDMF file.The results were almost identical to the first experiment (except that the gateway memory usage increase only started after the file had been completely received over SFTP). The peak RSS of the gateway was 1386 MiB, and the final usages were:
(i.e. 745.6 MiB RSS for the gateway). These also hadn't decreased several minutes later, or on closing the SFTP connection.
This doesn't support my contention in comment:3 or the original description, that the web-API usage is higher.
MDMF upload via web-API uses much more memory in the gateway process than updating the same file via SFTPto MDMF upload via web-API uses much more memory in the gateway process than expectedWhen uploading an immutable file using
bin/tahoe put zeros
(again after restarting the nodes), the memory usage did not increase above:(I tried again using data from
/dev/urandom
in case of some optimization involving zero memory pages, with the same results.)I don't understand how the gateway avoided holding the file in memory. Did somebody fix #320 while I wasn't looking? :-)
Replying to davidsarah:
Oh, it was holding the data in a temporary file.
For comparison, SDMF (still using 1.9alpha), has a peak gateway memory usage of 1314.5 MiB RSS, and then toward the end of the upload:
After the upload it went back down to
So, there is no memory leak in the gateway for SDMF, although it seems that the storage servers are taking a while to release their memory.
Edit: the memory usage of server1 and server3 was still this high after several hours.
For an upload using
tahoe put --mutable --mutable-type=mdmf
of a 100000000-byte file, the peak gateway memory usage is 713.8 MiB RSS, and the final usage is:(i.e. 395.2 MiB RSS for the gateway).
So, the major part of the leak is, unsurprisingly, roughly proportional to the filesize.
I repeated the SDMF test with the v1.8.2 release. There doesn't appear to be any memory regression in 1.9alpha upload (I haven't tried download) with respect to SDMF. The leak in MDMF is serious enough that we still might want to block 1.9beta to fix it, though.
Brian said on IRC that he didn't want to treat this as a blocker for v1.9. Presumably his reasoning is that it isn't a regression—nothing in MDMF can be a regression. :-) And also nothing in MDMF is turned on by default for v1.9. Still, this is pretty disappointing behavior. It means people cannot upload (put) MDMFs larger than their RAM (or actually than about 1/6 of their available RAM)? If we're going to ship v1.9 with this behavior, we certainly need to make sure source:docs/performance.rst tells about it!
Brian also said that he thought fixing it would require a new method added to the API of [RIStorageServer]source:trunk/src/allmydata/interfaces.py?annotate=blame&rev=5212#L91. I can't imagine why that would be, but I'm way too sleepy at this point to think about such things.
Duplicate of #1513. I'd forgotten that ticket when I filed this one.