why are so many helper files being abandoned? #395

New Issue

warner · 2008-04-25T00:09:22Z

warner commented

2008-04-25 00:09:22 +00:00

Our prodnet munin graphs show that our helper is receiving ciphertext for a lot of files which are then abandoned, either part-way through the ciphertext fetch phase, or during the encode+push phase. The clients then never try again.

add code (or admin tools) to delete old files from the helper's
directory after a while. We have something like 38GB of files in
there right now
figure out where these are coming from. It must be the windows
FUSE plugin (that's the only deployment we have on prodnet that
uses the helper), but what sorts of files are so large and then
abandoned? It's possible that encode+push takes so long that
the application gives up and tries again (with a new and
different file).

Our prodnet munin graphs show that our helper is receiving ciphertext for a lot of files which are then abandoned, either part-way through the ciphertext fetch phase, or during the encode+push phase. The clients then never try again. * add code (or admin tools) to delete old files from the helper's directory after a while. We have something like 38GB of files in there right now * figure out where these are coming from. It must be the windows FUSE plugin (that's the only deployment we have on prodnet that uses the helper), but what sorts of files are so large and then abandoned? It's possible that encode+push takes so long that the application gives up and tries again (with a new and different file).

warner added the

labels 2008-04-25 00:09:22 +00:00

warner added this to the eventually milestone 2008-04-25 00:09:22 +00:00

warner commented

2008-06-01 20:18:01 +00:00

I added cron jobs to delete files from CHK_encoding/ after one week, and from CHK_incoming/ after two weeks.

warner modified the milestone from eventually to undecided

2008-06-01 20:48:17 +00:00

davidsarah commented

2013-03-17 17:14:48 +00:00

According to source:git/docs/helper.rst#setting-up-a-helper, "future versions of tahoe" are supposed to delete abandoned upload files on the helper:

If a client disconnects while the ciphertext is being fetched, the
partial ciphertext will remain in CHK_incoming/ until they reconnect
and finish sending it. If a client disconnects while the ciphertext
is being encoded, the data will remain in CHK_encoding/ until they
reconnect and encoding is finished. For long-running and busy helpers,
it may be a good idea to delete files in these directories that have
not been modified for a week or two. Future versions of tahoe will
try to self-manage these files a bit better.

According to source:git/docs/helper.rst#setting-up-a-helper, "future versions of tahoe" are supposed to delete abandoned upload files on the helper: ``` If a client disconnects while the ciphertext is being fetched, the partial ciphertext will remain in CHK_incoming/ until they reconnect and finish sending it. If a client disconnects while the ciphertext is being encoded, the data will remain in CHK_encoding/ until they reconnect and encoding is finished. For long-running and busy helpers, it may be a good idea to delete files in these directories that have not been modified for a week or two. Future versions of tahoe will try to self-manage these files a bit better. ```

tahoe-lafs modified the milestone from undecided to eventually

2013-03-17 17:14:48 +00:00

Sign in to join this conversation.