cloud backend: crawlers are running too often #1886

New Issue

tahoe-lafs · 2012-12-05T03:49:05Z

davidsarah commented

2012-12-05 03:49:05 +00:00

I flogtool tailed the merged cloud backend and leasedb (as of 1819-cloud-merge/022796fb), and observed that the bucket crawler and accounting crawler were running much too often. This was confirmed by looking at the storage status page at /storage, which said that the bucket crawler and accounting crawler would run in 2 minutes and 7 minutes respectively, after they had run a few minutes ago.

The bucket crawler is supposed to have a minimum cycle time of one hour, and the accounting crawler 12 hours.

I `flogtool tail`ed the merged cloud backend and leasedb (as of [1819-cloud-merge/022796fb](https://github.com/davidsarah/tahoe-lafs/commit/022796fb7813c2f42d668a0ee3de9abae869deb5)), and observed that the bucket crawler and accounting crawler were running much too often. This was confirmed by looking at the storage status page at `/storage`, which said that the bucket crawler and accounting crawler would run in 2 minutes and 7 minutes respectively, after they had run a few minutes ago. The bucket crawler is supposed to have a minimum cycle time of one hour, and the accounting crawler 12 hours.

tahoe-lafs added the

labels 2012-12-05 03:49:05 +00:00

tahoe-lafs added this to the 1.11.0 milestone 2012-12-05 03:49:05 +00:00

tahoe-lafs modified the milestone from 1.11.0 to 1.12.0

2013-07-22 20:48:52 +00:00

zooko commented

2013-11-14 19:05:07 +00:00

This would be fixed by #1834.

daira commented

2013-11-21 22:55:06 +00:00

Replying to zooko:

This would be fixed by #1834.

Well, not unless we removed all use of crawlers, which I don't think #1834 proposed to do.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1886#issuecomment-90578): > This would be fixed by #1834. Well, not unless we removed *all* use of crawlers, which I don't think #1834 proposed to do.

zooko commented

2013-11-22 05:53:28 +00:00

Replying to [daira]comment:4:

Replying to zooko:

This would be fixed by #1834.

Well, not unless we removed all use of crawlers, which I don't think #1834 proposed to do.

IIRC, it proposes to remover all crawlers except for this one: #1835, and that one would be executed only in specific circumstances, not executing automatically over-and-over, so this ticket would almost certainly be obsolete.

Replying to [daira]comment:4: > Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1886#issuecomment-90578): > > This would be fixed by #1834. > > Well, not unless we removed *all* use of crawlers, which I don't think #1834 proposed to do. IIRC, it proposes to remover all crawlers except for this one: #1835, and that one would be executed only in specific circumstances, not executing automatically over-and-over, so this ticket would almost certainly be obsolete.

daira commented

2013-11-23 13:15:29 +00:00

Replying to [zooko]comment:5:

IIRC, it proposes to remover all crawlers except for this one: #1835, and that one would be executed only in specific circumstances, not executing automatically over-and-over, so this ticket would almost certainly be obsolete.

Oh, good point. Nevertheless I want to fix this bug before then, because I think it's shallow and could easily be fixed in an earlier release.

Replying to [zooko]comment:5: > IIRC, it proposes to remover all crawlers except for this one: #1835, and that one would be executed only in specific circumstances, not executing automatically over-and-over, so this ticket would almost certainly be obsolete. Oh, good point. Nevertheless I want to fix this bug before then, because I think it's shallow and could easily be fixed in an earlier release.

zooko commented

2014-12-03 04:23:17 +00:00

Hm, I wonder if this could have to do with the out-of-file-descriptors issue that my client ("WAG") is experiencing — #2342.

dquintela commented

2014-12-23 10:41:48 +00:00

Hello, first time tahoe user here,

I've testing cloud storage for 3 or 4 days (branch 2237-cloud-backend-s4) into my raspberry-pi, running raspbian.
Despite it having very long startup times, I noticed .tahoe/logs/twistd.log filled up of lines like this,
that seems indicating the shares crawler is being run to often. This has the nasty side effecting that on amazon billing page
I am already with 80000 requests. Rough estimate this could mean 3 to 5 USD per month on an idle storage node alone.

This seems to be related to #1835 and #1886 - sorry for the crossposting.

2014-12-22 15:48:37+0000 [-] Starting factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fod%2F>
2014-12-22 15:48:37+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fod%2F>
2014-12-22 15:48:37+0000 [-] Starting factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Foe%2F>
2014-12-22 15:48:38+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Foe%2F>
2014-12-22 15:48:38+0000 [-] Starting factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fof%2F>
2014-12-22 15:48:38+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fof%2F>
2014-12-22 15:48:38+0000 [-] Starting factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fog%2F>
2014-12-22 15:48:38+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fog%2F>
...

Hello, first time tahoe user here, I've testing cloud storage for 3 or 4 days (branch 2237-cloud-backend-s4) into my raspberry-pi, running raspbian. Despite it having very long startup times, I noticed .tahoe/logs/twistd.log filled up of lines like this, that seems indicating the shares crawler is being run to often. This has the nasty side effecting that on amazon billing page I am already with 80000 requests. Rough estimate this could mean 3 to 5 USD per month on an idle storage node alone. This seems to be related to #1835 and #1886 - sorry for the crossposting. 2014-12-22 15:48:37+0000 [-] Starting factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fod%2F>> 2014-12-22 15:48:37+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fod%2F>> 2014-12-22 15:48:37+0000 [-] Starting factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Foe%2F>> 2014-12-22 15:48:38+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Foe%2F>> 2014-12-22 15:48:38+0000 [-] Starting factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fof%2F>> 2014-12-22 15:48:38+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fof%2F>> 2014-12-22 15:48:38+0000 [-] Starting factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fog%2F>> 2014-12-22 15:48:38+0000 HTTPPageGetter,client Stopping factory <HTTPClientFactory: <http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fog%2F>> ...

warner commented

2016-03-22 05:02:25 +00:00

Milestone renamed

warner modified the milestone from 1.12.0 to 1.13.0

2016-03-22 05:02:25 +00:00

warner commented

2016-06-28 18:17:14 +00:00

renaming milestone

warner modified the milestone from 1.13.0 to 1.14.0

2016-06-28 18:17:14 +00:00

exarkun commented

2020-06-30 14:45:13 +00:00

Moving open issues out of closed milestones.

exarkun modified the milestone from 1.14.0 to 1.15.0

2020-06-30 14:45:13 +00:00

exarkun commented

2020-10-30 12:35:44 +00:00

The established line of development on the "cloud backend" branch has been abandoned. This ticket is being closed as part of a batch-ticket cleanup for "cloud backend"-related tickets.

If this is a bug, it is probably genuinely no longer relevant. The "cloud backend" branch is too large and unwieldy to ever be merged into the main line of development (particularly now that the Python 3 porting effort is significantly underway).

If this is a feature, it may be relevant to some future efforts - if they are sufficiently similar to the "cloud backend" effort - but I am still closing it because there are no immediate plans for a new development effort in such a direction.

Tickets related to the "leasedb" are included in this set because the "leasedb" code is in the "cloud backend" branch and fairly well intertwined with the "cloud backend". If there is interest in lease implementation change at some future time then that effort will essentially have to be restarted as well.

The established line of development on the "cloud backend" branch has been abandoned. This ticket is being closed as part of a batch-ticket cleanup for "cloud backend"-related tickets. If this is a bug, it is probably genuinely no longer relevant. The "cloud backend" branch is too large and unwieldy to ever be merged into the main line of development (particularly now that the Python 3 porting effort is significantly underway). If this is a feature, it may be relevant to some future efforts - if they are sufficiently similar to the "cloud backend" effort - but I am still closing it because there are no immediate plans for a new development effort in such a direction. Tickets related to the "leasedb" are included in this set because the "leasedb" code is in the "cloud backend" branch and fairly well intertwined with the "cloud backend". If there is interest in lease implementation change at some future time then that effort will essentially have to be restarted as well.

exarkun added the

wontfix

label 2020-10-30 12:35:44 +00:00

exarkun closed this issue

2020-10-30 12:35:44 +00:00

Sign in to join this conversation.