cloud backend spams twistd.log with two entries for each HTTP connection #1927

New Issue

This happened to a LeastAuthority customer server, exhausting its EBS disk space. The culprit appears to be the logging when the noisy attribute is set to True on an HTTPClientFactory (which it is by default), for example here in twisted/internet/protocol.py.

I think (but have no way to test from where I am now) that it can be fixed by changing this line in s3_container.py to:

    query = Query(*args, **kwargs)
    if hasattr(query.factory, 'noisy'):
        query.factory.noisy = False
    return query

Note that this only changes the S3 container; other cloud containers also use HTTPClientFactory and will need changing.

This happened to a LeastAuthority customer server, exhausting its EBS disk space. The culprit appears to be the logging when the `noisy` attribute is set to `True` on an `HTTPClientFactory` (which it is by default), for example [here in twisted/internet/protocol.py](https://github.com/twisted/twisted/blob/63be1f698a60d572fc6436cbc0af9e0fcff9713e/twisted/internet/protocol.py#L70). I think (but have no way to test from where I am now) that it can be fixed by changing [this line in s3_container.py](https://github.com/tahoe-lafs/tahoe-lafs/blob/2237-cloud-backend-s4/src/allmydata/storage/backends/cloud/s3/s3_container.py#L55) to: ``` query = Query(*args, **kwargs) if hasattr(query.factory, 'noisy'): query.factory.noisy = False return query ``` Note that this only changes the S3 container; other cloud containers also use `HTTPClientFactory` and will need changing.

daira commented

2014-06-28 16:24:19 +00:00

Replying to daira:

I think (but have no way to test from where I am now) that it can be fixed by changing this line in s3_container.py to:
    query = Query(*args, **kwargs)
    if hasattr(query.factory, 'noisy'):
        query.factory.noisy = False
    return query

Actually that won't quite work because that code is conditional on usertoken being not None (i.e. on DevPay being used). It would have to be made unconditional, or the noisy attribute reset to False in txAWS (just after here) instead.

Replying to [daira](/tahoe-lafs/trac-2024-07-25/issues/1927#issuecomment-91131): > I think (but have no way to test from where I am now) that it can be fixed by changing [this line in s3_container.py](https://github.com/tahoe-lafs/tahoe-lafs/blob/2237-cloud-backend-s4/src/allmydata/storage/backends/cloud/s3/s3_container.py#L55) to: > ``` > query = Query(*args, **kwargs) > if hasattr(query.factory, 'noisy'): > query.factory.noisy = False > return query > ``` Actually that won't quite work because that code is conditional on `usertoken` being `not None` (i.e. on DevPay being used). It would have to be made unconditional, or the `noisy` attribute reset to `False` in txAWS (just after [here](https://github.com/LeastAuthority/txaws/blob/master/txaws/client/base.py#L79)) instead.

daira commented

2016-09-20 13:49:27 +00:00

It needs to be like this:

[...]
endpoint = AWSServiceEndpoint(uri=url)

def make_query(*args, **kwargs):
    if usertoken is not None:
        amz_headers = kwargs.get("amz_headers", {})
        if producttoken is not None:
            amz_headers["security-token"] = (usertoken, producttoken)
        else:
            amz_headers["security-token"] = usertoken
        kwargs["amz_headers"] = amz_headers

    query = Query(*args, **kwargs)
    if hasattr(query.factory, 'noisy'):
        query.factory.noisy = False
    return query

self.client = S3Client(creds=creds, endpoint=endpoint, query_factory=make_query)
self.ServiceError = S3Error

It needs to be like this: ``` [...] endpoint = AWSServiceEndpoint(uri=url) def make_query(*args, **kwargs): if usertoken is not None: amz_headers = kwargs.get("amz_headers", {}) if producttoken is not None: amz_headers["security-token"] = (usertoken, producttoken) else: amz_headers["security-token"] = usertoken kwargs["amz_headers"] = amz_headers query = Query(*args, **kwargs) if hasattr(query.factory, 'noisy'): query.factory.noisy = False return query self.client = S3Client(creds=creds, endpoint=endpoint, query_factory=make_query) self.ServiceError = S3Error ```

vu3rdd commented

2016-09-27 05:38:15 +00:00

I just incorporated it in a pull request here: https://github.com/tahoe-lafs/tahoe-lafs/pull/355

I just incorporated it in a pull request here: <https://github.com/tahoe-lafs/tahoe-lafs/pull/355>

exarkun commented

2020-10-30 12:35:44 +00:00

The established line of development on the "cloud backend" branch has been abandoned. This ticket is being closed as part of a batch-ticket cleanup for "cloud backend"-related tickets.

If this is a bug, it is probably genuinely no longer relevant. The "cloud backend" branch is too large and unwieldy to ever be merged into the main line of development (particularly now that the Python 3 porting effort is significantly underway).

If this is a feature, it may be relevant to some future efforts - if they are sufficiently similar to the "cloud backend" effort - but I am still closing it because there are no immediate plans for a new development effort in such a direction.

Tickets related to the "leasedb" are included in this set because the "leasedb" code is in the "cloud backend" branch and fairly well intertwined with the "cloud backend". If there is interest in lease implementation change at some future time then that effort will essentially have to be restarted as well.

The established line of development on the "cloud backend" branch has been abandoned. This ticket is being closed as part of a batch-ticket cleanup for "cloud backend"-related tickets. If this is a bug, it is probably genuinely no longer relevant. The "cloud backend" branch is too large and unwieldy to ever be merged into the main line of development (particularly now that the Python 3 porting effort is significantly underway). If this is a feature, it may be relevant to some future efforts - if they are sufficiently similar to the "cloud backend" effort - but I am still closing it because there are no immediate plans for a new development effort in such a direction. Tickets related to the "leasedb" are included in this set because the "leasedb" code is in the "cloud backend" branch and fairly well intertwined with the "cloud backend". If there is interest in lease implementation change at some future time then that effort will essentially have to be restarted as well.

exarkun added the

wontfix

label 2020-10-30 12:35:44 +00:00

exarkun closed this issue

2020-10-30 12:35:44 +00:00

Sign in to join this conversation.