DeprecationWarning about passing non-bytes header values #2312

New Issue

tahoe-lafs · 2014-09-30T19:35:49Z

daira commented

2014-09-30 19:35:49 +00:00

(https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Marcus%20Cygwin%20WinXP/builds/26/steps/test/logs/stdio)

/home/marcus/bbtahoe/Marcus_Cygwin_WinXP/build/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-cygwin-1.7.9-i686.egg/twisted/web/server.py:220:
DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead.

I don't know why this is warning only on the "Marcus Cygwin WinXP" builder.

(https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Marcus%20Cygwin%20WinXP/builds/26/steps/test/logs/stdio) ``` /home/marcus/bbtahoe/Marcus_Cygwin_WinXP/build/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-cygwin-1.7.9-i686.egg/twisted/web/server.py:220: DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead. ``` I don't know why this is warning only on the "Marcus Cygwin WinXP" builder.

tahoe-lafs added the

labels 2014-09-30 19:35:49 +00:00

tahoe-lafs added this to the undecided milestone 2014-09-30 19:35:49 +00:00

daira commented

2014-09-30 19:41:17 +00:00

Ah, it is also happening on the Python 2.6 Travis build: https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/36701619

The common factor is Python 2.6.

Ah, it is also happening on the Python 2.6 Travis build: <https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/36701619> The common factor is Python 2.6.

tahoe-lafs modified the milestone from undecided to 1.11.0

2014-10-02 01:03:42 +00:00

warner commented

2015-01-20 17:49:16 +00:00

This is low-priority, but it'd be nice to fix. It appears that something is calling http.Request.write with a unicode string, but we need a larger stack trace to figure out what's happening.

warner added

minor

and removed

normal

labels 2015-01-20 17:49:16 +00:00

warner commented

2015-01-21 00:46:50 +00:00

A minimal STR is to test test_web.Web.test_GET_DIRURL_json under python2.6 .

It turns out that the twisted.web code is expecting the Content-Length header to contain bytes (like all other headers, who knew?). Three Tahoe code paths (in web/common.py and web/filenode.py) fill it with an integer, which takes advantage of a ("%s" % value) fallback.

A DeprecationWarning to complain about non-bytes was added in Twisted-12.3, probably to guard against unicode header values, but in this case it's triggering on our IntType. Twisted-12.3 is also when twisted.python.compat.intToBytes(n) was added to make this easier, but it isn't present in 11.0-12.2, which is what we use on windows. So we'll need our own intToBytes function.

A minimal STR is to test `test_web.Web.test_GET_DIRURL_json` under python2.6 . It turns out that the twisted.web code is expecting the `Content-Length` header to contain bytes (like all other headers, who knew?). Three Tahoe code paths (in web/common.py and web/filenode.py) fill it with an integer, which takes advantage of a `("%s" % value)` fallback. A DeprecationWarning to complain about non-bytes was added in Twisted-12.3, probably to guard against unicode header values, but in this case it's triggering on our `IntType`. Twisted-12.3 is also when `twisted.python.compat.intToBytes(n)` was added to make this easier, but it isn't present in 11.0-12.2, which is what we use on windows. So we'll need our own `intToBytes` function.

warner commented

2015-01-21 01:48:27 +00:00

Oh, and category=DeprecationWarning is globally disabled in python2.7, because(?) they assumed it would only be used for features implemented by python itself, and "2.7 is the last one". I think that's why this doesn't appear in py2.7 .

Sigh, that seems like a dangerous misfeature to me. Should Twisted (and other projects) use a subclass or alternative to DeprecationWarning?

Oh, and `category=DeprecationWarning` is globally disabled in python2.7, because(?) they assumed it would only be used for features implemented by python itself, and "2.7 is the last one". I think that's why this doesn't appear in py2.7 . Sigh, that seems like a dangerous misfeature to me. Should Twisted (and other projects) use a subclass or alternative to `DeprecationWarning`?

warner commented

2015-01-21 08:48:28 +00:00

After fixing that, I found one other instance, when various web requests use nevow.rend.Page's feature of self.addSlash==True. This code path causes setHeader("location", url) to be called with a nevow.url.URL instance, instead of a string. There's no good way for us to fix this from tahoe.. the fix really needs to be in Nevow itself. Filed as https://github.com/twisted/nevow/issues/52 .

After fixing that, I found one other instance, when various web requests use `nevow.rend.Page`'s feature of `self.addSlash==True`. This code path causes `setHeader("location", url)` to be called with a `nevow.url.URL` instance, instead of a string. There's no good way for us to fix this from tahoe.. the fix really needs to be in Nevow itself. Filed as <https://github.com/twisted/nevow/issues/52> .

warner commented

2015-01-21 18:33:42 +00:00

Ok, PR in https://github.com/tahoe-lafs/tahoe-lafs/pull/137 , ready for review. This will fix the immediate problem, but there will probably be others (including ones we can't fix because they're in Nevow or other libraries). I'll work on setting up a buildslave that will run tests with PYTHONWARNINGS=default::DeprecationWarning so we can at least see where they're happening.

Ok, PR in <https://github.com/tahoe-lafs/tahoe-lafs/pull/137> , ready for review. This will fix the immediate problem, but there will probably be others (including ones we can't fix because they're in Nevow or other libraries). I'll work on setting up a buildslave that will run tests with `PYTHONWARNINGS=default::DeprecationWarning` so we can at least see where they're happening.

warner commented

2015-01-21 18:44:23 +00:00

#2366 created to remind me about the warnings-scanning buildbot tool

daira commented

2015-01-22 02:55:46 +00:00

[6f74282688a871c5ef8cb0d472b2050edc8e6102/trunk] has the patch but it does not seem to have fixed the warnings: https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/47869702

[6f74282688a871c5ef8cb0d472b2050edc8e6102/trunk] has the patch but it does not seem to have fixed the warnings: <https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/47869702>

zooko commented

2015-01-22 17:29:06 +00:00

removing review-needed tag (which by the way was incorrectly spelled needs-review)

The next step is: figure out why the patch didn't fix the issue?

removing `review-needed` tag (which by the way was incorrectly spelled `needs-review`) The next step is: figure out why the patch didn't fix the issue?

warner commented

2015-01-27 08:38:25 +00:00

I suspect the remaining warnings are resulting from Nevow's addSlash=True problem. I don't think we can fix that within Tahoe, but I could be wrong.

I think we've gotten as much out of this ticket as we can for 1.10.1 . I propose we bump it to 1.11 at this week's triage.

I suspect the remaining warnings are resulting from Nevow's `addSlash=True` problem. I don't think we can fix that within Tahoe, but I could be wrong. I think we've gotten as much out of this ticket as we can for 1.10.1 . I propose we bump it to 1.11 at this week's triage.

daira commented

2015-01-27 18:39:02 +00:00

If we can't fix it, should we suppress the warning? We have done so for previous warnings that could only be fixed in dependencies.

warner commented

2015-03-10 16:36:35 +00:00

let's make sure that this has been reported upstream to Nevow, if possible

warner commented

2015-03-20 08:34:19 +00:00

Nevow bug filed: https://github.com/twisted/nevow/issues/52 . (oh, as I reported earlier in comment:95954, nevermind).

Buildbot is now scanning for deprecation warnings, https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Ubuntu%20trusty%2014.04/builds/45/steps/deprecations/logs/warnings has a list. This is a simple regexp scan, so some of the lines are coming from the tests that fail (due to stderr not being as quiet as it's supposed to be):

/home/bb-tahoe/bb-tahoe/Ubuntu_trusty_14_04/build/support/lib/python2.7/site-packages/Nevow-0.11.1-py2.7.egg/nevow/appserver.py:129: DeprecationWarning: twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0: Please use twisted.web.http.Request.requestHeaders instead.
/usr/lib/python2.7/dist-packages/twisted/web/server.py:217: DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead.
a = '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n  import sets as _sets\n'
twisted.trial.unittest.FailTest: ('allmydata-tahoe: 1.10.0.post287 [master: 681eb4d12e20d8818ee02baab7b55230a706e76d]\nfoolscap: 0.6.4\npycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958\nzfec: 1.4.5\nTwisted: 13.2.0\nNevow: 0.11.1\nzope.interface: unknown\npython: 2.7.6\nplatform: Linux-Ubuntu_14.04-x86_64-32bit_ELF\npyOpenSSL: 0.13\nsimplejson: 3.3.1\npycrypto: 2.6.1\npyasn1: 0.1.7\nmock: 1.0.1\nsetuptools: 0.6c16dev5\nservice-identity: 14.0.0\ncharacteristic: 14.3.0\npyasn1-modules: 0.0.5\n\n', '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n  import sets as _sets\n', 0)
twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated

So there are three real warnings:

nevow/appserver.py:129: "twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0"
twisted/web/server.py:217: "Passing non-bytes header values is deprecated since Twisted 12.3"
twisted/spread/jelly.py:92: "the sets module is deprecated; import sets as _sets"

The first is filed as https://github.com/twisted/nevow/issues/53 , and the second as https://github.com/twisted/nevow/issues/52 . I don't know if the third is filed anywhere.

Nevow bug filed: <https://github.com/twisted/nevow/issues/52> . (oh, as I reported earlier in [comment:95954](/tahoe-lafs/trac-2024-07-25/issues/2312#issuecomment-95954), nevermind). Buildbot is now scanning for deprecation warnings, <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Ubuntu%20trusty%2014.04/builds/45/steps/deprecations/logs/warnings> has a list. This is a simple regexp scan, so some of the lines are coming from the tests that fail (due to stderr not being as quiet as it's supposed to be): ``` /home/bb-tahoe/bb-tahoe/Ubuntu_trusty_14_04/build/support/lib/python2.7/site-packages/Nevow-0.11.1-py2.7.egg/nevow/appserver.py:129: DeprecationWarning: twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0: Please use twisted.web.http.Request.requestHeaders instead. /usr/lib/python2.7/dist-packages/twisted/web/server.py:217: DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead. a = '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n import sets as _sets\n' twisted.trial.unittest.FailTest: ('allmydata-tahoe: 1.10.0.post287 [master: 681eb4d12e20d8818ee02baab7b55230a706e76d]\nfoolscap: 0.6.4\npycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958\nzfec: 1.4.5\nTwisted: 13.2.0\nNevow: 0.11.1\nzope.interface: unknown\npython: 2.7.6\nplatform: Linux-Ubuntu_14.04-x86_64-32bit_ELF\npyOpenSSL: 0.13\nsimplejson: 3.3.1\npycrypto: 2.6.1\npyasn1: 0.1.7\nmock: 1.0.1\nsetuptools: 0.6c16dev5\nservice-identity: 14.0.0\ncharacteristic: 14.3.0\npyasn1-modules: 0.0.5\n\n', '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n import sets as _sets\n', 0) twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated ``` So there are three real warnings: * nevow/appserver.py:129: "twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0" * twisted/web/server.py:217: "Passing non-bytes header values is deprecated since Twisted 12.3" * twisted/spread/jelly.py:92: "the sets module is deprecated; import sets as _sets" The first is filed as <https://github.com/twisted/nevow/issues/53> , and the second as <https://github.com/twisted/nevow/issues/52> . I don't know if the third is filed anywhere.

warner commented

2015-03-26 01:56:41 +00:00

All the bugs (that might ever get fixed) are filed, a separate ticket will track the creation of a buildbot to look for more. I think this ticket is done.

warner added the

fixed

label 2015-03-26 01:56:41 +00:00

warner closed this issue

2015-03-26 01:56:41 +00:00

Sign in to join this conversation.