Unicode stdout/stderr replacement on Windows fails to print large strings #1232

New Issue

tahoe-lafs · 2010-10-27T02:29:48Z

davidsarah commented

2010-10-27 02:29:48 +00:00

While trying to debug #1045, I tried to print a ResponseCache object and got the following exception:

  File "d:\tahoe\tahoe-1.8.0c2\src\allmydata\windows\fixups.py", line 134, in write
    raise IOError("WriteConsoleW returned %r, n.value = %r" % (retval, n.value))
exceptions.IOError: WriteConsoleW returned 0, n.value = 0L

This is a bug in the Unicode stdout/stderr replacement in source:src/allmydata/windows/fixups.py that was added in Tahoe-LAFS v1.8 beta. On my machine it fails when trying to print more than 26608 characters at once to stdout or stderr. This appears to be an undocumented limitation of the Windows WriteConsoleW API call.

Technically this is a regression since v1.7.1, although we very rarely print a string that large. The fix is trivial; just limit the length of data passed in a single call to WriteConsoleW.

While trying to debug #1045, I tried to print a `ResponseCache` object and got the following exception: ``` File "d:\tahoe\tahoe-1.8.0c2\src\allmydata\windows\fixups.py", line 134, in write raise IOError("WriteConsoleW returned %r, n.value = %r" % (retval, n.value)) exceptions.IOError: WriteConsoleW returned 0, n.value = 0L ``` This is a bug in the Unicode stdout/stderr replacement in source:src/allmydata/windows/fixups.py that was added in Tahoe-LAFS v1.8 beta. On my machine it fails when trying to print more than 26608 characters at once to stdout or stderr. This appears to be an undocumented limitation of the Windows `WriteConsoleW` API call. Technically this is a regression since v1.7.1, although we very rarely print a string that large. The fix is trivial; just limit the length of data passed in a single call to `WriteConsoleW`.

tahoe-lafs added the

labels 2010-10-27 02:29:48 +00:00

tahoe-lafs added this to the 1.8.1 milestone 2010-10-27 02:29:48 +00:00

davidsarah commented

2010-10-27 02:36:12 +00:00

Correction: the [MSDN documentation for WriteConsoleW](http://msdn.microsoft.com/en-us/library/ms687401%28VS.85%29.aspx) does document a length limitation, but it says 64 KiB (which would be 32767 UTF-16 characters and a terminating NUL), not 26608 characters. The fix is the same regardless; I'll limit it to 10000 UTF-16 characters.

Correction: the [MSDN documentation for [WriteConsole](wiki/WriteConsole)W](http://msdn.microsoft.com/en-us/library/ms687401%28VS.85%29.aspx) does document a length limitation, but it says 64 KiB (which would be 32767 UTF-16 characters and a terminating NUL), not 26608 characters. The fix is the same regardless; I'll limit it to 10000 UTF-16 characters.

davidsarah commented

2010-10-27 02:42:39 +00:00

Attachment workaround-windows-writeconsole-suckage.darcs.patch (5408 bytes) added

windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.

**Attachment** workaround-windows-writeconsole-suckage.darcs.patch (5408 bytes) added windows/fixups.py: limit length of string passed in a single call to [WriteConsole](wiki/WriteConsole)W. fixes #1232.

workaround-windows-writeconsole-suckage.darcs.patch

5.3 KiB

warner commented

2010-10-29 09:13:19 +00:00

looks good to me.

david-sarah@jacaranda.org commented

2010-10-29 19:43:09 +00:00

In changeset:25d8103dde95d784:

windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.

In changeset:25d8103dde95d784: ``` windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232. ```

tahoe-lafs added the

fixed

label 2010-10-29 19:43:09 +00:00

tahoe-lafs closed this issue

2010-10-29 19:43:09 +00:00

david-sarah@jacaranda.org commented

2010-10-29 19:43:14 +00:00

In changeset:2610f8e0aa6e2221:

NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232

In changeset:2610f8e0aa6e2221: ``` NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232 ```

davidsarah commented

2011-01-09 08:47:29 +00:00

Thread about this issue that seems to suggest the limit varies between machines:

(@@http://www.mail-archive.com/log4net-dev@logging.apache.org/msg00661.html@@)

I have not seen any reports of it failing with < 10000 characters, so our current code should work. However, I'm attempting to file the bug with Microsoft's bug reporting system, such as it is.

Thread about this issue that seems to suggest the limit varies between machines: (@@http://www.mail-archive.com/log4net-dev@logging.apache.org/msg00661.html@@) I have not seen any reports of it failing with < 10000 characters, so our current code should work. However, I'm attempting to file the bug with Microsoft's bug reporting system, such as it is.

davidsarah commented

2011-03-27 14:13:00 +00:00

It is possible for calls to the console functions by multiple threads to cause this error even when the amount written in each call is limited to 10000 characters (20000 bytes). However Tahoe-LAFS is single-threaded (almost; there are a few uses of deferToThread, but I don't think we print from those threads), so this shouldn't affect us. I'm pointing it out here because I know this ticket is referenced from tickets in other projects.

# Warning: this test may DoS your system.

from threading import Thread
import sys
from ctypes import WINFUNCTYPE, windll, POINTER, byref
from ctypes.wintypes import BOOL, HANDLE, DWORD, LPVOID, LPCVOID

GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)(("GetStdHandle", windll.kernel32))
WriteFile = WINFUNCTYPE(BOOL, HANDLE, LPCVOID, DWORD, POINTER(DWORD), LPVOID) \
                        (("WriteFile", windll.kernel32))
GetLastError = WINFUNCTYPE(DWORD)(("GetLastError", windll.kernel32))
STD_OUTPUT_HANDLE = DWORD(-11)
INVALID_HANDLE_VALUE = DWORD(-1).value

hStdout = GetStdHandle(STD_OUTPUT_HANDLE)
assert hStdout is not None and hStdout != INVALID_HANDLE_VALUE

LENGTH = 20000 #bytes
THREADS = 10

data = b'\x08'*LENGTH

def run():
    n = DWORD(0)
    while True:
        ret = WriteFile(hStdout, data, LENGTH, byref(n), None)
        if ret == 0 or n.value != LENGTH:
            print("WriteFile returned %d, bytes written = %d, last error = %d"
                  % (ret, n.value, GetLastError()))
            sys.exit(1)

for i in range(THREADS):
    Thread(target=run).start()

It is possible for calls to the console functions by multiple threads to cause this error even when the amount written in each call is limited to 10000 characters (20000 bytes). However Tahoe-LAFS is single-threaded (almost; there are a few uses of `deferToThread`, but I don't think we print from those threads), so this shouldn't affect us. I'm pointing it out here because I know this ticket is referenced from tickets in other projects. ``` # Warning: this test may DoS your system. from threading import Thread import sys from ctypes import WINFUNCTYPE, windll, POINTER, byref from ctypes.wintypes import BOOL, HANDLE, DWORD, LPVOID, LPCVOID GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)(("GetStdHandle", windll.kernel32)) WriteFile = WINFUNCTYPE(BOOL, HANDLE, LPCVOID, DWORD, POINTER(DWORD), LPVOID) \ (("WriteFile", windll.kernel32)) GetLastError = WINFUNCTYPE(DWORD)(("GetLastError", windll.kernel32)) STD_OUTPUT_HANDLE = DWORD(-11) INVALID_HANDLE_VALUE = DWORD(-1).value hStdout = GetStdHandle(STD_OUTPUT_HANDLE) assert hStdout is not None and hStdout != INVALID_HANDLE_VALUE LENGTH = 20000 #bytes THREADS = 10 data = b'\x08'*LENGTH def run(): n = DWORD(0) while True: ret = WriteFile(hStdout, data, LENGTH, byref(n), None) if ret == 0 or n.value != LENGTH: print("WriteFile returned %d, bytes written = %d, last error = %d" % (ret, n.value, GetLastError())) sys.exit(1) for i in range(THREADS): Thread(target=run).start() ```

Sign in to join this conversation.