tahoe-lafs/trac-2024-07-25
2
0
Issues 1.1k Wiki

Error from 'tahoe cp' on Windows, due to a long path (IOError: Errno2 - no such file or dir.) #2235

New Issue
Closed
opened 2014-04-28 20:36:11 +00:00 by CyberAxe · 29 comments
CyberAxe commented 2014-04-28 20:36:11 +00:00
Owner
Copy Link
6449/61880 files, 169/2003 directories
6450/61880 files, 169/2003 directories
Traceback (most recent call last):
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 141, in runner
    rc = cli.dispatchcommand(so)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\cli.py", line 551, in cp
    rc = tahoe_cp.copy(options)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 770, in copy
    return Copier().do_copy(options)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 451, in do_copy

    status = self.try_copy()
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 512, in try_cop
y
    return self.copy_to_directory(sources, target)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 672, in copy_to
_directory
    self.copy_files_to_target(self.targetmaptarget, target)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 703, in copy_fi
les_to_target
    self.copy_file_into(source, name, target)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 748, in copy_fi
le_into
    target.put_file(name, f)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 158, in put_fil
e
    fileutil.put_file(pathname, inf)
> File "c:\allmydata-tahoe-1.10.0\src\allmydata\util\fileutil.py", line 274, in put_file
    outf = open(os.path.expanduser(pathname), "wb")
IOError: [2]Errno No such file or directory: u'c:\\IveBeenRestored\\Marketing\\Photos\\As
sets\\042056-abstract-red-and-gold-paint-splatter-icon-sports-hobbies-camping\\042056-abst
ract-red-and-gold-paint-splatter-icon-sports-hobbies-camping\\042056-abstract-red-and-gold
-paint-splatter-icon-sports-hobbies-camping.png'

C:\allmydata-tahoe-1.10.0\bin>python tahoe cp -r -v tahoe:BackedUp/Latest c:\IveBeenRestor
ed\ >> c:\tahoe-restore.log
``` 6449/61880 files, 169/2003 directories 6450/61880 files, 169/2003 directories Traceback (most recent call last): > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 156, in run rc = runner(sys.argv[1:], install_node_control=install_node_control) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 141, in runner rc = cli.dispatchcommand(so) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\cli.py", line 551, in cp rc = tahoe_cp.copy(options) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 770, in copy return Copier().do_copy(options) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 451, in do_copy status = self.try_copy() > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 512, in try_cop y return self.copy_to_directory(sources, target) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 672, in copy_to _directory self.copy_files_to_target(self.targetmaptarget, target) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 703, in copy_fi les_to_target self.copy_file_into(source, name, target) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 748, in copy_fi le_into target.put_file(name, f) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_cp.py", line 158, in put_fil e fileutil.put_file(pathname, inf) > File "c:\allmydata-tahoe-1.10.0\src\allmydata\util\fileutil.py", line 274, in put_file outf = open(os.path.expanduser(pathname), "wb") IOError: [2]Errno No such file or directory: u'c:\\IveBeenRestored\\Marketing\\Photos\\As sets\\042056-abstract-red-and-gold-paint-splatter-icon-sports-hobbies-camping\\042056-abst ract-red-and-gold-paint-splatter-icon-sports-hobbies-camping\\042056-abstract-red-and-gold -paint-splatter-icon-sports-hobbies-camping.png' C:\allmydata-tahoe-1.10.0\bin>python tahoe cp -r -v tahoe:BackedUp/Latest c:\IveBeenRestor ed\ >> c:\tahoe-restore.log ```
tahoe-lafs added the
unknown
normal
defect
1.10.0
 labels 2014-04-28 20:36:11 +00:00
tahoe-lafs added this to the undecided milestone 2014-04-28 20:36:11 +00:00
daira commented 2014-04-29 16:53:16 +00:00
Author
Owner
Copy Link

This may be a result of the length of the path name; Windows APIs do not normally accept paths longer than 259 Unicode characters, unless the "\\?\" prefix is used. The path in question has 262 characters.

Most Windows software doesn't use that prefix and just breaks on long paths. I have never understood the motivation for the design that supports longer paths but requires you to jump through an obscure hoop to use them.

This may be a result of the length of the path name; Windows APIs [do not normally accept paths longer than 259 Unicode characters](http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx#maxpath), unless the "`\\?\`" prefix is used. The path in question has 262 characters. Most Windows software doesn't use that prefix and just breaks on long paths. I have never understood the motivation for the design that supports longer paths but requires you to jump through an obscure hoop to use them.
tahoe-lafs added
code-frontend-cli
 and removed
unknown
 labels 2014-04-29 16:56:12 +00:00
tahoe-lafs changed title from Error while restoring data - IOError: Errno2 - no such file or dir. to Error from 'tahoe cp' on Windows, possibly due to a long path (IOError: Errno2 - no such file or dir.) 2014-04-29 16:56:12 +00:00
daira commented 2014-04-30 00:15:23 +00:00
Author
Owner
Copy Link

http://bugs.python.org/issue18199. This will not be fixed in Python 2.x, so the only way for us to work around it is either to:

  1. chdir to each subdirectory in turn using relative paths (this changes the limitation to 255 characters per directory component);

  2. on Windows, call the Win32 API functions with the "\\?\" prefix directly via ctypes or similar.

  3. seems like too much work.

<http://bugs.python.org/issue18199>. This will not be fixed in Python 2.x, so the only way for us to work around it is either to: 1. chdir to each subdirectory in turn using relative paths (this changes the limitation to 255 characters per directory component); 2. on Windows, call the Win32 API functions with the "`\\?\`" prefix directly via `ctypes` or similar. 2. seems like too much work.
zooko commented 2014-04-30 00:37:08 +00:00
Copy Link

Replying to daira:

http://bugs.python.org/issue18199. This will not be fixed in Python 2.x, so the only way for us to work around it is either to:
a. chdir to each subdirectory in turn using relative paths (this changes the limitation to 255 characters per directory component);
b. on Windows, call the Win32 API functions with the "\\?\" prefix directly via ctypes or similar.

Why can't we just prepend "\\?\" to the string before we pass it to open() on line 274 of [fileutil.py]source:trunk/src/allmydata/util/fileutil.py?rev=ff64a0fef5879d3651bc3db6ca0522d96b217d45?

Replying to [daira](/tahoe-lafs/trac-2024-07-25/issues/2235#issuecomment-95007): > <http://bugs.python.org/issue18199>. This will not be fixed in Python 2.x, so the only way for us to work around it is either to: > a. chdir to each subdirectory in turn using relative paths (this changes the limitation to 255 characters per directory component); > b. on Windows, call the Win32 API functions with the "`\\?\`" prefix directly via `ctypes` or similar. Why can't we just prepend "`\\?\`" to the string before we pass it to `open()` on line 274 of [fileutil.py]source:trunk/src/allmydata/util/fileutil.py?rev=ff64a0fef5879d3651bc3db6ca0522d96b217d45?
daira commented 2014-04-30 00:45:21 +00:00
Author
Owner
Copy Link

Replying to [zooko]comment:4:

Why can't we just prepend "\\?\" to the string before we pass it to open() on line 274 of [fileutil.py]source:trunk/src/allmydata/util/fileutil.py?rev=ff64a0fef5879d3651bc3db6ca0522d96b217d45?

It's not quite as simple as that:

  • Paths prefixed with "\\?\" must be absolute, but CLI commands accept relative paths. Also, they don't canonicalize in the same way, e.g. ".." and "." are not supported.
  • "\\?\" is only supported by the Unicode Win32 file APIs. I think Python does use those APIs (indirectly, via msvcrt) when given a unicode string, though.
  • For consistency you'd need to change many other places in Tahoe-LAFS that access files, even to just support long paths in CLI commands.
Replying to [zooko]comment:4: > Why can't we just prepend "`\\?\`" to the string before we pass it to `open()` on line 274 of [fileutil.py]source:trunk/src/allmydata/util/fileutil.py?rev=ff64a0fef5879d3651bc3db6ca0522d96b217d45? It's not quite as simple as that: * Paths prefixed with "`\\?\`" must be absolute, but CLI commands accept relative paths. Also, they don't canonicalize in the same way, e.g. "`..`" and "`.`" are not supported. * "`\\?\`" is only supported by the Unicode Win32 file APIs. I think Python does use those APIs (indirectly, via msvcrt) when given a `unicode` string, though. * For consistency you'd need to change many other places in Tahoe-LAFS that access files, even to just support long paths in CLI commands.
daira commented 2014-04-30 00:47:33 +00:00
Author
Owner
Copy Link

Also see http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx. (Parts 2 and 3 are not so interesting.)

Also see <http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx>. (Parts 2 and 3 are not so interesting.)
daira commented 2014-04-30 01:10:23 +00:00
Author
Owner
Copy Link

Replying to [daira]comment:5:

  • Paths prefixed with "\\?\" must be absolute, but CLI commands accept relative paths. Also, they don't canonicalize in the same way, e.g. ".." and "." are not supported.

Given this constraint, the best place to prepend "\\?\" would probably be in fileutil.abspath_expanduser_unicode.

Replying to [daira]comment:5: > * Paths prefixed with "`\\?\`" must be absolute, but CLI commands accept relative paths. Also, they don't canonicalize in the same way, e.g. "`..`" and "`.`" are not supported. Given this constraint, the best place to prepend "`\\?\`" would probably be in `fileutil.abspath_expanduser_unicode`.
zooko commented 2014-05-01 15:54:27 +00:00
Copy Link

Replying to [daira]comment:7:

Replying to [daira]comment:5:

  • Paths prefixed with "\\?\" must be absolute, but CLI commands accept relative paths. Also, they don't canonicalize in the same way, e.g. ".." and "." are not supported.

Given this constraint, the best place to prepend "\\?\" would probably be in fileutil.abspath_expanduser_unicode.

Okay, I don't see any reason not to prepend "\\?\" to all paths when reading files.

We should probably think about it a bit more when creating files i.e. during a tahoe get, because if we do this then the resulting files wouldn't be accessible via tools that don't support long paths, such as the Explorer or cmd.exe, for example, if I understand correctly.

Replying to [daira]comment:7: > Replying to [daira]comment:5: > > * Paths prefixed with "`\\?\`" must be absolute, but CLI commands accept relative paths. Also, they don't canonicalize in the same way, e.g. "`..`" and "`.`" are not supported. > > Given this constraint, the best place to prepend "`\\?\`" would probably be in `fileutil.abspath_expanduser_unicode`. Okay, I don't see any reason *not* to prepend "`\\?\`" to all paths when reading files. We should probably think about it a bit more when *creating* files i.e. during a `tahoe get`, because if we do this then the resulting files wouldn't be accessible via tools that don't support long paths, such as the Explorer or cmd.exe, for example, if I understand correctly.
daira commented 2014-05-03 23:53:02 +00:00
Author
Owner
Copy Link

Prepending "\\?\" in fileutil.abspath_expanduser_unicode would also have the effect of making all file accesses underneath a node base directory use "long" paths on Windows. I don't know whether or not that is what we want.

Prepending "`\\?\`" in `fileutil.abspath_expanduser_unicode` would also have the effect of making all file accesses underneath a node base directory use "long" paths on Windows. I don't know whether or not that is what we want.
daira commented 2014-05-03 23:54:17 +00:00
Author
Owner
Copy Link

I created a branch https://github.com/daira/tahoe-lafs/commit/2235-long-paths-on-windows. It results in the following failures of existing tests, some of which are obviously shallow. The others I haven't had time to investigate.

[FAIL]
Traceback (most recent call last):
  File "c:\tahoe\git\trunk\src\allmydata\test\test_cli.py", line 389, in test_catalog_shares_error
    "didn't see 'error processing' in '%s'" % err)
twisted.trial.unittest.FailTest: didn't see 'error processing' in ''

allmydata.test.test_cli.CLI.test_catalog_shares_error
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "c:\tahoe\git\trunk\src\allmydata\test\test_cli.py", line 3826, in test_basedir
    ".tahoe"))
  File "C:\Python27\lib\site-packages\twisted-12.2.0-py2.7-win32.egg\twisted\trial\unittest.py", lin
e 271, in assertEqual
    % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = u'\\\\?\\C:\\Documents and Settings\\David-Sarah\\.tahoe'
b = 'C:\\Documents and Settings\\David-Sarah\\.tahoe'


allmydata.test.test_cli.Options.test_basedir
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "C:\tahoe\git\trunk\support\Lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1201, in pa
tched
    return func(*args, **keywargs)
  File "c:\tahoe\git\trunk\src\allmydata\test\test_client.py", line 48, in test_error_on_old_config_
files
    self.failUnlessIn(os.path.abspath(os.path.join(basedir, "introducer.furl")), e.args[0])
twisted.trial.unittest.FailTest: 'C:\\tahoe\\git\\trunk\\_trial_temp\\test_client.Basic.test_error_o
n_old_config_files\\introducer.furl' not in set([u'\\\\?\\C:\\tahoe\\git\\trunk\\_trial_temp\\test_c
lient.Basic.test_error_on_old_config_files\\readonly_storage', u'\\\\?\\C:\\tahoe\\git\\trunk\\_tria
l_temp\\test_client.Basic.test_error_on_old_config_files\\introducer.furl', u'\\\\?\\C:\\tahoe\\git\
\trunk\\_trial_temp\\test_client.Basic.test_error_on_old_config_files\\no_storage', u'\\\\?\\C:\\tah
oe\\git\\trunk\\_trial_temp\\test_client.Basic.test_error_on_old_config_files\\debug_discard_storage
'])

allmydata.test.test_client.Basic.test_error_on_old_config_files
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "c:\tahoe\git\trunk\src\allmydata\test\test_system.py", line 1374, in _test_runner
    self.failUnlessEqual(len(sharefiles), 10)
  File "C:\Python27\lib\site-packages\twisted-12.2.0-py2.7-win32.egg\twisted\trial\unittest.py", lin
e 271, in assertEqual
    % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = 0
b = 10


allmydata.test.test_system.SystemTest.test_filesystem
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "c:\tahoe\git\trunk\src\allmydata\test\test_util.py", line 491, in test_abspath_expanduser_un
icode
    self.failUnlessEqual(abspath_cwd, saved_cwd)
  File "C:\Python27\lib\site-packages\twisted-12.2.0-py2.7-win32.egg\twisted\trial\unittest.py", lin
e 271, in assertEqual
    % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = u'\\\\?\\C:\\tahoe\\git\\trunk\\_trial_temp'
b = u'C:\\tahoe\\git\\trunk\\_trial_temp'


allmydata.test.test_util.FileUtil.test_abspath_expanduser_unicode
-------------------------------------------------------------------------------
Ran 1137 tests in 878.831s

FAILED (skips=15, expectedFailures=3, failures=5, successes=1114)
I created a branch <https://github.com/daira/tahoe-lafs/commit/2235-long-paths-on-windows>. It results in the following failures of existing tests, some of which are obviously shallow. The others I haven't had time to investigate. ``` [FAIL] Traceback (most recent call last): File "c:\tahoe\git\trunk\src\allmydata\test\test_cli.py", line 389, in test_catalog_shares_error "didn't see 'error processing' in '%s'" % err) twisted.trial.unittest.FailTest: didn't see 'error processing' in '' allmydata.test.test_cli.CLI.test_catalog_shares_error =============================================================================== [FAIL] Traceback (most recent call last): File "c:\tahoe\git\trunk\src\allmydata\test\test_cli.py", line 3826, in test_basedir ".tahoe")) File "C:\Python27\lib\site-packages\twisted-12.2.0-py2.7-win32.egg\twisted\trial\unittest.py", lin e 271, in assertEqual % (msg, pformat(first), pformat(second))) twisted.trial.unittest.FailTest: not equal: a = u'\\\\?\\C:\\Documents and Settings\\David-Sarah\\.tahoe' b = 'C:\\Documents and Settings\\David-Sarah\\.tahoe' allmydata.test.test_cli.Options.test_basedir =============================================================================== [FAIL] Traceback (most recent call last): File "C:\tahoe\git\trunk\support\Lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1201, in pa tched return func(*args, **keywargs) File "c:\tahoe\git\trunk\src\allmydata\test\test_client.py", line 48, in test_error_on_old_config_ files self.failUnlessIn(os.path.abspath(os.path.join(basedir, "introducer.furl")), e.args[0]) twisted.trial.unittest.FailTest: 'C:\\tahoe\\git\\trunk\\_trial_temp\\test_client.Basic.test_error_o n_old_config_files\\introducer.furl' not in set([u'\\\\?\\C:\\tahoe\\git\\trunk\\_trial_temp\\test_c lient.Basic.test_error_on_old_config_files\\readonly_storage', u'\\\\?\\C:\\tahoe\\git\\trunk\\_tria l_temp\\test_client.Basic.test_error_on_old_config_files\\introducer.furl', u'\\\\?\\C:\\tahoe\\git\ \trunk\\_trial_temp\\test_client.Basic.test_error_on_old_config_files\\no_storage', u'\\\\?\\C:\\tah oe\\git\\trunk\\_trial_temp\\test_client.Basic.test_error_on_old_config_files\\debug_discard_storage ']) allmydata.test.test_client.Basic.test_error_on_old_config_files =============================================================================== [FAIL] Traceback (most recent call last): File "c:\tahoe\git\trunk\src\allmydata\test\test_system.py", line 1374, in _test_runner self.failUnlessEqual(len(sharefiles), 10) File "C:\Python27\lib\site-packages\twisted-12.2.0-py2.7-win32.egg\twisted\trial\unittest.py", lin e 271, in assertEqual % (msg, pformat(first), pformat(second))) twisted.trial.unittest.FailTest: not equal: a = 0 b = 10 allmydata.test.test_system.SystemTest.test_filesystem =============================================================================== [FAIL] Traceback (most recent call last): File "c:\tahoe\git\trunk\src\allmydata\test\test_util.py", line 491, in test_abspath_expanduser_un icode self.failUnlessEqual(abspath_cwd, saved_cwd) File "C:\Python27\lib\site-packages\twisted-12.2.0-py2.7-win32.egg\twisted\trial\unittest.py", lin e 271, in assertEqual % (msg, pformat(first), pformat(second))) twisted.trial.unittest.FailTest: not equal: a = u'\\\\?\\C:\\tahoe\\git\\trunk\\_trial_temp' b = u'C:\\tahoe\\git\\trunk\\_trial_temp' allmydata.test.test_util.FileUtil.test_abspath_expanduser_unicode ------------------------------------------------------------------------------- Ran 1137 tests in 878.831s FAILED (skips=15, expectedFailures=3, failures=5, successes=1114) ```
daira commented 2014-05-04 23:03:55 +00:00
Author
Owner
Copy Link

I fixed the test failures apart from the ones in allmydata.test.test_cli.CLI.test_catalog_shares_error and allmydata.test.test_system.SystemTest.test_filesystem.

I fixed the test failures apart from the ones in `allmydata.test.test_cli.CLI.test_catalog_shares_error` and `allmydata.test.test_system.SystemTest.test_filesystem`.
tahoe-lafs changed title from Error from 'tahoe cp' on Windows, possibly due to a long path (IOError: Errno2 - no such file or dir.) to Error from 'tahoe cp' on Windows, due to a long path (IOError: Errno2 - no such file or dir.) 2014-05-12 22:35:28 +00:00
zooko commented 2014-05-20 15:18:51 +00:00
Copy Link

The next step on this should be to write a test, perhaps in source:trunk/src/allmydata/test/test_util.py, that attempts to create a file under a long filename in the (real, not faked) local filesystem, and if the file is successfully created then the test passes. That test should be red on current trunk when run on Windows, and go green when daira's patch or another fixit-patch is applied.

I don't understand this not path.startswith(u"\\\\") clause in Daira's patch, and I'd like to see a second test added to the one described above, which verifies that whatever behavior is intended by that clause is what actually happens.

Other than those two additional tests, just getting all of the current tests to pass with the patch applied will be sufficient! ☺

The next step on this should be to write a test, perhaps in source:trunk/src/allmydata/test/test_util.py, that attempts to create a file under a long filename in the (real, not faked) local filesystem, and if the file is successfully created then the test passes. That test should be red on current trunk when run on Windows, and go green when [daira's patch](https://github.com/daira/tahoe-lafs/commit/2a2a1adee5f814fa8018fe4119e4daf843dc229a) or another fixit-patch is applied. I don't understand this `not path.startswith(u"\\\\")` clause in Daira's patch, and I'd like to see a second test added to the one described above, which verifies that whatever behavior is intended by that clause is what actually happens. Other than those two additional tests, just getting all of the *current* tests to pass with the patch applied will be sufficient! ☺
zooko commented 2014-05-20 15:19:52 +00:00
Copy Link

P.S. If you don't know why that not path.startswith(u"\\\\") clause is in there, and Daira isn't around to explain it, then try removing that clause. If all the tests still pass, then great — you've removed unnecessary code! If tests start failing then examining the failing tests will probably show you why that clause is there.

P.S. If you don't know why that `not path.startswith(u"\\\\")` clause is in there, and Daira isn't around to explain it, then try removing that clause. If all the tests still pass, then great — you've removed unnecessary code! If tests start failing then examining the failing tests will probably show you why that clause is there.
daira commented 2014-05-20 16:41:18 +00:00
Author
Owner
Copy Link

Because prepending \\?\ is not correct for a UNC path ( http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx#maxpath ):

The "\\?\" prefix can also be used with paths constructed according to the universal naming convention (UNC). To specify such a path using UNC, use the "\\?\UNC\" prefix. For example, "\\?\UNC\server\share", where "server" is the name of the computer and "share" is the name of the shared folder. These prefixes are not used as part of the path itself. They indicate that the path should be passed to the system with minimal modification, which means that you cannot use forward slashes to represent path separators, or a period to represent the current directory, or double dots to represent the parent directory. Because you cannot use the "\\?\" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters.

It is also not correct for a path that already starts with \\?\ (or, more obscurely, \\.\).

I could have written code like this to cover all cases:

if sys.platform == "win32":
    path = to_windows_long_path(path)
    [...]

def to_windows_long_path(path):
    if path.startswith(u"\\\\?\\") or path.startswith(u"\\\\.\\"):
        return path
    elif path.startswith(u"\\\\"):
        return u"\\\\?\\UNC\\" + path[2 :]
    else:
        return u"\\\\?\\" + path

but I don't actually care about long UNC paths; I just wanted to avoid doing something obviously incorrect for them.

Because prepending `\\?\` is not correct for a UNC path ( <http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx#maxpath> ): > The `"\\?\"` prefix can also be used with paths constructed according to the universal naming convention (UNC). To specify such a path using UNC, use the `"\\?\UNC\"` prefix. For example, `"\\?\UNC\server\share"`, where "server" is the name of the computer and "share" is the name of the shared folder. These prefixes are not used as part of the path itself. They indicate that the path should be passed to the system with minimal modification, which means that you cannot use forward slashes to represent path separators, or a period to represent the current directory, or double dots to represent the parent directory. Because you cannot use the `"\\?\"` prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters. It is also not correct for a path that already starts with `\\?\` (or, more obscurely, `\\.\`). I could have written code like this to cover all cases: ``` if sys.platform == "win32": path = to_windows_long_path(path) [...] def to_windows_long_path(path): if path.startswith(u"\\\\?\\") or path.startswith(u"\\\\.\\"): return path elif path.startswith(u"\\\\"): return u"\\\\?\\UNC\\" + path[2 :] else: return u"\\\\?\\" + path ``` but I don't actually care about long UNC paths; I just wanted to avoid doing something obviously incorrect for them.
daira commented 2014-05-20 17:35:49 +00:00
Author
Owner
Copy Link

Branch updated to use the code in comment:95019 : https://github.com/tahoe-lafs/tahoe-lafs/commits/2235-long-paths-on-windows

Branch updated to use the code in [comment:95019](/tahoe-lafs/trac-2024-07-25/issues/2235#issuecomment-95019) : <https://github.com/tahoe-lafs/tahoe-lafs/commits/2235-long-paths-on-windows>
daira commented 2014-05-20 17:38:11 +00:00
Author
Owner
Copy Link

https://github.com/tahoe-lafs/tahoe-lafs/pull/93. Note: this doesn't pass all tests on Windows and shouldn't be merged yet.

<https://github.com/tahoe-lafs/tahoe-lafs/pull/93>. Note: this doesn't pass all tests on Windows and shouldn't be merged yet.
CyberAxe commented 2014-05-22 05:24:52 +00:00
Author
Owner
Copy Link

I'm not sure this hasn't been covered but this error didn't show up during backup, but only has shown up during recovery with IIRC -CP -R shown above.

I'll start testing with a single file name with 300 char long.

I'm not sure this hasn't been covered but this error didn't show up during backup, but only has shown up during recovery with IIRC -CP -R shown above. I'll start testing with a single file name with 300 char long.
zooko commented 2014-05-26 20:59:32 +00:00
Copy Link

Replying to CyberAxe:

I'm not sure this hasn't been covered but this error didn't show up during backup, but only has shown up during recovery with IIRC -CP -R shown above.

I'll start testing with a single file name with 300 char long.

CyberAxe: have you tested this yet? What did you learn?

Replying to [CyberAxe](/tahoe-lafs/trac-2024-07-25/issues/2235#issuecomment-95022): > I'm not sure this hasn't been covered but this error didn't show up during backup, but only has shown up during recovery with IIRC -CP -R shown above. > > I'll start testing with a single file name with 300 char long. CyberAxe: have you tested this yet? What did you learn?
daira commented 2014-05-28 21:40:45 +00:00
Author
Owner
Copy Link

Replying to zooko:

The next step on this should be to write a test, perhaps in source:trunk/src/allmydata/test/test_util.py, that attempts to create a file under a long filename in the (real, not faked) local filesystem, and if the file is successfully created then the test passes. That test should be red on current trunk when run on Windows, and go green when daira's patch or another fixit-patch is applied.

I agree that we need such a test.

I don't understand this not path.startswith(u"\\\\") clause in Daira's patch, and I'd like to see a second test added to the one described above, which verifies that whatever behavior is intended by that clause is what actually happens.

This has been done.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/2235#issuecomment-95017): > The next step on this should be to write a test, perhaps in source:trunk/src/allmydata/test/test_util.py, that attempts to create a file under a long filename in the (real, not faked) local filesystem, and if the file is successfully created then the test passes. That test should be red on current trunk when run on Windows, and go green when [daira's patch](https://github.com/daira/tahoe-lafs/commit/2a2a1adee5f814fa8018fe4119e4daf843dc229a) or another fixit-patch is applied. I agree that we need such a test. > I don't understand this `not path.startswith(u"\\\\")` clause in Daira's patch, and I'd like to see a second test added to the one described above, which verifies that whatever behavior is intended by that clause is what actually happens. This has been done.
daira commented 2014-08-05 01:59:49 +00:00
Author
Owner
Copy Link

The test failures on Windows have been fixed. The problem was that when using the \\?\ prefix, Windows is pickier about path syntax and does not accept / as a path component separator.

The test failures on Windows have been fixed. The problem was that when using the `\\?\` prefix, Windows is pickier about path syntax and does not accept `/` as a path component separator.
tahoe-lafs modified the milestone from undecided to 1.11.0 2014-08-14 20:42:11 +00:00
daira commented 2014-08-21 02:06:39 +00:00
Author
Owner
Copy Link

Pull request https://github.com/tahoe-lafs/tahoe-lafs/pull/100 has a test and is now ready for review.

Pull request <https://github.com/tahoe-lafs/tahoe-lafs/pull/100> has a [test](https://github.com/tahoe-lafs/tahoe-lafs/commit/abece3d27aab9edc491ba477a9f160a089837893) and is now ready for review.
daira commented 2014-08-21 08:50:56 +00:00
Author
Owner
Copy Link

Replying to daira:

Pull request https://github.com/tahoe-lafs/tahoe-lafs/pull/100 has a test and is now ready for review.

Please hold off merging this; there are instances of os.path.abspath and os.path.expanduser that don't go through fileutil.abspath_expanduser_unicode and should be changed to do so. That includes the call in fileutil.put_file that motivated this ticket, so the pull request doesn't actually fix the bug. See the results of egrep -Rn --include='*.py' 'expanduser|abspath' src |egrep -v 'abspath_expanduser_unicode|argv_to_abspath' for other cases.

Replying to [daira](/tahoe-lafs/trac-2024-07-25/issues/2235#issuecomment-95027): > Pull request <https://github.com/tahoe-lafs/tahoe-lafs/pull/100> has a [test](https://github.com/tahoe-lafs/tahoe-lafs/commit/abece3d27aab9edc491ba477a9f160a089837893) and is now ready for review. Please hold off merging this; there are instances of `os.path.abspath` and `os.path.expanduser` that don't go through `fileutil.abspath_expanduser_unicode` and should be changed to do so. That includes the call in `fileutil.put_file` that motivated this ticket, so the pull request doesn't actually fix the bug. See the results of `egrep -Rn --include='*.py' 'expanduser|abspath' src |egrep -v 'abspath_expanduser_unicode|argv_to_abspath'` for other cases.
daira commented 2014-08-21 08:56:28 +00:00
Author
Owner
Copy Link

Omitting some spurious hits:

src/allmydata/introducer/server.py:67:        staticdir = os.path.expanduser(staticdir)
src/allmydata/stats.py:261:                       quote_output(os.path.abspath(self.picklefile)))
src/allmydata/util/fileutil.py:274:    outf = open(os.path.expanduser(pathname), "wb")
src/allmydata/util/fileutil.py:299:    path = os.path.expanduser(path)
src/allmydata/manhole.py:66:        self.authorized_keys_file = os.path.expanduser(authorized_keys_file)
src/allmydata/manhole.py:253:        # TODO: expanduser this, and make it relative to the buildmaster's
src/allmydata/frontends/auth.py:27:        for line in open(os.path.expanduser(accountfile), "r"):
src/allmydata/frontends/drop_upload.py:22:            local_dir_u = os.path.expanduser(local_dir_utf8.decode('utf-8'))
src/allmydata/scripts/tahoe_get.py:29:            outf = open(os.path.expanduser(to_file), "wb")
src/allmydata/scripts/tahoe_put.py:76:        infileobj = open(os.path.expanduser(from_file), "rb")
src/allmydata/scripts/tahoe_cp.py:71:        return open(os.path.expanduser(self.pathname), "rb")
src/allmydata/client.py:454:        staticdir = os.path.expanduser(staticdir)
src/allmydata/test/test_mutable.py:3113:            fso.nodedirs = [unicode(os.path.dirname(os.path.abspath(storedir)))
src/allmydata/test/test_encodingutil.py:277:        expanded = os.path.expanduser("~/" + lumiere_nfc)
src/allmydata/test/check_grid.py:67:        self.nodedir = os.path.expanduser(nodedir)
src/allmydata/test/check_grid.py:68:        self.tahoe = os.path.abspath(os.path.expanduser(tahoe))
Omitting some spurious hits: ``` src/allmydata/introducer/server.py:67: staticdir = os.path.expanduser(staticdir) src/allmydata/stats.py:261: quote_output(os.path.abspath(self.picklefile))) src/allmydata/util/fileutil.py:274: outf = open(os.path.expanduser(pathname), "wb") src/allmydata/util/fileutil.py:299: path = os.path.expanduser(path) src/allmydata/manhole.py:66: self.authorized_keys_file = os.path.expanduser(authorized_keys_file) src/allmydata/manhole.py:253: # TODO: expanduser this, and make it relative to the buildmaster's src/allmydata/frontends/auth.py:27: for line in open(os.path.expanduser(accountfile), "r"): src/allmydata/frontends/drop_upload.py:22: local_dir_u = os.path.expanduser(local_dir_utf8.decode('utf-8')) src/allmydata/scripts/tahoe_get.py:29: outf = open(os.path.expanduser(to_file), "wb") src/allmydata/scripts/tahoe_put.py:76: infileobj = open(os.path.expanduser(from_file), "rb") src/allmydata/scripts/tahoe_cp.py:71: return open(os.path.expanduser(self.pathname), "rb") src/allmydata/client.py:454: staticdir = os.path.expanduser(staticdir) src/allmydata/test/test_mutable.py:3113: fso.nodedirs = [unicode(os.path.dirname(os.path.abspath(storedir))) src/allmydata/test/test_encodingutil.py:277: expanded = os.path.expanduser("~/" + lumiere_nfc) src/allmydata/test/check_grid.py:67: self.nodedir = os.path.expanduser(nodedir) src/allmydata/test/check_grid.py:68: self.tahoe = os.path.abspath(os.path.expanduser(tahoe)) ```
daira commented 2014-08-21 09:30:55 +00:00
Author
Owner
Copy Link

Making all user-expansions go through fileutil.abspath_expanduser_unicode will make it easier to fix #1674.

Making all user-expansions go through `fileutil.abspath_expanduser_unicode` will make it easier to fix #1674.
daira commented 2014-10-27 02:40:27 +00:00
Author
Owner
Copy Link

Currently working on https://github.com/tahoe-lafs/tahoe-lafs/commits/2235-long-paths-on-windows-5.

Currently working on <https://github.com/tahoe-lafs/tahoe-lafs/commits/2235-long-paths-on-windows-5>.
daira commented 2015-01-20 17:38:08 +00:00
Author
Owner
Copy Link

I need to rebase this and check that it is complete and passes tests.

I need to rebase this and check that it is complete and passes tests.
zooko commented 2015-01-22 17:26:02 +00:00
Copy Link

removing review-needed keyword

The next step is for Daira to do the changes she described in comment:28.

removing `review-needed` keyword The next step is for Daira to do the changes she described in comment:28.
daira commented 2015-01-29 19:32:27 +00:00
Author
Owner
Copy Link

Done. Review needed for https://github.com/tahoe-lafs/tahoe-lafs/pull/138 (which also fixes #1674 and #2027).

Done. Review needed for <https://github.com/tahoe-lafs/tahoe-lafs/pull/138> (which also fixes #1674 and #2027).
zooko commented 2015-02-03 18:59:14 +00:00
Copy Link

Reviewed all the patches on https://github.com/tahoe-lafs/tahoe-lafs/pull/138 and posted comments.

Reviewed all the patches on <https://github.com/tahoe-lafs/tahoe-lafs/pull/138> and posted comments.
daira commented 2015-02-05 02:09:07 +00:00
Author
Owner
Copy Link

Fixed on the pr138 branch which was merged in [3afe827ad0cbdb41b2928d17c8bfbbaf5102acc8/trunk].

(Woohoo! This set of bugs has been irking me for some time.)

Fixed on the pr138 branch which was merged in [3afe827ad0cbdb41b2928d17c8bfbbaf5102acc8/trunk]. (Woohoo! This set of bugs has been irking me for some time.)
tahoe-lafs added the
fixed
 label 2015-02-05 02:09:07 +00:00
daira closed this issue 2015-02-05 02:09:07 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
0.2.0

   
0.3.0

   
0.4.0

   
0.5.0

   
0.5.1

   
0.6.0

   
0.6.1

   
0.7.0

   
0.8.0

   
0.9.0

   
1.0.0

   
1.1.0

   
1.10.0

   
1.10.1

   
1.10.2

   
1.10a2

   
1.11.0

   
1.12.0

   
1.12.1

   
1.13.0

   
1.14.0

   
1.15.0

   
1.15.1

   
1.2.0

   
1.3.0

   
1.4.1

   
1.5.0

   
1.6.0

   
1.6.1

   
1.7.0

   
1.7.1

   
1.7β

   
1.8.0

   
1.8.1

   
1.8.2

   
1.8.3

   
1.8β

   
1.9.0

   
1.9.0-s3branch

   
1.9.0a1

   
1.9.0a2

   
1.9.0b1

   
1.9.1

   
1.9.2

   
1.9.2a1

   
LeastAuthority.com automation

   
blocker

   
cannot reproduce

   
cloud-branch

   
code

catch-all for anything that can be fixed by writing code but isn't classified better

  
code-dirnodes

anything involving tahoe directories

  
code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
code-frontend-cli

CLI tools

  
code-frontend-ftp-sftp

   
code-frontend-magic-folder

   
code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
code-mutable

anything involving mutable files: encoding/decoding, share format

  
code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
contrib

problems with non-core things that live in the source:contrib/ directory

  
critical

   
defect

   
dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
documentation

docs items. Also use 'docs' keyword on tickets in more specific components.

  
duplicate

   
enhancement

   
fixed

   
invalid

   
major

   
minor

   
n/a

   
normal

   
operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
somebody else's problem

   
supercritical

   
task

   
trivial

   
unknown

Uncategorized. All tickets default to this component until someone updates it.

  
was already fixed

   
website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
wontfix

   
worksforme
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
Milestone
Clear milestone
No items
No Milestone
1.10.1
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#2235
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?