tahoe cp copies a source directory's children rather than the directory itself #712

New Issue

zooko · 2009-05-23T20:52:08Z

zooko commented

2009-05-23 20:52:08 +00:00

I had two entries in my local filesystem, a file named "22brain.html" and a directory named "22brain_files", which was full of other files and subdirectories. I ran {{tahoe cp 22brain* tahoe:22brain/}}}. Afterward I expected to find tahoe:22brain/22brain.html and tahoe:22brain/22brain_files/, but instead I found the contents of the local "22brain_files" directory copied directly into the tahoe:22brain directory. This breaks all the links in the "22brain.html" file and means I need to delete this directory and try again.

I had two entries in my local filesystem, a file named "22brain.html" and a directory named "22brain_files", which was full of other files and subdirectories. I ran {{tahoe cp 22brain* tahoe:22brain/}}}. Afterward I expected to find `tahoe:22brain/22brain.html` and `tahoe:22brain/22brain_files/`, but instead I found the *contents* of the local "22brain_files" directory copied directly into the `tahoe:22brain` directory. This breaks all the links in the "22brain.html" file and means I need to delete this directory and try again.

zooko added the

labels 2009-05-23 20:52:08 +00:00

zooko added this to the eventually milestone 2009-05-23 20:52:08 +00:00

warner was assigned by zooko

2009-05-23 20:52:08 +00:00

zooko commented

2009-05-23 21:15:33 +00:00

I see that the same thing happens if I run tahoe cp -r 22brain_files tahoe:22brain/.

See also #705 ("tahoe mv" unlinks the target even when it is a directory).

I see that the same thing happens if I run `tahoe cp -r 22brain_files tahoe:22brain/`. See also #705 ("tahoe mv" unlinks the target even when it is a directory).

davidsarah commented

2012-11-26 00:16:50 +00:00

zooko forgot he had filed this ticket :-) and wrote in the duplicate ticket #1854 :

$ ls -al
total 1740
drwxrwxr-x 3 zooko zooko   4096 Nov 11 04:56 .
drwxrwxr-x 3 zooko zooko   4096 Nov 11 04:52 ..
-rw-rw-r-- 1 zooko zooko  80898 Aug 21 19:23 276a292eebb111e186fe22000a1c9ebd_7.jpg
-rw-rw-r-- 1 zooko zooko  96881 Nov 10 03:48 causes-of-death-2010.png
-rw-rw-r-- 1 zooko zooko  20450 Nov 11 04:52 hackers-2012-slides.html
-rw-rw-r-- 1 zooko zooko  11458 Nov 10 14:57 hackers-2012-slides.rst
-rw-rw-r-- 1 zooko zooko 948821 Nov 10 03:46 Murphy-2012-Deaths__Preliminary_Data_For_2010.pdf
-rw-rw-r-- 1 zooko zooko 113191 Nov 10 11:51 Neal-2008-Table4.png
-rw-rw-r-- 1 zooko zooko 117367 Nov 10 10:48 Neal-2008-The_ketogenic_diet_for_the_treatment_of_childhood_epilepsy__a_randomised_controlled_trial.pdf
-rw-rw-r-- 1 zooko zooko 148251 Nov 10 10:37 Neal-2010-Efficacy_Of_Dietary_Treatments_For_Epilepsy.pdf
-rw-rw-r-- 1 zooko zooko 219436 Nov 10 10:42 Neal-2010-Table2.png
drwxrwxr-x 4 zooko zooko   4096 Nov 10 14:04 ui
$ time tahoe cp -r * URI:DIR2-MDMF:qda3lf42dkxclwbuh5h7qwygne:obhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a:
Success: files copied

real    0m26.085s
user    0m1.003s
sys     0m0.140s
HACK zompu:~/book/gitworld/presentations/hackers-2012$ tahoe ls -l URI:DIR2-MDMF:qda3lf42dkxclwbuh5h7qwygne:obhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a:
-r--  80898 Nov 11 04:58                                                                  276a292eebb111e186fe22000a1c9ebd_7.jpg
-r-- 948821 Nov 11 04:58                                                       Murphy-2012-Deaths__Preliminary_Data_For_2010.pdf
-r-- 113191 Nov 11 04:58                                                                                    Neal-2008-Table4.png
-r-- 117367 Nov 11 04:58 Neal-2008-The_ketogenic_diet_for_the_treatment_of_childhood_epilepsy__a_randomised_controlled_trial.pdf
-r-- 148251 Nov 11 04:58                                               Neal-2010-Efficacy_Of_Dietary_Treatments_For_Epilepsy.pdf
-r-- 219436 Nov 11 04:58                                                                                    Neal-2010-Table2.png
-r--  96881 Nov 11 04:58                                                                                causes-of-death-2010.png
drwx      - Nov 11 04:56                                                                                                 default
-r--  20450 Nov 11 04:58                                                                                hackers-2012-slides.html
-r--  11458 Nov 11 04:58                                                                                 hackers-2012-slides.rst
drwx      - Nov 11 04:56                                                                                             small-white

This is wrong, causing my slides to be unable to load their CSS when viewed from Tahoe-LAFS (e.g.it looks like https://zooko.com/uri/URI%3ADIR2-MDMF-RO%3Appnrefnrnovjpoiv3jirjnpoim%3Aobhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a/hackers-2012-slides.html when it should look like this: https://zooko.com/uri/URI:DIR2-MDMF-RO:nmrgdbrmwdta2bfzke4pfttsei:vlijvwz4kncumxkob5um45sni7btqelu4q5oltcl5nqxwjrif2iq/hackers-2012-slides.html).

The result should have been that the directory-and-file structure in the tahoe-lafs dir matched that in the local dir.

zooko forgot he had filed this ticket :-) and wrote in the duplicate ticket #1854 : > ``` > $ ls -al > total 1740 > drwxrwxr-x 3 zooko zooko 4096 Nov 11 04:56 . > drwxrwxr-x 3 zooko zooko 4096 Nov 11 04:52 .. > -rw-rw-r-- 1 zooko zooko 80898 Aug 21 19:23 276a292eebb111e186fe22000a1c9ebd_7.jpg > -rw-rw-r-- 1 zooko zooko 96881 Nov 10 03:48 causes-of-death-2010.png > -rw-rw-r-- 1 zooko zooko 20450 Nov 11 04:52 hackers-2012-slides.html > -rw-rw-r-- 1 zooko zooko 11458 Nov 10 14:57 hackers-2012-slides.rst > -rw-rw-r-- 1 zooko zooko 948821 Nov 10 03:46 Murphy-2012-Deaths__Preliminary_Data_For_2010.pdf > -rw-rw-r-- 1 zooko zooko 113191 Nov 10 11:51 Neal-2008-Table4.png > -rw-rw-r-- 1 zooko zooko 117367 Nov 10 10:48 Neal-2008-The_ketogenic_diet_for_the_treatment_of_childhood_epilepsy__a_randomised_controlled_trial.pdf > -rw-rw-r-- 1 zooko zooko 148251 Nov 10 10:37 Neal-2010-Efficacy_Of_Dietary_Treatments_For_Epilepsy.pdf > -rw-rw-r-- 1 zooko zooko 219436 Nov 10 10:42 Neal-2010-Table2.png > drwxrwxr-x 4 zooko zooko 4096 Nov 10 14:04 ui > $ time tahoe cp -r * URI:DIR2-MDMF:qda3lf42dkxclwbuh5h7qwygne:obhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a: > Success: files copied > > real 0m26.085s > user 0m1.003s > sys 0m0.140s > HACK zompu:~/book/gitworld/presentations/hackers-2012$ tahoe ls -l URI:DIR2-MDMF:qda3lf42dkxclwbuh5h7qwygne:obhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a: > -r-- 80898 Nov 11 04:58 276a292eebb111e186fe22000a1c9ebd_7.jpg > -r-- 948821 Nov 11 04:58 Murphy-2012-Deaths__Preliminary_Data_For_2010.pdf > -r-- 113191 Nov 11 04:58 Neal-2008-Table4.png > -r-- 117367 Nov 11 04:58 Neal-2008-The_ketogenic_diet_for_the_treatment_of_childhood_epilepsy__a_randomised_controlled_trial.pdf > -r-- 148251 Nov 11 04:58 Neal-2010-Efficacy_Of_Dietary_Treatments_For_Epilepsy.pdf > -r-- 219436 Nov 11 04:58 Neal-2010-Table2.png > -r-- 96881 Nov 11 04:58 causes-of-death-2010.png > drwx - Nov 11 04:56 default > -r-- 20450 Nov 11 04:58 hackers-2012-slides.html > -r-- 11458 Nov 11 04:58 hackers-2012-slides.rst > drwx - Nov 11 04:56 small-white > ``` > > This is wrong, causing my slides to be unable to load their CSS when viewed from Tahoe-LAFS (e.g.it looks like <https://zooko.com/uri/URI%3ADIR2-MDMF-RO%3Appnrefnrnovjpoiv3jirjnpoim%3Aobhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a/hackers-2012-slides.html> when it should look like this: <https://zooko.com/uri/URI:DIR2-MDMF-RO:nmrgdbrmwdta2bfzke4pfttsei:vlijvwz4kncumxkob5um45sni7btqelu4q5oltcl5nqxwjrif2iq/hackers-2012-slides.html>). > > The result should have been that the directory-and-file structure in the tahoe-lafs dir matched that in the local dir.

tahoe-lafs modified the milestone from eventually to 1.11.0

2012-11-26 00:16:50 +00:00

davidsarah commented

2012-11-26 00:21:39 +00:00

It isn't clear from the bug description or from comment:71239 whether it is just that a directory specified in the command line won't be created as a directory in the destination, or whether all subdirectories will be flattened.

It isn't clear from the bug description or from [comment:71239](/tahoe-lafs/trac-2024-07-25/issues/712#issuecomment-71239) whether it is just that a directory specified in the command line won't be created as a directory in the destination, or whether *all* subdirectories will be flattened.

markberger commented

2013-08-02 17:09:04 +00:00

Replying to davidsarah:

It isn't clear from the bug description or from comment:71239 whether it is just that a directory specified in the command line won't be created as a directory in the destination, or whether all subdirectories will be flattened.

From my testing it appears that only the top level directories are flattened:

Here is my example folder on the local drive:

tahoe-cp-test/
    readme.txt
    pics/
        rio.jpg
        seattle/
            seattle.jpg

When in the directory tahoe-cp-test/, the command tahoe cp -r * tahoe:tahoe-cp-test/ will output the following structure on the grid:

tahoe-cp-test/
    readme.txt
    rio.jpg
    seattle/
        seattle.jpg

This error appears to only occur when you are in the directory you want to upload. cp -r ~/tahoe-cp-test tahoe:tahoe-cp-test uploads the files with the correct structure.

Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/712#issuecomment-71241): > It isn't clear from the bug description or from [comment:71239](/tahoe-lafs/trac-2024-07-25/issues/712#issuecomment-71239) whether it is just that a directory specified in the command line won't be created as a directory in the destination, or whether *all* subdirectories will be flattened. From my testing it appears that only the top level directories are flattened: Here is my example folder on the local drive: ``` tahoe-cp-test/ readme.txt pics/ rio.jpg seattle/ seattle.jpg ``` When in the directory tahoe-cp-test/, the command `tahoe cp -r * tahoe:tahoe-cp-test/` will output the following structure on the grid: ``` tahoe-cp-test/ readme.txt rio.jpg seattle/ seattle.jpg ``` This error appears to only occur when you are in the directory you want to upload. `cp -r ~/tahoe-cp-test tahoe:tahoe-cp-test` uploads the files with the correct structure.

markberger commented

2013-08-02 18:26:12 +00:00

This error doesn't occur when ./ is supplied instead of *.

I believe this ticket ultimately arises from tahoe's interface design and not necessarily from a bug in the code. Because tahoe cp can take multiple arguments, * is supplying both files and directories. The CLI breaks down the list of arguments into individual copy actions, so each folder and file can be viewed to have its own tahoe cp call.

In the instance of tahoe cp -r example_dir/ tahoe:example_dir/, the contents of example_dir/ should be placed into tahoe:example_dir/ without preserving the top level directory (by this I mean we want tahoe:example_dir/contents, not tahoe:example_dir/example_dir/contents).

However with this ticket, we want the opposite behavior. By supplying *, we are telling tahoe to copy multiple directories to the grid, but in this instance we want to retain the top level directory (we want tahoe to create tahoe:example_dir/example_dir/ if example_dir/ is in *).

In order to have the desired behavior outlined in this ticket, we need to distinguish between these two scenarios. I believe the current behavior makes sense, although it is counter intuitive to what users probably expect. I'm not sure how we can distinguish between these two behaviors while keeping support for multiple arguments since it seems like * is populated by the console (I could be wrong about this).

This error doesn't occur when `./` is supplied instead of `*`. I believe this ticket ultimately arises from tahoe's interface design and not necessarily from a bug in the code. Because `tahoe cp` can take multiple arguments, `*` is supplying both files and directories. The CLI breaks down the list of arguments into individual copy actions, so each folder and file can be viewed to have its own `tahoe cp` call. In the instance of `tahoe cp -r example_dir/ tahoe:example_dir/`, the contents of example_dir/ should be placed into tahoe:example_dir/ without preserving the top level directory (by this I mean we want tahoe:example_dir/contents, not tahoe:example_dir/example_dir/contents). However with this ticket, we want the opposite behavior. By supplying `*`, we are telling tahoe to copy multiple directories to the grid, but in this instance we want to retain the top level directory (we want tahoe to create tahoe:example_dir/example_dir/ if example_dir/ is in *). In order to have the desired behavior outlined in this ticket, we need to distinguish between these two scenarios. I believe the current behavior makes sense, although it is counter intuitive to what users probably expect. I'm not sure how we can distinguish between these two behaviors while keeping support for multiple arguments since it seems like `*` is populated by the console (I could be wrong about this).

daira commented

2013-08-03 00:37:09 +00:00

Wildcards are indeed expanded by the Unix shell. I don't think that's the problem, though; the problem is that tahoe cp is doing the wrong thing with individual source arguments that are directories. Compare with Unix cp:

$ mkdir foo
$ mkdir bar
$ cp foo bar
cp: omitting directory `foo'
$ ls
bar  foo
$ cp -R foo bar
$ ls bar
foo

I.e. cp -R foo bar should create a copy of foo itself, not the children of foo, under bar. (It doesn't matter whether or not foo is specified with a trailing slash.)

Wildcards are indeed expanded by the Unix shell. I don't think that's the problem, though; the problem is that `tahoe cp` is doing the wrong thing with individual source arguments that are directories. Compare with `Unix cp`: ``` $ mkdir foo $ mkdir bar $ cp foo bar cp: omitting directory `foo' $ ls bar foo $ cp -R foo bar $ ls bar foo ``` I.e. `cp -R foo bar` should create a copy of `foo` itself, not the children of `foo`, under `bar`. (It doesn't matter whether or not `foo` is specified with a trailing slash.)

tahoe-lafs changed title from ~~tahoe cp flattens directories~~ to tahoe cp copies a source directory's children rather than the directory itself

2013-08-03 00:40:34 +00:00

markberger commented

2013-08-05 12:27:12 +00:00

Thanks for clearing that up Daira! I'll start working on this.

markberger commented

2013-08-05 15:57:59 +00:00

Here is the pull request: https://github.com/tahoe-lafs/tahoe-lafs/pull/56

Here is the pull request: <https://github.com/tahoe-lafs/tahoe-lafs/pull/56>

daira commented

2013-08-06 23:26:43 +00:00

Needs more work, removing 'review-needed'.

daira commented

2013-08-07 00:53:48 +00:00

Reviewed, +1. I added #2047 for a possible cleanup that can be done separately.

daira commented

2013-08-07 01:05:47 +00:00

Still needs docs and a NEWS entry noting the non-backward-compatible change.

zooko commented

2013-08-27 15:20:15 +00:00

We have inconsistent policies about NEWS at this point. I prefer the Twisted process, where each ticket that makes a change comes with a NEWS entry. Brian prefers — last time we talked about it, IIRC — for the Released Manager to read back through the closed tickets and write a NEWS entry for each one.

But the missing docs should definitely be fixed for this ticket. Removing the reviewed keyword.

We have inconsistent policies about NEWS at this point. I prefer the Twisted process, where each ticket that makes a change comes with a NEWS entry. Brian prefers — last time we talked about it, IIRC — for the Released Manager to read back through the closed tickets and write a NEWS entry for each one. But the missing docs should definitely be fixed for this ticket. Removing the `reviewed` keyword.

markberger commented

2013-08-27 15:55:35 +00:00

I couldn't find any existing documentation about the expected behavior for tahoe cp -r so I added an example to the docs.

I couldn't find any existing documentation about the expected behavior for `tahoe cp -r` so I added an example to the docs.

tahoe-lafs modified the milestone from soon to 1.11.0

2013-08-28 16:46:10 +00:00

daira commented

2014-09-02 16:31:57 +00:00

warner will rebase this.

Brian Warner <warner@lothar.com> commented

2014-09-02 20:14:37 +00:00

In /tahoe-lafs/trac-2024-07-25/commit/b88d8b0b3ea345dec3833fdef0a98326c30572f8:

Merge branch markberger/712-tahoe-cp-doesnt-copy-dir

Closes ticket:712 (trac).
Closes tahoe-lafs/tahoe-lafs#56 (github PR).

In [/tahoe-lafs/trac-2024-07-25/commit/b88d8b0b3ea345dec3833fdef0a98326c30572f8](/tahoe-lafs/trac-2024-07-25/commit/b88d8b0b3ea345dec3833fdef0a98326c30572f8): ``` Merge branch markberger/712-tahoe-cp-doesnt-copy-dir Closes ticket:712 (trac). Closes tahoe-lafs/tahoe-lafs#56 (github PR). ```

tahoe-lafs added the

fixed

label 2014-09-02 20:14:37 +00:00

Brian Warner <warner@lothar.com> closed this issue

2014-09-02 20:14:37 +00:00

daira commented

2014-11-25 18:10:35 +00:00

This fix may have caused a regression: #2329

Sign in to join this conversation.