command-line interface #53

New Issue

zooko · 2007-05-25T03:39:47Z

zooko commented

2007-05-25 03:39:47 +00:00

I met a nice guy named Jeff Hehner at Caffe Sole tonight, who said that he wanted a command-line interface to a tool like Tahoe.

I agreed with him that command-line interfaces are very good to have.

I'm marking this one as "priority: major". Along with the XML-RPC interface (#48) this might become my next primary focus. Basically, Tahoe already works well enough for several valuable use cases, and now we just need to enable people to use it. (Meaning, of course, people who are enabled by CLI and XML-RPC... People who are enabled by Windows filesystem integration come next...)

This is already mentioned in source:roadmap.txt .

I met a nice guy named Jeff Hehner at Caffe Sole tonight, who said that he wanted a command-line interface to a tool like Tahoe. I agreed with him that command-line interfaces are very good to have. I'm marking this one as "priority: major". Along with the XML-RPC interface (#48) this might become my next primary focus. Basically, Tahoe already works well enough for several valuable use cases, and now we just need to enable people to use it. (Meaning, of course, people who are enabled by CLI and XML-RPC... People who are enabled by Windows filesystem integration come next...) This is already mentioned in source:roadmap.txt .

zooko added the

labels 2007-05-25 03:39:47 +00:00

zooko commented

2007-05-28 23:11:31 +00:00

I'm motivated to work on this. The first question is: does the process that you execute block until the operation completes or do we do some funky trick such as it producing a randomly-named temporary file that it later populates when the operation completes? Tahoe (and all of our thinking) is strongly in the non-blocking, event-based paradigm, but I think this is an exception. The purpose of this component is to interface with Unix users and programmers, and Unix users and programmers expect their long-running command-lines to block.

Raising the priority, not because it is "critical", but to indicate that I'm more motivated by this than by some other tickets. Actually perhaps it is "critical". A word like "critical" begs the question "critical for what?", and this ticket is critical for getting certain Unix system administrators to give tahoe a spin.

I'm motivated to work on this. The first question is: does the process that you execute block until the operation completes or do we do some funky trick such as it producing a randomly-named temporary file that it later populates when the operation completes? Tahoe (and all of our thinking) is strongly in the non-blocking, event-based paradigm, but I think this is an exception. The purpose of this component is to interface with Unix users and programmers, and Unix users and programmers expect their long-running command-lines to block. Raising the priority, not because it is "critical", but to indicate that I'm more motivated by this than by some other tickets. Actually perhaps it is "critical". A word like "critical" begs the question "critical for *what*?", and this ticket is critical for getting certain Unix system administrators to give tahoe a spin.

zooko added

critical

and removed

major

labels 2007-05-28 23:11:31 +00:00

warner commented

2007-05-29 19:46:50 +00:00

I think the CLI process should block until the operation completes. I'd like to see a 'tahoe put LOCALFILE VDRIVEFILENAME', 'tahoe get VDRIVEFILENAME LOCALFILE', and then a collection of mv/rm/cp type operations. Actually an interface like scp's would be great (which would imply 'tahoe-cp ~/stuff.txt vdrive:here.txt', or something).

I'm assuming that there is a long-running daemon, and that the CLI tool is a short-lived process that connects to the daemon, tells it to do something, waits for it to indicate completion, emits a message, then quits. If this is the case, both Foolscap and XMLRPC are useful approaches (with either a FURL or a webfront-style "account handle" as the secret authentication bootstrapping pieces). Another technique I've used the past (on buildbot) is to have a special "command directory", like an inbox, which is a Maildir where you can drop in specially-formatted uniquely-named files. A file might have "upload" on the first line, a local filename on the second line, and a vdrive filename on the third line. The daemon could poll this directory every few seconds (or use inotify), execute the commands it finds there, and delete the file when it is done. This reduces the authentication-bootstrap issue to writability of a specific directory, and doesn't require any network for the CLI tool at all.

(incidentally, I'm unsure about using trac's 'priority' field to indicate which tickets we're currently interested in.. we aren't confined to resolving these tickets in any particular order. maybe we need another field named 'coolness' or 'sexyness' for this purpose..)

I think the CLI process should block until the operation completes. I'd like to see a 'tahoe put LOCALFILE VDRIVEFILENAME', 'tahoe get VDRIVEFILENAME LOCALFILE', and then a collection of mv/rm/cp type operations. Actually an interface like scp's would be great (which would imply 'tahoe-cp ~/stuff.txt vdrive:here.txt', or something). I'm assuming that there is a long-running daemon, and that the CLI tool is a short-lived process that connects to the daemon, tells it to do something, waits for it to indicate completion, emits a message, then quits. If this is the case, both Foolscap and XMLRPC are useful approaches (with either a FURL or a webfront-style "account handle" as the secret authentication bootstrapping pieces). Another technique I've used the past (on buildbot) is to have a special "command directory", like an inbox, which is a Maildir where you can drop in specially-formatted uniquely-named files. A file might have "upload" on the first line, a local filename on the second line, and a vdrive filename on the third line. The daemon could poll this directory every few seconds (or use inotify), execute the commands it finds there, and delete the file when it is done. This reduces the authentication-bootstrap issue to writability of a specific directory, and doesn't require any network for the CLI tool at all. (incidentally, I'm unsure about using trac's 'priority' field to indicate which tickets we're currently interested in.. we aren't confined to resolving these tickets in any particular order. maybe we need another field named 'coolness' or 'sexyness' for this purpose..)

zooko commented

2007-05-30 16:43:01 +00:00

Re: trac "priority" field, let's discuss it in #60.

Re: scp interface: Perfect! Thanks.

Re: command-line backend -- how about if we implement #48 -- XML-RPC interface -- and then use that for the command-line client?

Re: trac "priority" field, let's discuss it in #60. Re: scp interface: Perfect! Thanks. Re: command-line backend -- how about if we implement #48 -- XML-RPC interface -- and then use that for the command-line client?

warner commented

2007-06-06 22:39:32 +00:00

in thinking about this more, I think I want to see the client provide a virtual drive (composed of a variety of filetrees) in exactly the same way that a kernel provides a single filesystem (composed of a variety of volumes) to applications. The basic ls/cp/rm/mkdir interface can treat this vdrive as a single filesystem, without being too aware of the pieces that it comprises.

More advanced tools can use additional APIs which expose the filetree pieces, and provide ways to manipulate them (splitting a single filetree into multiple pieces so one can be shared without the other, creating a "virtual CD" snapshot from a given root, etc). But frontends that want to think of the vdrive as just a regular filesystem are free to do so.

in thinking about this more, I think I want to see the client provide a virtual drive (composed of a variety of filetrees) in exactly the same way that a kernel provides a single filesystem (composed of a variety of volumes) to applications. The basic ls/cp/rm/mkdir interface can treat this vdrive as a single filesystem, without being too aware of the pieces that it comprises. More advanced tools can use additional APIs which expose the filetree pieces, and provide ways to manipulate them (splitting a single filetree into multiple pieces so one can be shared without the other, creating a "virtual CD" snapshot from a given root, etc). But frontends that want to think of the vdrive as just a regular filesystem are free to do so.

warner commented

2007-06-07 18:12:51 +00:00

robk rightly points out that we should look at xuilcli (from amdlib), and see if we can adapt that.

warner commented

2007-07-13 03:10:41 +00:00

we have 'tahoe ls' and 'tahoe get' working right now. I think we should declare our 0.5.0 goals to be 'tahoe get -r' and 'tahoe put -r' as well, and then call it a release. Maybe add 'rm' as well. But 'cp' and 'mv' can probably wait for 0.5.1 .

zooko commented

2007-07-23 18:09:52 +00:00

This is a blocker for the v0.5 release.

zooko added

blocker

and removed

critical

labels 2007-07-23 18:09:52 +00:00

zooko commented

2007-08-03 16:52:09 +00:00

Let's make it so that "allmydata-tahoe ls foo/bar.txt" lists bar.txt (if it exists). Currently ls ignores arguments which aren't directories.

zooko commented

2007-08-07 21:35:34 +00:00

Nathan had a good question: why do we have the "-r" flag? If someone asks to "tahoe get" a directory, then what else could they mean other than that they want to recursively get the directory's contents?

warner commented

2007-08-07 22:26:02 +00:00

I suspect the usual answer is that there's a significant probability that they referred to a directory by mistake, and might be surprised to find themselves downloading hundreds of files instead of just one.

A weaker answer is "because all the other tools do it", and consistency has some merits.

And the historical answer is probably that you can actually open and read directories, although the results are not normally useful, and I think more recent unix kernels removed this ability.

I suspect the usual answer is that there's a significant probability that they referred to a directory by mistake, and might be surprised to find themselves downloading hundreds of files instead of just one. A weaker answer is "because all the other tools do it", and consistency has some merits. And the historical answer is probably that you can actually open and read directories, although the results are not normally useful, and I think more recent unix kernels removed this ability.

zooko commented

2007-08-08 02:23:23 +00:00

Good enough for me.

zooko commented

2007-08-09 20:49:36 +00:00

Interesting, "tahoe get ADIRECTORY" currently emits an html page describing that directory. :-)

warner added

code-frontend

and removed

code

labels 2007-08-14 18:52:21 +00:00

zooko commented

2007-08-15 18:46:27 +00:00

Okay, so far we have:

allmydata-tahoe ls
allmydata-tahoe get

We would like to add

allmydata-tahoe put

And that will be good enough for v0.5! (Note that we won't yet support "-r", nor a bunch of features for "ls", etc.)

Okay, so far we have: ``` allmydata-tahoe ls allmydata-tahoe get ``` We would like to add ``` allmydata-tahoe put ``` And that will be good enough for v0.5! (Note that we won't yet support "-r", nor a bunch of features for "ls", etc.)

zooko removed their assignment 2007-08-15 18:46:27 +00:00

warner was assigned by zooko

2007-08-15 18:46:27 +00:00

warner was unassigned by zooko

2007-08-15 19:17:03 +00:00

zooko self-assigned this 2007-08-15 19:17:03 +00:00

zooko commented

2007-08-16 21:51:43 +00:00

Almost done with "put"...