FTP frontend needs tests #512

New Issue

warner · 2008-09-10T20:22:45Z

warner commented

2008-09-10 20:22:45 +00:00

Peter says it would be really useful for us to offer an FTP frontend. This could be done outside the node (in some sort of bridge process, or via a FUSE layer), but I'm currently thinking that it would work best if it were inside the node.

Twisted has some ftp server support, but I haven't looked at it in a long time. My thoughts are:

add a config knob to the tahoe node that turns on FTP support.
add an authorization mechanism: a table that maps username/password to rootcap. It should be possible to configure this to use the allmydata PHP-based login mechanism instead.

Peter says it would be really useful for us to offer an FTP frontend. This could be done outside the node (in some sort of bridge process, or via a FUSE layer), but I'm currently thinking that it would work best if it were inside the node. Twisted has some ftp server support, but I haven't looked at it in a long time. My thoughts are: * add a config knob to the tahoe node that turns on FTP support. * add an authorization mechanism: a table that maps username/password to rootcap. It should be possible to configure this to use the allmydata PHP-based login mechanism instead.

warner added the

labels 2008-09-10 20:22:45 +00:00

warner added this to the undecided milestone 2008-09-10 20:22:45 +00:00

warner commented

2008-09-10 20:25:38 +00:00

zooko suggests we look into pyftpdlib (which is asyncore-based), since it's mature and well-supported

warner commented

2008-10-02 01:21:33 +00:00

cool, Twisted's FTP support is pretty good (although this is in SVN trunk, and it hasn't been released yet). I was able to get a little server running inside a Tahoe node this afternoon, with LIST and GET functionality working.

Next steps:

make GET be streaming instead of the current hack of holding everything in RAM
add PUT, and the other commands (MKDIR, RM, MV)
add an authentication mechanism that either uses a local file of (username, password, rootcap) tuples, or a URL that points to the allmydata.com -style PHP server which accepts (username,password) and returns a rootcap
add unit tests, ftp-client-to-self, disabled unless twisted.vfs is available

And then I want to consider adding per-user dirnode caching, with some simple safety rules like 'flush the whole cache if a MV command is performed'. That should make things pretty snappy.

cool, Twisted's FTP support is pretty good (although this is in SVN trunk, and it hasn't been released yet). I was able to get a little server running inside a Tahoe node this afternoon, with LIST and GET functionality working. Next steps: * make GET be streaming instead of the current hack of holding everything in RAM * add PUT, and the other commands (MKDIR, RM, MV) * add an authentication mechanism that either uses a local file of (username, password, rootcap) tuples, or a URL that points to the allmydata.com -style PHP server which accepts (username,password) and returns a rootcap * add unit tests, ftp-client-to-self, disabled unless twisted.vfs is available And then I want to consider adding per-user dirnode caching, with some simple safety rules like 'flush the whole cache if a MV command is performed'. That should make things pretty snappy.

zooko commented

2008-10-02 01:33:02 +00:00

Sweet! Do you know when to expect the next Twisted release?

warner commented

2008-10-07 01:57:58 +00:00

Nope, unfortunately. http://twistedmatrix.com/trac/roadmap suggests January for the 8.2 release (which would really make it the 9.1 release), but I suspect that their roadmap is about as unrelated to reality as our own...

incidentally, I just pushed the code to trunk. It has full upload/download/mkdir/rm/mv support, and offers two different authentication mechanisms: a file of username/password/rootcap, or a URL of a server that works like peter's native_client.php (the one that Allmydata uses for customer authentication).

What's missing:

unit tests
possible caching, but I'm not going to implement that until we get some experience with it and can decide if it is Fast Enough. Local testing (on my workstation, against the prodnet grid) makes me think that it might be fast enough, although it's worth noting that my 'ncftp' client is doing some fairly aggressive caching of its own.

Nope, unfortunately. <http://twistedmatrix.com/trac/roadmap> suggests January for the 8.2 release (which would really make it the 9.1 release), but I suspect that their roadmap is about as unrelated to reality as our own... incidentally, I just pushed the code to trunk. It has full upload/download/mkdir/rm/mv support, and offers two different authentication mechanisms: a file of username/password/rootcap, or a URL of a server that works like peter's native_client.php (the one that Allmydata uses for customer authentication). What's missing: * unit tests * possible caching, but I'm not going to implement that until we get some experience with it and can decide if it is Fast Enough. Local testing (on my workstation, against the prodnet grid) makes me think that it might be fast enough, although it's worth noting that my 'ncftp' client is doing some fairly aggressive caching of its own.

warner commented

2008-10-09 00:36:54 +00:00

An update on the FTP server: we're running a test instance on a public machine against the prodnet grid. BSD ftp and ncftp appear to work well, but we're having problems getting the Mac OS-X Finder's "Connect To Server" ftp client to work: it looks like the client is not correctly connecting to the PASV-mode data port.

It would be really cool to get OS-X to connect to this, since that would make native access to files work really well.

An update on the FTP server: we're running a test instance on a public machine against the prodnet grid. BSD ftp and ncftp appear to work well, but we're having problems getting the Mac OS-X Finder's "Connect To Server" ftp client to work: it looks like the client is not correctly connecting to the PASV-mode data port. It would be really cool to get OS-X to connect to this, since that would make native access to files work really well.

warner commented

2008-10-09 19:03:53 +00:00

The wait-for-upload patch is in Twisted ticket #3462 .. if someone else could review it, we could probably get it committed, and the next release of Twisted will make deploying the tahoe FTP server a lot easier.

The wait-for-upload patch is in Twisted ticket [#3462](http://twistedmatrix.com/trac/ticket/3462) .. if someone else could review it, we could probably get it committed, and the next release of Twisted will make deploying the tahoe FTP server a lot easier.

warner commented

2008-10-09 19:07:43 +00:00

An update on compatibility: Apparently the OS-X Finder works if you don't have an "@" character in the username. Since Allmydata uses email addresses as account names, we might need to provide some sort of username-mangling function, like, translate "*" or "#" or "~" or something into an "@" before sending the username to the native_client.php service. We want to make sure that ftp: URLs work too, if there is even a spec for them, since "@" is probably used to separate the username from the hostname.

However, Peter and Zandr report that the OS-X FTP mounting code is read-only! Bummer! This was looking so promising. There are plenty of good FTP clients for the Mac, but I was hoping that we could use FTP to provide a simple virtual-drive that was available for all applications.

The Windows FTP mounting code is read-write, and it sounds like it works pretty well.

An update on compatibility: Apparently the OS-X Finder works if you don't have an "@" character in the username. Since Allmydata uses email addresses as account names, we might need to provide some sort of username-mangling function, like, translate "*" or "#" or "~" or something into an "@" before sending the username to the native_client.php service. We want to make sure that ftp: URLs work too, if there is even a spec for them, since "@" is probably used to separate the username from the hostname. However, Peter and Zandr report that the OS-X FTP mounting code is read-only! Bummer! This was looking so promising. There are plenty of good FTP clients for the Mac, but I was hoping that we could use FTP to provide a simple virtual-drive that was available for all applications. The Windows FTP mounting code is read-write, and it sounds like it works pretty well.

zandr commented

2008-10-09 19:50:11 +00:00

So, some digging around this morning, I've learned the following:

The Mac OS X mount_ftp command is read only. This has been a source of much wailing and gnashing of teeth on the intarwebs, but apparently not enough to cause either apple or anyone else to write a r/w replacement. :)

Indeed, changing my email address in the database from zandr@allmydata.com to zandr-allmydata-com allowed the mac to mount the tahoe/ftp server. I'm not sure this is tied to the URL handling stuff, since the mac uses anonymous / cfnetwork@apple.com as a user/pass combo for anonymous FTP.

So, if FTP doesn't work for the mac, that leaves us a couple of alternatives:

WebDAV: Horrid. Run Away. Has the benefit of being well supported by Apple for iDisk.

SFTP: Interesting. Well supported by standalone ftp clients, and it appears that sshfs is actually more like sftpfs. Also, as RobK pointed out, easy to implement alongside FTP in twisted.

I fired up MacFuse and sshfs on my machine pointed at my linux box in colo. Aside from some oddities about getting the drive to show up in the finder (has to be mounted -o local to appear, though I think RobK might know some other magic) it seems pretty solid.

Looking at top on the linux box, it appears that sshfs is using SFTP for the heavy lifting:

5891 n6mod 20 0 69464 3104 1052 S 0 0.2 0:01.00 sshd: n6mod@notty
5892 n6mod 20 0 40036 1992 1300 S 0 0.1 0:00.19 /usr/lib/openssh/sftp-server

So, I think the next step is going to be to run sshfs against an sshd with logging turned up to 11 to see how sshfs works. But I'm out of time for today.

So, some digging around this morning, I've learned the following: The Mac OS X mount_ftp command is read only. This has been a source of much wailing and gnashing of teeth on the intarwebs, but apparently not enough to cause either apple or anyone else to write a r/w replacement. :) Indeed, changing my email address in the database from zandr@allmydata.com to zandr-allmydata-com allowed the mac to mount the tahoe/ftp server. I'm not sure this is tied to the URL handling stuff, since the mac uses anonymous / cfnetwork@apple.com as a user/pass combo for anonymous FTP. So, if FTP doesn't work for the mac, that leaves us a couple of alternatives: WebDAV: Horrid. Run Away. Has the benefit of being well supported by Apple for iDisk. SFTP: Interesting. Well supported by standalone ftp clients, and it appears that sshfs is actually more like sftpfs. Also, as RobK pointed out, easy to implement alongside FTP in twisted. I fired up [MacFuse](wiki/MacFuse) and sshfs on my machine pointed at my linux box in colo. Aside from some oddities about getting the drive to show up in the finder (has to be mounted -o local to appear, though I think RobK might know some other magic) it seems pretty solid. Looking at top on the linux box, it appears that sshfs is using SFTP for the heavy lifting: 5891 n6mod 20 0 69464 3104 1052 S 0 0.2 0:01.00 sshd: n6mod@notty 5892 n6mod 20 0 40036 1992 1300 S 0 0.1 0:00.19 /usr/lib/openssh/sftp-server So, I think the next step is going to be to run sshfs against an sshd with logging turned up to 11 to see how sshfs works. But I'm out of time for today.

zandr commented

2008-10-09 20:05:13 +00:00

OK, I lied. I had to look.

sshfs connects and immediately requests the SFTP subsystem.

So I think that's a win, actually.

OK, I lied. I had to look. sshfs connects and immediately requests the SFTP subsystem. So I think that's a win, actually.

warner commented

2008-10-09 21:33:31 +00:00

zandr: so can the Finder natively mount SFTP as a virtual disk? Or do you have to install MacFuse and sshfs to enable that?

Peter reports that OS-X claims to support the following prefixes in the "Connect To Server" list:

afp, at, file, ftp, http, https, nfs, smb, cifs, vnc

Which I guess excludes SFTP natively.

Hrm, maybe it's time to implement an Appletalk File Protocol server.. how card could it be? :-)

zandr: so can the Finder natively mount SFTP as a virtual disk? Or do you have to install MacFuse and sshfs to enable that? Peter reports that OS-X claims to support the following prefixes in the "Connect To Server" list: * afp, at, file, ftp, http, https, nfs, smb, cifs, vnc Which I guess excludes SFTP natively. Hrm, maybe it's time to implement an Appletalk File Protocol server.. how card could it be? :-)

zandr commented

2008-10-10 17:14:22 +00:00

warner: No, Yes.

SFTP does not work natively. If we want native write access from Macs, I vote for WebDAV. Yeah, it sucks, but not as badly as trying to wire Tahoe into the back of Samba.

But it would be trivial to build up an installer that dropped in MacFuse and sshfs. In fact, the stock MacFuse installer works great, we'd just need to add the sshfs binary, which is currently separate. Add some gratuitous icons and a mount script, and you're good to go. :)

warner: No, Yes. SFTP does not work natively. If we want native write access from Macs, I vote for WebDAV. Yeah, it sucks, but not as badly as trying to wire Tahoe into the back of Samba. But it would be trivial to build up an installer that dropped in [MacFuse](wiki/MacFuse) and sshfs. In fact, the stock [MacFuse](wiki/MacFuse) installer works great, we'd just need to add the sshfs binary, which is currently separate. Add some gratuitous icons and a mount script, and you're good to go. :)

zandr commented

2008-10-10 17:19:31 +00:00

I should add for those playing along at home, this is not the same thing as building a FUSE<>Tahoe interface. This is using FUSE to present sftp to the Finder. sshfs dates back to LUFS, and is pretty robust these days. It was quite responsive connecting to my box in colo from home, much more so than iDisk, for example.

warner commented

2009-01-23 20:37:10 +00:00

Incidentally, rfc-2640 describes how internationalization ought to be done for FTP. We currently have problems handling non-ascii characters in the FTP interface: Tahoe expects/provides UTF-8, but I imagine that without the "FEAT" feature test, a correctly-written client will not provide/handle it. I don't know if the FEAT command is something that's feasible for us to add, or if we'd need to wait for Twisted to provide it.

Incidentally, [rfc-2640](http://www.ietf.org/rfc/rfc2640.txt) describes how internationalization ought to be done for FTP. We currently have problems handling non-ascii characters in the FTP interface: Tahoe expects/provides UTF-8, but I imagine that without the "FEAT" feature test, a correctly-written client will not provide/handle it. I don't know if the FEAT command is something that's feasible for us to add, or if we'd need to wait for Twisted to provide it.

davidsarah commented

2010-01-15 02:44:00 +00:00

(http://twistedmatrix.com/trac/ticket/3462) is waiting for unit tests.

zooko commented

2010-02-06 04:13:31 +00:00

Ed Pimentl and Jack Lloyd are offering bounties for SFTP: http://allmydata.org/pipermail/tahoe-dev/2010-February/003807.html

Ed Pimentl and Jack Lloyd are offering bounties for SFTP: <http://allmydata.org/pipermail/tahoe-dev/2010-February/003807.html>

zooko commented

2010-02-06 04:14:49 +00:00

Ticket #531 is about SFTP, this ticket is about FTP.

tahoe-lafs added

defect

and removed

enhancement

labels 2010-02-07 16:43:10 +00:00

tahoe-lafs modified the milestone from undecided to 1.7.0

2010-02-07 16:43:10 +00:00

tahoe-lafs changed title from ~~FTP frontend~~ to FTP frontend needs tests

2010-02-07 16:43:10 +00:00

davidsarah commented

2010-04-13 17:55:08 +00:00

Not a priority for 1.7, but the SFTP tests (#531) are. We can then port those to FTP for 1.8.

tahoe-lafs modified the milestone from 1.7.0 to 1.8.0

2010-04-13 17:55:08 +00:00

tahoe-lafs modified the milestone from 1.8.0 to 1.7.1

2010-06-21 00:24:29 +00:00

tahoe-lafs modified the milestone from 1.7.1 to soon

2010-07-11 21:32:17 +00:00

lebek commented

2012-03-31 00:07:06 +00:00

The beginnings of a unit test for the FTP frontend are contained in a patch at #1688, but this patch only tests timestamps returned by the FTP LIST command. It does not test that a file added via FTP (rather than by the internal Python API) is created with the right timestamp, so this ticket should incorporate that (along with tests for all the remaining commands).

tahoe-lafs modified the milestone from soon to 1.11.0

2012-04-01 00:57:16 +00:00

tahoe-lafs modified the milestone from 1.11.0 to 1.12.0

2013-08-13 22:52:12 +00:00

warner added

code-frontend-ftp-sftp

and removed

code-frontend

labels 2014-12-02 19:40:38 +00:00

warner commented

2016-03-22 05:02:25 +00:00

Milestone renamed