UPnP #49

New Issue

zooko · 2007-05-22T19:48:13Z

zooko commented

2007-05-22 19:48:13 +00:00

I would like it if the Tahoe node would talk to a UPnP-enabled firewall and automatically arrange for port-forwarding as needed.

zooko added the

code

minor

enhancement

labels 2007-05-22 19:48:13 +00:00

zooko commented

2007-05-22 20:06:43 +00:00

Here is a free-software UPnP implementation:

https://coherence.beebits.net/wiki

But it doesn't have the internet-gateway-client functionality yet.

Frank Sholz, the author of Coherence, writes:

for the port-forwarding we need to add a internet-gateway-client	
http://upnp.org/standardizeddcps/igd.asp				
it is in the file WANIPConnection					
it looks not difficult						
we need something like this
      https://coherence.beebits.net/browser/trunk/coherence/upnp/devices/media_server_client.py
but with the idg services
but that's then already the device, won't be only afew more lines of python code   
https://coherence.beebits.net/browser/trunk/coherence/upnp/services/clients/av_transport_client.py
and that's the second thing					
a simple mapping of python methods to the soap calls

Here is a free-software UPnP implementation: <https://coherence.beebits.net/wiki> But it doesn't have the internet-gateway-client functionality yet. Frank Sholz, the author of Coherence, writes: ``` for the port-forwarding we need to add a internet-gateway-client http://upnp.org/standardizeddcps/igd.asp it is in the file WANIPConnection it looks not difficult we need something like this https://coherence.beebits.net/browser/trunk/coherence/upnp/devices/media_server_client.py but with the idg services but that's then already the device, won't be only afew more lines of python code https://coherence.beebits.net/browser/trunk/coherence/upnp/services/clients/av_transport_client.py and that's the second thing a simple mapping of python methods to the soap calls ```

warner added

code-network

and removed

code

labels 2007-08-14 18:57:13 +00:00

ghazel commented

2007-10-09 04:03:36 +00:00

Unrelated to the Coherence route, I've had good luck with a python-wrapped version of MiniUPnP, http://miniupnp.free.fr/
If all you're looking for is port mapping, it's probably the smallest way out. If you plan to provide data to set-top boxes and the like, Coherence looks like it is definitely a better way to go. Adding IGD support will get you basic compatability, but there are a lot of ugly parts of dealing with improper implementations on consumer routers.
Just my 2c.

Unrelated to the Coherence route, I've had good luck with a python-wrapped version of MiniUPnP, <http://miniupnp.free.fr/> If all you're looking for is port mapping, it's probably the smallest way out. If you plan to provide data to set-top boxes and the like, Coherence looks like it is definitely a better way to go. Adding IGD support will get you basic compatability, but there are a lot of ugly parts of dealing with improper implementations on consumer routers. Just my 2c.

zooko added this to the undecided milestone 2008-03-21 22:37:07 +00:00

warner modified the milestone from eventually to undecided

2008-06-01 20:53:47 +00:00

zooko commented

2009-07-11 18:03:36 +00:00

I was just trying to make my storage servers for the volunteer grid more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in.

So this is just a note to remind myself that I wish someone would implement this. :-)

I was just trying to make my storage servers for [the volunteer grid](wiki/VolunteerGrid) more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in. So this is just a note to remind myself that I wish someone would implement this. :-)

swillden commented

2009-07-13 01:02:22 +00:00

Replying to zooko:

I was just trying to make my storage servers for the volunteer grid more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in.

So this is just a note to remind myself that I wish someone would implement this. :-)

I've fiddled a little bit with miniupnpc, which ghazel already mentioned. It's a C library but includes a Python wrapper. It's pretty simple. The following code was all it took to tell my gateway (a Linux box running linux-igd) to set up a forwarding for my Tahoe node:

import miniupnpc

u = miniupnpc.UPnP()
u.discoverdelay = 200
u.discover()
u.selectigd()

external_port = 3456
r = u.getspecificportmapping(external_port, 'TCP')
while r != None and external_port < 65536:
    external_port += 1
    r = u.getspecificportmapping(external_port, 'TCP')
u.addportmapping(external_port, 'TCP', u.lanaddr, 3456, "Tahoe %u" % external_port, '')

Obviously, that needs a lot of error checking, but it's still really straightforward. I ran that script from the command line, and watched the log on my gateway to verify that the mapping was set up. I then verified that I can connect to my local Tahoe server from outside. Works fine.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/49#issuecomment-60152): > I was just trying to make my storage servers for [the volunteer grid](wiki/VolunteerGrid) more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in. > > So this is just a note to remind myself that I wish someone would implement this. :-) I've fiddled a little bit with miniupnpc, which ghazel already mentioned. It's a C library but includes a Python wrapper. It's pretty simple. The following code was all it took to tell my gateway (a Linux box running linux-igd) to set up a forwarding for my Tahoe node: ``` import miniupnpc u = miniupnpc.UPnP() u.discoverdelay = 200 u.discover() u.selectigd() external_port = 3456 r = u.getspecificportmapping(external_port, 'TCP') while r != None and external_port < 65536: external_port += 1 r = u.getspecificportmapping(external_port, 'TCP') u.addportmapping(external_port, 'TCP', u.lanaddr, 3456, "Tahoe %u" % external_port, '') ``` Obviously, that needs a lot of error checking, but it's still really straightforward. I ran that script from the command line, and watched the log on my gateway to verify that the mapping was set up. I then verified that I can connect to my local Tahoe server from outside. Works fine.

zooko commented

2009-07-13 12:37:16 +00:00

Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-)

swillden commented

2009-07-13 14:02:45 +00:00

Replying to zooko:

Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-)

Writing tests for it might be challenging, since good tests would require the presence of a UPnP IGD to respond to the registration/de-registration, which would require providing a rudimentary (at least) IGD daemon in the code base just for testing.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/49#issuecomment-60154): > Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-) Writing tests for it might be challenging, since good tests would require the presence of a UPnP IGD to respond to the registration/de-registration, which would require providing a rudimentary (at least) IGD daemon in the code base just for testing.

warner commented

2009-07-15 07:21:01 +00:00

does anyone know what percentage of home users have functional UPnP firewalls? I seem to remember reading somebody's report on the p2p-hackers mailing list that suggested that a large fraction (half?) of the boxes which claimed to support UPnP were in fact completely broken.

zooko commented

2009-07-15 12:43:08 +00:00

(http://zgp.org/pipermail/p2p-hackers/2006-June/003880.html)

Alex Pankratov gave that 50% number on p2p-hackers in 2006. Around the same time allmydata.com implemented UPnP in their proprietary product (the one that preceded Tahoe) and measured the success rate among allmydata.com customers. Unfortunately I don't have or remember the numbers. We should ask Jim McCoy and Rob Kinninmont if they have or remember the results.

I assume that deployment and robustness of UPnP has improved since then. ;-)

Because of foolscap's nice design of using multiple connection hints in parallel and requiring strong SSL authentication, a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right?

(http://zgp.org/pipermail/p2p-hackers/2006-June/003880.html) Alex Pankratov gave that 50% number on p2p-hackers in 2006. Around the same time allmydata.com implemented UPnP in their proprietary product (the one that preceded Tahoe) and measured the success rate among allmydata.com customers. Unfortunately I don't have or remember the numbers. We should ask Jim [McCoy](wiki/McCoy) and Rob Kinninmont if they have or remember the results. I assume that deployment and robustness of UPnP has improved since then. ;-) Because of foolscap's nice design of using multiple connection hints in parallel and requiring strong SSL authentication, a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right?

warner commented

2009-07-15 17:04:04 +00:00

I assume that deployment and robustness of UPnP has improved since then. ;-)

Thanks for the reference. I admire your confidence :)

a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right?

correct. Worst case, the node thinks it is externally visible but is not.

Implementation-wise, we probably need some improvements in the overall IntroducerClient behavior, since the current code assumes that it can get its location hints quickly at startup, and that they never change after that. We need something to handle changes to the location hints at runtime (including the initial determination, which usually requires running /sbin/ifconfig to compute and is thus asynchronous) and then send a new announcement to the Introducer each time. Foolscap is not entirely happy with changing location hints either: FURLs are computed once at startup (specifically at tub.registerReference time), and not updated afterwards, so we'd need some more work to make that behave smoothly.

We should probably implement #445 (relay) and #50 (STUNT) eventually, too.

> I assume that deployment and robustness of UPnP has improved since then. ;-) Thanks for the reference. I admire your confidence :) > a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right? correct. Worst case, the node thinks it is externally visible but is not. Implementation-wise, we probably need some improvements in the overall IntroducerClient behavior, since the current code assumes that it can get its location hints quickly at startup, and that they never change after that. We need something to handle changes to the location hints at runtime (including the initial determination, which usually requires running /sbin/ifconfig to compute and is thus asynchronous) and then send a new announcement to the Introducer each time. Foolscap is not entirely happy with changing location hints either: FURLs are computed once at startup (specifically at `tub.registerReference` time), and not updated afterwards, so we'd need some more work to make that behave smoothly. We should probably implement #445 (relay) and #50 (STUNT) eventually, too.

zooko commented

2009-07-15 17:23:50 +00:00

Thanks for the reference. I admire your confidence :)

Waaataminute, by "admire" you mean "mock", don't you?

As to the testing question that Shawn raised, I think we could write a fake IGD daemon in the test code, for example one which just returns a canned answer in response to the UPnP "are you there" and another canned answer in response to the UPnP "can I have an external port please?", and so on for all the UPnP IGD client functionality that we need. That is slightly complicated (especially if the UPnP IGD client we are using uses irreproducible randomness in its queries and expects the server to echo that randomness), but probably not trickier than the other "network server fakery" that is already in the TahoeLAFS unit tests.

> Thanks for the reference. I admire your confidence :) Waaataminute, by "admire" you mean "mock", don't you? As to the testing question that Shawn raised, I think we could write a fake IGD daemon in the test code, for example one which just returns a canned answer in response to the UPnP "are you there" and another canned answer in response to the UPnP "can I have an external port please?", and so on for all the UPnP IGD client functionality that we need. That is slightly complicated (especially if the UPnP IGD client we are using uses irreproducible randomness in its queries and expects the server to echo that randomness), but probably not trickier than the other "network server fakery" that is already in the TahoeLAFS unit tests.

zooko commented

2009-07-15 17:27:24 +00:00

Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as miniupnp, right? I'm not sure about this.

Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as `miniupnp`, right? I'm not sure about this.

swillden commented

2009-07-16 20:49:01 +00:00

Replying to zooko:

Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as miniupnp, right?

Makes sense to me. If it were necessary to write test cases to validate the correct functioning of every library used, you'd never have time to write any code.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/49#issuecomment-60160): > Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as `miniupnp`, right? Makes sense to me. If it were necessary to write test cases to validate the correct functioning of every library used, you'd never have time to write any code.

zooko commented

2009-09-19 22:14:42 +00:00

If you like this ticket, you may also like #50 (STUNT/ICE), #169 (tcp hole-punching!), #445 (implement relay: allow storage servers behind NAT), and #754 (merge manually specified tub location with autodetected tub location).

jrydberg commented

2009-10-09 16:58:28 +00:00

A simple UPNP library, that also aims at supporting NAT-PMP: http://github.com/jrydberg/natmap

A simple UPNP library, that also aims at supporting NAT-PMP: <http://github.com/jrydberg/natmap>

USSJoin commented

2010-02-21 19:41:05 +00:00

Just another vote (over the three years this ticket has been outstanding) for this. Additional use case, that I know applies to Zooko (as well as many other users): laptops that move from home to work should ideally tell the firewall in each place to let connections through, each time they move. UPnP would make this immediate and automatic. (Just run it on startup, naturally-- not just a one-time configuration.)

Just another vote (over the *three years* this ticket has been outstanding) for this. Additional use case, that I know applies to Zooko (as well as many other users): laptops that move from home to work should ideally tell the firewall in each place to let connections through, each time they move. UPnP would make this immediate and automatic. (Just run it on startup, naturally-- not just a one-time configuration.)

zooko commented

2010-06-17 04:16:12 +00:00

This came up on the mailing list again, as it does every year, and this time I found the CoralCDN stats:

http://tahoe-lafs.org/pipermail/tahoe-dev/2010-June/004469.html

CoralCDN did some large-scale
stats gathering: http://illuminati.coralcdn.org/stats/ . They say that about 75% of the kind of people
who participate in CoralCDN surveys are behind NAT. There's no
timestamp on that page, but judging from the late of their last
publication I guess it was 2007. They also did some measurements of
how much added latency you would incur by going through a relay, which
is something that you might want to take a look at since you've been
hacking on foolscap relay.

This came up on the mailing list again, as it does every year, and this time I found the CoralCDN stats: <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-June/004469.html> CoralCDN did some large-scale stats gathering: <http://illuminati.coralcdn.org/stats/> . They say that about 75% of the kind of people who participate in CoralCDN surveys are behind NAT. There's no timestamp on that page, but judging from the late of their last publication I guess it was 2007. They also did some measurements of how much added latency you would incur by going through a relay, which is something that you might want to take a look at since you've been hacking on foolscap relay.

zooko commented

2010-09-30 05:02:09 +00:00

jrydberg's http://github.com/jrydberg/natmap looks good. A possible next-step for this ticket would be to add a dependency on natmap and invoke natmap some start-up code, possibly from where-ever iputil gets invoked to determine your current IP address(es).

jrydberg's <http://github.com/jrydberg/natmap> looks good. A possible next-step for this ticket would be to add a dependency on natmap and invoke natmap some start-up code, possibly from where-ever `iputil` gets invoked to determine your current IP address(es).

Sign in to join this conversation.