download: speak to as few servers as possible #448

New Issue

warner · 2008-06-03T05:15:28Z

warner commented

2008-06-03 05:15:28 +00:00

Our old roadmap.txt pointed out, in "Download Peer Selection step 2", that the current simplistic download code asks all known peers for the shares that they hold, whereas we really only need to ask enough peers to get "k" shares (plus a few extra in case we lose some during download, see #287).

This will probably occur about the same time we do a large-scale refactoring of the download code.

Our old roadmap.txt pointed out, in "Download Peer Selection step 2", that the current simplistic download code asks all known peers for the shares that they hold, whereas we really only need to ask enough peers to get "k" shares (plus a few extra in case we lose some during download, see #287). This will probably occur about the same time we do a large-scale refactoring of the download code.

warner added the

labels 2008-06-03 05:15:28 +00:00

warner added this to the undecided milestone 2008-06-03 05:15:28 +00:00

zooko modified the milestone from undecided to 1.7.0

2010-02-27 06:42:15 +00:00

warner was assigned by zooko

2010-02-27 06:44:32 +00:00

zooko changed title from ~~download: only speak to minimal number of peers~~ to download: speak to as few servers as possible

2010-02-27 17:48:44 +00:00

zooko commented

2010-05-08 19:43:50 +00:00

#605 (two-hour delay to connect to a grid from Win32, if there are many storage servers unreachable) is blocked on this ticket because I want to see if fixing this ticket (#448) improves (or otherwise changes) the behavior of the #605 issue.

zooko commented

2010-05-08 19:45:44 +00:00

If you like this ticket, you might also like the "Brian's New Downloader" bundle of tickets: #800 (improve alacrity by downloading only the part of the Merkle Tree that you need), #798 (improve random-access download to retrieve/decrypt less data), #809 (Measure how segment size affects upload/download speed.), #287 (download: tolerate lost or missing servers), and #605 (two-hour delay to connect to a grid from Win32, if there are many storage servers unreachable).

zooko commented

2010-05-08 22:48:46 +00:00

Brian's New Downloader is now planned for v1.8.0.

zooko modified the milestone from 1.7.0 to 1.8.0

2010-05-08 22:48:46 +00:00

davidsarah commented

2010-08-10 04:17:32 +00:00

To what extent does the New Downloader address this ticket?

warner commented

2010-08-14 18:24:05 +00:00

#798 resolves this ticket. The ShareFinder logic will send out multiple queries at once, so we'll wind up with more shares (and talk to more servers) that we strictly need, but it's limited to a window of 10 outstanding requests, which felt reasonable. (using something like 3*k might be more adaptive). And it will ask for more if it loses servers during the download. But it won't talk to the whole grid, which was the concern about this ticket.

So I'm closing this one.

#798 resolves this ticket. The `ShareFinder` logic will send out multiple queries at once, so we'll wind up with more shares (and talk to more servers) that we strictly need, but it's limited to a window of 10 outstanding requests, which felt reasonable. (using something like `3*k` might be more adaptive). And it will ask for more if it loses servers during the download. But it won't talk to the whole grid, which was the concern about this ticket. So I'm closing this one.

warner added the

fixed

label 2010-08-14 18:24:05 +00:00

warner closed this issue

2010-08-14 18:24:05 +00:00

zooko commented

2010-08-15 06:14:40 +00:00

I created #1180 (put more DYHBs into flight at once when K is larger).

Sign in to join this conversation.