Measure how segment size affects upload/download speed. #809

Open
opened 2009-09-24 06:16:06 +00:00 by zooko · 7 comments

#392 has measurements from Brian showing that reducing segment size from 1 MiB to 128 KiB reduced upload/download speed drastically on a high-speed LAN. See if this is still true. Maybe make a little graph of upload/download speed at different segments sizes. There is a script being run by the buildbot to measure upload/download speed, which you could use:

http://allmydata.org/buildbot/builders/speed-colo

#392 has measurements from Brian showing that reducing segment size from 1 MiB to 128 KiB reduced upload/download speed drastically on a high-speed LAN. See if this is still true. Maybe make a little graph of upload/download speed at different segments sizes. There is a script being run by the buildbot to measure upload/download speed, which you could use: <http://allmydata.org/buildbot/builders/speed-colo>
zooko added the
code-network
major
enhancement
1.5.0
labels 2009-09-24 06:16:06 +00:00
zooko added this to the undecided milestone 2009-09-24 06:16:06 +00:00
Author

If you love this ticket, you might also like #398 (allow users to disable use of helper: direct uploads might be faster) and #320 (add streaming (on-line) upload to HTTP interface).

If you love this ticket, you might also like #398 (allow users to disable use of helper: direct uploads might be faster) and #320 (add streaming (on-line) upload to HTTP interface).
zooko modified the milestone from undecided to 1.7.0 2010-02-27 06:41:46 +00:00
warner was assigned by zooko 2010-02-27 06:45:44 +00:00
Author

If you like this ticket, you might also like the "Brian's New Downloader" bundle of tickets: #800 (improve alacrity by downloading only the part of the Merkle Tree that you need), #798 (improve random-access download to retrieve/decrypt less data), #448 (download: speak to as few servers as possible), #287 (download: tolerate lost or missing servers), and #605 (two-hour delay to connect to a grid from Win32, if there are many storage servers unreachable).

If you like this ticket, you might also like the "Brian's New Downloader" bundle of tickets: #800 (improve alacrity by downloading only the part of the Merkle Tree that you need), #798 (improve random-access download to retrieve/decrypt less data), #448 (download: speak to as few servers as possible), #287 (download: tolerate lost or missing servers), and #605 (two-hour delay to connect to a grid from Win32, if there are many storage servers unreachable).
Author

Brian's New Downloader is now planned for v1.8.0.

Brian's New Downloader is now planned for v1.8.0.
zooko modified the milestone from 1.7.0 to 1.8.0 2010-05-08 22:46:18 +00:00
Author

By the way Brian has big plans to make some sort of beautiful graphical visualization of network behavior/performance. I can't wait! For one thing, I intend to work on upload code in v1.8 and such visualization would help tremendously.

By the way Brian has big plans to make some sort of beautiful graphical visualization of network behavior/performance. I can't wait! For one thing, I intend to work on upload code in v1.8 and such visualization would help tremendously.
Author

Kyle Markley did some good work on this but the results are inconclusive:

More measurements and more analysis of Kyle's measurements (especially his packet traces) is warranted!

See also Nathan Eisenberg's benchmarks of 1.7.1 (standard pipeline and segment size settings) with 1.8.0c1 (standard pipeline and segment size settings):

Kyle Markley did some good work on this but the results are inconclusive: * <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-July/004776.html> * <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-July/004781.html> * <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-July/004782.html> * <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-July/004786.html> * <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-August/004853.html> More measurements and more analysis of Kyle's measurements (especially his packet traces) is warranted! See also Nathan Eisenberg's benchmarks of 1.7.1 (standard pipeline and segment size settings) with 1.8.0c1 (standard pipeline and segment size settings): * <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-August/004930.html>
zooko modified the milestone from 1.8.0 to soon 2010-08-12 20:59:31 +00:00
davidsarah commented 2010-09-14 02:03:29 +00:00
Owner

If #1110 or #1187 (download pipelining) were implemented, then the download performance should in theory be less sensitive to segment size. The segment size could then be reduced to improve memory usage.

If #1110 or #1187 (download pipelining) were implemented, then the download performance should in theory be less sensitive to segment size. The segment size could then be reduced to improve memory usage.
Author

Trunk now has the new visualizer in it, so you could run downloads with different segment sizes and then look at the visualizations to see how the segments sizes changed the network behavior.

Trunk now has the new visualizer in it, so you could run downloads with different segment sizes and then look at the visualizations to see how the segments sizes changed the network behavior.
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#809
No description provided.