downloader: avoid redundant Share.loop calls #1186

New Issue

warner · 2010-08-25T15:01:50Z

warner commented

2010-08-25 15:01:50 +00:00

In #1170 (c92)#comment:92 I mentioned a performance problem in which the downloader (specifically Share._got_response) queues a new call to Share.loop for each message received (which is linear with the number of segments, roughly k3numsegs total). Given the way that messages arrive in the pipeline, most of these calls have no work to do, and each one takes about 250us on my laptop. In a purely local download (running at 4.4MBps), this wastes about 8% of the download time.

The fix is to set a flag when the loop needs to be run, queue a call if-and-only-iff the flag was not already set, and clear the flag when the loop is run.

In [#1170 (c92)](/tahoe-lafs/trac-2024-07-25/issues/6232)#comment:92 I mentioned a performance problem in which the downloader (specifically `Share._got_response`) queues a new call to `Share.loop` for each message received (which is linear with the number of segments, roughly k*3*numsegs total). Given the way that messages arrive in the pipeline, most of these calls have no work to do, and each one takes about 250us on my laptop. In a purely local download (running at 4.4MBps), this wastes about 8% of the download time. The fix is to set a flag when the loop needs to be run, queue a call if-and-only-iff the flag was not already set, and clear the flag when the loop is run.

warner added the

labels 2010-08-25 15:01:50 +00:00

warner added this to the soon milestone 2010-08-25 15:01:50 +00:00

zooko commented

2010-09-06 13:39:31 +00:00

I think you mean [ShareFinder._got_response]source:trunk/src/allmydata/immutable/downloader/finder.py@4707#L89#L167 instead of Share._got_response. Let's see, how does it trigger ShareFinder.loop? It calls [ShareFinder._deliver_shares]source:trunk/src/allmydata/immutable/downloader/finder.py@4707#L89#L217 which eventually calls [DownloadNode.got_shares]source:trunk/src/allmydata/immutable/downloader/node.py@4707#L222 which calls [SegmentFetcher.add_shares]source:trunk/src/allmydata/immutable/downloader/fetcher.py@4707#L65 which eventually calls [SegmentFetcher.loop]source:trunk/src/allmydata/immutable/downloader/fetcher.py@4707#L80.

I think you mean [ShareFinder._got_response]source:trunk/src/allmydata/immutable/downloader/finder.py@4707#L89#L167 instead of `Share._got_response`. Let's see, how does it trigger `ShareFinder.loop`? It calls [ShareFinder._deliver_shares]source:trunk/src/allmydata/immutable/downloader/finder.py@4707#L89#L217 which eventually calls [DownloadNode.got_shares]source:trunk/src/allmydata/immutable/downloader/node.py@4707#L222 which calls [SegmentFetcher.add_shares]source:trunk/src/allmydata/immutable/downloader/fetcher.py@4707#L65 which eventually calls [SegmentFetcher.loop]source:trunk/src/allmydata/immutable/downloader/fetcher.py@4707#L80.

warner commented

2011-09-15 16:20:18 +00:00

Nope, ShareFinder does the DYHB queries, so it's only involved once at the beginning of the download. It looks like I really meant Share._got_data, which is called multiple times per fetched block: one for the block data itself, another couple times for hashes, made worse by the fact that we're doing independent (although pipelined) foolscap remote_read()s, since we don't have an immutable readv storage API.

But it turns out that this is a dup: #1268 is the real ticket, and it was fixed in changeset:c9def7697757bdae.

Nope, `ShareFinder` does the DYHB queries, so it's only involved once at the beginning of the download. It looks like I really meant `Share._got_data`, which is called multiple times per fetched block: one for the block data itself, another couple times for hashes, made worse by the fact that we're doing independent (although pipelined) foolscap `remote_read()`s, since we don't have an immutable `readv` storage API. But it turns out that this is a dup: #1268 is the real ticket, and it was fixed in changeset:c9def7697757bdae.

warner added the

duplicate

label 2011-09-15 16:20:18 +00:00

warner modified the milestone from soon to 1.9.0

2011-09-15 16:20:18 +00:00

warner closed this issue

2011-09-15 16:20:18 +00:00

Sign in to join this conversation.