MODE_WRITE mapupdate: maybe increase epsilon to handle large batches of new servers better #549

Open
opened 2008-12-05 07:18:13 +00:00 by warner · 9 comments

Part of the problem described in #546 is due to adding a large number of new servers. This inserts large gaps in the permuted server list, and the mapupdate code that handles MODE_WRITE may not look far enough to get past the gaps.

It might be a good idea to increase the "epsilon" used for MODE_WRITE from its current value of "k" (usually 3) to "N" (usually 10). As noted at the bottom of #546, if all the servers are writeable, this should be enough to make sure we don't need to send any shares to unqueried servers.

Part of the problem described in #546 is due to adding a large number of new servers. This inserts large gaps in the permuted server list, and the mapupdate code that handles MODE_WRITE may not look far enough to get past the gaps. It might be a good idea to increase the "epsilon" used for MODE_WRITE from its current value of "k" (usually 3) to "N" (usually 10). As noted at the bottom of #546, if all the servers are writeable, this should be enough to make sure we don't need to send any shares to unqueried servers.
warner added the
code-mutable
major
defect
1.2.0
labels 2008-12-05 07:18:13 +00:00
warner added this to the undecided milestone 2008-12-05 07:18:13 +00:00
tahoe-lafs modified the milestone from undecided to 1.7.0 2010-03-25 01:14:59 +00:00

It's really bothering me that mutable file upload and download behavior is so finicky, buggy, inefficient, hard to understand, different from immutable file upload and download behavior, etc. So I'm putting a bunch of tickets into the "1.8" Milestone. I am not, however, at this time, volunteering to work on these tickets, so it might be a mistake to put them into the 1.8 Milestone, but I really hope that someone else will volunteer or that I will decide to do it myself. :-)

It's really bothering me that mutable file upload and download behavior is so finicky, buggy, inefficient, hard to understand, different from immutable file upload and download behavior, etc. So I'm putting a bunch of tickets into the "1.8" Milestone. I am not, however, at this time, volunteering to work on these tickets, so it might be a mistake to put them into the 1.8 Milestone, but I really hope that someone else will volunteer or that I will decide to do it myself. :-)
zooko modified the milestone from 1.7.0 to 1.8.0 2010-05-27 22:09:07 +00:00
tahoe-lafs modified the milestone from 1.8.0 to soon 2010-08-10 04:10:09 +00:00
bazuka commented 2013-11-01 08:53:36 +00:00
Owner

review-needed

review-needed
bazuka commented 2013-11-01 08:56:29 +00:00
Owner
(https://github.com/basu123/tahoe-lafs)
bazuka commented 2013-11-01 08:58:41 +00:00
Owner

Keywords review-needed

Keywords review-needed
daira commented 2013-11-01 12:08:51 +00:00
Owner

The name EPSILON implies that it's a constant. If it's no longer a constant then I think the attribute is no longer needed.

Also a test would be needed. If you're not sure how to write one, please ask us on IRC.

The name EPSILON implies that it's a constant. If it's no longer a constant then I think the attribute is no longer needed. Also a test would be needed. If you're not sure how to write one, please ask us on IRC.
daira commented 2013-11-01 12:21:15 +00:00
Owner

Hmm, it appears that self.EPSILON is already not a constant (and there's some dubious code around source:src/allmydata/mutable/servermap.py#L491 that sets k and N based on assuming encoding defaults of 3/10, which I think should never be reached).

Hmm, it appears that self.EPSILON is already not a constant (and there's some dubious code around source:src/allmydata/mutable/servermap.py#L491 that sets k and N based on assuming encoding defaults of 3/10, which I think should never be reached).

Replying to daira:

Hmm, it appears that self.EPSILON is already not a constant (and there's some dubious code around source:src/allmydata/mutable/servermap.py#L491 that sets k and N based on assuming encoding defaults of 3/10, which I think should never be reached).

Yes, [that code]source:src/allmydata/mutable/servermap.py?annotate=blame&rev=4751a468408d8cc8c0e7d8165237d128741ce08a#L491 should be removed (to remove it is the goal of ticket #1847). That its absence would make it easier for us to evaluate this patch is a good example of why #1847 would be useful.

Replying to [daira](/tahoe-lafs/trac-2024-07-25/issues/549#issuecomment-68511): > Hmm, it appears that self.EPSILON is already not a constant (and there's some dubious code around source:src/allmydata/mutable/servermap.py#L491 that sets k and N based on assuming encoding defaults of 3/10, which I think should never be reached). Yes, [that code]source:src/allmydata/mutable/servermap.py?annotate=blame&rev=4751a468408d8cc8c0e7d8165237d128741ce08a#L491 should be removed (to remove it is the goal of ticket #1847). That its absence would make it easier for us to evaluate this patch is a good example of why #1847 would be useful.

Hey bazuka, how would you like to fix #1847 now? It doesn't require writing new unit tests! Remove the fallback code and make sure all the current unit tests stay green, and you're done.

Hey bazuka, how would you like to fix #1847 now? It doesn't require writing new unit tests! Remove the fallback code and make sure all the current unit tests stay green, and you're done.
bazuka commented 2013-11-01 17:17:53 +00:00
Owner

Replying to zooko:

Hey bazuka, how would you like to fix #1847 now? It doesn't require writing new unit tests! Remove the fallback code and make sure all the current unit tests stay green, and you're done.
Sure Zooko, I'll assign it to myself.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/549#issuecomment-68513): > Hey bazuka, how would you like to fix #1847 now? It doesn't require writing new unit tests! Remove the fallback code and make sure all the current unit tests stay green, and you're done. Sure Zooko, I'll assign it to myself.
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#549
No description provided.