make docs/performance.rst more precise and accurate #1398

New Issue

zooko · 2011-05-08T12:29:46Z

zooko commented

2011-05-08 12:29:46 +00:00

In comment:13:/tahoe-lafs/trac-2024-07-25/issues/6457, Brian wrote, concerning the "Performing a file-verify on an A-byte file":

to be "N/KS times a small multiple". I think the multiple is currently about 2 or 3. During encryption, we hold both a plaintext share and a ciphertext share in RAM at the same time (so 2S), then we drop the plaintext. During erasure-coding, we hold a whole S of ciphertext in memory at the same time as the N/K*S shares, then we drop the ciphertext before pushing. We also pipeline the sends a little bit, I think 10kB or 50kB per server, to get better utilization out of a non-zero-latency wire.

Also Python's memory-management strategy interacts weirdly. Dropping the plaintext segment may not be enough: Python might not re-use that memory space for anything else right away. Although I'd expect it to de-fragment or coalesce free blocks before asking the OS for so much memory that it crashed.

Although I wonder if Brian was thinking of repair rather than verify since he talks about encrypting, which is not done in verify.

Subsequently I reviewed the document and I see a bunch of things I'm not sure are right. (Note that I myself am mostly responsible for the current state of this document.)

"Publishing an A-byte immutable file" / "when the file is already uploaded": memory footprint: N/K*S. Shouldn't that just be memory footprint: S? All it does is read each S-byte segment in turn and hash it. K and N shouldn't come into it. This is probably just a cut-and-paste error of think-o error on my part originally, so unless someone else knows of a better reason why I wrote that then I'm going to change it to memory footprint: S.
"Publishing an A-byte immutable file" / "when the file is not already uploaded": if we're going to make the memory footprint more precise as Brian suggests above then this one should be changed too. Also network: ~N + ~A should actually be network: N/K*~A, right?
"Downloading B bytes of an A-byte immutable file": cpu: ~A. What? The CPU usage for downloading B bytes of an A-byte immutable file is ~A? I really hope it is actually ~B (plus an amount of CPU logarithmic in A for Merkle Tree verification, but I'm not sure we should try to include that much precision in this document). Unless someone tells me I'm wrong now and was right then, I'm going to change this to cpu: ~B.
"Repairing an A-byte file" network: variable; up to around ~A: surely that should say network: variable; up to around N/K*A

In comment:13:[/tahoe-lafs/trac-2024-07-25/issues/6457](/tahoe-lafs/trac-2024-07-25/issues/6457), Brian wrote, concerning the ["Performing a file-verify on an A-byte file"](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/performance.rst?rev=3523b50716e65575#performing-a-file-verify-on-an-a-byte-file): to be "N/K*S times a small multiple". I think the multiple is currently about 2 or 3. During encryption, we hold both a plaintext share and a ciphertext share in RAM at the same time (so 2*S), then we drop the plaintext. During erasure-coding, we hold a whole S of ciphertext in memory at the same time as the N/K*S shares, then we drop the ciphertext before pushing. We also pipeline the sends a little bit, I think 10kB or 50kB per server, to get better utilization out of a non-zero-latency wire. Also Python's memory-management strategy interacts weirdly. Dropping the plaintext segment may not be enough: Python might not re-use that memory space for anything else right away. Although I'd expect it to de-fragment or coalesce free blocks before asking the OS for so much memory that it crashed. Although I wonder if Brian was thinking of repair rather than verify since he talks about encrypting, which is not done in verify. Subsequently I reviewed the document and I see a bunch of things I'm not sure are right. (Note that I myself am mostly responsible for the current state of this document.) * ["Publishing an A-byte immutable file" / "when the file is already uploaded"](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/performance.rst?rev=3523b50716e65575#publishing-an-a-byte-immutable-file): `memory footprint: N/K*S`. Shouldn't that just be `memory footprint: S`? All it does is read each `S`-byte segment in turn and hash it. `K` and `N` shouldn't come into it. This is probably just a cut-and-paste error of think-o error on my part originally, so unless someone else knows of a better reason why I wrote that then I'm going to change it to `memory footprint: S`. * ["Publishing an A-byte immutable file" / "when the file is not already uploaded"](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/performance.rst?rev=3523b50716e65575#publishing-an-a-byte-immutable-file): if we're going to make the `memory footprint` more precise as Brian suggests above then this one should be changed too. Also `network: ~N + ~A` should actually be `network: N/K*~A`, right? * ["Downloading B bytes of an A-byte immutable file"](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/performance.rst?rev=3523b50716e65575#downloading-b-bytes-of-an-a-byte-immutable-file): `cpu: ~A`. What? The CPU usage for downloading `B` bytes of an `A`-byte immutable file is `~A`? I really hope it is actually `~B` (plus an amount of CPU logarithmic in `A` for Merkle Tree verification, but I'm not sure we should try to include that much precision in this document). Unless someone tells me I'm wrong now and was right then, I'm going to change this to `cpu: ~B`. * ["Repairing an A-byte file"](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/performance.rst?rev=3523b50716e65575#repairing-an-a-byte-file-mutable-or-immutable) `network: variable; up to around ~A`: surely that should say `network: variable; up to around N/K*A`

zooko added the

labels 2011-05-08 12:29:46 +00:00

zooko added this to the undecided milestone 2011-05-08 12:29:46 +00:00

zooko commented

2011-05-08 12:44:11 +00:00

Attachment update-performance.rst.darcs.patch (13143 bytes) added

**Attachment** update-performance.rst.darcs.patch (13143 bytes) added

update-performance.rst.darcs.patch

13 KiB

zooko commented

2011-05-08 12:46:07 +00:00

Please review update-performance.rst.darcs.patch . It fixes all of the issues that I raised in the original ticket description except for the ones that I'm not sure how precise Brian wants it to get or what exact numbers would be correct for the finer precision. It also fixed another issue: the costs of repair have been updated to show both lower bounds (just as cheap as a download) and upper (just as expensive as a full initial upload).

Please review [update-performance.rst.darcs.patch](/tahoe-lafs/trac-2024-07-25/attachments/000078ac-ee02-c292-75e8-39294e3cad62) . It fixes all of the issues that I raised in the original ticket description except for the ones that I'm not sure how precise Brian wants it to get or what exact numbers would be correct for the finer precision. It also fixed another issue: the costs of repair have been updated to show both lower bounds (just as cheap as a download) and upper (just as expensive as a full initial upload).

davidsarah commented

2011-07-28 23:24:30 +00:00

Should this go into 1.9?

zooko commented

2011-07-28 23:52:18 +00:00

+1 from me for this to go into 1.9.

davidsarah commented

2011-07-29 02:54:20 +00:00

It looks OK to me, but Brian is more familiar with the performance characteristics.

tahoe-lafs modified the milestone from undecided to 1.9.0

2011-07-29 02:54:20 +00:00

warner commented

2011-08-26 15:46:15 +00:00

I think I was thinking of encryption/upload in that original
description, not verify: good catch.

publishing immutable when the file is already uploaded: our

Uploader doesn't short-circuit the upload when it discovers that
enough shares are already in place. The upload sequence runs
(including encryption and erasure-coding of each share), but the
results are discarded. (this is arguably an inefficiency, but I'm not
convinced the additional code paths would be worth it). So the memory
footprint is the same as for not-already-uploaded

publishing immutable when the file is not uploaded: do you want

to include the constant multiple of memory footprint that results from
the breadth/overlap of the pipeline? Specifically, we hold both
plaintext and ciphertext in memory briefly, and then hold both
ciphertext and encoded shares simultaneously. That makes it more like
(1+N/K)*S.

publishing mutable file: network is the same as for immutable:
N/K*A, not just A (SDMF has one segment, not one
share). Also, this should probably refer specifically to SDMF, since

MDMF has different characteristics.

Other than that, it looks accurate. Some entries are missing memory
footprints, and most memory footprints could take the constant-multiple
(1+N/K)*S overlap into account if you want to get that detailed.

I think I *was* thinking of encryption/upload in that original description, not verify: good catch. * **publishing immutable when the file is already uploaded**: our > Uploader doesn't short-circuit the upload when it discovers that > enough shares are already in place. The upload sequence runs (including encryption and erasure-coding of each share), but the > results are discarded. (this is arguably an inefficiency, but I'm not > convinced the additional code paths would be worth it). So the memory > footprint is the same as for not-already-uploaded * **publishing immutable when the file is not uploaded**: do you want > to include the constant multiple of memory footprint that results from > the breadth/overlap of the pipeline? Specifically, we hold both > plaintext and ciphertext in memory briefly, and then hold both > ciphertext and encoded shares simultaneously. That makes it more like `(1+N/K)*S`. * **publishing mutable file**: network is the same as for immutable: `N/K*A`, not just `A` (SDMF has one *segment*, not one *share*). Also, this should probably refer specifically to SDMF, since > MDMF has different characteristics. Other than that, it looks accurate. Some entries are missing memory footprints, and most memory footprints could take the constant-multiple `(1+N/K)*S` overlap into account if you want to get that detailed.

warner was unassigned by zooko

2011-08-26 16:57:57 +00:00

zooko self-assigned this 2011-08-26 16:57:57 +00:00

zooko commented

2011-08-30 18:39:54 +00:00

Note to self:

figure out what to do about Brian's comment in the original description
consistentify "Checking", "Verifying", "Repairing"
add references for people who wonder "what the heck is checking/verifying/repairing?"
separate out mutable and immutable for check/verify/repair
separate out (S)FTP access to mutables
add note about fix to memory footprint in 1.9.0
s/memory footprint/memory high water mark/ ?

Note to self: * figure out what to do about Brian's comment in the original description * consistentify "Checking", "Verifying", "Repairing" * add references for people who wonder "what the heck is checking/verifying/repairing?" * separate out mutable and immutable for check/verify/repair * separate out (S)FTP access to mutables * add note about fix to memory footprint in 1.9.0 * s/memory footprint/memory high water mark/ ?

zooko commented

2011-08-30 18:42:47 +00:00

FAQify sickness's question "How big are SDMF's and how big are MDMF's and will there be LDMF's?"
reference FAQ from [performance.rst]source:docs/performance.rst?
reference FAQ and/or performance.rst from [mutable.rst]source:docs/specifications/mutable.rst

* [FAQify](wiki/FAQ) sickness's question "How big are SDMF's and how big are MDMF's and will there be LDMF's?" * reference FAQ from [performance.rst]source:docs/performance.rst? * reference FAQ and/or performance.rst from [mutable.rst]source:docs/specifications/mutable.rst

davidsarah commented

2011-08-30 23:20:57 +00:00

Replying to zooko:

separate out (S)FTP access to mutables

Note that FTP does not support mutables at all (#680).

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1398#issuecomment-83493): > * separate out (S)FTP access to mutables Note that FTP does not support mutables at all (#680).

zooko@zooko.com commented

2011-10-13 19:58:53 +00:00

In changeset:3dc491758daad9df:

docs: fix several imprecise or inaccurate values in performance.rst
add cpu values for each operation
sort the list of values into the same order in each operation
refs #1398

In changeset:3dc491758daad9df: ``` docs: fix several imprecise or inaccurate values in performance.rst add cpu values for each operation sort the list of values into the same order in each operation refs #1398 ```

davidsarah commented

2011-10-13 19:59:43 +00:00

I will fix the points in comment:83490.

I will fix the points in [comment:83490](/tahoe-lafs/trac-2024-07-25/issues/1398#issuecomment-83490).

zooko@zooko.com commented

2011-10-20 20:24:29 +00:00

In [5506/ticket999-S3-backend]:

docs: fix several imprecise or inaccurate values in performance.rst
add cpu values for each operation
sort the list of values into the same order in each operation
refs #1398

In [5506/ticket999-S3-backend]: ``` docs: fix several imprecise or inaccurate values in performance.rst add cpu values for each operation sort the list of values into the same order in each operation refs #1398 ```

warner commented

2011-10-28 04:44:47 +00:00

the remainder of this won't happen in time for 1.9, bumping

warner modified the milestone from 1.9.0 to 1.10.0

2011-10-28 04:44:47 +00:00

warner commented

2012-05-13 02:12:53 +00:00

looks like this is currently owned by davidsarah, and the next step is to edit the patch (to incorporate both my comments and zooko's additional notes). Removing the review-needed flag.

zooko commented

2012-05-13 02:16:19 +00:00

davidsarah: please assign it to me once you've updated the doc or decided that you won't prioritize doing it. Once it is assigned to me, I'll go through all the notes I wrote to myself saying "note to self: do $X; do $Y".

davidsarah: please assign it to me once you've updated the doc or decided that you won't prioritize doing it. Once it is assigned to me, I'll go through all the notes I wrote to myself saying "note to self: do *$X*; do *$Y*".

davidsarah commented

2012-05-13 04:32:01 +00:00

I'm not prioritizing this.

zooko commented

2012-05-13 18:59:04 +00:00

Brian:

I don't entirely understand the more precise estimate of memory footprint during repair that you've suggested (original ticket contents), and I'm not sure it is worth trying to document that level of precision if the absolute difference is going to be only a hundred kilobytes or so.

Would you please either post a patch (or just new text suitable for pasting into source:docs/performance.rst with your suggested improvement, or tell me that the current docs of that are good enough?

Brian: I don't entirely understand the more precise estimate of memory footprint during repair that you've suggested (original ticket contents), and I'm not sure it is worth trying to document that level of precision if the absolute difference is going to be only a hundred kilobytes or so. Would you please either post a patch (or just new text suitable for pasting into source:docs/performance.rst with your suggested improvement, or tell me that the current docs of that are good enough?

zooko removed their assignment 2012-05-13 18:59:04 +00:00

warner was assigned by zooko

2012-05-13 18:59:04 +00:00

Brian Warner <warner@lothar.com> commented

2012-05-13 21:08:06 +00:00

In changeset:e850b54772f4303d:

performance.rst: small updates, mention (lack of) MDMF

refs #1398

In changeset:e850b54772f4303d: ``` performance.rst: small updates, mention (lack of) MDMF refs #1398 ```

Brian Warner <warner@lothar.com> commented

2012-05-13 21:08:32 +00:00

In changeset:e850b54772f4303d:

performance.rst: small updates, mention (lack of) MDMF

refs #1398

In changeset:e850b54772f4303d: ``` performance.rst: small updates, mention (lack of) MDMF refs #1398 ```

warner commented

2012-05-13 21:10:16 +00:00

I added a line about constant factors, that should cover what I mentioned without getting into unnecessary detail.

warner removed their assignment 2012-05-13 21:10:16 +00:00

zooko was assigned by warner

2012-05-13 21:10:16 +00:00

zooko modified the milestone from 1.11.0 to 1.10.0

2012-05-16 22:05:17 +00:00

zooko modified the milestone from 1.10.0 to 1.9.2

2012-05-16 22:05:33 +00:00

zooko was unassigned by tahoe-lafs

2012-06-12 16:57:48 +00:00

warner was assigned by tahoe-lafs

2012-06-12 16:57:48 +00:00

warner commented

2012-06-12 22:46:48 +00:00

No complaints from zooko, so I'm considering this one closed.

warner added the

fixed

label 2012-06-12 22:46:48 +00:00

warner closed this issue

2012-06-12 22:46:48 +00:00

zooko commented

2012-06-13 18:02:14 +00:00

Looks good to me -- thanks!

Sign in to join this conversation.

No Label

LeastAuthority.com automation

code-frontend-ftp-sftp

code-frontend-magic-folder

somebody else's problem

No Milestone

No Assignees

3 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#1398