Make HTTP no slower than Foolscap for mutable uploads and downloads #3957

New Issue

itamarst · 2022-12-14T16:59:02Z

itamarst commented

2022-12-14 16:59:02 +00:00

It seems like the fix for #3954 mostly helps with this, but I am having hard time testing larger files due to #3956, so need to wait until that's fixed to do more testing.

itamarst added the

labels 2022-12-14 16:59:02 +00:00

itamarst added this to the HTTP Storage Protocol milestone 2022-12-14 16:59:02 +00:00

itamarst commented

2023-01-06 16:18:43 +00:00

For a tens-of-MB mutable SDMF upload, with no network latency, HTTP upload is about twice as slow as Foolscap. Downloads are slower, but not as much. The issue seems to mostly be CPU.

With network latency of 60ms, the difference between HTTP and Foolscap becomes less marked. The latency penalty is not terrible.

Conclusion: As a first pass, for SDMF figure out why HTTP is using more CPU.

For a tens-of-MB mutable SDMF upload, with no network latency, HTTP upload is about twice as slow as Foolscap. Downloads are slower, but not as much. The issue seems to mostly be CPU. With network latency of 60ms, the difference between HTTP and Foolscap becomes less marked. The latency penalty is not terrible. **Conclusion:** As a first pass, for SDMF figure out why HTTP is using more CPU.

itamarst commented

2023-01-20 16:44:16 +00:00

Looking at profiling results. 10-15% of CPU time is at least CDDL validation and CBOR decoding. And there's some HTTP parsing. So that's probably a big part of it.

itamarst commented

2023-01-23 13:37:59 +00:00

My first attempt to speed up CDDL validation was a failure. As a short-term solution (and probably a good idea anyway) I'm going to move CDDL validation and CBOR parsing into a thread, so at least it doesn't block the event loop (or block it as much). CDDL validation is the bulk of the time at the moment (which is embarassing, but OK) and it will release the GIL. This won't improve latency, but it will help server parallelism: #3968

itamarst modified the milestone from HTTP Storage Protocol to HTTP Storage Protocol v2

2023-01-23 16:44:54 +00:00

Sign in to join this conversation.