Performance hit by a million cuts #4079

New Issue

itamarst · 2023-12-08T15:34:38Z

itamarst commented

2023-12-08 15:34:38 +00:00

Tahoe uses a decent amount of CPU.

Profiling shows just lots and lots and lots of tiny little actions. Question is, is there any obvious culprint beyond "it's Python", and if so, can we do anything about it.

Things already investigated:

I measured overhead of using plain Deferreds. I'd estimate it's maybe 1% of CPU usage? So probably not that. In future Python versions this might become more significant depending how good the theoretical proposed JIT ends up being, as callbacks may prevent optimizations at that level. But that's not quite the same thing as overhead.

Things to investigate:

inlineCallbacks probably has overhead too. I didn't measure it yet.
HTTP overhead. Twisted's HTTP parser probably isn't the fastest.
Chunk size. If chunking is done in too small of a unit, this increases all the overhead. Larger chunk sizes will help. See #4080

Tahoe uses a decent amount of CPU. Profiling shows just lots and lots and lots of tiny little actions. Question is, is there any obvious culprint beyond "it's Python", and if so, can we do anything about it. Things already investigated: * I measured overhead of using plain Deferreds. I'd estimate it's maybe 1% of CPU usage? So probably not that. In future Python versions this might become more significant depending how good the theoretical proposed JIT ends up being, as callbacks may prevent optimizations at that level. But that's not quite the same thing as overhead. Things to investigate: * inlineCallbacks probably has overhead too. I didn't measure it yet. * HTTP overhead. Twisted's HTTP parser probably isn't the fastest. * Chunk size. If chunking is done in too small of a unit, this increases all the overhead. Larger chunk sizes will help. See #4080