Performance hit by a million cuts #4079

Open
opened 2023-12-08 15:34:38 +00:00 by itamarst · 0 comments

Tahoe uses a decent amount of CPU.

Profiling shows just lots and lots and lots of tiny little actions. Question is, is there any obvious culprint beyond "it's Python", and if so, can we do anything about it.

Things already investigated:

  • I measured overhead of using plain Deferreds. I'd estimate it's maybe 1% of CPU usage? So probably not that. In future Python versions this might become more significant depending how good the theoretical proposed JIT ends up being, as callbacks may prevent optimizations at that level. But that's not quite the same thing as overhead.

Things to investigate:

  • inlineCallbacks probably has overhead too. I didn't measure it yet.
  • HTTP overhead. Twisted's HTTP parser probably isn't the fastest.
  • Chunk size. If chunking is done in too small of a unit, this increases all the overhead. Larger chunk sizes will help. See #4080
Tahoe uses a decent amount of CPU. Profiling shows just lots and lots and lots of tiny little actions. Question is, is there any obvious culprint beyond "it's Python", and if so, can we do anything about it. Things already investigated: * I measured overhead of using plain Deferreds. I'd estimate it's maybe 1% of CPU usage? So probably not that. In future Python versions this might become more significant depending how good the theoretical proposed JIT ends up being, as callbacks may prevent optimizations at that level. But that's not quite the same thing as overhead. Things to investigate: * inlineCallbacks probably has overhead too. I didn't measure it yet. * HTTP overhead. Twisted's HTTP parser probably isn't the fastest. * Chunk size. If chunking is done in too small of a unit, this increases all the overhead. Larger chunk sizes will help. See #4080
itamarst added the
unknown
normal
defect
n/a
labels 2023-12-08 15:34:38 +00:00
itamarst added this to the Performance and Benchmarking milestone 2023-12-08 15:34:38 +00:00
Sign in to join this conversation.
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#4079
No description provided.