flogtool scalability/performance #325

New Issue

zooko · 2008-02-28T12:17:37Z

zooko commented

2008-02-28 12:17:37 +00:00

I'm trying to diagnose problems on the current tahoe production grid, and here are some observations:

The logs.pickle file is 2 GB in size, and can't be web-viewed as flogtool web-viewer logs.pickle eventually dies after running out of RAM.
7zipping logs.pickle takes 15 minutes and results in a logs.pickle.7z of 115 MB. (I chose 7zip because I didn't want to use up too much CPU on the tahoe production grid server, and 7zip is one of the most CPU-efficient among the compressors that also give good ratio on a large file.)
flogtool dump logs.pickle &> logs.dump.txt on my Macbook Pro takes 30 minutes and results in a logs.dump.txt of size 415 MB.

So for even a little-used and small grid like the current tahoe production grid, flogtool has some scalability/performance problems.

By the way, in order to estimate how much redundancy there is in the resulting text log file compared to the pickle, I 7zipped the logs.dump.txt, which resulted in a logs.dump.txt.7z of size 40 MB.

I'm trying to diagnose problems on the current tahoe production grid, and here are some observations: 1. The `logs.pickle` file is 2 GB in size, and can't be web-viewed as `flogtool web-viewer logs.pickle` eventually dies after running out of RAM. 2. 7zipping `logs.pickle` takes 15 minutes and results in a `logs.pickle.7z` of 115 MB. (I chose 7zip because I didn't want to use up too much CPU on the tahoe production grid server, and 7zip is one of the most CPU-efficient among the compressors that also give good ratio on a large file.) 3. `flogtool dump logs.pickle &> logs.dump.txt` on my Macbook Pro takes 30 minutes and results in a `logs.dump.txt` of size 415 MB. So for even a little-used and small grid like the current tahoe production grid, flogtool has some scalability/performance problems. By the way, in order to estimate how much redundancy there is in the resulting text log file compared to the pickle, I 7zipped the `logs.dump.txt`, which resulted in a `logs.dump.txt.7z` of size 40 MB.