I notice the number of apache processes starts tailing off around half to 2/3 of the way through the log, and then just steadily decline until only 1 recorder is left. one gets a lot with non-loggable lines it'd finish sooner). Regarding the steady tailing-off, what I'm wondering is: when you specify lots of recorders, do they each grab an equal number of log lines at the start then work through them? That would explain why some finish earlier than others (if e.g. I may set up a separate VM with a tweaked kernel and optimised apache to deal with log imports anyway, so I'm sure I can improve on that figure. I also tried hacking the script to run a PHP wrapper script that called piwik.php directly on the command line, but it went horribly slowly, presumably because of the lag in loading up PHP.Īnyway, I'm happy with 250-300r/sec. However, running with -dry-run (or just removing the line which actually calls the script) means the python script runs at around 4000r/sec, so I can only conclude the limit is in apache/php (putting in APC definitely helped). gif without doing any processing, but weirdly the performance was about the same. I also tried a few experiments to see where the bottleneck might lie, for instance I stuck in a mod_rewrite to send the importer to a basic PHP file that just returned the. A single recorder manages around 6-7r/sec. if I run at 32, I get more like 200r/sec rather than 250+r/sec. All I know is that the number of records/sec increases pretty much linearly with -recorders up until 40. This is a VM running on a high powered Dell R710, so although the OS only thinks it has 4 CPUs I don't know how things actually pan out. I did quite a few experiments, and eventually found that 40 is about right. You shouldn't have to exceed the number of cores in your system, even a bit lower (as the import script and MySQL will run at the same time). Oliverhumpage: 48 is almost certainly too high, unless you have a 48-core machines.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |