CMSMS Performance Comparison

[calguy1000]

So... just to get some benchmarks as to how much performance has improved I threw together a little script that would use wget to request a CMSMS page, and grab the performance statistics. And do this 100 times (with a 5 second delay).

I cleared the cache on my stock (virgin) CSMSMS 1.9.4.2 install, and on my stock new install of CMSMS 1.10 SVN. then ran my script on both installs and saved the results. Both of these installs are on my ubuntu LAMP VM running from my desktop machine... so it's not 'quite' a real world experience, but it is fairly close. There is no eaccelerator or other opcode cache on this machine it's a 'stock' ubuntu 10.04 server with the necessary php modules installed to run my various CMSMS modules. My VM is open to the internet, and I have multiple CMSMS installs on it, but it gets very little traffic from the outside (usually only when somebody is helping me to test something).

I then entered the admin interface on both installs, and turned on menu manager caching, and then completely cleared the cache, and re-ran the scripts. When they were done I cleaned up the data (the ascii files needed some parsing to remove html comments etc, and convert it into a CSV file for passing into open office calc.

After all the data was in open office calc I figured out how to create some charts, and created two. The first one shows the amount of php time required to generate the page. The second chart shows the peak memory used to generate the page.

You can see here: http://www.cmsmadesimple.org/uploads/charts.pdf (simple PDF file) that there has been a steady, and significant reduction in processing time required, and a steady and very significant reduction in the memory required to generate a stock page. This is great news for CMSMS and finally starts to quantify the improvements that have been made over the past few versions.

I can share the simple script I used, and the original spreadsheet used to generate the graphs if anybody is interested.

Your thoughts?

Edit:

I modified my spreadsheet today to output some statistics so that I can get a few simple numbers.

PHP Time (over 100 requests)
CMSMS 1.9 (Stock) - Median Time = 0.145353
CMSMS 1.9 With MenuManager Cache - Median Time = 0.124565
CMSMS 1.10 (Stock) - Median Time = 0.136485
CMSMS 1.10 With MenuManager cache - Median Time = 0.1186975
CMSMS 1.10 Stock improved performance over 1.9.4.2 stock by 6.10%
CMSMS 1.10 With MenuManager cache improved performance over 1.9 Stock by 18.34%

Memory Usage:
CMSMS 1.9 Stock - Median Value = 8933560
CMSMS 1.9 With MenuManager Cache - Median Value = 8057768
CMSMS 1.10 Stock - Median Value = 7982184
CMSMS 1.10 With MenuManager Cache - Median Value = 7135060
CMSMS 1.10 Stock improvement over 1.9 Stock = 10.65%
CMSMS 1.10 with MenuManager cache improvement over 1.9 Stock = 20.13%

These are significant performance improvements, again on a (virtual) machine without other performance improvements like Eaccelerator, mod_cache or anything like that. This is a big deal, hope you will enjoy CMSMS 1.10

[otelo]

I would love to have the script you've created. I need to run some tests myself on a news website we implemented and manage.

The site served 801,480 pages the last 30 days with 344,055 visits and 141,704 unique visitors. It's the biggest site we have made so far. It uses the news module and your blog module as well (and many more modules, but the bulk of the site is served through these two modules)

We have had some problems with the server, last Saturday for instance it went down and there were less than 50 people connected. We've had over 400 people on the same time at the site before without any problems.

The site is running on a dedicated Linux server with 8GB of RAM. It's also running an OpenX server to deliver the sites banners and a WordPress site for a side project.

I'm getting a lot of heat for these server crashes and the client is suggesting that it has to do with CMS Made Simple. I love CMS Made Simple and I wouldn't wan to migrate to another CMS, if I could just pinpoint what the problem is.

I greatly appreciate your help!

[faglork]

if I could just pinpoint what the problem is.

Did you look into the server's error_log?
Did you try the debug option of CMSMS?

Cheers,
Alex

[otelo]

I just looked at the error log, the access from the cPanel brings me the last 300 errors and goes back just three days, I haven't had any issues the last 3 days.
Next time I have a problem with the server I'll be sure to go through the error log right away.
As for the debug option I'll try it on a test server and see what shows up there.

[janaf]

calguy1000, nice to see a 10-20% improvement on the 1.9 which was snappy start with. Great!