Please start automated testing
Posted: Tue Mar 27, 2012 5:51 pm
I am a professional Software Engineer, but on the side I do some web stuff for fun and profit. I use CMS Made Simple for 3 years now for a couple of clients. It is much easier for them to use than typo3, joomla, or else and that is the first thing that counts for them.
Unfortunately most updates on my way from 1.5 to 1.10 over the years broke major things for me and forced me to debug the cms code, submit bug reports and locally patch my copy or revert to an older version. I completely stopped using new major versions and started waiting for at least 2 bugfix releases. Now after upgrading from 1.9.4.3 to 10.3 several things broke again. Unfortunately I was forced to stick to the new version, as the 1.9.4.3 tinyMCE is broken in Firefox 11. I managed to fix the problems now (and report again), but this isn't the way it should be. Bugs in releases happen, but frequently having major breaks in releases is not good.
Where I am heading with this post is a suggestion that I believe CMSMS needs better quality assurance, especially in order to catch regressions, things that worked before and suddenly stopped working in a new version. Doing such testing by hand is a lot of work, but automating it can take away a large part of that.
I personally use http://www.simpletest.org/ for that in my own PHP projects and comparable libraries in python, etc. It comes with a programmable browser, which can be used to write test cases, which browse around the site, login as some user, edits some pages, checks if everything works. This is very useful. Even with low effort it already finds a lot of problems. There could be a few test installations of cmsms with different setings and a script which automatically upgrades them and checks for errors after.
I think this could improve the software quality of CMSMS significantly. Usually, even few simple cases can already increase confidence in the software a lot. And that is what automated testing is all about.
Unfortunately most updates on my way from 1.5 to 1.10 over the years broke major things for me and forced me to debug the cms code, submit bug reports and locally patch my copy or revert to an older version. I completely stopped using new major versions and started waiting for at least 2 bugfix releases. Now after upgrading from 1.9.4.3 to 10.3 several things broke again. Unfortunately I was forced to stick to the new version, as the 1.9.4.3 tinyMCE is broken in Firefox 11. I managed to fix the problems now (and report again), but this isn't the way it should be. Bugs in releases happen, but frequently having major breaks in releases is not good.
Where I am heading with this post is a suggestion that I believe CMSMS needs better quality assurance, especially in order to catch regressions, things that worked before and suddenly stopped working in a new version. Doing such testing by hand is a lot of work, but automating it can take away a large part of that.
I personally use http://www.simpletest.org/ for that in my own PHP projects and comparable libraries in python, etc. It comes with a programmable browser, which can be used to write test cases, which browse around the site, login as some user, edits some pages, checks if everything works. This is very useful. Even with low effort it already finds a lot of problems. There could be a few test installations of cmsms with different setings and a script which automatically upgrades them and checks for errors after.
I think this could improve the software quality of CMSMS significantly. Usually, even few simple cases can already increase confidence in the software a lot. And that is what automated testing is all about.