I am a professional Software Engineer, but on the side I do some web stuff for fun and profit. I use CMS Made Simple for 3 years now for a couple of clients. It is much easier for them to use than typo3, joomla, or else and that is the first thing that counts for them.

Unfortunately most updates on my way from 1.5 to 1.10 over the years broke major things for me and forced me to debug the cms code, submit bug reports and locally patch my copy or revert to an older version. I completely stopped using new major versions and started waiting for at least 2 bugfix releases. Now after upgrading from 1.9.4.3 to 10.3 several things broke again. Unfortunately I was forced to stick to the new version, as the 1.9.4.3 tinyMCE is broken in Firefox 11. I managed to fix the problems now (and report again), but this isn't the way it should be. Bugs in releases happen, but frequently having major breaks in releases is not good.

Where I am heading with this post is a suggestion that I believe CMSMS needs better quality assurance, especially in order to catch regressions, things that worked before and suddenly stopped working in a new version. Doing such testing by hand is a lot of work, but automating it can take away a large part of that.

I personally use http://www.simpletest.org/ for that in my own PHP projects and comparable libraries in python, etc. It comes with a programmable browser, which can be used to write test cases, which browse around the site, login as some user, edits some pages, checks if everything works. This is very useful. Even with low effort it already finds a lot of problems. There could be a few test installations of cmsms with different setings and a script which automatically upgrades them and checks for errors after.

I think this could improve the software quality of CMSMS significantly. Usually, even few simple cases can already increase confidence in the software a lot. And that is what automated testing is all about.

Like you, I can recommend things for others to do:

I am sure if you volunteered to spend the time to setup and maintain a comprehensive and thorough set of tests for the CMS Made Simple core, and for the third party modules that you are interested in, that our quality assurance people would at least take a look at it.

Alternatively, if you don't want to invest the time to manage all of this stuff, you could get involved in the beta process that we have had for numerous years running.

_________________
Follow me on twitter
--
if you can't bother explaining your problem well, you shouldn't expect much in the way of assistance.
----------------
Don't make me angry..... you won't like me when I'm angry....

Ok, go ahead, recommend something .

But honestly, I didn't mean to sound like a bugger, suggesting work to other people I wouldn't do myself. I am involved in voluntary projects, too and I know there are often those people. I am not one of them. I was merely throwing an idea on the table looking for responses.

If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.

I believe we would already gain a lot with only a few very high-level "acceptance" tests operating on the browser level. No need for a comprehensive test suite right away as it would also increase maintenance work. Having a low effort test suite could overall actually save time, as there may be less bugs to fix later.

Btw. I mostly do software engineering, but I am also an ISTQB certified software tester, using tests in multiple projects.

I think that DEV Team welcomes any help by community, so if you are willing to setup and maintain a testing environment, report results to DEV Team, then why not.
Go ahead create a testing environment, run tests, report to DEV Team.

There is also for as good as each major release a Beta cycle, do'nt miss out a chance to report any issues.
I have been reading topics like "Oh they cant release a single release without bugs" but written by people that did not bother to test and report a single issue during beta cycle.
A OpenSource Product can only be as good as it's community, if community doesn't bother to run tests and report issues during Beta cycles then why expect it from few DEV Team members or those few Beta testers that do care and try to test different cases, after all just as everyone else these Members also have their life and daily jobs.

The summary of my reply is:
Sure go ahead setup whatever you think might help, report back, keep it up to date.
If other community members can help you in any test cases, let them know, announce a Forum topic.

But don't let it be like many cases that can be found here in forum, where People start a topic with 110% motivation, but then when it comes to maintaining, reporting results, setting up something actually there is no trace of that motivation and those "motivated" People vanish.

_________________
CMSMS Blog - I do this!
Forge profile
Github profile
Twitter
=============================================
Support CMSMS

Ok, I'll give it a shot and put something together. I'll report back when there are questions or there is anything to look at. Will take a few weeks, depending on how much time I find.

I would love to have this. Would make my life a lot easier

Grtz. Rolf

_________________
Dutch CMSMS community website cmsms.nl and Wiki wiki.cmsms.nl
--------------------------
My CMSMS Tips and Tricks Weblog and website about Pneumatic Tube Systems / Rohrpost
My other (Dutch) website: Smakelijk eten zonder zout (Design: Compufairy)