Please start automated testing

A place to discuss the testing process in beta cycles or against SVN for the CMS Made Simple CORE package.
Locked
cvogt
New Member
New Member
Posts: 8
Joined: Sun Jul 12, 2009 11:10 pm

Please start automated testing

Post by cvogt »

I am a professional Software Engineer, but on the side I do some web stuff for fun and profit. I use CMS Made Simple for 3 years now for a couple of clients. It is much easier for them to use than typo3, joomla, or else and that is the first thing that counts for them.

Unfortunately most updates on my way from 1.5 to 1.10 over the years broke major things for me and forced me to debug the cms code, submit bug reports and locally patch my copy or revert to an older version. I completely stopped using new major versions and started waiting for at least 2 bugfix releases. Now after upgrading from 1.9.4.3 to 10.3 several things broke again. Unfortunately I was forced to stick to the new version, as the 1.9.4.3 tinyMCE is broken in Firefox 11. I managed to fix the problems now (and report again), but this isn't the way it should be. Bugs in releases happen, but frequently having major breaks in releases is not good.

Where I am heading with this post is a suggestion that I believe CMSMS needs better quality assurance, especially in order to catch regressions, things that worked before and suddenly stopped working in a new version. Doing such testing by hand is a lot of work, but automating it can take away a large part of that.

I personally use http://www.simpletest.org/ for that in my own PHP projects and comparable libraries in python, etc. It comes with a programmable browser, which can be used to write test cases, which browse around the site, login as some user, edits some pages, checks if everything works. This is very useful. Even with low effort it already finds a lot of problems. There could be a few test installations of cmsms with different setings and a script which automatically upgrades them and checks for errors after.

I think this could improve the software quality of CMSMS significantly. Usually, even few simple cases can already increase confidence in the software a lot. And that is what automated testing is all about.
calguy1000
Support Guru
Support Guru
Posts: 8169
Joined: Tue Oct 19, 2004 6:44 pm
Location: Fernie British Columbia, Canada

Re: Please start automated testing

Post by calguy1000 »

Like you, I can recommend things for others to do:

I am sure if you volunteered to spend the time to setup and maintain a comprehensive and thorough set of tests for the CMS Made Simple core, and for the third party modules that you are interested in, that our quality assurance people would at least take a look at it.

Alternatively, if you don't want to invest the time to manage all of this stuff, you could get involved in the beta process that we have had for numerous years running.
Follow me on twitter
Please post system information from "Extensions >> System Information" (there is a bbcode option) on all posts asking for assistance.
--------------------
If you can't bother explaining your problem well, you shouldn't expect much in the way of assistance.
cvogt
New Member
New Member
Posts: 8
Joined: Sun Jul 12, 2009 11:10 pm

Re: Please start automated testing

Post by cvogt »

Ok, go ahead, recommend something :P.

But honestly, I didn't mean to sound like a bugger, suggesting work to other people I wouldn't do myself. I am involved in voluntary projects, too and I know there are often those people. I am not one of them. I was merely throwing an idea on the table looking for responses.

If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.

I believe we would already gain a lot with only a few very high-level "acceptance" tests operating on the browser level. No need for a comprehensive test suite right away as it would also increase maintenance work. Having a low effort test suite could overall actually save time, as there may be less bugs to fix later.

Btw. I mostly do software engineering, but I am also an ISTQB certified software tester, using tests in multiple projects.
uniqu3

Re: Please start automated testing

Post by uniqu3 »

cvogt wrote: If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.
I think that DEV Team welcomes any help by community, so if you are willing to setup and maintain a testing environment, report results to DEV Team, then why not.
Go ahead create a testing environment, run tests, report to DEV Team.

There is also for as good as each major release a Beta cycle, do'nt miss out a chance to report any issues.
I have been reading topics like "Oh they cant release a single release without bugs" but written by people that did not bother to test and report a single issue during beta cycle.
A OpenSource Product can only be as good as it's community, if community doesn't bother to run tests and report issues during Beta cycles then why expect it from few DEV Team members or those few Beta testers that do care and try to test different cases, after all just as everyone else these Members also have their life and daily jobs.

The summary of my reply is:
Sure go ahead setup whatever you think might help, report back, keep it up to date.
If other community members can help you in any test cases, let them know, announce a Forum topic.

But don't let it be like many cases that can be found here in forum, where People start a topic with 110% motivation, but then when it comes to maintaining, reporting results, setting up something actually there is no trace of that motivation and those "motivated" People vanish.
cvogt
New Member
New Member
Posts: 8
Joined: Sun Jul 12, 2009 11:10 pm

Re: Please start automated testing

Post by cvogt »

Ok, I'll give it a shot and put something together. I'll report back when there are questions or there is anything to look at. Will take a few weeks, depending on how much time I find.
User avatar
Rolf
Dev Team Member
Dev Team Member
Posts: 7825
Joined: Wed Apr 23, 2008 7:53 am
Location: The Netherlands
Contact:

Re: Please start automated testing

Post by Rolf »

I would love to have this. Would make my life a lot easier ;)

Grtz. Rolf
- + - + - + - + - + - + -
LATEST TUTORIAL AT CMS CAN BE SIMPLE:
Migrating Company Directory module to LISE
- + - + - + - + - + - + -
Image
User avatar
Rolf
Dev Team Member
Dev Team Member
Posts: 7825
Joined: Wed Apr 23, 2008 7:53 am
Location: The Netherlands
Contact:

Re: Please start automated testing

Post by Rolf »

cvogt wrote:Ok, I'll give it a shot and put something together. I'll report back when there are questions or there is anything to look at. Will take a few weeks, depending on how much time I find.
Any news?
- + - + - + - + - + - + -
LATEST TUTORIAL AT CMS CAN BE SIMPLE:
Migrating Company Directory module to LISE
- + - + - + - + - + - + -
Image
cvogt
New Member
New Member
Posts: 8
Joined: Sun Jul 12, 2009 11:10 pm

Re: Please start automated testing

Post by cvogt »

Unfortunately no (like uniqu3 feared). Lack of time and not important to my current job :-\. I still think would significantly help reducing regressions and not particularly hard to do, but I don't see myself doing it these days. Sorry.
Locked

Return to “[locked] Quality Assurance”