• twitter image
  • facebook image
  • youtube image
  • linkedin image
Language: CMS Made Simple Czech CMS Made Simple France CMS Made Simple Hungary CMS Made Simple Russia CMS Made Simple Netherlands

All times are UTC




Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 8 posts ] 
Author Message
 Post subject: Please start automated testing
PostPosted: Tue Mar 27, 2012 5:51 pm 
Offline
New Member
New Member

Joined: Sun Jul 12, 2009 11:10 pm
Posts: 8
I am a professional Software Engineer, but on the side I do some web stuff for fun and profit. I use CMS Made Simple for 3 years now for a couple of clients. It is much easier for them to use than typo3, joomla, or else and that is the first thing that counts for them.

Unfortunately most updates on my way from 1.5 to 1.10 over the years broke major things for me and forced me to debug the cms code, submit bug reports and locally patch my copy or revert to an older version. I completely stopped using new major versions and started waiting for at least 2 bugfix releases. Now after upgrading from 1.9.4.3 to 10.3 several things broke again. Unfortunately I was forced to stick to the new version, as the 1.9.4.3 tinyMCE is broken in Firefox 11. I managed to fix the problems now (and report again), but this isn't the way it should be. Bugs in releases happen, but frequently having major breaks in releases is not good.

Where I am heading with this post is a suggestion that I believe CMSMS needs better quality assurance, especially in order to catch regressions, things that worked before and suddenly stopped working in a new version. Doing such testing by hand is a lot of work, but automating it can take away a large part of that.

I personally use http://www.simpletest.org/ for that in my own PHP projects and comparable libraries in python, etc. It comes with a programmable browser, which can be used to write test cases, which browse around the site, login as some user, edits some pages, checks if everything works. This is very useful. Even with low effort it already finds a lot of problems. There could be a few test installations of cmsms with different setings and a script which automatically upgrades them and checks for errors after.

I think this could improve the software quality of CMSMS significantly. Usually, even few simple cases can already increase confidence in the software a lot. And that is what automated testing is all about.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Please start automated testing
PostPosted: Tue Mar 27, 2012 6:06 pm 
Offline
Dev Team Member
Dev Team Member
User avatar

Joined: Tue Oct 19, 2004 6:44 pm
Posts: 7960
Location: Fernie British Columbia, Canada
Like you, I can recommend things for others to do:

I am sure if you volunteered to spend the time to setup and maintain a comprehensive and thorough set of tests for the CMS Made Simple core, and for the third party modules that you are interested in, that our quality assurance people would at least take a look at it.

Alternatively, if you don't want to invest the time to manage all of this stuff, you could get involved in the beta process that we have had for numerous years running.

_________________
Follow me on twitter
Please post system information from "Extensions >> System Information" (there is a bbcode option) on all posts asking for assistance.
--------------------
If you can't bother explaining your problem well, you shouldn't expect much in the way of assistance.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Please start automated testing
PostPosted: Tue Mar 27, 2012 7:27 pm 
Offline
New Member
New Member

Joined: Sun Jul 12, 2009 11:10 pm
Posts: 8
Ok, go ahead, recommend something :P.

But honestly, I didn't mean to sound like a bugger, suggesting work to other people I wouldn't do myself. I am involved in voluntary projects, too and I know there are often those people. I am not one of them. I was merely throwing an idea on the table looking for responses.

If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.

I believe we would already gain a lot with only a few very high-level "acceptance" tests operating on the browser level. No need for a comprehensive test suite right away as it would also increase maintenance work. Having a low effort test suite could overall actually save time, as there may be less bugs to fix later.

Btw. I mostly do software engineering, but I am also an ISTQB certified software tester, using tests in multiple projects.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Please start automated testing
PostPosted: Sat Mar 31, 2012 7:48 am 
Offline
Power Poster
Power Poster
User avatar

Joined: Tue Aug 12, 2008 9:30 pm
Posts: 2071
Location: Feldkirchen in Kärnten, Austria
cvogt wrote:
If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.


I think that DEV Team welcomes any help by community, so if you are willing to setup and maintain a testing environment, report results to DEV Team, then why not.
Go ahead create a testing environment, run tests, report to DEV Team.

There is also for as good as each major release a Beta cycle, do'nt miss out a chance to report any issues.
I have been reading topics like "Oh they cant release a single release without bugs" but written by people that did not bother to test and report a single issue during beta cycle.
A OpenSource Product can only be as good as it's community, if community doesn't bother to run tests and report issues during Beta cycles then why expect it from few DEV Team members or those few Beta testers that do care and try to test different cases, after all just as everyone else these Members also have their life and daily jobs.

The summary of my reply is:
Sure go ahead setup whatever you think might help, report back, keep it up to date.
If other community members can help you in any test cases, let them know, announce a Forum topic.

But don't let it be like many cases that can be found here in forum, where People start a topic with 110% motivation, but then when it comes to maintaining, reporting results, setting up something actually there is no trace of that motivation and those "motivated" People vanish.

_________________
CMSMS Blog - I do this!
Forge profile
Github profile
Twitter
=============================================
Support CMSMS


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Please start automated testing
PostPosted: Sat Mar 31, 2012 9:16 am 
Offline
New Member
New Member

Joined: Sun Jul 12, 2009 11:10 pm
Posts: 8
Ok, I'll give it a shot and put something together. I'll report back when there are questions or there is anything to look at. Will take a few weeks, depending on how much time I find.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Please start automated testing
PostPosted: Sat Mar 31, 2012 5:10 pm 
Offline
Dev Team Member
Dev Team Member
User avatar

Joined: Wed Apr 23, 2008 7:53 am
Posts: 7586
Location: The Netherlands
I would love to have this. Would make my life a lot easier ;)

Grtz. Rolf

_________________
Image

Did my post help you solving a problem at your (customers) website and it saved you many hours of work? Great!! Consider buying me a cup of coffee in return! [ Click here ]



Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Please start automated testing
PostPosted: Fri Oct 11, 2013 9:28 pm 
Offline
Dev Team Member
Dev Team Member
User avatar

Joined: Wed Apr 23, 2008 7:53 am
Posts: 7586
Location: The Netherlands
cvogt wrote:
Ok, I'll give it a shot and put something together. I'll report back when there are questions or there is anything to look at. Will take a few weeks, depending on how much time I find.
Any news?

_________________
Image

Did my post help you solving a problem at your (customers) website and it saved you many hours of work? Great!! Consider buying me a cup of coffee in return! [ Click here ]



Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Please start automated testing
PostPosted: Sat Oct 12, 2013 12:59 am 
Offline
New Member
New Member

Joined: Sun Jul 12, 2009 11:10 pm
Posts: 8
Unfortunately no (like uniqu3 feared). Lack of time and not important to my current job :-\. I still think would significantly help reducing regressions and not particularly hard to do, but I don't see myself doing it these days. Sorry.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 8 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
A2 Hosting