I am a professional Software Engineer, but on the side I do some web stuff for fun and profit. I use CMS Made Simple for 3 years now for a couple of clients. It is much easier for them to use than typo3, joomla, or else and that is the first thing that counts for them.
Unfortunately most updates on my way from 1.5 to 1.10 over the years broke major things for me and forced me to debug the cms code, submit bug reports and locally patch my copy or revert to an older version. I completely stopped using new major versions and started waiting for at least 2 bugfix releases. Now after upgrading from 1.9.4.3 to 10.3 several things broke again. Unfortunately I was forced to stick to the new version, as the 1.9.4.3 tinyMCE is broken in Firefox 11. I managed to fix the problems now (and report again), but this isn't the way it should be. Bugs in releases happen, but frequently having major breaks in releases is not good.
Where I am heading with this post is a suggestion that I believe CMSMS needs better quality assurance, especially in order to catch regressions, things that worked before and suddenly stopped working in a new version. Doing such testing by hand is a lot of work, but automating it can take away a large part of that.
I personally use http://www.simpletest.org/ for that in my own PHP projects and comparable libraries in python, etc. It comes with a programmable browser, which can be used to write test cases, which browse around the site, login as some user, edits some pages, checks if everything works. This is very useful. Even with low effort it already finds a lot of problems. There could be a few test installations of cmsms with different setings and a script which automatically upgrades them and checks for errors after.
I think this could improve the software quality of CMSMS significantly. Usually, even few simple cases can already increase confidence in the software a lot. And that is what automated testing is all about.
Please start automated testing
-
- Support Guru
- Posts: 8169
- Joined: Tue Oct 19, 2004 6:44 pm
- Location: Fernie British Columbia, Canada
Re: Please start automated testing
Like you, I can recommend things for others to do:
I am sure if you volunteered to spend the time to setup and maintain a comprehensive and thorough set of tests for the CMS Made Simple core, and for the third party modules that you are interested in, that our quality assurance people would at least take a look at it.
Alternatively, if you don't want to invest the time to manage all of this stuff, you could get involved in the beta process that we have had for numerous years running.
I am sure if you volunteered to spend the time to setup and maintain a comprehensive and thorough set of tests for the CMS Made Simple core, and for the third party modules that you are interested in, that our quality assurance people would at least take a look at it.
Alternatively, if you don't want to invest the time to manage all of this stuff, you could get involved in the beta process that we have had for numerous years running.
Follow me on twitter
Please post system information from "Extensions >> System Information" (there is a bbcode option) on all posts asking for assistance.
--------------------
If you can't bother explaining your problem well, you shouldn't expect much in the way of assistance.
Please post system information from "Extensions >> System Information" (there is a bbcode option) on all posts asking for assistance.
--------------------
If you can't bother explaining your problem well, you shouldn't expect much in the way of assistance.
Re: Please start automated testing
Ok, go ahead, recommend something .
But honestly, I didn't mean to sound like a bugger, suggesting work to other people I wouldn't do myself. I am involved in voluntary projects, too and I know there are often those people. I am not one of them. I was merely throwing an idea on the table looking for responses.
If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.
I believe we would already gain a lot with only a few very high-level "acceptance" tests operating on the browser level. No need for a comprehensive test suite right away as it would also increase maintenance work. Having a low effort test suite could overall actually save time, as there may be less bugs to fix later.
Btw. I mostly do software engineering, but I am also an ISTQB certified software tester, using tests in multiple projects.
But honestly, I didn't mean to sound like a bugger, suggesting work to other people I wouldn't do myself. I am involved in voluntary projects, too and I know there are often those people. I am not one of them. I was merely throwing an idea on the table looking for responses.
If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.
I believe we would already gain a lot with only a few very high-level "acceptance" tests operating on the browser level. No need for a comprehensive test suite right away as it would also increase maintenance work. Having a low effort test suite could overall actually save time, as there may be less bugs to fix later.
Btw. I mostly do software engineering, but I am also an ISTQB certified software tester, using tests in multiple projects.
Re: Please start automated testing
I think that DEV Team welcomes any help by community, so if you are willing to setup and maintain a testing environment, report results to DEV Team, then why not.cvogt wrote: If you guys support the idea, I am happy to setup a testing framework for cmsms based on simpletest and start adding a few tests. But that only makes sense if there is a general interest for it in the dev/QA team.
Go ahead create a testing environment, run tests, report to DEV Team.
There is also for as good as each major release a Beta cycle, do'nt miss out a chance to report any issues.
I have been reading topics like "Oh they cant release a single release without bugs" but written by people that did not bother to test and report a single issue during beta cycle.
A OpenSource Product can only be as good as it's community, if community doesn't bother to run tests and report issues during Beta cycles then why expect it from few DEV Team members or those few Beta testers that do care and try to test different cases, after all just as everyone else these Members also have their life and daily jobs.
The summary of my reply is:
Sure go ahead setup whatever you think might help, report back, keep it up to date.
If other community members can help you in any test cases, let them know, announce a Forum topic.
But don't let it be like many cases that can be found here in forum, where People start a topic with 110% motivation, but then when it comes to maintaining, reporting results, setting up something actually there is no trace of that motivation and those "motivated" People vanish.
Re: Please start automated testing
Ok, I'll give it a shot and put something together. I'll report back when there are questions or there is anything to look at. Will take a few weeks, depending on how much time I find.
Re: Please start automated testing
I would love to have this. Would make my life a lot easier
Grtz. Rolf
Grtz. Rolf
- + - + - + - + - + - + -
LATEST TUTORIAL AT CMS CAN BE SIMPLE:
Migrating Company Directory module to LISE
Migrating Company Directory module to LISE
- + - + - + - + - + - + -
Re: Please start automated testing
Any news?cvogt wrote:Ok, I'll give it a shot and put something together. I'll report back when there are questions or there is anything to look at. Will take a few weeks, depending on how much time I find.
- + - + - + - + - + - + -
LATEST TUTORIAL AT CMS CAN BE SIMPLE:
Migrating Company Directory module to LISE
Migrating Company Directory module to LISE
- + - + - + - + - + - + -
Re: Please start automated testing
Unfortunately no (like uniqu3 feared). Lack of time and not important to my current job . I still think would significantly help reducing regressions and not particularly hard to do, but I don't see myself doing it these days. Sorry.