CMSMS Forge Rewrite

[Ted]

Hey all,

I have to admit, I don't post in the forums very often these days.  The majority of you non-irc people probably think I've disappeared, but alas, you're not getting rid of me that easily. 

One of the key projects that CMSMS needs to do in order to progress involves taking over the module "situation".  By situation, I mean that it's hard to know what works, what doesn't, where issues lie, etc.  Gforge (which is what the current dev.cmsmadesimple.org is running) is great if you're trying to recreate a sourceforge clone, but it doesn't help us with the more "social" aspects of software development and usage.

We decided well over a year ago that we need to move away from gforge and onto something more custom.  That original idea was to create modules in CMSMS 2.0 using some of the new frontend user features and piecing a new forge clone together.  However, this has proved to not work because 2.0 is taking a long time to develop, and we need something now.

The other issue is that gforge uses an ancient version of postgreSQL which requires us to stay on an older linux distribution... which I'm sure is causing the issues we've been seeing lately with the dev site going down.  The quicker we get over to mysql and a new Linux distro, the better.

So, we've decided to rewrite the parts of gforge we need in a new application written from scratch.  We've pulled the parts we use out, added some "social" features and called it our own.  Again, most of the layout is lifted from gforge with a few additions like:

- Project ratings
- Project comments
- Web 2.0 style project tagging
- Git SCM support
- Simplified trackers
- Project types (so we know what's a language, what's a module etc without needing the complicated categories)

At this point, it's about 80% done and just about ready to go into a beta testing phase.  I'm not totally sure how I'm going to proceed with that yet, because of some incompatibility with the old forge's svn gatekeeper, but I'll come up with something...  or we'll just disable it with proper announcements before hand.

I need about 2 more weeks to finish the development and go into a real beta testing phase, but if you want to look at what's existing so far, the url is http://devnew.cmsmadesimple.org  I make no guarantees it'll be always up or bug free yet, but you're welcome to take a look.  All the automated emails are still going to me, to be gentle. 

There is plenty of room for new features and additions.  Right now the layout and workflow is very similar to gforge...  this was in an effort to save time and not make a lot of huge decisions all at once.

The code is written in a very easily updatable framework.  It allows us to quickly develop new features and roll them out pretty much instantly.  Feature suggestions can go into the Feature Request track on the CMS Forge project here: http://dev.cmsmadesimple.org/tracker/?a ... unc=browse

[Pierre M.]

Hello Ted and all,

This forge rework is nice news : I think the success of a free software project is closely tied to its community and the tools available to it (robust distributed workflow). Hence the importance of the "social" aspects. Successfull free software projects have packages tracking systems, features tracking systems, bugs tracking systems, patches/code tracking systems, translations tracking systems, documentation tracking systems, artwork tracking systems and so on an integrated QA and smooth communication.

If git is the contribution pivot I'd like a very simple tutorial about it in the context of the new forge. Ted had written one in the past but it was about SVN bridging. I'd like a very simple use case like correcting a typo in a translation or submitting an alternative atwork icon. So that anybody (like me) could understand how to contribute easyly (to any part). And maybe a directory of subsystems maintainers would be usefull (who shall I bump to pull my branch?)

There are interesting out-of-mainline hacking tips published in the forum. I'm hoping the new forge is going to lessen the barrier to pull them for mainline inclusion.

And congrats for the rating and comments systems too. I'm hoping it is going to encourage maintainers to communicate and get even better feedback : announces, release notes, tips, documentation updates...
Best wishes to the 20% left.

Pierre M.

[nhaack]

Hello dev-team!

Now, after a year of usage, I pretty much "know" what is available, what works and what not. But as a CMSMS beginner, I found it very difficult to check all the info and try the stuff out. I had a look at the new version and I find it very promissing. I think its handling will ease the use for many starters. I like it

Just an idea, but what do you think about starting with an empty new forge? So only active projects would be showcased in the future. For unmaintained modules, probably the one or other will step in to continue - or the item will be sorted out.

Best
Nils

[Ted]

If git is the contribution pivot I'd like a very simple tutorial about it in the context of the new forge. Ted had written one in the past but it was about SVN bridging. I'd like a very simple use case like correcting a typo in a translation or submitting an alternative atwork icon. So that anybody (like me) could understand how to contribute easyly (to any part). And maybe a directory of subsystems maintainers would be usefull (who shall I bump to pull my branch?)

Git isn't necessarily the contribution pivot, but will be another SCM option.  Granted, it'll be the one I usually pick for my projects, but it's not everyone's cup of tea.

But, assuming people did start to use git, it would be great if we could have a github like system where anyone could easily fork a repository and easily be able to pull branches.  But, it's probably beyond me to try and recreate that.  It would be a good idea to have a "bump devs" kind of button on the Code page for a project with git set as it's SCM, in case someone does have a public repo.  Or at least some decent instructions on how to use git-send-email so that contributing back changes is easy.  Maybe even have the forge show the command line commands required to do it, customized to the project.

And congrats for the rating and comments systems too. I'm hoping it is going to encourage maintainers to communicate and get even better feedback : announces, release notes, tips, documentation updates...

Well, the idea here is to have an early warning system before installing a module.  If a module isn't good and others have used it, you'll know it before you even download it.

Just an idea, but what do you think about starting with an empty new forge? So only active projects would be showcased in the future. For unmaintained modules, probably the one or other will step in to continue - or the item will be sorted out.

One of the first additions I want to put in after it goes live and I figure out how to graph the SCM repository history is have it mark projects as "old" or "deprecated" or something.  If a project hasn't committed or released a file in the last 6 months, then we put some kind of warning graphic up on the project's page.  Hopefully that both allows people to know right away if code is old and maybe look for alternatives, but I'm hoping it'll also encourage other developers to take up old projects and get them up to snuff.

Thanks for the feedback, guys!

[Ted]

Well, fate has thrown in my lap the necessity to expedite this project.  As of this morning, the gforge database has decided to corrupt one of the tables necessary for the tracker to work properly to the point that it's not recoverable.  My newest backup is unfortunately too old to be terribly useful without knocking out some new projects and releases.

Because I had already bought a new VPS for the forge and eventual transition to Ubuntu Server, I have the hardware in place to do the migration.  The forge itself is about 80% done, but could run fine for most of our needs with only a few hours of work.  I've already switched the dev forge posted earlier to the new server, so that part is ready to go.

My goal is have this in "production" by Monday.  Here is a proposed timeline:

Thursday:
- repository creation scripts (done)
- finish up svn gatekeeper (done)
- test migration of good remaining data

Friday:
- sync file release files to S3 (done)
- Clean up some existing admin pages (done)

Saturday and Sunday:
- sync file release files to S3 (again) (done)
- Add way to add version fields to the bug tracker (will automate later from file release creation -- but this is a stop-gap) (done)
- open beta test!!!
- bug fixes and handle feedback
- put in ticket to up VPS to "full size"

Monday:
- do one more database sync
- do one more s3 sync
- point dev.cmsmadesimple.org to the new site
- create redirect rule so that download.php points to s3 for legacy links

This is obviously a "perfect world" scendario, but I think it's doable.  Because we'd be transitioning early, some of the features I wanted to add will have to be pushed out at a later date.  The only major thing I see us losing is download statistics until I write something to parse the S3 logs.  There are probably a couple of other little things, but they'll show up pretty quickly in open beta.

If people have time this weekend, I'd love some testing.  Try and break it if you can so that I can clean up the big bugs before we go live.  If people create projects, I'll make sure to approve them as quick as I can.  Do whatever you'd like, because this will all be reset on Sunday night...  I'll toss all new repositories and database entries.  If you see any bugs, either add them to the cmsforge's project trackers in the new forge (I'll read them before i reset the database), send me an email, or hit me up on irc.  I'll definitely be around the majority of the weekend.

I'll keep this thread up to date as I do stuff and will give the all clear to go crazy on the beta testing.

Thanks!

[Pierre M.]

Hello again,

If git is the contribution pivot I'd like a very simple tutorial about it in the context of the new forge. (...). I'd like a very simple use case like correcting a typo in a translation or submitting an alternative atwork icon. So that anybody (like me) could understand how to contribute easyly (to any part). And maybe a directory of subsystems maintainers would be usefull (who shall I bump to pull my branch?)

Here is an example of what I mean by being able to submit very light contributions : bug #1983.
http://dev.cmsmadesimple.org/tracker/in ... 6&atid=101
This was mid-term blahblah, I didn't want to disturb Ted in the migration process happening this week end.

Pierre M.

[Ted]

In case people haven't noticed, the new forge was setup yesterday.  Please let me know of issues you see by leaving bug reports on the cmsforge project (click the "We're in Beta.  Read more." link that's on every page).

I know people are having issues with IE7 on certain pages.  That's a bit out of my element, so I'm open to suggestions on how to fix it. 

Thanks

[Nick Smart]

Looking good, but... how do I get a list of modules? I'm on the forge homepage, what do I press? The links on the 'Downloads' menu display error pages.

Nick

[Ted]

I don't have a browse feature up yet, because we don't have everything categorized.  The problem with the categorization on the old forge was that it was terribly complicated, people never did it, and it didn't match up very well.

I'm going to put together a project admin today that we can go through and quickly categorize everything and make a browse scenario work.  In the meantime, I'll remove those links from the menu and you can use the project search to find what you're looking for.

Thanks

EDIT: If module devs want to go into the forge and categorize/tag their stuff, that would be super helpful.  Thanks!

[nhaack]

Hi Ted,

I realized that the download counter for current versions was reset to zero. Not a big deal though. However, some users might use this as a guide whether they should download it... (you know... popularity and such)

But anyway... Good job.. thanks
Nils

[Ted]

Yeah, unfortunately, one of the other tables affected by postgresql screwing up was the download stats.  In fact, running a query against it would just cause postgres to restart...

So all download stats are from Oct 30, even though new file releases were able to be rescued.

I won't get to be able to get the counters updating until later this week (or weekend) because since they files live on S3 now, I need to write some kind of log parser thing and periodically run it.  Should be a fun project. 

[duclet]

Just checked this out, and it looked really good. Much cleaner. Can't wait until this goes live. By the way Ted, is that really a picture of you? I swear, you look so much like my Director of Technology at my job.

[nuno]

I know people are having issues with IE7 on certain pages.  That's a bit out of my element, so I'm open to suggestions on how to fix it. 

*cough*

[duclet]

I know people are having issues with IE7 on certain pages.  That's a bit out of my element, so I'm open to suggestions on how to fix it. 

*cough*

Which page?

[Ted]

Quick Update:

Module, plugin and project list are all working.  As more projects get categorized, this will become more useful (obviously).  Reneh is doing a great job categorizing as best he can, but if you want to update your own projects, feel free.  Less work for him.

Download stats from S3 are now being properly logged and stored.  I will write the script to pull those results back into the database (nightly) this week.

We've fixed some other bugs in the last week as well.  It's starting to level out, which is great.  It means that I can go back to 2.0 this week.

Thanks for everyone's help so far in trying to keep this transition smooth.  We're almost done.