Dynamically generated but statically served pages

[Pierre M.]

Hello,

I've tried CMSms quickly and I like it much. It is efficient because of simplicity, enforcing the KISS programming philosophy. (I speak more as a user rather than a code reviewer because I haven't looked at the code yet.) And it is GPL Thank you developers of this nice product.

I'm trying to contribute back with this little post about cache/caching, static export, pretty URLs and rewriting. I have overviewed Apache's URL rewriting guide. Please read "On-the-fly Content-Regeneration" on http://httpd.apache.org/docs/2.0/misc/r ... ml#content (scroll down a bit). This tip may be a nice way to easyly get/improve (if needed) 3 powerfull features of CMSms : advanced caching, static export and cool readable URLs. Extract :

"Dynamically generated but statically served pages, i.e. pages should be delivered as pure static pages (read from the filesystem and just passed through), but they have to be generated dynamically by the webserver if missing. This is done via the following ruleset:
[if the static page doesn't exist, dynamically create it and serve the static page.]
RewriteCond %{REQUEST_FILENAME}  !-s
RewriteRule ^page\.html$          page.cgi  [T=application/x-httpd-cgi,L]

Here a request to page.html leads to a internal run of a corresponding page.cgi if page.html is still missing or has filesize null. The trick here is that page.cgi is a usual CGI script which (additionally to its STDOUT) writes its output to the file page.html. Once it was run, the server sends out the data of page.html. When the webmaster wants to force a refresh the contents, he just removes page.html."

OK, it has to be worked "from cgi to PHP" (and to keep CMSms querystring) but it looks feasable, doesn't it ?

I suppose it isn't a big programming task to achieve this. The drawback is that it needs mod_rewrite. The advantages I see are a bugproof implementation of advanced caching and a powerfull scalability because once a page has been served, all subsequent requests for the same page just download a static file (no PHP runtime, no DB access...)

What do you think about that ? Is there any mistake ? a stopper drawback ?

PM

PS : I apologize if this subject has already been posted. I was not able to find it by "search".

[Ted]

What happens if the page changes?  Do you generate a seperate static page for every news article?

I think it would get very tricky, but yes, I can see the value in a high traffic site, that's for sure.  Though, I can also see CMSMS handling this function instead of mod_rewrite.  Do something like movable type does and generate the static pages when some content changes....  Of course, Movable Type is 5x harder than CMSMS to setup. 

[cyberman]

Have everyone done this task? If yes is there a guide available?

[faglork]

This would be the absolute killer feature. Right now it is too slow for many commercial purposes.

Wouldn't it actually be quite simple?

- whenever a page gets changed, write a static (if the page's  "cache" option is turned on)
- when a page gets requested, check if  page is set to "cacheable".
        Yes: get it from cache, if it exists, if not generate and serve from cache
        No: generate and serve

This would even allow the mixing of static/dynamic content. And it would be so much faster.

Alex

[reviewum]

Wow, this seems to be a pretty popular topic.  I didn't know which thread to add to, but I think I'd like to add my thoughts to this one.

Here is why "Dynamically generated but statically served pages" really appeals to me:

1)  Server Resources:  I use really lean applications on my sites, for example, I love PunBB for forums because of how quick and lean it runs.  I see a lot of people having to make huge upgrades in their server environments because their really popular DB intensive scripts are eating up CPU and RAM.

2)  It would be nice to be able to export my whole site (content / images) to static pages.  I'd use these for:

• Backups
• Running a whole site from a CD
• A copy that I could use solely as a static site (if content is no longer being updated, the whole site could be converted to a static site).

Looking forward to your additional thoughts!

[cyberman]

Hmm, perhaps it could be done with playing on smarty variable $cache_lifetime  ...

[moorezilla]

For what it's worth, I would love this feature; it would be the best of both worlds. CMSMS would have the features made possible by a database backend and yet never lose the speed and portability of flatfiles.

Will the devs consider this if I email each of them a case of beer?

[Pierre M.]

Will the devs consider this if I email each of them a case of beer?

As a GPL'ed product, CMSms is "free beer" for sure
I wonder if it is possible to Email beer, but if true, please consider a little bottle for the initial submitter of the idea even if I'm not a dev (that was one year ago

I don't know anything about smarty. Here are my "implementation thoughts", but be aware that I've never read a single ligne of CMSms code 8-#

1-the standard output of the rendered content (result of a hit to a -pretty- URL) should not be send via HTTP but written to a file at content keyboarding time. And may be smarty can do it. May be this is a light patch : write(urlfilename, renderedpage) instead of echo renderedpage.

2-Each time a "dynamic" content is changed (page, news, calendar entry, bloc, template...) all the pages and templates including it should be removed from the filesystem (well, archived...) and rewritten each once, to disk. And there should not be any need of smarty's cache (assuming the webserver or the operating system has its own cache). This should be a more tricky patch.

And please, no heavy export feature : just import static generated pages using one of tar, cp, rsync, wget..

PM

[Piratos]

You can do a lot with smarty:

        $smarty->caching = true;
$smarty->cache_lifetime = 7200;
$smarty->compile_check = false;
                $smarty->use_sub_dir = true;

[moorezilla]

My bad, Pierre. I'll email you one too.

[Pierre M.]

@Piratos : do you mean that my "part 1" (light patch) is just a matter of configuring smarty and it would generate the rendered pages to disk ? Very nice.

@moorezilla : thank you very much. We will Ecelebrate this lovely feature together with the community.

PM

[Kavrick]

I'm not much of a PHP developer but was thinking you could go down this route...

Attach a Custom User Tag to the ContentEditPost Event which basically grabs the output of the PHP page and saves it as a static file. You can use PHP CURL to grab the output really easily and just save it as an HTML File.

That would basically re-create the static page every time it's updated through the CMS. Sounds pretty clean and easy to me.

If somebody (not me) can figure out how to automagically fetch the Title of the page (content_name) in the Custom User Tag, and generate a SEO friendly URL to save the outputted content as, we'd be in business.

Automatically updating static pages with SEO friendly URLS.

David

PS - Also, I think if you pointed the dynamic pages to a sub-directory and then made sure via Robots.txt that those weren't indexed, that would stop Google etc. from thingking you had duplicate content on the server.

[faglork]

That would basically re-create the static page every time it's updated through the CMS. Sounds pretty clean and easy to me.

You also have to check if the page was moved in the page hierarchy. Or if the whole structure of the heirarchy has been changed. In that case, it may be necessary to rereite all pages.

I suggest an option to rewrite the whole shebang either per click or per cron job.

(Shouldnt this thread get moved to "feature requests"?)


Cheers,
Alex

[Piratos]

Make it with smarty - take a look in the smarty manual.

That is easy - you can make it in some minutes.

[faglork]

Fine for you that you can do it. I don't have a clue about smarty. Care to share your wisdom? Or if not, how much would you charge for it?

Cheers,
Alex