utf-8 urls / page aliases for SEO and other issues

[kurashiki_ben]

Hi all,

I am running a site in Japanese.
CMSMS is set up to only accept "Letters and numbers" (latin characters and arabic numerals)
(This is from the CheckAliasError function in class.contentoperations.inc.php)

This means that I can only set page aliases in latin characters, which is a bit dissapointing.

I can understand why it was done like this.
If you had a page alias called "ページ" in Japanese, then the url:
http://www.mysite.com/index.php?page=ページ
is displayed:
http://www.mysite.com/index.php?page=ページ in some browsers, which some people might think is ugly.

However, utf-8 urls are a reality in today's internet.
These days there are more and more browsers that can read utf-8 characters in urls provided you have the correct language pack installed. (My browser firefox 3 beta 5 on ubuntu displays Japanese characters in urls perfectly).
You can even have utf-8 domain names now like www.私のサイト.jp

The major disadvantage of being forced to use latin characters for page aliases is SEO.
One of the most important factors in SEO is having your keyword in the url.
It is unfortunte that CMSMS can't do this for non latin-based scripts.

If you look at wikipedia Japan, you can see that all the urls are set up with the page title in the url, (e.g. http://ja.wikipedia.org/wiki/ページ ), which means that it is easy to find the page from a search engine.

Wikipedia are one of the only sites in Japan to have realised the value of using Japanese characters in urls (most people don't even know that it's possible and would romanise "ページ" to "peeji" or use the english equivalent "page").
So it's no suprise that if you search for pratically any general term on google, wikipedia almost always comes out top.　Look at:
http://www.google.co.jp/search?hl=ja&q=ページ
I also typed just "ページ" into my address bar and firefox redirected me straight to http://ja.wikipedia.org/wiki/ページ the power of utf-8!

Converting accented characters in latin based scripts to the non-accented equivalents may make sense, (search engines can recognise "équivalent" as the same as "equivalent"), but as as a webmaster of a site using non-latin based script it is in my opinion a major limitation.
A search engine cannot recognise "peeji" as "ページ", so my page would be outranked by one containing "ページ" in the url if all other factors were the same.
The same goes for russian and greek. I think that there must be many more CMSMS in russian and greek than japanese. I'm sure they would be more than delighted to have the ability to use utf-8 characters in the page alias.
It is a shame that a great product like CMS MS is not currently up to speed. Perhaps it limits its internationability.


A second issue has been brought about by not being able to have utf-8 characters in the url.
The news module automatically takes the article title and puts it in the url.
It strips out non-utf-8 characters and replaces them with equivalents if there are any.
As my news article titles are all utf-8 characters that means I get nothing in the url except a slash.
So I have urls like this http://www.mysite.com/news/2/66//
(Two trailing slashes using pretty urls!)

I know that it is possible to set up equivalents, but there are thousands and thousands of japanese kanji, so it would take me a month to do it!


I saw thay somebody came up with a hack to bypass the check.
http://forum.cmsmadesimple.org/index.ph ... 394.0.html
Is this a decent solution?
Is there a better one?

Overall, I think that it would be better to have no limitation, or at least an option to use utf-8 characters in future versions.
I would apreciate peoples thoughts on this.

regards

Ben

[kurashiki_ben]

simplesimon has posted this possible solution
http://forum.cmsmadesimple.org/index.ph ... 762.0.html