Strange characters on all pages (over 1000 pages) after 1.6 upgrade!

[kurtf]

I have some non-standard or strange characters appearing on all our pages of the CMS (over 1000) after doing an upgrade from 1.51 to 1.6.  I haven't seen a thread that quite matches this dilemma but figure it has something to do with UTF-8 character encoding.  Here is an example of the characters on the pages:

’

I cut and pasted the above from one of the pages which should be an apostrophe and instead is those characters.  I can't go back and manually edit each page, so how do I get TinyMCE to change all of our pages back to normal?  I've uninstalled and re-installed it multiple times through the Module manager and it hasn't changed anything.  It shows version 2.5 as being installed.

This is a huge problem for us, so any help/suggestions are appreciated!

[alby]

I have some non-standard or strange characters appearing on all our pages of the CMS (over 1000) after doing an upgrade from 1.51 to 1.6.  I haven't seen a thread that quite matches this dilemma but figure it has something to do with UTF-8 character encoding.  Here is an example of the characters on the pages:

’

Very strange on upgrade ....
Have you switched some param in config.php?
Look for $config['set_names'] = true; and switch to false

Alby

[kurtf]

Thanks for the quick reply.  That variable was set for true and I changed it to false, but is there a way to get it to take all those characters out of the pages now that they have all been changed?

[jmcgin51]

Kurt,

I had a very similar situation on 2 upgraded sites, except there were injected  characters sprinkled throughout the site.  Not changed from other characters, but new characters added in.  I could not find them via a database text search, using the HTML code (Â) or the actual rendered character (Â), so I had to manually find and replace them in all my pages.  Luckily for me the sites are fairly small.  I have a couple of other sites that I'm reluctant to upgrade because they have a large number of News items which will all need checking.

Can't help you with a fix, but at least you know you're not alone.

[kurtf]

We have that same  character sprinkled throughout in many cases.  That character has replaced the non-breaking space   and inside the HTML code, this is what it now shows:

 

We're going to try and write a script that searches for all of that and replaces it, but we don't want to screw up the database, so here's hoping it all goes smoothly

[mfal55]

Hi Kurtf -
Funny you should post this - I've been dealing with the same issue as well.  I've been using phpmyadmin to search for each item and doing a replace by using:

Code: Select all

update <table_name>
set <field> = replace(<field>,'’','\'');

The unfortunate thing is that it's riddled in many tables with field names that are all labeled differently.  I wish there was a way to do a find/replace for every table and every field in the database - I just haven't found it yet if it is possible.

I hope all goes well with the script.  I would be interested in donating to it if it works! 

Good luck!

[kurtf]

We're now able to make the change so it's correct in the database using the script, but the page is still printing out the wrong code.  Anyone have any ideas?

[kurtf]

Clearing the site-cache did it.  We'll be posting the script shortly for those of you who have also run into this problem.

[Golf Gti]

Nice! I am going through this same problem! I am a database rookie so this is really annoying.

[kenton]

hey everyone,

i've got a working script to fix these characters. mfal55 was right. there are multiple tables that need to be edited. i'm going through and making sure i get all of the default tables (only module is the news module). i'll update this later this afternoon when the script is done.

*** update ***

here's the script for repairing these weird characters. it replaces the weird stuff with what used to be there in numeric form. so instead of   it's   if you use global content blocks for your footer, you will have to replace the © symbol.

http://kg-assets.com/utf-fix.php.zip

the only thing you should modify is whether or not to include the news module. if you want to exclude it, set $news to false in the top of the file. upload it to the same directory as config.php and head to yoursite.com/utf-fix.php

after you've run the script and gotten the okay message. empty your site cache (admin > site admin > advanced setup > clear cache)

that's it. again this worked for us locally and on our live site, but use at your own risk. make sure to backup your database before using it.

[Pierre M.]

We'll be posting the script shortly for those of you who have also run into this problem.

Thanks in advance. This makes a script contest.

Pierre M.

[kenton]

Thanks in advance. This makes a script contest.

Pierre M.

i work with kurtf 

[mfal55]

Hi! Thank you so so much for this!  You guys are brilliant!! This has been incredibly helpful.  A couple of questions:

I see the script only handles the issue of  and â, but my site is riddled with many other characters, for example: ’, �, …, –, “, â€, ‘.  Additionally, they appear in more modules than the regular content and news modules (calendar, cguserdirectory, etc).

Can this script be easily modified to handle those other characters and be replaced across the other modules?  Unfortunately, I am not a developer, but can copy/paste/modify with some direction.

Thanks again for everyone's help on this! 

[kenton]

if you can give me all of the modules that have the weird characters... and what needs to replace the characters, i'll update the script for you.

[mfal55]

Oh you are wonderful!  I'll get right back to you - the site is so large, so I've been finding them as I go along.  I'll compile the list and post back.

Thanks so much!