MicroTiny and Emojis

For questions and problems with the CMS core. This board is NOT for any 3rd party modules, addons, PHP scripts or anything NOT distributed with the CMS made simple package itself.
Post Reply
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

MicroTiny and Emojis

Post by JamesT »

Whenever I enter emojis into MicroTiny they get converted to a bunch of question marks.

Does MicroTiny support UTF-8 / Unicode?
scooper
Forum Members
Forum Members
Posts: 242
Joined: Fri Dec 09, 2005 12:36 pm
Location: Marlow, UK

Re: MicroTiny and Emojis

Post by scooper »

I've got emojis working in TinyMCE on other projects so I assume MicroTiny will support them ok.

You probably will need to update your database charset - it's going to need to be utf8mb4 (I think the default for CMSM is utf8).

If you have a dev environment you can play around with then you could try updating the cms_content_props table (or just the cms_content_props content field) to utf8mb4 and see if that works. There might be other places you need to change the charset but that's probably a good place to start.
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

Re: MicroTiny and Emojis

Post by JamesT »

Thanks, I have changed the table collation for cms_content_props to utf8mb4 but emojis are still showing up as question marks when submitting with MicroTiny.

Is there anything else I can try?

Screenshot 2021-07-19 190332.jpg
Screenshot 2021-07-19 190332.jpg (8.21 KiB) Viewed 2937 times
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

Re: MicroTiny and Emojis

Post by JamesT »

I've just set up a completely fresh CMSMS installation and emojis come out as question marks when entered into the Content Manager. Something is wrong if emoji support is broken out of the box.

Can a dev look at this please?
User avatar
PinkElephant
Forum Members
Forum Members
Posts: 169
Joined: Fri Feb 06, 2009 2:08 pm

Re: MicroTiny and Emojis

Post by PinkElephant »

JamesT wrote: Wed Jul 21, 2021 10:16 amSomething is wrong if emoji support is broken out of the box.
Speaking as a non-dev, looking at the files... TinyMCE handles the functionality via its emoticons plugin (where you can customise the emoticon database, etc) but it doesn't ship with MicroTiny. The MicroTiny module Help outlines at length why it's been deliberately cut down but arguably, you're unlucky in this case, as emoticons are pure content. Whether the devs would want to support it or not is unknown. For now, TinyMCE looks like the way.
User avatar
Jo Morg
Dev Team Member
Dev Team Member
Posts: 1921
Joined: Mon Jan 29, 2007 4:47 pm

Re: MicroTiny and Emojis

Post by Jo Morg »

To the best of my knowledge this is not a CMSMS core or module issue. The used fonts have to support the Unicode character codes, and that seems to be perfectly explicit on the TinyMCE docs:
TinyMCE Docs wrote:By default, the emoticon plugin inserts Unicode character codes, such as \ud83d\ude03 for the smiley emoji. How emoji are rendered is dependent on the web browser and operating system of the user. As a result of this, some emoji may be rendered in black and white, or may not render. To ensure emoji render consistently across browsers and operating systems, Tiny recommends adding an emoji-compatible web font to the default font-family using content_css.
This affects both MicroTiny and TinyMCE.

HTH.
"There are 10 types of people in this world, those who understand binary... and those who don't."
* by the way: English is NOT my native language (sorry for any mistakes...).
Code of Condut | CMSMS Docs | Help Support CMSMS
My developer Page on the Forge
GeekMoot 2015 in Ghent, Belgium: I was there!
GeekMoot 2016 in Leicester, UK: I was there!
DevMoot 2023 in Cynwyd, Wales: I was there!
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

Re: MicroTiny and Emojis

Post by JamesT »

@Jo Morg

Thanks for your reply. For reference, I am using the latest versions of Windows 10 and Chrome.

I have created a page with just one entry in the stylesheet:

Code: Select all

body { font-family: Arial; }

To the best of my knowledge, Arial supports emojis, so why then are emojis still appearing as question marks when entered into MicroTiny?
User avatar
Jo Morg
Dev Team Member
Dev Team Member
Posts: 1921
Joined: Mon Jan 29, 2007 4:47 pm

Re: MicroTiny and Emojis

Post by Jo Morg »

To clarify, web fonts are installed server side, the default ones are usually part of the server OS installation, and extra can be used via CSS (there is online documentation about how to proceed with their installation). So, most likely, the issues you are experiencing are server side related, not specific to the local device (whether it is a computer or a mobile device) i.e. on one hand the installed fonts on the server OS should be up to date and support the Unicode set of characters needed, but on the other the additional optional web fonts, should they be used, have to be uploaded on the server and the CSS rules have to include these explicitly for them to take effect.
The browser role on all this depends on how they are set up to use fonts, and whether, in some rare cases, they support or not web fonts.
HTH.
"There are 10 types of people in this world, those who understand binary... and those who don't."
* by the way: English is NOT my native language (sorry for any mistakes...).
Code of Condut | CMSMS Docs | Help Support CMSMS
My developer Page on the Forge
GeekMoot 2015 in Ghent, Belgium: I was there!
GeekMoot 2016 in Leicester, UK: I was there!
DevMoot 2023 in Cynwyd, Wales: I was there!
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

Re: MicroTiny and Emojis

Post by JamesT »

I was under the impression that "web safe fonts" (as opposed to "web fonts") are fonts which can be reasonably expected to be pre-installed on the vast majority of web clients. Therefore if Arial is specified, as I did so in my case, there should be no need to upload anything to the server for the font to function correctly in web clients.

Unfortunately, in the case of MicroTiny / CMSMS, this does not seem to be the case.
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

Re: MicroTiny and Emojis

Post by JamesT »

Further to this, consider these two images taken from Chrome:


test1.jpg
test1.jpg (4.38 KiB) Viewed 2823 times


test2.jpg
test2.jpg (4.37 KiB) Viewed 2823 times


The source code for these two pages is identical. The first image was generated by submitting text with inserted emojis via MicroTiny.

The second image was from copy-pasting the raw HTML from the first into a new file, except the question marks were replaced with the same emojis I submitted via MicroTiny. As you can see it displays just fine.

Surely this is enough proof to lay the finger on CMSMS?
scooper
Forum Members
Forum Members
Posts: 242
Joined: Fri Dec 09, 2005 12:36 pm
Location: Marlow, UK

Re: MicroTiny and Emojis

Post by scooper »

I'd still lay the blame at not using utf8mb4 - you'd probably need to make sure your myslqi connection is using the right charset as well.

Posting code in the forums is (rightly) frowned upon but you could try setting the charset (something like $this->_mysql->set_charset("utf8mb4") ) after the database connection is created to see if that helps.

If that is the issue then it is one for the devs to implement, probably via a config option.
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

Re: MicroTiny and Emojis

Post by JamesT »

I would have no idea where to start messing with mysql connection code.

Either way, I have a self-made script which enters data into a utf8 (not utf8mb4) database, and emojis are stored and shown on a web page just fine, with no special character set instructions. Data is entered via a form input field, no WYSIWYG extensions.

So, I do not believe this is a mysql character set issue, I believe the emojis are lost before data is sent to a mysql connection.
User avatar
Dr.CSS
Moderator
Moderator
Posts: 12709
Joined: Thu Mar 09, 2006 5:32 am
Location: Arizona

Re: MicroTiny and Emojis

Post by Dr.CSS »

Try using TinyMCE as microtiny is rather stripped if you ask me...
JamesT
Forum Members
Forum Members
Posts: 175
Joined: Tue Sep 08, 2015 10:41 am

Re: MicroTiny and Emojis

Post by JamesT »

TinyMCE exhibits the same issue, emojis come out as question marks when submitted.
User avatar
velden
Dev Team Member
Dev Team Member
Posts: 3483
Joined: Mon Nov 28, 2011 9:29 am
Location: The Netherlands

Re: MicroTiny and Emojis

Post by velden »

I've been doing some tests and think it may be a CMSMS issue too. We'll look into that.

You've already filed a BR so it's on the todo list to investigate.
http://dev.cmsmadesimple.org/bug/view/12462
Post Reply

Return to “CMSMS Core”