CMS Made Simple Forums

Whenever I enter emojis into MicroTiny they get converted to a bunch of question marks.

Does MicroTiny support UTF-8 / Unicode?

I've got emojis working in TinyMCE on other projects so I assume MicroTiny will support them ok.

You probably will need to update your database charset - it's going to need to be utf8mb4 (I think the default for CMSM is utf8).

If you have a dev environment you can play around with then you could try updating the cms_content_props table (or just the cms_content_props content field) to utf8mb4 and see if that works. There might be other places you need to change the charset but that's probably a good place to start.

Thanks, I have changed the table collation for cms_content_props to utf8mb4 but emojis are still showing up as question marks when submitting with MicroTiny.

Is there anything else I can try?

I've just set up a completely fresh CMSMS installation and emojis come out as question marks when entered into the Content Manager. Something is wrong if emoji support is broken out of the box.

Can a dev look at this please?

JamesT wrote: ↑Wed Jul 21, 2021 10:16 amSomething is wrong if emoji support is broken out of the box.

Speaking as a non-dev, looking at the files... TinyMCE handles the functionality via its emoticons plugin (where you can customise the emoticon database, etc) but it doesn't ship with MicroTiny. The MicroTiny module Help outlines at length why it's been deliberately cut down but arguably, you're unlucky in this case, as emoticons are pure content. Whether the devs would want to support it or not is unknown. For now, TinyMCE looks like the way.

To the best of my knowledge this is not a CMSMS core or module issue. The used fonts have to support the Unicode character codes, and that seems to be perfectly explicit on the TinyMCE docs:

TinyMCE Docs wrote:By default, the emoticon plugin inserts Unicode character codes, such as \ud83d\ude03 for the smiley emoji. How emoji are rendered is dependent on the web browser and operating system of the user. As a result of this, some emoji may be rendered in black and white, or may not render. To ensure emoji render consistently across browsers and operating systems, Tiny recommends adding an emoji-compatible web font to the default font-family using content_css.

This affects both MicroTiny and TinyMCE.

HTH.

@Jo Morg

Thanks for your reply. For reference, I am using the latest versions of Windows 10 and Chrome.

I have created a page with just one entry in the stylesheet:


To the best of my knowledge, Arial supports emojis, so why then are emojis still appearing as question marks when entered into MicroTiny?

To clarify, web fonts are installed server side, the default ones are usually part of the server OS installation, and extra can be used via CSS (there is online documentation about how to proceed with their installation). So, most likely, the issues you are experiencing are server side related, not specific to the local device (whether it is a computer or a mobile device) i.e. on one hand the installed fonts on the server OS should be up to date and support the Unicode set of characters needed, but on the other the additional optional web fonts, should they be used, have to be uploaded on the server and the CSS rules have to include these explicitly for them to take effect.
The browser role on all this depends on how they are set up to use fonts, and whether, in some rare cases, they support or not web fonts.
HTH.

I was under the impression that "web safe fonts" (as opposed to "web fonts") are fonts which can be reasonably expected to be pre-installed on the vast majority of web clients. Therefore if Arial is specified, as I did so in my case, there should be no need to upload anything to the server for the font to function correctly in web clients.

Unfortunately, in the case of MicroTiny / CMSMS, this does not seem to be the case.

Further to this, consider these two images taken from Chrome:






The source code for these two pages is identical. The first image was generated by submitting text with inserted emojis via MicroTiny.

The second image was from copy-pasting the raw HTML from the first into a new file, except the question marks were replaced with the same emojis I submitted via MicroTiny. As you can see it displays just fine.

Surely this is enough proof to lay the finger on CMSMS?

I'd still lay the blame at not using utf8mb4 - you'd probably need to make sure your myslqi connection is using the right charset as well.

Posting code in the forums is (rightly) frowned upon but you could try setting the charset (something like $this->_mysql->set_charset("utf8mb4") ) after the database connection is created to see if that helps.

If that is the issue then it is one for the devs to implement, probably via a config option.

I would have no idea where to start messing with mysql connection code.

Either way, I have a self-made script which enters data into a utf8 (not utf8mb4) database, and emojis are stored and shown on a web page just fine, with no special character set instructions. Data is entered via a form input field, no WYSIWYG extensions.

So, I do not believe this is a mysql character set issue, I believe the emojis are lost before data is sent to a mysql connection.

Try using TinyMCE as microtiny is rather stripped if you ask me...

TinyMCE exhibits the same issue, emojis come out as question marks when submitted.

I've been doing some tests and think it may be a CMSMS issue too. We'll look into that.

You've already filed a BR so it's on the todo list to investigate.
http://dev.cmsmadesimple.org/bug/view/12462