Page 1 of 1

v3.0.0.8 mailwatch unicode support

Posted: 15 Aug 2015 09:30
by pdwalker
Hi Shawn,

The mailwatch upgrade in 3.0.0.8 is supposed to support unicode.

When I send myself a message, the message list shows the same improperly decoded gibberish it always used to show.
e.g.
Subject: chinese test: 中国
but I get
Subject: chinese test: 谢谢
I was kinda hoping to see the original Chinese characters in the message list.

Am I doing something wrong, or do I have something misconfigured? Could it be my mysql databases are not utf8?

Code: Select all

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| efa                |
| mailscanner        |
| mysql              |
| sa_bayes           |
| sbtest             |
| sbtest1            |
| sqlgrey            |
+--------------------+
show create database (*)

Code: Select all

+-------------+----------------------------------------------------------------------------------------------+
| Database    | Create Database                                                                              |
+-------------+----------------------------------------------------------------------------------------------+
| efa         | CREATE DATABASE `efa` /*!40100 DEFAULT CHARACTER SET latin1 */                               |
| mailscanner | CREATE DATABASE `mailscanner` /*!40100 DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci */ |
| sa_bayes    | CREATE DATABASE `sa_bayes` /*!40100 DEFAULT CHARACTER SET latin1 */                          |
| sqlgrey     | CREATE DATABASE `sqlgrey` /*!40100 DEFAULT CHARACTER SET latin1 */                           |
+-------------+----------------------------------------------------------------------------------------------+
Will the non UTF8 databases cause this problem?

The https://efa.local/mailscanner/status.php page is in UTF8 encoding.

(if anyone else has working unicode displaying foreign characters in your mailscanner interface, I'd love to compare notes with you)

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 08:26
by shawniverson
Confirmed issue.

The non utf8 databases are not the issue. The data is stored in mailscanner db. which shows as utf8.

It appears that MailWatch is not rendering in unicode. Checking source....

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 08:57
by shawniverson
Looks like problem is in functions.php under db_colorised_table...

If I query the db and display raw data, the characters render properly...

Still digging....

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 09:04
by pdwalker
need access to my system for a comparison?

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 09:13
by shawniverson
This is going to take some work up in mailwatch development....

The new getUTF8String function in the latest mailwatch code isn't doing its job...so even updating mailwatch doesn't seem to resolve the issue.

Ironically, if I remove getUTF8String, it works.

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 09:15
by shawniverson

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 10:02
by EndelWar
@pdwalker: do you have mbstring extension installed on your system? and also: do you see these strange chars only indetail view or also in frontpage (status.php)? is the maillog table updated to utf8?

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 10:07
by pdwalker
Good question, let me find out.

Mbstring is a php extension, correct?

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 10:25
by pdwalker
In detail.php, the subject is the same as in status.php

In viewmail.php,

the subject is the same gibberish,
The text/plain rendering is the same gibberish
The text/htmlNow rendering is correct

Phpinfo() doesn't show mbstring installed. Confirmed, not installed. Should I install it?

Maillog is in UTF8 format

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 10:28
by shawniverson
do you have mbstring extension installed on your system? and also: do you see these strange chars only indetail view or also in frontpage (status.php)? is the maillog table updated to utf8?
I have mbstring installed ...text is still not displaying correctly for me. Can you confirm pdwalker? I also updated mailwatch to latest snapshot.

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 10:30
by shawniverson
Wait, restarted httpd, now its golden!

I think this may be resolved.

Code: Select all

sudo yum install php-mbstring
sudo service httpd restart

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 12:16
by pdwalker
I'm an hour away from my computer. I'll try those two steps shortly.

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 13:29
by EndelWar
mbstring extension makes life easier, but a solution that works also without extension should be tested, I've done some research and I think that I can do some experiments with Portable-utf8 (https://github.com/voku/portable-utf8) and Patchwork UTF8 (https://github.com/tchwork/utf8) (the latter is used in Laravel framework)

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 15:14
by pdwalker
EndelWar wrote:mbstring extension makes life easier, but a solution that works also without extension should be tested, I've done some research and I think that I can do some experiments with Portable-utf8 (https://github.com/voku/portable-utf8) and Patchwork UTF8 (https://github.com/tchwork/utf8) (the latter is used in Laravel framework)
One difficulty with using these packages is that they are not in the public repositories which makes the system harder to maintain. It is almost always better to go with prebuilt packages.

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 15:30
by pdwalker

Code: Select all

sudo yum install php-mbstring
sudo service httpd restart
results:
mbstring successfully installed.

status.php -> most message subjects displayed correctly. One message appeared like this:
商業電話專線 每月只需HK$60起 直\xE9\xA7\x81手提/飛線國內號碼
Maybe mbstring cannot decode everything. No matter. Much improved.

detail.php - subject: ok

viewdetail.php
- subject: gibberish
- text/plain: gibberish (this is really viewpart.php) - page is encoded in utf8.
- text/html: ok

We're now a fair bit closer.

Re: v3.0.0.8 mailwatch unicode support

Posted: 16 Aug 2015 20:04
by shawniverson
Looks like an update to mailwatch (specifically functions.php and viewdetail.php) will take care of this.

Although it may break other things, so use caution if you update those two files. I am working on adding latest mailwatch to next update.