v3.0.0.8 mailwatch unicode support

General eFa discussion
Post Reply
User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

v3.0.0.8 mailwatch unicode support

Post by pdwalker » 15 Aug 2015 09:30

Hi Shawn,

The mailwatch upgrade in 3.0.0.8 is supposed to support unicode.

When I send myself a message, the message list shows the same improperly decoded gibberish it always used to show.
e.g.
Subject: chinese test: 中国
but I get
Subject: chinese test: 谢谢
I was kinda hoping to see the original Chinese characters in the message list.

Am I doing something wrong, or do I have something misconfigured? Could it be my mysql databases are not utf8?

Code: Select all

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| efa                |
| mailscanner        |
| mysql              |
| sa_bayes           |
| sbtest             |
| sbtest1            |
| sqlgrey            |
+--------------------+
show create database (*)

Code: Select all

+-------------+----------------------------------------------------------------------------------------------+
| Database    | Create Database                                                                              |
+-------------+----------------------------------------------------------------------------------------------+
| efa         | CREATE DATABASE `efa` /*!40100 DEFAULT CHARACTER SET latin1 */                               |
| mailscanner | CREATE DATABASE `mailscanner` /*!40100 DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci */ |
| sa_bayes    | CREATE DATABASE `sa_bayes` /*!40100 DEFAULT CHARACTER SET latin1 */                          |
| sqlgrey     | CREATE DATABASE `sqlgrey` /*!40100 DEFAULT CHARACTER SET latin1 */                           |
+-------------+----------------------------------------------------------------------------------------------+
Will the non UTF8 databases cause this problem?

The https://efa.local/mailscanner/status.php page is in UTF8 encoding.

(if anyone else has working unicode displaying foreign characters in your mailscanner interface, I'd love to compare notes with you)

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by shawniverson » 16 Aug 2015 08:26

Confirmed issue.

The non utf8 databases are not the issue. The data is stored in mailscanner db. which shows as utf8.

It appears that MailWatch is not rendering in unicode. Checking source....
Version eFa 4.0.2 now available!

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by shawniverson » 16 Aug 2015 08:57

Looks like problem is in functions.php under db_colorised_table...

If I query the db and display raw data, the characters render properly...

Still digging....
Version eFa 4.0.2 now available!

User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

Re: v3.0.0.8 mailwatch unicode support

Post by pdwalker » 16 Aug 2015 09:04

need access to my system for a comparison?

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by shawniverson » 16 Aug 2015 09:13

This is going to take some work up in mailwatch development....

The new getUTF8String function in the latest mailwatch code isn't doing its job...so even updating mailwatch doesn't seem to resolve the issue.

Ironically, if I remove getUTF8String, it works.
Version eFa 4.0.2 now available!

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by shawniverson » 16 Aug 2015 09:15

Version eFa 4.0.2 now available!

EndelWar
Posts: 3
Joined: 11 Feb 2015 18:12
Location: Italy
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by EndelWar » 16 Aug 2015 10:02

@pdwalker: do you have mbstring extension installed on your system? and also: do you see these strange chars only indetail view or also in frontpage (status.php)? is the maillog table updated to utf8?

User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

Re: v3.0.0.8 mailwatch unicode support

Post by pdwalker » 16 Aug 2015 10:07

Good question, let me find out.

Mbstring is a php extension, correct?

User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

Re: v3.0.0.8 mailwatch unicode support

Post by pdwalker » 16 Aug 2015 10:25

In detail.php, the subject is the same as in status.php

In viewmail.php,

the subject is the same gibberish,
The text/plain rendering is the same gibberish
The text/htmlNow rendering is correct

Phpinfo() doesn't show mbstring installed. Confirmed, not installed. Should I install it?

Maillog is in UTF8 format

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by shawniverson » 16 Aug 2015 10:28

do you have mbstring extension installed on your system? and also: do you see these strange chars only indetail view or also in frontpage (status.php)? is the maillog table updated to utf8?
I have mbstring installed ...text is still not displaying correctly for me. Can you confirm pdwalker? I also updated mailwatch to latest snapshot.
Version eFa 4.0.2 now available!

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by shawniverson » 16 Aug 2015 10:30

Wait, restarted httpd, now its golden!

I think this may be resolved.

Code: Select all

sudo yum install php-mbstring
sudo service httpd restart
Version eFa 4.0.2 now available!

User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

Re: v3.0.0.8 mailwatch unicode support

Post by pdwalker » 16 Aug 2015 12:16

I'm an hour away from my computer. I'll try those two steps shortly.

EndelWar
Posts: 3
Joined: 11 Feb 2015 18:12
Location: Italy
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by EndelWar » 16 Aug 2015 13:29

mbstring extension makes life easier, but a solution that works also without extension should be tested, I've done some research and I think that I can do some experiments with Portable-utf8 (https://github.com/voku/portable-utf8) and Patchwork UTF8 (https://github.com/tchwork/utf8) (the latter is used in Laravel framework)

User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

Re: v3.0.0.8 mailwatch unicode support

Post by pdwalker » 16 Aug 2015 15:14

EndelWar wrote:mbstring extension makes life easier, but a solution that works also without extension should be tested, I've done some research and I think that I can do some experiments with Portable-utf8 (https://github.com/voku/portable-utf8) and Patchwork UTF8 (https://github.com/tchwork/utf8) (the latter is used in Laravel framework)
One difficulty with using these packages is that they are not in the public repositories which makes the system harder to maintain. It is almost always better to go with prebuilt packages.

User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

Re: v3.0.0.8 mailwatch unicode support

Post by pdwalker » 16 Aug 2015 15:30

Code: Select all

sudo yum install php-mbstring
sudo service httpd restart
results:
mbstring successfully installed.

status.php -> most message subjects displayed correctly. One message appeared like this:
商業電話專線 每月只需HK$60起 直\xE9\xA7\x81手提/飛線國內號碼
Maybe mbstring cannot decode everything. No matter. Much improved.

detail.php - subject: ok

viewdetail.php
- subject: gibberish
- text/plain: gibberish (this is really viewpart.php) - page is encoded in utf8.
- text/html: ok

We're now a fair bit closer.

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: v3.0.0.8 mailwatch unicode support

Post by shawniverson » 16 Aug 2015 20:04

Looks like an update to mailwatch (specifically functions.php and viewdetail.php) will take care of this.

Although it may break other things, so use caution if you update those two files. I am working on adding latest mailwatch to next update.
Version eFa 4.0.2 now available!

Post Reply