Slow Spam Reporting

Questions and answers about how to do stuff
pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Slow Spam Reporting

Post by pipjo » 12 Oct 2017 22:13

Hi,

When I click the link to report a mail as spam the browser opens and one of the following happens.
Page load times out
Page opens after around a minute
Page keeps loading, but nothing actually loads

This happens on all browsers and from PCs and Mobile Phones.

Other pages (login, portal, etc) all load and work fine.

Is there anything I can do to improve this?

Thanks

TPJ

User avatar
pdwalker
Posts: 1149
Joined: 18 Mar 2015 09:16

Re: Slow Spam Reporting

Post by pdwalker » 13 Oct 2017 07:33

What link is the browser trying to open up?

I'm guessing the link is unresolvable from your mail client so that needs checking/verifying.

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 13 Oct 2017 09:05

pdwalker wrote:
13 Oct 2017 07:33
What link is the browser trying to open up?

I'm guessing the link is unresolvable from your mail client so that needs checking/verifying.
It’s the same url as for the portal which is working fine. The mail client instantly opens the browser and that’s where the problem seems to start.

User avatar
pdwalker
Posts: 1149
Joined: 18 Mar 2015 09:16

Re: Slow Spam Reporting

Post by pdwalker » 13 Oct 2017 09:35

I would verify that link.

As I cannot see your system or the messages, I cannot confirm it.

Also remember that links can show something different than what it really is, e.g.
<a href="http://this.is.the.real.link/">http://t ... ng.else</a>

If your urls are correct, and you access the link, then accessing the same link from an email - assuming the email client is on the same network, should be no problem at all.

Good luck.

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 13 Oct 2017 17:02


pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 14 Oct 2017 18:35

Anyone have any ideas?

TheGr8Wonder
Posts: 97
Joined: 01 Jul 2017 02:32

Re: Slow Spam Reporting

Post by TheGr8Wonder » 14 Oct 2017 23:23

What version EFA are you running?

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 15 Oct 2017 01:06

3.0.2.5

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 15 Oct 2017 15:20

I’ve thrown an extra 2 GB memory to the VM but this hasn’t made any difference. Are there any logs I can take a look at.

I have noticed that the page displayed if a mail has already been submitted as Spam loads at normal speed.

TheGr8Wonder
Posts: 97
Joined: 01 Jul 2017 02:32

Re: Slow Spam Reporting

Post by TheGr8Wonder » 15 Oct 2017 15:25

Is there a reason you're only using http and not https? One of the changes in 3.0.2.5 was changing the default url for quarantine reports to https, but that shouldn't effect reporting within the main mailwatch gui.

What happens when you enable https (you can use the kets encrypt feature for simplicity) and try again.

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 15 Oct 2017 15:45

Just done that but need to change the port. I only have a single ip and 443 is already in use. Can this port be changed?

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 15 Oct 2017 19:12

I’ve undone Let’s Encrypt to get things working. I’ve also uploaded a video of what happen. I’d advise not watching it as it’s about 6:30 waiting for a mail to be logged as Spam and then about 3 seconds getting the screen up saying that the mail has already been submitted.
https://drive.google.com/open?id=0B0Rex ... zk3TFR6Ykk

Help please, spam reporting is making this unusable.

TheGr8Wonder
Posts: 97
Joined: 01 Jul 2017 02:32

Re: Slow Spam Reporting

Post by TheGr8Wonder » 15 Oct 2017 23:20

When a message gets reported, its updating your local DB, and reporting it to pyzor.

Based on your video, it looks like your web service is stopping during the report. Do you get message from EFA monitor regarding httpd stopping and restarting unexpectedly?

Have you done any customization to httpd on the appliance? Do you have modsecurity enabled?

Have you reviewed the httpd or mariadb logs of the system when you click report and wait?

If you want me to review the logs of your system, please shoot me a PM for further instructions.

Thanks!

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 16 Oct 2017 01:49

Thanks for the reply

Only mod is that I’m running on port 5272 instead of port 80.

I’ll grab the logs tomorrow.

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 16 Oct 2017 19:56

I was able to get the log for Httpd but not MariaDB.

Log is httpd/logs/error_log

Code: Select all

[Sun Oct 15 16:40:04 2017] [notice] Digest: generating secret for digest authentication ...
[Sun Oct 15 16:40:04 2017] [notice] Digest: done
[Sun Oct 15 16:40:05 2017] [notice] Apache/2.2.15 (Unix) PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Sun Oct 15 16:40:05 2017] [notice] caught SIGTERM, shutting down
[Sun Oct 15 16:40:06 2017] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Oct 15 16:40:07 2017] [notice] Digest: generating secret for digest authentication ...
[Sun Oct 15 16:40:07 2017] [notice] Digest: done
[Sun Oct 15 16:40:08 2017] [notice] Apache/2.2.15 (Unix) PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Sun Oct 15 17:02:13 2017] [warn] child process 47815 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:13 2017] [warn] child process 47816 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:13 2017] [warn] child process 47817 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:13 2017] [warn] child process 47818 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:13 2017] [warn] child process 47819 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:13 2017] [warn] child process 47820 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:13 2017] [warn] child process 47821 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:13 2017] [warn] child process 47823 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47815 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47816 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47817 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47818 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47819 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47820 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47821 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:15 2017] [warn] child process 47823 still did not exit, sending a SIGTERM
[Sun Oct 15 17:02:18 2017] [notice] caught SIGTERM, shutting down
[Sun Oct 15 17:03:33 2017] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Oct 15 17:03:46 2017] [notice] Digest: generating secret for digest authentication ...
[Sun Oct 15 17:03:46 2017] [notice] Digest: done
[Sun Oct 15 17:03:58 2017] [notice] Apache/2.2.15 (Unix) PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Sun Oct 15 17:04:48 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon-120x120-precomposed.png
[Sun Oct 15 17:04:48 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon-120x120.png
[Sun Oct 15 17:04:49 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon.png
[Sun Oct 15 17:04:49 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon.png
[Sun Oct 15 18:22:33 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon-120x120-precomposed.png
[Sun Oct 15 18:22:34 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon-120x120.png
[Sun Oct 15 18:22:34 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon.png
[Sun Oct 15 18:22:34 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/apple-touch-icon.png
[Sun Oct 15 18:59:56 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/robots.txt
[Sun Oct 15 19:42:12 2017] [error] [client 86.29.132.218] File does not exist: /var/www/html/robots.txt

TheGr8Wonder
Posts: 97
Joined: 01 Jul 2017 02:32

Re: Slow Spam Reporting

Post by TheGr8Wonder » 16 Oct 2017 20:21

Just to rule out modsecuirty, main menu 11 (apache) -> 2 (mod security) and type N when prompted if you want to enable it. It will stop/start apache and will then display a message saying its disabled. Does reporting still present same behavior?

If so...
When you did your custom port for the mailwatch interface, did you open that port in the firewall?

Did you also disable the https selection under menu 11 as well?

Did you modify your httpd config for the new port on the virtual directory?

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 17 Oct 2017 00:44

Mod security was disabled but I have re disabled it. This has made no difference. Same for https

I believe that the port is set up correctly. I am able to access the GUI and receive the Spam already reported screen, both on the same port.

Are there any other logs worth checking?

User avatar
shawniverson
Posts: 2842
Joined: 13 Jan 2014 23:30
Location: Rushville, Indiana, USA
Contact:

Re: Slow Spam Reporting

Post by shawniverson » 17 Oct 2017 22:08

If you are on port 5272, did you modify the report inline sigs to use port 5272?

Code: Select all

<br /><a href="http://$hostname:5272/cgi-bin/learn-msg.cgi?id=$id&token=$token">Click here to report this message as spam.</a>
Version eFa 4.0.0 RC3 now available in testing repo. Come join us in advancing eFa!

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 18 Oct 2017 00:18

Thanks for taking a look.
Unfortunately yes I did change the sigs

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 18 Oct 2017 11:29

I think I might have found something. There were 132600 mails showing in Webmin for the Postfix Queues. I have cleared these down but I am sill seeing the mailserver.mtalog table constantly growing with the following:

Code: Select all

 said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 08:29:16 |
|    900791 | 2017-10-17 09:21:48 | spam | relay | 6632756965F | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 06:43:42 |
|    900792 | 2017-10-17 09:21:49 | spam | relay | 7876F562C7E | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 14:01:31 |
|    900793 | 2017-10-17 09:21:49 | spam | relay | 77AA556D466 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 09:52:54 |
|    900794 | 2017-10-17 09:21:50 | spam | relay | 9681C56FA55 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 07:15:31 |
|    900795 | 2017-10-17 09:21:50 | spam | relay | 5FA01569595 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 09:00:36 |
|    900796 | 2017-10-17 09:21:52 | spam | relay | 851C3560F52 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 11:31:03 |
|    900797 | 2017-10-17 09:21:53 | spam | relay | B12C656A243 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 13:09:13 |
|    900798 | 2017-10-17 09:21:54 | spam | relay | F19D756304F | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 06:30:47 |
|    900799 | 2017-10-17 09:21:55 | spam | relay | BD4E256E338 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 08:50:21 |
|    900800 | 2017-10-17 09:21:55 | spam | relay | 069BB565B12 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 00:42:50 |
|    900801 | 2017-10-17 09:21:56 | spam | relay | 974C6569FB8 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 13:18:17 |
|    900802 | 2017-10-17 09:21:56 | spam | relay | 4919E566283 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 11:39:56 |
|    900803 | 2017-10-17 09:21:57 | spam | relay | A3336562DC9 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 13:43:27 |
|    900804 | 2017-10-17 09:21:58 | spam | relay | 5FBB256E55A | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 08:42:55 |
|    900805 | 2017-10-17 09:21:58 | spam | relay | 5C06F56BB0E | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 11:29:33 |
|    900806 | 2017-10-17 09:21:58 | spam | relay | 8E7BD566541 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 11:12:01 |
|    900807 | 2017-10-17 09:21:59 | spam | relay | 408D5569B3A | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 13:36:59 |
|    900808 | 2017-10-17 09:21:59 | spam | relay | 27C0456F1B4 | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 07:49:51 |
|    900809 | 2017-10-17 09:22:00 | spam | relay | 23D8F56BB2A | 192.168.1.8 | 5.1.1 | bounced (host 192.168.1.8[192.168.1.8] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) | 11:29:40 |
I can (and have cleared this down but within a couple of hours it's up to 50000 rows again.

Any idea how I can stop this?

jamerson
Posts: 136
Joined: 19 Aug 2017 18:57
Location: kaaskop

Re: Slow Spam Reporting

Post by jamerson » 18 Oct 2017 13:10

I have seen this before in one box,
I notice the email that are sent from the system using http and not https
make sure you have both ports open 80 and 443 forwarded to the EFA.
it works fine here from my side.
Version eFa 4.0.0 RC1 now available in testing repo. Come join us in advancing eFa!

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 19 Oct 2017 22:17

I managed to clear down mailscanner.mtalog yesterday afternoon and things seemed to be ok. All stopped again today, just checked and I have 230000 rows in the mtalog, all looking like they are bounce messages,

1 - How can I stop this
2 - Is there anyway I can display some of the messages listed in mtalog so I can try to see where they are from and where they are going?

Thanks

User avatar
shawniverson
Posts: 2842
Joined: 13 Jan 2014 23:30
Location: Rushville, Indiana, USA
Contact:

Re: Slow Spam Reporting

Post by shawniverson » 20 Oct 2017 21:46

I think your appliance is crashing and looping emails through MailScanner. This causes the log to grow very fast. You'll probably find they are all identical.
Version eFa 4.0.0 RC3 now available in testing repo. Come join us in advancing eFa!

pipjo
Posts: 43
Joined: 06 Jul 2017 09:08

Re: Slow Spam Reporting

Post by pipjo » 21 Oct 2017 01:06

I managed to locate the relevant Exchange log and these mails are being rejected with sender unknown.
I have created a mailbox with these addresses as aliases and set up a transport rule to delete them.
I can see that the messages have now been processed by Exchange and that there are now no attempts to deliver them but mtalog still keeps filling. There must be some way to flush whatever feeds into mtalog. Any ideas?

User avatar
shawniverson
Posts: 2842
Joined: 13 Jan 2014 23:30
Location: Rushville, Indiana, USA
Contact:

Re: Slow Spam Reporting

Post by shawniverson » 21 Oct 2017 10:32

Whoops my bad, I read maillog, not mtalog. Disregard my previous post. Yeah, the mtalog is very busy, depending on your mail volume.

The db_clean task at /usr/local/bin/mailwatch/tools/Cron_jobs/mailwatch_db_clean.php manages the mtalog as well.

The /etc/cron.daily/mailwatch cron job should be calling it daily.

It will clean the mtalog based on this setting.

/var/www/html/mailscanner/conf.php

Code: Select all

// Define how many days of emails to keep
define('RECORD_DAYS_TO_KEEP', 60);
Reduce this number to reduce the overall growth of mtalog.
Version eFa 4.0.0 RC3 now available in testing repo. Come join us in advancing eFa!

Post Reply