Improving my EFA Performance

Questions and answers about how to do stuff
toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Improving my EFA Performance

Post by toddh » 26 Mar 2015 13:21

Hello,

I am wondering if there is something amiss with my EFA box or this is normal. I have been adjusting hardware and setting trying to improve speed/performance. Not in accuracy but rather how many messages EFA can handle per min/hr.

We processed 23588 messages yesterday total.

Yesterday morning the inbound queue started to back up. My config at the time was 8 Cores, 8 Gb Ram, 8 Mailscanner threads.

Load was showing just under 3 across the board, ie 2.83, 2.78, 2.88. MailScanner threads showed 11.

I though the load should be higher with 8 cores. I changed the MailScanner threads from 8 to 10 and load went to just under 8, ie 7.89, 7.83, 7.66 and MS Threads went to 18. The queue came down within 15 min.

Still new to EFA so trying to get a handle on performance and loading.

Why did it back up? 8 Cores, 8Gb and 8 MS Threads seems like it should handle this volume? And why with 8 MS Threads was it only loading the CPU at less than 3?

Not sure why all the Processing came down and CPU resources went up so significantly when I changed the MS Threads from 8 to 10?

Todd

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 26 Mar 2015 16:52

I'd say something is wrong. That's just way too high.

run top, display all the CPUs.

Is the machine running out of memory and swapping?

Is the machine stuck with a high % of IOWAIT? (waiting on disk io)

Which process is getting hung up and consuming cpu?

Without more information, it's very difficult to say.

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Improving my EFA Performance

Post by shawniverson » 26 Mar 2015 17:05

With 8 cores, you may want to try around 16 mailscanner threads.

EFA-Configure has a limit of 10, so edit /etc/MailScanner/MailScanner.conf and change it there.
Version eFa 4.0.2 now available!

User avatar
darky83
Site Admin
Posts: 535
Joined: 30 Sep 2012 11:03
Location: eFa
Contact:

Re: Improving my EFA Performance

Post by darky83 » 26 Mar 2015 20:17

One overlooked item can sometimes be disk IO do you see any issues there on the VM hosts?

If you look at top also try to see if there is any wait time, it should be close to 0, if higher your system is waiting for something and in most cases its disk IO.

top - 21:14:22 up 39 days, 1:12, 1 user, load average: 0.62, 0.55, 0.58
Tasks: 156 total, 2 running, 154 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.6 us, 2.0 sy, 0.0 ni, 87.0 id, 0.1 wa, 0.0 hi, 2.3 si, 0.0 st
Version eFa 4.0.1 now available!

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 26 Mar 2015 22:17

Is the VM running out of Memory and swapping - I don't think so. I started with 2 Gb Ram and have been slowly incrementing by 2GB so now I am up to 8 Gb. So far whatever I give it seems to use. However when the slowness occurred several days ago I bumped ram from 6 to 8 Gb and it made no difference. Adding cores at the time did.

Top processes rotate but mostly MailScanner, Clam, ImageCerbius, and MySQL. I would like to move the MySQL DBs to a dedicated MySQL server we have and offload some resources.

Top after its been running about 35 hrs
EFA1 Top.PNG
EFA1 Top.PNG (51.83 KiB) Viewed 7724 times
I am seeing wa times from .0 - 12.% at the moment, which is a slower time right now - 5 pm and everyone is leaving for the day. I will look at my disks and see where this VM is running. Its on a cluster so I have to trace the CSV to the SAN and Disk Volume.

.

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 03:07

Press '1' in top to give a per CPU usage breakdown and send a screenshot of that.

Next, try iotop and get a feel of what is happening with io and how much

How did you setup your system? By hand, or from the prepared vm images?

What about the host machine? Is it busy, or lacking ram?

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 05:41

As a wild ass guess, I'd suspect mysql being the bottleneck to mailscanner.

Perhaps there is a way to debug the time it takes for Mailscanner to handle requests. That information could give you a better indication of where Mailscanner is being held up. Once you know that, then solving the problem becomes much easier.

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 12:55

Hmmm...

Interesting. I am getting slower performance now. I am seeing a lot of iowait.

Let me see if I can figure this out...

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance - findings

Post by pdwalker » 27 Mar 2015 13:24

SA Learn causes a lot of IO wait for me

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 27 Mar 2015 13:36

VM is vhd from EFA.

Morning business is starting here so this is normal/light load
EFA1 Top1A.PNG
EFA1 Top1A.PNG (71.68 KiB) Viewed 7714 times
Host machine has nothing else significant running. Dell c6100, 2x Xeon L5520( 16 Cores ) 48 Gb ram. The SAN and disk array that the VM sits on runs a number of other VMs so the storage is not idea for this type of load. I enabled LSI Cachecade on the CSV last night to see if that helps.


.

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 13:50

It looks like the mailscanner process is getting hung up on i/o somewhere.

I have an idea. I'll test it in a couple of hours and let you know if it makes a difference for me.

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 27 Mar 2015 14:26

I think part of my confusion at the high load is that we were previously using a similar box, Postfix, SA, Clam, and it processed the same load on 8 year old hardware with only a fraction of the load. Ran single CPU with 2 or 4 cores and 4gb of ram, averaged around 15% cpu and rarely used 2gb with 20 Amavis/SA processes running.

Was a project called Maia Mailguard and it off loaded everything to a MySQL server. All SA config, Spam and Ham as well as Bayes were kept in MySQL db which we ran on a separate server.

I know things change so I was trying to get a feel for if this is normal in EFA or is there something off with my system. And if there was a way to improve performance without adding more resources.

.

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 15:46

It should be plenty of hardware, that's not the problem. The problem is elsewhere.

I'm doing some testing now to see if I can isolate the issue.

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Improving my EFA Performance

Post by shawniverson » 27 Mar 2015 15:48

I am curious toddh. If you halt mail flow to your appliance, does your CPU trend down to idle? Or does it still show a high load (especially MailScanner)

Even with the amount of mail you are processing, I agree with pdwalker that something may be amiss.

Also, I would be particularly interested in whether you have your Maximum Processing Attempts in MailScanner set to a non-zero value.
Version eFa 4.0.2 now available!

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 16:55

Shawn, I expect if he does as you say, the load will drop to near 0.

I'm now almost certain that the problem is because of an unoptimal mysql configuration. I'm playing with the configurations now to see if I can improve the performance without making too many changes.

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 17:11

todd,

Do you get a lot of non English language mail into your system?

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 27 Mar 2015 18:12

If I remove mail does cpu to to 0. Yes. It drops in the evenings to a load of .3 or so. I also have a 2nd EFA, clone of this VM, that runs as 2nd MX and processed only 400 messages today, load is 0, 0, 0.

Do we get a lot of non English? I would say a few but not a lot. Probably a relatively small percentage.

The Cerbius Image plugin seems to use a lot of cpu when it kicks in, anywhere from 100 - 350%.

Appreciate the help!

.

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 27 Mar 2015 18:23

Shawn,

Maximum Processing Attempts is set to 0

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 27 Mar 2015 19:01

Can you run the following command a couple of times and tell me what kind of queries to which database you are seeing?

mysqladmin -uroot -p<EFA mysql root password from /etc/EFA-Config> processlist

run it 5 or 10 times. what ones in the "db" column show active queries? ("Command" column = "Query")

The output will appear something like

Code: Select all

+-----+-----------+-----------+-------------+---------+------+---------------+------------------------------------------------------------------------------------------------------+
| Id  | User      | Host      | db          | Command | Time | State         | Info                                                                                                 |
+-----+-----------+-----------+-------------+---------+------+---------------+------------------------------------------------------------------------------------------------------+
| 3   | mailwatch | localhost | mailscanner | Sleep   | 269  |               |                                                                                                      |
| 7   | sqlgrey   | localhost | sqlgrey     | Sleep   | 1592 |               |                                                                                                      |
| 26  | root      | localhost |             | Sleep   | 0    |               |                                                                                                      |
| 53  | root      | localhost | sa_bayes    | Sleep   | 797  |               |                                                                                                      |
| 111 | mailwatch | localhost | mailscanner | Sleep   | 38   |               |                                                                                                      |
| 445 | sa_user   | localhost | sa_bayes    | Query   | 0    | freeing items | UPDATE bayes_token
                     SET spam_count = spam_count + '1'
                   WHERE i |
| 448 | root      | localhost |             | Query   | 0    |               | show processlist                                                                                     |
+-----+-----------+-----------+-------------+---------+------+---------------+------------------------------------------------
What I see on my system is a lot of is a large number of quick queries happening to the sa_bayes table (I use the spam/ham training for a few messages in the Message Operations page)

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 27 Mar 2015 20:50

Went to look for the pw in /etc/EFA-Conf and found

INTERFACE:eth0

But I am using eth1.

I get this error when booting. Could this be related
EFA eth0 error.PNG
EFA eth0 error.PNG (50.05 KiB) Viewed 7699 times
I believe this happened after I moved or copied the VM, not sure which. I simply assigned IP info to eth1 when it happened and everything was working so I left it.

.

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 27 Mar 2015 20:54

MySQL query. Ran it a number of times, most responses were like the last 2.
Last edited by toddh on 28 Mar 2015 03:57, edited 1 time in total.

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 28 Mar 2015 03:32

First, edit your post and remove the password.

What was the load average during that time? And was top showing iowait during that time?

If so, we're both experiencing the same thing.

I'm still looking.

toddh
Posts: 56
Joined: 16 Feb 2015 18:52

Re: Improving my EFA Performance

Post by toddh » 30 Mar 2015 14:15

When I run the command a number of times quickly i get the f0llowig. Not much change in Top. Load was around 2.17, 1.79, 1.58, not a large variance.
EFA MySQL Query1.PNG
EFA MySQL Query1.PNG (41.23 KiB) Viewed 7671 times

User avatar
pdwalker
Posts: 1213
Joined: 18 Mar 2015 09:16

Re: Improving my EFA Performance

Post by pdwalker » 31 Mar 2015 04:54

You're seeing the same thing I am.

While the system is running sa-learn, the rest of the system grinds to a halt, or becomes very slow and unresponsive. It is like the whole system waits until sa-learn completes. It seems to take about 30 seconds per message for sa-learn to run.

It's not an SQL table lock contention as I get the same performance when using InnoDB tables as I do with MyISAM tables. I get no slow queries (slow queries longer than 1 second in my configuration). Also, the sa-learn sends hundreds of update queries for that time. My database goes from 0qps to over 100qps.

Also, the UI blocks while sa-learn is running, which it shouldn't do.

The databases are tiny, and should be running out of cache in my configuration.

It's puzzling.

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Improving my EFA Performance

Post by shawniverson » 01 Apr 2015 17:02

This seems way too high.

Timed mine a 6 sec/message.
Version eFa 4.0.2 now available!

Post Reply