Multiple VM's in a cluster or more CPU and RAM on 1 VM

[drewmorris]

Hi guys... I am just curious which do you think is more advantageous... running 1 machine with 18GB RAM and 6 VCPUs or 3 VMs with 6GB Ram and 2 VCPUs?

I am struggling with this as I am scaling up because in my current model I get pretty decent performance from the 6GB RAM 2 VCPU setup (about 150,000 msgs delivered per day) and I simply run lots of VMs to scale. As it get to very large installations (5+ million msgs/day) it would seem to require 30+ servers which makes administration and monitoring a little tough (though not impossible).

I have designed a mechanism for clustering these and mostly providing a central administration via a cron-job that synchronized the configurations and a policy service that centralizes all the logic for delivery and user lookups but still... 30+ servers is a lot.

What are your thoughts/experiences? I'd love to hear them.

[darky83]

Not the answer you will be looking for:

'it depends'

Do you intend to run the three VM's on 1 physical node? if so then you would be better off using one big VM and configure the mailscanner children as needed (needs some playing around to find the right amount in your case).

Also the load of other VM's on the physical hosts may influence your decision.

So it all depends on what your hardware can handle, best methode would be to increase amount of resources on a single EFA VM as you go and see at which point you see processing slowing down or you see resource issues on your physical hardware.

One big note here is that you should look at your available resources the right way, don't just look at memory and cpu usage.
It might be that your physical host has just 5% CPU but over 50% CPU Ready time, this means you have major performance issues while it looks like the host is just about idle..

So check your physical host for CPU ready time's (should be below 5%), load, disk latency, memory overcommitment, network latency etc.

Spam filtering is a 'different' kind of resource usage VM than your typical VM (especially under high load) and it all depends on your physical hardware and how you use that hardware.

Also note that you might need to look at physical hardware.
I run a few physical hardware node's that do 2 milion+ per day with ease

[drewmorris]

So what are the specs of the physical hardware you use to achieve the 2mil+ throughput?

Not the answer you will be looking for:

'it depends'

Do you intend to run the three VM's on 1 physical node? if so then you would be better off using one big VM and configure the mailscanner children as needed (needs some playing around to find the right amount in your case).

Also the load of other VM's on the physical hosts may influence your decision.

So it all depends on what your hardware can handle, best methode would be to increase amount of resources on a single EFA VM as you go and see at which point you see processing slowing down or you see resource issues on your physical hardware.

One big note here is that you should look at your available resources the right way, don't just look at memory and cpu usage.
It might be that your physical host has just 5% CPU but over 50% CPU Ready time, this means you have major performance issues while it looks like the host is just about idle..

So check your physical host for CPU ready time's (should be below 5%), load, disk latency, memory overcommitment, network latency etc.

Spam filtering is a 'different' kind of resource usage VM than your typical VM (especially under high load) and it all depends on your physical hardware and how you use that hardware.

Also note that you might need to look at physical hardware.
I run a few physical hardware node's that do 2 milion+ per day with ease

[darky83]

They are 'heavy':

Proliant DL380 G8
2x E5-2690v2
512GB mem
8x 1TB SSD
2x 10GB uplink

[drewmorris]

Wow you weren't kidding on heavy... 20 cores + 512GB Ram and 8 SSDs... that is some serious muscle.

Why do you need 8TB for an EFA server though? That seems excessive if you are using it as a gateway server.

They are 'heavy':

Proliant DL380 G8
2x E5-2690v2
512GB mem
8x 1TB SSD
2x 10GB uplink

[darky83]

I store up to 30 day's and I keep lots of logs

(but yes 8TB is still overkill, could do with about 3 to 4 probably)

But hey, hardware is cheap compared to all the other infrastructure costs we run, we rather spend a few bucks more knowing we have enough hardware capacity than buying scaled to size and have performance issues on the long run (troubleshooting man hours, replacing, loss of production etc costs way more than the few 1000$ you spend extra on a machine with more capacity)

It has proven us that investing more on the hardware saves on the long run, thus we categorise all work flow tasks.
Spam filtering is categorised as heavy thus we buy one of the fastest server we can find.

Running the same workload shared over a whole bunch of VM's would probably cause performance issues with other servers running on the VM platform or the shared storage or network, to save us from all those issues and as we already figured out the workload is heavy we used dedicated hardware.
We do the same for high performance database servers.

As long as the software can run on multiple servers, and the downtime of a single server does not affect the working condition of the product hardware based is good enough, no need to use vmware's vmotion etc just to get 100% uptime. Spam filtering is a product that works good enough for this.
Shutting down a machine won't affect much, traffic will flow to the other server within a few seconds or will be retried by the sending mailserver.