Page 1 of 1
Date Rollback Condition is BAD
Posted: 02 Mar 2014 04:23
by shawniverson
My NTP server was rebuilt today, and for a brief time was handing out a time in the past.
EFA saw it and synced to the past.
It appears to be walking through all emails in the quarantine????
Processed: 2,082,528 90.8Gb
Clean: 2,081,286 99.9%
Viruses: 0 0.0%
Top Virus: None
Blocked files: 1,242 0.1%
Others: 0 0.0%
Spam: 0 0.0%
High Scoring Spam: 0 0.0%
MCP: 0 0.0%
High Scoring MCP: 0 0.0%
Oh no!!!!!
Re: Date Rollback Condition is BAD
Posted: 02 Mar 2014 04:36
by shawniverson
Okay, it looks like the system is stabilizing now that the time is correct...
CPU utilization is still high due to incoming queue being backlogged...
Monitoring to see what happens...
Re: Date Rollback Condition is BAD
Posted: 02 Mar 2014 05:16
by shawniverson
Something wrong...this appears to have started prior to the NTP server rebuild.
MailWatch shows the system looping through previously quarantined messages and marking them as clean repeatedly...
System is rapidly declining on disk space.
Watching incoming queue...it is trending downward slowly.
Re: Date Rollback Condition is BAD
Posted: 02 Mar 2014 05:36
by shawniverson
Ok, found clamd dead. Was retrying scanning on a batch of messages over and over.
Started clamd...
MailScanner processes are turning zombie...
Re: Date Rollback Condition is BAD
Posted: 02 Mar 2014 05:41
by shawniverson
Moved /var/spool/postfix/hold --> /var/spool/postfix/hold.chk
Created new hold
System now normal...something corrupt in hold queue? Checking...
Re: Date Rollback Condition is BAD
Posted: 02 Mar 2014 07:05
by shawniverson
Nothing corrupt in hold queue...all messages eventually processed.
However, clamd went down again part way....as soon as it did, the looping started again.
I had to stop MailScanner, start clamd, and start MailScanner.
Will be monitoring this very closely.
Re: Date Rollback Condition is BAD
Posted: 02 Mar 2014 07:19
by ramtech
Sounds less than fun.
Looking forward reading to the progress.
Re: Date Rollback Condition is BAD
Posted: 02 Mar 2014 08:26
by shawniverson
It appears that a combination of CPU overload and too many MailScanner threads may be at fault...I think that clamd is getting pushed to its limit with the large number of incoming messages.
I have reduced my MailScanner threads to 5. Will test with a large queue again and see what happens.
Re: Date Rollback Condition is BAD
Posted: 03 Mar 2014 11:18
by shawniverson
Yep, too many MailScanner children was the culprit.
Re: Date Rollback Condition is BAD
Posted: 26 Mar 2014 20:26
by shawniverson