My NTP server was rebuilt today, and for a brief time was handing out a time in the past.
EFA saw it and synced to the past.
It appears to be walking through all emails in the quarantine????
Processed: 2,082,528 90.8Gb
Clean: 2,081,286 99.9%
Viruses: 0 0.0%
Top Virus: None
Blocked files: 1,242 0.1%
Others: 0 0.0%
Spam: 0 0.0%
High Scoring Spam: 0 0.0%
MCP: 0 0.0%
High Scoring MCP: 0 0.0%
Oh no!!!!!
Date Rollback Condition is BAD
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Date Rollback Condition is BAD
Okay, it looks like the system is stabilizing now that the time is correct...
CPU utilization is still high due to incoming queue being backlogged...
Monitoring to see what happens...
CPU utilization is still high due to incoming queue being backlogged...
Monitoring to see what happens...
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Date Rollback Condition is BAD
Something wrong...this appears to have started prior to the NTP server rebuild.
MailWatch shows the system looping through previously quarantined messages and marking them as clean repeatedly...
System is rapidly declining on disk space.
Watching incoming queue...it is trending downward slowly.
MailWatch shows the system looping through previously quarantined messages and marking them as clean repeatedly...
System is rapidly declining on disk space.
Watching incoming queue...it is trending downward slowly.
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Date Rollback Condition is BAD
Ok, found clamd dead. Was retrying scanning on a batch of messages over and over.
Started clamd...
MailScanner processes are turning zombie...
Started clamd...
MailScanner processes are turning zombie...
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Date Rollback Condition is BAD
Moved /var/spool/postfix/hold --> /var/spool/postfix/hold.chk
Created new hold
System now normal...something corrupt in hold queue? Checking...
Created new hold
System now normal...something corrupt in hold queue? Checking...
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Date Rollback Condition is BAD
Nothing corrupt in hold queue...all messages eventually processed.
However, clamd went down again part way....as soon as it did, the looping started again.
I had to stop MailScanner, start clamd, and start MailScanner.
Will be monitoring this very closely.
However, clamd went down again part way....as soon as it did, the looping started again.
I had to stop MailScanner, start clamd, and start MailScanner.
Will be monitoring this very closely.
Re: Date Rollback Condition is BAD
Sounds less than fun.
Looking forward reading to the progress.
Looking forward reading to the progress.
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Date Rollback Condition is BAD
It appears that a combination of CPU overload and too many MailScanner threads may be at fault...I think that clamd is getting pushed to its limit with the large number of incoming messages.
I have reduced my MailScanner threads to 5. Will test with a large queue again and see what happens.
I have reduced my MailScanner threads to 5. Will test with a large queue again and see what happens.
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Date Rollback Condition is BAD
Yep, too many MailScanner children was the culprit.
- shawniverson
- Posts: 3649
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact: