First Crash - Newby Needs Help

General eFa discussion
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

First Crash - Newby Needs Help

Post by Archer »

Well it happened.....We had our first crash. NO real response....NO ssl, but I did fin on the Webmin page a way to restart the whole thing. So thinking life is ok.....until it tells me that the MySQL database needs repair. So SSL connection, option 10, and ...... it stops on the restart.

.........

MyISAM file: /var/lib/mysql/sqlgrey/optout_email.MYI is already checked
Press [Enter] key to continue...
Attempting to start MySQL
Starting mysqld: [ OK ]
Performing additional database checks
efa.tokens OK
mailscanner.audit_log OK
mailscanner.blacklist OK
mailscanner.geoip_country OK
mailscanner.inq OK

...........

So, now what? How do we get back to the happy life?

Any and all help is welcomed.
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

Not enough information.

When you say "crash", what do you mean exactly? Did the server reboot? Was there a power failure? Did you power cycle the machine? Did errors show up on screen

Are there any relevant errors in the log files? Can you log in?

Can you reboot the machine?
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: First Crash - Newby Needs Help

Post by shawniverson »

Have you rebooted?

Is this the last thing you saw at the end of recovery?

Code: Select all

MailScanner and SQLgrey
Starting MailScanner daemons:
         incoming postfix:                                 [  OK  ]
         outgoing postfix:                                 [  OK  ]
         MailScanner:                                      [  OK  ]
Starting SQLgrey:                                          [  OK  ]
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

shawniverson wrote:Have you rebooted?

Is this the last thing you saw at the end of recovery?

Code: Select all

MailScanner and SQLgrey
Starting MailScanner daemons:
         incoming postfix:                                 [  OK  ]
         outgoing postfix:                                 [  OK  ]
         MailScanner:                                      [  OK  ]
Starting SQLgrey:                                          [  OK  ]

You are correct. That is the last thing I saw after the recovery. I left it about 30 minutes, without change. I then rebooted again from the SSL interface. Email is coming and going...that is the good part.
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

When you say "crash", what do you mean exactly?
>>The only thing it would do is allow part of the Webmin site. (Not all of it worked.) SSL was no responsive. Email was not allowed in or out. EFA was non responsive to anything that I tried.

Did the server reboot?
>>Only after I did the reboot in the Webmin interface.

Was there a power failure?
>>No

Did you power cycle the machine?
>>Power Cycle, no........reboot yes.

Did errors show up on screen
>>No

Are there any relevant errors in the log files?
>>Sorry to much of a Newby to find them to answer this question

Can you log in?
>>Yes

Can you reboot the machine?
>>Yes
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

Other Information I have seen or found:
>>>>>>>>From several emails<<<<<<<
Sent: Wednesday, April 22, 2015 2:58 PM
To: >>NAME REMOVED<<
Subject: Re: club help May 7

MailScanner was attacked by a Denial Of Service attack, and has therefore deleted this part of the message. Please contact your e-mail providers for more information if you need it, giving them the whole of this report. Attack in: /var/spool/MailScanner/incoming/22177/EE8A461687.A488A/nmsg-22177-7.html
>>>>>>>>End Email<<<<<<<<


>>>>>>>>Login in to Mailwatch shows<<<<<
Error executing query:

144: Table './mailscanner/maillog' is marked as crashed and last (automatic?) repair failed

SQL:
SELECT
COUNT(*) AS processed,
SUM(
CASE WHEN (
(virusinfected=0 OR virusinfected IS NULL)
AND (nameinfected=0 OR nameinfected IS NULL)
AND (otherinfected=0 OR otherinfected IS NULL)
AND (isspam=0 OR isspam IS NULL)
AND (ishighspam=0 OR ishighspam IS NULL)
AND (ismcp=0 OR ismcp IS NULL)
AND (ishighmcp=0 OR ishighmcp IS NULL)
) THEN 1 ELSE 0 END
) AS clean,
ROUND((
SUM(
CASE WHEN (
(virusinfected=0 OR virusinfected IS NULL)

>>>>>>>>>>Actual display truncated for space here<<<<<<<<<<<


>>>>>>>>>>Also in Mailwatch Display<<<<<<<<<<<<<<
Please verify read permissions on /var/spool/postfix/hold and /var/spool/postfix/incoming
>>>>>>>>>>>>End<<<<<<<<<<<<<<<<<<<<<<<<<<


Thank you again!!
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

--------------------------------------------------------------
--- Welcome to the EFA Configuration program ---
--- http://www.efa-project.org ---
--------------------------------------------------------------


MySQL Daemon and Database Recovery

Description:
This tool will assist with mysql recovery
after a system crash or power failure.

If recovery fails, it is recommended that
you restore your mysql database from a
recent backup.

[EFA] Do you wish to continue? (y/n): y

Beginning recovery...

Stopping MySQL, MailScanner, and sqlgrey
Shutting down SQLgrey: [ OK ]
Shutting down MailScanner daemons:
MailScanner: [ OK ]
incoming postfix: [ OK ]
outgoing postfix: [ OK ]
Stopping mysqld: [ OK ]
Removing socket if present
Performing MyISAM checks
Checking MyISAM file: /var/lib/mysql/efa/tokens.MYI
Data records: 13087 Deleted blocks: 0
myisamchk: warning: Table is marked as crashed
- check file-size
myisamchk: warning: Size of datafile is: 536576 Should be: 536567
- check record delete-chain
- check key delete-chain
- check index reference
- recovering (with keycache) MyISAM-table '/var/lib/mysql/efa/tokens.MYI'
Data records: 13087

---------

MyISAM file: /var/lib/mysql/mailscanner/audit_log.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/blacklist.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/geoip_country.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/inq.MYI is already checked

---------

Checking MyISAM file: /var/lib/mysql/mailscanner/maillog.MYI
Data records: 0 Deleted blocks: 0
myisamchk: warning: Table is marked as crashed and last repair failed
- check file-size
myisamchk: warning: Size of indexfile is: 25330688 Should be: 1024
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check data record references index: 2
- check data record references index: 3
- check data record references index: 4
- check data record references index: 5
- check data record references index: 6
- check data record references index: 7
- check data record references index: 8
- check data record references index: 9
- check record links
myisamchk: error: Record-count is not ok; is 277396 Should be: 0
myisamchk: warning: Found 277396 key parts. Should be: 0
- recovering (with sort) MyISAM-table '/var/lib/mysql/mailscanner/maillog.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
- Fixing index 7
- Fixing index 8
- Fixing index 9
Data records: 277396

---------

MyISAM file: /var/lib/mysql/mailscanner/mcp_rules.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/mtalog_ids.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/mtalog.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/outq.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/sa_rules.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/saved_filters.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/spamscores.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/user_filters.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/users.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mailscanner/whitelist.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/columns_priv.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/db.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/event.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/func.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/help_category.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/help_keyword.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/help_relation.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/help_topic.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/host.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/ndb_binlog_index.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/plugin.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/proc.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/procs_priv.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/servers.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/tables_priv.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/time_zone_leap_second.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/time_zone.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/time_zone_name.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/time_zone_transition.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/time_zone_transition_type.MYI is already checked

---------

MyISAM file: /var/lib/mysql/mysql/user.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/config.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/connect.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/domain_awl.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/from_awl.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/optin_domain.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/optin_email.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/optout_domain.MYI is already checked

---------

MyISAM file: /var/lib/mysql/sqlgrey/optout_email.MYI is already checked
Press [Enter] key to continue...
Attempting to start MySQL


Timeout error occurred trying to start MySQL Daemon.
Starting mysqld: [FAILED]
Performing additional database checks
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) when trying to connect
Press [Enter] key to continue...Runing database optimization
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) when trying to connect
Press [Enter] key to continue...MailScanner and SQLgrey
Starting MailScanner daemons:
incoming postfix: [ OK ]
outgoing postfix: [ OK ]
MailScanner: [ OK ]
Starting SQLgrey: [ OK ]
Press [Enter] key to continue...
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

Ok, that's more helpful.

So, mysql is not starting and we need to solve that problem first.

Get back to the EFA Configuration program and select "1) Shell"

Next, type in this command:

Code: Select all

sudo less /var/log/mysqld.log
press G (shift g) to go to the bottom of the file

You use "b" to go back one page and " " <space> to go forward a page. See if you can find anything interesting in the mysql log file and post it back here.

"q" will quit the less text browsing program.

"exit" will log you out once you have quit "less"
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

Well I "THINK" i know what the issue is.........but not sure how to solve this:

150423 14:33:00 mysqld_safe Number of processes running now: 0
150423 14:33:00 mysqld_safe mysqld restarted
150423 14:33:00 InnoDB: Initializing buffer pool, size = 8.0M
150423 14:33:00 InnoDB: Completed initialization of buffer pool
150423 14:33:00 InnoDB: Started; log sequence number 9 101171141
150423 14:33:00 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
150423 14:33:00 [ERROR] Can't start server: can't create PID file: No space left on device
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

150423 14:33:00 [ERROR] Can't start server: can't create PID file: No space left on device
And there it is, in black and white.

Do one more thing, go back to the shell and get the results of the "df -H" command to post here. Once you've done that, then we can can solve your disk space problem.
Last edited by pdwalker on 23 Apr 2015 18:48, edited 1 time in total.
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

To me the indication of out of space....is contradicted by info below:


Real memory 801.95 MB used, 1.83 GB total
Local disk space 22.48 GB used, 29.51 GB total
Package updates 30 package updates are available
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

[administrator@acs-mailscan ~]$ df -H
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_00-lv_root
8.5G 1.9G 6.2G 24% /
tmpfs 985M 0 985M 0% /dev/shm
/dev/sda1 508M 62M 420M 13% /boot
/dev/mapper/vg_00-lv_tmp
1.1G 48M 956M 5% /tmp
/dev/mapper/vg_00-lv_var
22G 21G 0 100% /var
none 985M 21M 964M 3% /var/spool/MailScanner/incoming
[administrator@acs-mailscan ~]$


Looks like:
tmpfs 985M 0 985M 0% /dev/shm

is the issue......If I am reading that correctly....is that a temp file????
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

formatted for legibility

Code: Select all

Filesystem                Size Used Avail Use% Mounted on
/dev/mapper/vg_00-lv_root 8.5G 1.9G  6.2G  24% /
tmpfs                     985M 0     985M   0% /dev/shm
/dev/sda1                 508M 62M   420M  13% /boot
/dev/mapper/vg_00-lv_tmp  1.1G 48M   956M   5% /tmp
/dev/mapper/vg_00-lv_var   22G 21G      0 100% /var
none                      985M 21M   964M   3% /var/spool/MailScanner/incoming
Is your your /var partition which is filled up.

Let's find out where the space is hiding. Run this command next

Code: Select all

sudo du -sch /var/* | grep -v [0-9]K
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

pdwalker wrote:formatted for legibility
Well now....that did help.


[administrator@acs-mailscan /]$ sudo du -sch /var/* | grep -v [0-9]K
[sudo] password for administrator:
51M /var/cache
283M /var/clamav
1.6G /var/dcc
778M /var/EFA
740M /var/lib
221M /var/log
0 /var/mail
16G /var/spool
13M /var/www
20G total
[administrator@acs-mailscan /]$
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

16 gb /var/spool/MailScanner
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

79M /var/spool/MailScanner/incoming
16G /var/spool/MailScanner/quarantine
16G total
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

[administrator@acs-mailscan quarantine]$ sudo du -sch /var/spool/MailScanner/quarantine/* | grep -v [0-9]K
534M /var/spool/MailScanner/quarantine/20150323
517M /var/spool/MailScanner/quarantine/20150324
646M /var/spool/MailScanner/quarantine/20150325
572M /var/spool/MailScanner/quarantine/20150326
500M /var/spool/MailScanner/quarantine/20150327
142M /var/spool/MailScanner/quarantine/20150328
219M /var/spool/MailScanner/quarantine/20150329
501M /var/spool/MailScanner/quarantine/20150330
467M /var/spool/MailScanner/quarantine/20150331
503M /var/spool/MailScanner/quarantine/20150401
484M /var/spool/MailScanner/quarantine/20150402
305M /var/spool/MailScanner/quarantine/20150403
170M /var/spool/MailScanner/quarantine/20150404
220M /var/spool/MailScanner/quarantine/20150405
483M /var/spool/MailScanner/quarantine/20150406
493M /var/spool/MailScanner/quarantine/20150407
897M /var/spool/MailScanner/quarantine/20150408
1.2G /var/spool/MailScanner/quarantine/20150409
617M /var/spool/MailScanner/quarantine/20150410
145M /var/spool/MailScanner/quarantine/20150411
176M /var/spool/MailScanner/quarantine/20150412
2.1G /var/spool/MailScanner/quarantine/20150413
496M /var/spool/MailScanner/quarantine/20150414
423M /var/spool/MailScanner/quarantine/20150415
479M /var/spool/MailScanner/quarantine/20150416
393M /var/spool/MailScanner/quarantine/20150417
188M /var/spool/MailScanner/quarantine/20150418
330M /var/spool/MailScanner/quarantine/20150419
488M /var/spool/MailScanner/quarantine/20150420
541M /var/spool/MailScanner/quarantine/20150421
473M /var/spool/MailScanner/quarantine/20150422
123M /var/spool/MailScanner/quarantine/20150423
16G total
[administrator@acs-mailscan quarantine]$
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

Ok, you have most of your space taken up by your quarantine directory. Now the question is, how come it hasn't been cleaned out?

Did you suddenly got a flood of mail into your system?

And here is a question for others: How does the quarantine directory get cleared out? Do we have to do that by hand, or is there a process that goes and takes care of it?
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

also do

Code: Select all

sudo du -sch /var/lib/mysql/* | grep -v [0-9]K
sudo du -sch /var/dcc/* | grep -v [0-9]K
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

pdwalker wrote:Ok, you have most of your space taken up by your quarantine directory. Now the question is, how come it hasn't been cleaned out?

Did you suddenly got a flood of mail into your system?

And here is a question for others: How does the quarantine directory get cleared out? Do we have to do that by hand, or is there a process that goes and takes care of it?

Yes, we get hit hard from time to time..........., just got to love the spammers....

On clearing it out....I know (or think I do) that there is a way to set the quarantine period to shorter than the 30 days.
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

pdwalker wrote:also do

Code: Select all

sudo du -sch /var/lib/mysql/* | grep -v [0-9]K
sudo du -sch /var/dcc/* | grep -v [0-9]K

[administrator@acs-mailscan quarantine]$ sudo du -sch /var/lib/mysql/* | grep -v [0-9]K
[sudo] password for administrator:
75M /var/lib/mysql/ibdata1
5.0M /var/lib/mysql/ib_logfile0
5.0M /var/lib/mysql/ib_logfile1
565M /var/lib/mysql/mailscanner
2.7M /var/lib/mysql/sqlgrey
653M total
[administrator@acs-mailscan quarantine]$ sudo du -sch /var/dcc/* | grep -v [0-9]K
0 /var/dcc/dccifd
2.3M /var/dcc/libexec
1.6G /var/dcc/log
1.6G total
[administrator@acs-mailscan quarantine]$
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

You've got 32 days there, so your cleaner hasn't been cleaning.

in /etc/cron.daily is a script called mailwatch which appears to handle the cleaning. I'm looking through it now.

I'm also curious as you why the /var/dcc directory takes up so much space.
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

/var/www/html/mailscanner/conf.php

Code: Select all

132 define('QUARANTINE_DAYS_TO_KEEP', 30);
You may also wish to consider reducing these slightly

Code: Select all

113 // Define how many days of emails to keep
114 define('RECORD_DAYS_TO_KEEP', 60);
115
116 // Define how many days to audit logs to keep
117 define('AUDIT_DAYS_TO_KEEP', 60);
edit: after changing those values, run the quarantine_maint.php script

Code: Select all

sudo /usr/local/bin/mailwatch/tools/Cron_jobs/quarantine_maint.php --clean
df -h
Is there any chance you can increase the size of your /var partition? Consider the tip from here: Increasing Partition Sizes

shawniverson - feature request: could these configurable either during the setup and/or through the EFA-Configure program?
Archer
Posts: 27
Joined: 30 Oct 2014 13:38

Re: First Crash - Newby Needs Help

Post by Archer »

pdwalker wrote:/var/www/html/mailscanner/conf.php

Code: Select all

132 define('QUARANTINE_DAYS_TO_KEEP', 30);
You may also wish to consider reducing these slightly

Code: Select all

113 // Define how many days of emails to keep
114 define('RECORD_DAYS_TO_KEEP', 60);
115
116 // Define how many days to audit logs to keep
117 define('AUDIT_DAYS_TO_KEEP', 60);
edit: after changing those values, run the quarantine_maint.php script

Code: Select all

sudo /usr/local/bin/mailwatch/tools/Cron_jobs/quarantine_maint.php --clean
df -h
My setting match those.
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: First Crash - Newby Needs Help

Post by pdwalker »

So you'll need to change the first one at least. Say 21 days?

Changing the second is optional. You won't save very much changing those.
Post Reply