efa-project.org

Posted: **21 Dec 2020 11:52**

My server is under attack, my antivirus is unable to scan incoming messages and they all get stuck on /var/spool/MailScanner/milterout/

On my logs things are crazy, this is a sample of what I see:

Code: Select all

Dec 21 10:43:58 filtro MailScanner[31637]: Virus Scanning: Denial Of Service attack detected!
Dec 21 10:44:01 filtro MailScanner[32217]: Virus Scanning: Denial Of Service attack detected!
Dec 21 10:46:59 filtro MailScanner[31637]: Unpacking 7zip archive: nmsg-31637-71.txt
Dec 21 10:46:59 filtro MailScanner[32217]: Unpacking 7zip archive: nmsg-32217-67.txt
Dec 21 10:46:59 filtro MailScanner[32217]: Unpacking 7zip archive: nmsg-32217-64.txt
Dec 21 10:46:59 filtro MailScanner[31637]: Unpacking 7zip archive: nmsg-31637-77.txt
Dec 21 10:46:59 filtro MailScanner[31656]: Unpacking 7zip archive: nmsg-31656-69.txt
Dec 21 10:46:59 filtro MailScanner[31656]: Unpacking 7zip archive: nmsg-31656-73.txt
Dec 21 10:47:00 filtro MailScanner[31656]: Unpacking 7zip archive: nmsg-31656-78.txt
Dec 21 10:47:00 filtro MailScanner[31656]: Unpacking 7zip archive: nmsg-31656-81.txt
Dec 21 10:47:01 filtro MailScanner[31637]: Unpacking 7zip archive: nmsg-31637-100.txt
Dec 21 10:47:01 filtro MailScanner[31637]: Unpacking 7zip archive: nmsg-31637-102.txt
Dec 21 10:47:01 filtro MailScanner[31637]: Unpacking 7zip archive: nmsg-31637-106.txt
Dec 21 10:47:01 filtro MailScanner[31656]: Unpacking 7zip archive: nmsg-31656-90.txt
Dec 21 10:47:01 filtro MailScanner[31656]: Unpacking 7zip archive: nmsg-31656-94.txt
Dec 21 10:47:02 filtro MailScanner[31656]: Unpacking 7zip archive: nmsg-31656-98.txt

Also there are some lines indicating that MailScanner cannot open a file:

Code: Select all

Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF90qh8z6Y78: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF90qh8z6Y78 for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF91WgQz6Y6q: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF91WgQz6Y6q for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF922VZz6Y7F: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF922VZz6Y7F for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF92gCMz6Y6X: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF92gCMz6Y6X for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF92h3jz1qNr: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF92h3jz1qNr for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF92kfvz6Y73: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF92kfvz6Y73 for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF935Ycz6Y7H: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF935Ycz6Y7H for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF93GYFz6Y6v: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF93GYFz6Y6v for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF95B4Yz1qNp: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF95B4Yz1qNp for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxF95F7Pz6Y6x: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxF95F7Pz6Y6x for relaying, will try again later
Dec 21 11:55:01 filtro MailScanner[26468]: Could not open file +</var/spool/MailScanner/milterout/4CzxFB07KSz6Y6l: No such file or directory
Dec 21 11:55:01 filtro MailScanner[26468]: Cannot open /var/spool/MailScanner/milterout/4CzxFB07KSz6Y6l for relaying, will try again later

This is the detail for one message id, I see the message is requeued from previous ID (4Czv0N2KY7z2xS5S) to new ID (4CzvF14vS4z66Qk) buth then, MailScanner cannot open the file:

Code: Select all

[host]# grep "4CzvF14vS4z66Qk" /var/log/maillog
Dec 21 10:24:37 filtro MailScanner[1770]: Requeue: 4Czv0N2KY7z2xS5S to 4CzvF14vS4z66Qk
Dec 21 10:24:43 filtro MailScanner[4354]: Could not open file +</var/spool/MailScanner/milterout/4CzvF14vS4z66Qk: No such file or directory
Dec 21 10:24:43 filtro MailScanner[4354]: Cannot open /var/spool/MailScanner/milterout/4CzvF14vS4z66Qk for relaying, will try again later

Then on clamav logs (/var/log/clamd.scan):

Code: Select all

WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Czvnw4Lpgz2WxWD.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4CzthQ29GCz2vkdY.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Czvgb4g1Gz2WxWD.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Czvnc6pBmz2vkdK.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Czvfn0rH7z36lkG.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Czvn24c6mz31YC5.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Czvtp1KnXz31YCp.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Cztl13DVpz2vkdT.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/14575/4Czthx4870z2vkdT.message
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4CzvmL3SV6z2WxWD.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4Czvhj2Zq1z2vkdP.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4Czvhm0bsrz2vkdK.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4Czvmf114Hz2vkdK.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4Czvj44DdHz2vkdT.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4CzvjB2mTtz2vkdT.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4Czvmh3qXLz366Qj.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4CzvjM06n5z2vkdK.header
WARNING: Failed to determine real path for: /var/spool/MailScanner/incoming/15224/4CzvjQ0sjWz2vkdK.header

I am reading that this could be a zip-bomb, I am not shure about this and I am now shure how to protect me from this attack.
Any help would be apreciated.

Posted: **21 Dec 2020 13:14**

From the docs:

In order to apply filename and filetype checks on the contents of Zip, Rar and UU-encoded archives, they are all unpacked. The Rar archives are unpacked using an external "unrar" program, while Zip and UU-encoded archives are handled internally. Any archive found nested deeper than the “Maximum Archive Depth” will make MailScanner reject the message.

In order to stop MailScanner checking the filenames and filetypes of the contents of Zip and Rar archives, the "Maximum Archive Depth" option must be set to 0. Setting this to 0 stops MailScanner unpacking Zip and Rar archives itself. Virus scanners do their own archive unpacking, and so setting this to 0 does not stop MailScanner finding viruses in archives.

There are 2 common settings for this option:

•0 stops MailScanner unpacking Zip and Rar archives. Viruses inside Zip and Rar archives will still be found. No filename checking or filetype checking will be done on the contents of Zip and Rar archives. Checks on Archive Contents• Contents of Zip and Rar archives checked against rules for valid filenames and filetypes• Recursively unpacked to a preset depth• Rar version 3 archives supported

•3 makes MailScanner check the filenames and filetypes of files within archives within archives within the message, which is as deeply nested as any average user will create. Any archive nested deeper than this will cause MailScanner to reject the attachment as a potential denial-of-service attack and it will be replaced by a warning message.

So to summarize any archive that it more then 3 layers deep will be rejected and a warning message will appear for potential denial-of-service attack. It's up to you to investigate and determine whether it is or isn't legitimate from there.

Posted: **22 Dec 2020 07:03**

Thanks for your help smyers119.
I was able to track the problem down to a certain degree, please read my question at the end...

This post clamd@scan.service start operation timed out. Terminating had similar processor behavior than my server.

and cpu is going through the roof
clamscan 87.2 % /usr/sbin/clamd -c /etc/clamd.d/scan.conf

So I followed shawniverson sugestion and disabled clamav-unofficial-sigs.

Code: Select all

sudo sed -i "/^user_configuration_complete=/ c\user_configuration_complete=no" /etc/clamav-unofficial-sigs/user.conf
sudo rm -rf /var/lib/clamav/*
sudo freshclam
sudo systemctl start clamd@scan

This are the databases I had before in /var/lib/clamav/:

Code: Select all

badmacro.ndb                 CVE-2017-11882.yar                malware.expert.hdb         shelter.ldb
bank_rule.yar                CVE-2018-20250.yar                malware.expert.ldb         sigwhitelist.ign2
blurl.ndb                    CVE-2018-4878.yar                 malware.expert.ndb         spamattach.hdb
bofhland_cracked_URL.ndb     daily.cld                         malwarehash.hsb            spamimg.hdb
bofhland_malware_attach.hdb  EK_BleedingLife.yar               MiscreantPunch099-Low.ldb  spam.ldb
bofhland_malware_URL.ndb     EMAIL_Cryptowall.yar              phish.ndb                  spearl.ndb
bofhland_phishing_URL.ndb    email_Ukraine_BE_powerattack.yar  phishtank.ndb              spear.ndb
bytecode.cvd                 foxhole_filename.cdb              porcupine.hsb              urlhaus.ndb
CVE-2010-0805.yar            foxhole_generic.cdb               porcupine.ndb              winnow.attachments.hdb
CVE-2010-0887.yar            foxhole_js.cdb                    rfxn.hdb                   winnow_bad_cw.hdb
CVE-2010-1297.yar            foxhole_js.ndb                    rfxn.ndb                   winnow.complex.patterns.ldb
CVE-2012-0158.yar            hackingteam.hsb                   rfxn.yara                  winnow_extended_malware.hdb
CVE-2013-0074.yar            JJencode.yar                      rogue.hdb                  winnow_extended_malware_links.ndb
CVE-2013-0422.yar            junk.ndb                          sanesecurity.ftm           winnow_malware.hdb
CVE-2015-1701.yar            jurlbla.ndb                       Sanesecurity_sigtest.yara  winnow_malware_links.ndb
CVE-2015-2426.yar            jurlbl.ndb                        Sanesecurity_spam.yara     winnow_phish_complete_url.ndb
CVE-2015-2545.yar            lott.ndb                          scamnailer.ndb             winnow_spam_complete.ndb
CVE-2015-5119.yar            main.cvd                          scam.ndb                   WShell_ASPXSpy.yar
CVE-2016-5195.yar            malware.expert.fp                 scam.yar                   WShell_Drupalgeddon2_icos.yar

After deleting all files and running freshclam, this is whai I have now

Code: Select all

bytecode.cvd  daily.cvd  main.cvd

This is my server load before and after:

: system load clamav unoficial sigs.png (27.13 KiB) Viewed 35672 times

Now my question is: I would like to use those databases, is there a method to debug which one is causing the processor

bottleneck?

Posted: **25 Dec 2020 08:36**

I have a similar issue since 23rd November. Genuine mails randomly getting blocked under Denial of Service attack detected! (same mail gets delivered fine to other users). CPU utilization by clamd is also high. I have disabled virus scanning for now. May be I should try disabling unofficial sigs.

Posted: **25 Dec 2020 11:55**

I am preparing another update to clamav-unofficial-sigs to hopefully provide better stability

Posted: **06 Jan 2021 02:45**

Hello Community,

I am suffering from the same issue here where a huge number of our legit outgoing emails are being reported as DOS attack by clamd since the recent update. I have to temporarily disable scanning of outgoing messages using my rulesets for the time being.

Shawn,

Is there a github issue that we can follow on the progress of this case?

Thx in advance.

Posted: **14 Jan 2021 15:00**

is there a way to get the email that was blocked by the Denial of service
as this is causing problem and a very important email has been stop

so how can i recover

thanks
Trident

Posted: **20 Jan 2021 17:31**

From what I have tested, you will not be able to get those emails back because they were not quarantined. I think you need to set your config in mailscanner.conf to save silent viruses in order to quarantine this kind of email. If you do that and test it, you should be able to release that kind of email if this happens again to you in the future. This happened to me as well and after making that change, I am now able to release them.

Posted: **22 Jan 2021 18:09**

This is still a major problem for us here. I have now disabled the unofficial sigs as well.
I am also interested if there is a place I can monitor to see if/when progress is made on solving these issues?

Posted: **23 Jan 2021 12:15**

https://github.com/E-F-A/v4/issues/256

Posted: **05 Feb 2021 07:02**

this is still happening also after the last update of efa 4.0.4-7

Posted: **05 Feb 2021 11:40**

Did you remove and disable the yara ruleset?

Posted: **05 Feb 2021 16:36**

Yes I did....

05/02/21 16:18:38 bounce-mc.us11_42424841.4820294-978f611f74@mail34.... p.vanderhall@xxxxxx Blijf je in ons netwerk? 61.26kB
Virus (Denial of Service attack in message! )
05/02/21 16:15:39 klantinfo@rabobank.nl.stlc.online info@xxxxxxx Lopende zaken (ref. 1-103-107926120) 19.02kB
Virus (Denial of Service attack in message! )

Posted: **06 Feb 2021 10:47**

meanwhile I did a full re-install till now the DOS notifications are gone but the next issue is showing in the logs":

NOQUEUE: reject: RCPT from mail.brookparko.com[46.165.236.29]: 450 4.1.2 <xxxx@xxxxx>: Recipient address rejected: Domain not found; from=<bounce@brookparko.com> to=<xxxxx@xxxxxx> proto=ESMTP helo=<mail.brookparko.com>

the email address is valid as it is mine

.
enabling and disabling recipient verification does nothing ( still showing in the logs)

Posted: **06 Feb 2021 11:41**

Check that you can resolve your domain on the appliance and that unbound is operational.

Posted: **06 Feb 2021 16:35**

yes I can, besides that the DOS with Virus scanning is back again

found this in the logs...
AV engine clamd timed out
Feb 6 17:25:38 MailScanner[4196]: clamd: Failed to complete, timed out
Feb 6 17:25:38 MailScanner[4196]: Virus Scanning: Denial Of Service attack is in message 4DXy6s6fBkzB5v9w

Posted: **07 Feb 2021 00:09**

I recommend you disable your clamav-unofficial-sigs rulesets and purge them as a troubleshooting step. Your clamd can't keep up for some reason, so this would be a starting point to make sure your clam runs properly.

If your dns is fine, and you can resolve the domain, make sure your transport rules are set up correctly.

Posted: **07 Feb 2021 16:13**

how do I do that? ( sorry for the NOOB question),

Andwhat has changed in recent version (reinstall ) of EFA that this DOS is happening as I never had it before on the same hardware?

Posted: **07 Feb 2021 16:40**

Not sure yet, that why we need to test things. I suspect clamav-unofficial-sigs though because it is a new version.

/etc/clamav-unofficial-sigs/user.conf

Code: Select all

user_configuration_complete="no"

Enable maintenance mode in eFa-Configure

Code: Select all

sudo systemctl stop clamd@scan
mkdir ~/sigsbackup
sudo mv /var/lib/clamav/* ~/sigsbackup
sudo freshclam
sudo systemctl start clamd@scan

Disable maintenance mode in eFa-Configure

Posted: **10 Feb 2021 13:32**

I also have that problem, sigh

Posted: **10 Feb 2021 14:47**

nicola.piazzi wrote: ↑10 Feb 2021 13:32 I also have that problem, sigh

Can you help me isolate in on what is causing clam to crash here?

Posted: **10 Feb 2021 14:58**

is very difficoult to do, i can try

Posted: **10 Feb 2021 17:30**

Hi Guys,

I'm running into the same issue. It just started happening today.
I was seeing a lot of 'Denial of service attack detected!' errors and Clamd/scan was running the cpu at an all time high.

So I updated eFa-project to the latest version
I disabled the clamav-unofficial-sigs (upon Shawn's recommendation)

I restarted the box and see that out of 2980 emails stuck in MilternIN, there are now 2100.
It was delivering mail for a few minutes, then it hangs....for a minute or two, then continues to deliver.

I don't see any other errors in the Maillogs, but if I do a 'tail -f /var/log/maillog' I can see it scrolling by and then stop.
When It stops scrolling, is when email is stagnant. However, after awhile it will continue.

Posted: **11 Feb 2021 07:35**

Yesterday i put in /etc/clamd.d/scan.conf
OfficialDatabaseOnly yes

this is log today 11/02 at 08.27 and after this no more problem

[root@EFA42 batch]# grep "Denial of Service" /var/log/maillog
Feb 9 17:33:36 EFA42 MailScanner[131547]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 12:31:55 EFA42 MailScanner[492234]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 12:54:53 EFA42 MailScanner[492209]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:05:02 EFA42 MailScanner[491589]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:05:14 EFA42 MailScanner[492198]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:05:35 EFA42 MailScanner[491184]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:06:03 EFA42 MailScanner[491109]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:08:03 EFA42 MailScanner[492234]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:08:03 EFA42 MailScanner[492234]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:09:51 EFA42 MailScanner[491140]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:09:52 EFA42 MailScanner[491201]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:09:53 EFA42 MailScanner[492209]: Viruses marked as silent: Denial of Service attack in message!
Feb 10 13:15:28 EFA42 MailScanner[491589]: Viruses marked as silent: Denial of Service attack in message!

This is differences in signatures between un-official and official (I added also securiteinfo in unofficial)
/var/log/messages:42035:Feb 10 11:54:19 EFA42 clamd[2874]: Database correctly reloaded (12900151 signatures)
/var/log/messages:44985:Feb 10 15:07:53 EFA42 clamd[3053]: Database correctly reloaded (8655826 signatures)

Posted: **11 Feb 2021 13:24**

I wonder if we can reintroduce various signatures from clamav-unofficial-sigs rulesets and try to isolate the offending ruleset(s)

efa-project.org

Virus Scanning: Denial Of Service attack detected!

Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!

Re: Virus Scanning: Denial Of Service attack detected!