clamd scan pdf cpu

Bugs in eFa 4
Post Reply
nicola.piazzi
Posts: 388
Joined: 23 Apr 2015 09:45

clamd scan pdf cpu

Post by nicola.piazzi »

clamd take an excessive cpu time scanning pdf, and we always have pdf to scan so system hangs
as you can see whith clandtop we have files that take minutes to scan (and they are normal files not large)

disabling unofficial signatures seems to be ok
older efa with signatures was ok


COMMAND QUEUEDSINCE FILE
MULTISCAN 46.654s
MULTISCANFILE 46.654s /var/spool/MailScanner/incoming/40557/4DG9bW3Rl0z1LQSMh.message
MULTISCANFILE 46.654s /var/spool/MailScanner/incoming/40557/4DG9bW3Rl0z1LQSMh/nBVP130 LED260-4S 740 S.pdf
MULTISCANFILE 46.652s /var/spool/MailScanner/incoming/40557/4DG9bW3Rl0z1LQSMh/nBVP140 LED420-4S 36K1 740 PSU S.pdf
MULTISCANFILE 46.652s /var/spool/MailScanner/incoming/40557/4DG9bW3Rl0z1LQSMh/nBVP125 LED120-4S 740 S.pdf
MULTISCAN 1.857s
MULTISCANFILE 1.852s /var/spool/MailScanner/incoming/40369/4DG9cM6bBJz1LQSN2.message
MULTISCANFILE 1.425s /var/spool/MailScanner/incoming/40369/4DG9cM6bBJz1LQSN2/n20210113_154818.jpg
IDLE 0.151s
STATS 0.000s
nicola.piazzi
Posts: 388
Joined: 23 Apr 2015 09:45

Re: clamd scan pdf cpu

Post by nicola.piazzi »

no more problems moving these, now mission is to find problem signature

mv CVE-2010-0805.yar park
mv CVE-2010-0887.yar park
mv CVE-2010-1297.yar park
mv CVE-2012-0158.yar park
mv CVE-2013-0074.yar park
mv CVE-2013-0422.yar park
mv CVE-2015-1701.yar park
mv CVE-2015-2426.yar park
mv CVE-2015-2545.yar park
mv CVE-2015-5119.yar park
mv CVE-2016-5195.yar park
mv CVE-2017-11882.yar park
mv CVE-2018-20250.yar park
mv CVE-2018-4878.yar park

mv EK_BleedingLife.yar park
mv EMAIL_Cryptowall.yar park
mv email_Ukraine_BE_powerattack.yar park
mv foxhole_js.cdb park
mv foxhole_js.ndb park
mv javascript.ndb park
mv JJencode.yar park
mv jurlbla.ndb park
mv lott.ndb park
mv MiscreantPunch099-Low.ldb park
mv rfxn.yara park
mv Sanesecurity_sigtest.yara park
mv Sanesecurity_spam.yara park
mv scamnailer.ndb park
mv scam.yar park
mv shelter.ldb park
mv spam.ldb park
mv spearl.ndb park
mv spear.ndb park
mv urlhaus.ndb park
mv winnow_bad_cw.hdb park
mv winnow.complex.patterns.ldb park
mv winnow_phish_complete_url.ndb park
mv winnow_spam_complete.ndb park
mv WShell_ASPXSpy.yar park
mv WShell_Drupalgeddon2_icos.yar park
jon doe
Posts: 19
Joined: 07 Feb 2017 16:26
Location: Canada

Re: clamd scan pdf cpu

Post by jon doe »

I am experiencing the same issues (lots of clamd issues lately).
I was wondering what would be considered a good amount of scan time for a normal size PDF (1-3mb)?
Currently, I have moved all of these rules out as well but I still have some PDFs taking over 60 seconds to scan.
If a lot get queued, then it sends the clamd into a tailspin with timeouts and falsely marking email as "Virus (Denial of Service attack in message!)".


I currently have a simple script that is monitoring the milterin to notify me if it gets too high so I can look and fix these issues before they cause a major interruption.
nicola.piazzi
Posts: 388
Joined: 23 Apr 2015 09:45

Re: clamd scan pdf cpu

Post by nicola.piazzi »

can you send me that script :-) ?
nicola.piazzi
Posts: 388
Joined: 23 Apr 2015 09:45

Re: clamd scan pdf cpu

Post by nicola.piazzi »

I am testing clamd cpu usage and i found that this is caused by JJencode.yar
Is possible to remove it from installation ?
nicola.piazzi
Posts: 388
Joined: 23 Apr 2015 09:45

Re: clamd scan pdf cpu

Post by nicola.piazzi »

In these tests i run 10 minutes of mailserver activity with and without JJencode.yar in signature
As you can see with JJencode.yar it takes more than 10 minutes of cpu and without it takes 1 minute !!!!






Now test 1st time CLAMD CPU USAGE IN 10 MINUTES JJencode.yar PRESENT
--------------------------------------------------------------------
echo "* t e s t s t a r t";echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar;echo "* stopping services";systemctl stop clamd@scan.service;systemctl stop mailscanner;sleep 5;echo "* starting services";systemctl start clamd@scan.service;systemctl start mailscanner;echo "* clamscan cpu after start";ps -ef | grep clamscan;echo "* sleep 10 minutes";sleep 600;echo "* clamscan cpu after 1 minute";ps -ef | grep clamscan;echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar
* t e s t s t a r t
* check if present JJencode.yar
/var/lib/clamav/JJencode.yar
* stopping services
* starting services
* clamscan cpu after start
clamscan 243476 1 0 09:50 ? 00:00:00 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 244079 186146 0 09:50 pts/0 00:00:00 grep --color=auto clamscan
* sleep 10 minute
* clamscan cpu after 1 minute
clamscan 243476 1 99 09:50 ? 00:21:25 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 249184 186146 0 10:00 pts/0 00:00:00 grep --color=auto clamscan
* check if present JJencode.yar
/var/lib/clamav/JJencode.yar
in 10 minutes clamd used more that 21 minutes of cpu !


Now test 2nd time CLAMD CPU USAGE IN 10 MINUTES JJencode.yar PRESENT
--------------------------------------------------------------------
echo "* t e s t s t a r t";echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar;echo "* stopping services";systemctl stop clamd@scan.service;systemctl stop mailscanner;sleep 5;echo "* starting services";systemctl start clamd@scan.service;systemctl start mailscanner;echo "* clamscan cpu after start";ps -ef | grep clamscan;echo "* sleep 10 minutes";sleep 600;echo "* clamscan cpu after 1 minute";ps -ef | grep clamscan;echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar
* t e s t s t a r t
* check if present JJencode.yar
/var/lib/clamav/JJencode.yar
* stopping services
* starting services
* clamscan cpu after start
clamscan 249526 1 0 10:01 ? 00:00:00 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 250129 186146 0 10:01 pts/0 00:00:00 grep --color=auto clamscan
* sleep 10 minute
* clamscan cpu after 1 minute
clamscan 249526 1 99 10:01 ? 00:15:10 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 255188 186146 0 10:11 pts/0 00:00:00 grep --color=auto clamscan
* check if present JJencode.yar
/var/lib/clamav/JJencode.yar
in 10 minutes clamd used more that 15 minutes of cpu !

When JJencode.yar PRESENT clamdtop have files that stays some seconds and sometimes lot of seconds
--------------------------------------------------------------------------------------------------
COMMAND QUEUEDSINCE FILE
MULTISCAN 200.440s
MULTISCANFILE 200.440s /var/spool/MailScanner/incoming/244221/4DGdRT5Gypz1LQPF5.message
IDLE 3.910s
IDLE 3.909s

When JJencode.yar PRESENT clamscan process can take more than 100% of a single cpu
----------------------------------------------------------------------------------
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
249526 clamscan 20 0 2714012 1.7g 6872 S 183.7 14.9 6:00.28 clamd



Now we remove JJencode.yar
--------------------------
systemctl stop clamd@scan.service;systemctl stop mailscanner
mkdir /var/lib/clamav/park
mv /var/lib/clamav/JJencode.yar /var/lib/clamav/park/JJencode.yar



Now test 1st time CLAMD CPU USAGE IN 10 MINUTES JJencode.yar ABSENT
-------------------------------------------------------------------
echo "* t e s t s t a r t";echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar;echo "* stopping services";systemctl stop clamd@scan.service;systemctl stop mailscanner;sleep 5;echo "* starting services";systemctl start clamd@scan.service;systemctl start mailscanner;echo "* clamscan cpu after start";ps -ef | grep clamscan;echo "* sleep 10 minutes";sleep 600;echo "* clamscan cpu after 1 minute";ps -ef | grep clamscan;echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar
* t e s t s t a r t
* check if present JJencode.yar
ls: cannot access '/var/lib/clamav/JJencode.yar': No such file or directory
* stopping services
* starting services
* clamscan cpu after start
clamscan 256406 1 0 10:15 ? 00:00:00 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 257006 186146 0 10:15 pts/0 00:00:00 grep --color=auto clamscan
* sleep 10 minutes
* clamscan cpu after 1 minute
clamscan 256406 1 11 10:15 ? 00:01:10 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 263011 186146 0 10:25 pts/0 00:00:00 grep --color=auto clamscan
* check if present JJencode.yar
ls: cannot access '/var/lib/clamav/JJencode.yar': No such file or directory
in 10 minutes clamd used about 1 minute of cpu, and consider that a lot of this is used when start !


Now test 2nd time CLAMD CPU USAGE IN 10 MINUTES JJencode.yar ABSENT
-------------------------------------------------------------------
echo "* t e s t s t a r t";echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar;echo "* stopping services";systemctl stop clamd@scan.service;systemctl stop mailscanner;sleep 5;echo "* starting services";systemctl start clamd@scan.service;systemctl start mailscanner;echo "* clamscan cpu after start";ps -ef | grep clamscan;echo "* sleep 10 minutes";sleep 600;echo "* clamscan cpu after 1 minute";ps -ef | grep clamscan;echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar
* t e s t s t a r t
* check if present JJencode.yar
ls: cannot access '/var/lib/clamav/JJencode.yar': No such file or directory
* stopping services
* starting services
* clamscan cpu after start
clamscan 263384 1 0 10:26 ? 00:00:00 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 263987 186146 0 10:26 pts/0 00:00:00 grep --color=auto clamscan
* sleep 10 minutes
* clamscan cpu after 1 minute
clamscan 263384 1 7 10:26 ? 00:00:46 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 269333 186146 0 10:36 pts/0 00:00:00 grep --color=auto clamscan
* check if present JJencode.yar
ls: cannot access '/var/lib/clamav/JJencode.yar': No such file or directory
in 10 minutes clamd used less than 1 minute of cpu, and consider that a lot of this is used when start !


Now we restore JJencode.yar
----------------------------
systemctl stop clamd@scan.service;systemctl stop mailscanner
mv /var/lib/clamav/park/JJencode.yar /var/lib/clamav/JJencode.yar


Now test 3rdnd time CLAMD CPU USAGE IN 10 MINUTES JJencode.yar PRESENT
--------------------------------------------------------------------
echo "* t e s t s t a r t";echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar;echo "* stopping services";systemctl stop clamd@scan.service;systemctl stop mailscanner;sleep 5;echo "* starting services";systemctl start clamd@scan.service;systemctl start mailscanner;echo "* clamscan cpu after start";ps -ef | grep clamscan;echo "* sleep 10 minutes";sleep 600;echo "* clamscan cpu after 1 minute";ps -ef | grep clamscan;echo "* check if present JJencode.yar";ls /var/lib/clamav/JJencode.yar
* t e s t s t a r t
* check if present JJencode.yar
/var/lib/clamav/JJencode.yar
* stopping services
* starting services
* clamscan cpu after start
clamscan 286388 1 0 11:10 ? 00:00:00 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 286991 186146 0 11:10 pts/0 00:00:00 grep --color=auto clamscan
* sleep 10 minutes
* clamscan cpu after 1 minute
clamscan 286388 1 99 11:10 ? 00:10:13 /usr/sbin/clamd -c /etc/clamd.d/scan.conf
root 292047 186146 0 11:20 pts/0 00:00:00 grep --color=auto clamscan
* check if present JJencode.yar
/var/lib/clamav/JJencode.yar
in 10 minutes clamd used more that 10 minutes of cpu !
jon doe
Posts: 19
Joined: 07 Feb 2017 16:26
Location: Canada

Re: clamd scan pdf cpu

Post by jon doe »

nicola.piazzi wrote: 14 Jan 2021 07:26 can you send me that script :-) ?
Of course.
Here is the script attached.

Few things to note;
1. I run this from crontab every 15 minutes but you can obviously choose how often you check.
2. It's important to use a third party mail server. If you use localhost to send the mail, you could possibly have delays in sending the warning if the server has high CPU and not sending email quickly.
Attachments
miltercheck.zip
basic script to check milterin
(507 Bytes) Downloaded 164 times
henk
Posts: 517
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: clamd scan pdf cpu

Post by henk »

.

Disabled JJencode.yar

Code: Select all

/etc/clamav-unofficial-sigs/master.conf

Code: Select all

# Detect well-known software packers, that can be used by malware to hide itself.
#packers/JJencode.yar|MEDIUM
packers/JJencode.yar|DISABLED
“We are stuck with technology when what we really want is just stuff that works.” -Douglas Adams
jon doe
Posts: 19
Joined: 07 Feb 2017 16:26
Location: Canada

Re: clamd scan pdf cpu

Post by jon doe »

henk wrote: 17 Jan 2021 18:15 .

Disabled JJencode.yar

Code: Select all

/etc/clamav-unofficial-sigs/master.conf

Code: Select all

# Detect well-known software packers, that can be used by malware to hide itself.
#packers/JJencode.yar|MEDIUM
packers/JJencode.yar|DISABLED
If we make this change in master.conf will it overwrite on the next update? Should we be doing this in user.conf instead or will that do the same?
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: clamd scan pdf cpu

Post by shawniverson »

Use user.conf
Post Reply