Page 1 of 1

[FRQ] Integration of Malware Patrol

Posted: 20 Apr 2015 21:43
by Patschi
Hello,

I would wish me an integration of Malware Patrol in the great email filter appliance with SpamAssassin and ClamAV to beat the spam even better. What you think about this idea?

Regards,
Patrik

Re: [FR] Integration of Malware Patrol

Posted: 20 Apr 2015 21:56
by shawniverson
Sounds great, but it looks like it requires a subscription to use it, including the free version?

Re: [FR] Integration of Malware Patrol

Posted: 20 Apr 2015 22:05
by Patschi
Yes, both requires a registration at this site. But the free version is really free, you just need to sign up and you get a access to the free version of this service (I am currently using that, but thinking about to switch to an payed subscription). After that you receive something like an API key where you can retrieve the according data via an URL like lists.malwarepatrol.net/cgi/getfile?receipt=API_KEY&product=8&list=spamassassin.

This information have to be downloaded in an specific interval, parsed and integrated with the services. Cybercizi for example provided a simple way how I can be integrated with postfix (but the malwarepatrol URL is not working anymore in this article): http://www.cyberciti.biz/tips/howto-set ... lware.html

Re: [FR] Integration of Malware Patrol

Posted: 21 Apr 2015 06:32
by pdwalker
[edit: warning - stream of consciousness posting]

If you subscribe to the free version, you'll get access to to the block lists.

They provide preconfigured files for:
  • - ClamAV
    - Postfix MTA
    - SpamAssassin
(there are other files for other programs, but these are the three relevant to efa)

They also provide a script (https://www.malwarepatrol.net/MalwarePatrolDownload.sh - you may have to log in to get the script) to aid you in downloading the data. You only have to configure two things in the script before you run it.

As the free data is only updated every 2 to 3 days, don't abuse their hospitality and run it more often than that.

Having tried it, it now takes spamassassin lint from 5 seconds to over 735, so I won't be doing that check too often with the rules enabled.

---

Looking further, it appears that ClamAV used to use the Malware Patrol db. In fact, I can see in the log /var/log/clamav-unofficial-sigs.log that it tries to download a block list to: /usr/unofficial-dbs/mbl-dbs/mbl.ndb with the following content:
###################################################################
# WARNING: this block list was discontinued on Oct/07/2013
# according to our previous announcements.
# Users are advised to visit https://www.malwarepatrol.net/
# for information on how to continue using our data feed.
###################################################################
So, I guess that ClamAV should be the way to integrate the MalwarePatrol block list.

Other info:

cron job:
/etc/cron.d/clamav-unofficial-sigs-cron

script file (https://sourceforge.net/projects/unofficial-sigs/files/ - not updated since 2013-12-02)
/usr/local/bin/clamav-unofficial-sigs.sh

configuration file
/usr/local/etc/clamav-unofficial-sigs.conf

We should comment out lines 184-186 to avoid trying to download a list we cannot get until the "new" way of getting the data is working.

Also, we should set mbl_update_hours from 6 to 60.

---

The /usr/local/bin/clamav-unofficial-sigs.sh script will need updating to allow for the new way of downloading the MalwarePatrol block list. The relevant lines are between 1265 and 1380.

Re: [FR] Integration of Malware Patrol

Posted: 21 Apr 2015 06:48
by pdwalker
solved.

writing up the details now

[HOWTO] Enable Integration of Malware Patrol in efa 3.0.0.9

Posted: 21 Apr 2015 08:51
by pdwalker
ok, I have it working, and I fixed a minor problem with the ClamAV installation.

First, the minor problem

/etc/clamd.conf has the database directory set to:

Code: Select all

DatabaseDirectory /var/lib/clamav
/usr/local/etc/clamav-unofficial-sigs.conf has it set to:

Code: Select all

clam_dbs="/var/clamav"
Change the latter to match the former so the unofficial databases will be used, like so:

Code: Select all

clam_dbs="/var/lib/clamav"
(that's how it was configured in my installation - check yours to see what you have configured - the problem just may have been with me - do let me know, I'm curious)
Moderator: The correct path is /var/clamav . EPEL update tries to configure clamd to /var/lib/clamav.

Second, re-enable MailPatrol support with ClamAV

subscribe to the MailPatrol service
  1. go to https://www.malwarepatrol.net/lists.shtml and choose either the free or premium subscription
  2. you will get a receipt/password string.
  3. save this.
fix your /usr/local/etc/clamav-unofficial-sigs.conf configuration

diff clamav-unofficial-sigs.conf clamav-unofficial-sigs.conf.org
Moderator: The correct path is /var/clamav . EPEL update tries to configure clamd to /var/lib/clamav.

Code: Select all

53c53
< clam_dbs="/var/lib/clamav"
---
> clam_dbs="/var/clamav"
175c175
< si_update_hours="167"   # Most of these are hardly updated anymore.
---
> si_update_hours="4"   # Default is 4 hours (6 update checks daily).
188,189d187
< mbl_receipt="YOUR RECEIPT PASSWORD HERE"
<
198c196
< mbl_update_hours="60"   # Every 2.5 days.
---
> mbl_update_hours="6"   # Default is 6 hours (4 downloads daily).
Don't forget to update "YOUR RECEIPT PASSWORD HERE" with your proper receipt id for the Malware Patrol service

fix /usr/local/bin/clamav-unofficial-sigs.sh update script

diff clamav-unofficial-sigs.sh clamav-unofficial-sigs.sh.org

Code: Select all

19c19
< default_config="/usr/local/etc/clamav-unofficial-sigs.conf"
---
> default_config="/etc/clamav-unofficial-sigs.conf"
756c756
< mbl_url="lists.malwarepatrol.net"
---
> mbl_url="www.malwarepatrol.net"
1284d1283
<             # [Note: new format DB not available to free subscriptions]
1287c1286
<             # test -e $clam_dbs/$db_file -o -e $clam_dbs/$db_file-bak && rm -f -- "$clam_dbs/mbl.d*"
---
>             test -e $clam_dbs/$db_file -o -e $clam_dbs/$db_file-bak && rm -f -- "$clam_dbs/mbl.d*"
1294,1295c1293
<                --max-time "$curl_max_time" -o $mbl_dir/$db_file --compressed \
<                "https://$mbl_url/cgi/getfile?receipt=$mbl_receipt&product=8&list=clamav_basic"
---
>                --max-time "$curl_max_time" -o $mbl_dir/$db_file http://$mbl_url/cgi/submit?action=list_clamav_ext
test the command by running it and looking for errors

Code: Select all

/usr/local/bin/clamav-unofficial-sigs.sh
send yourself a bad message

Code: Select all

To: me
Subject: clamd test
Body:
this is an evil url:
http://update.ss838.com/
and send it to your efa protected account. You should see this message flag (make sure you use a current evil url)

Fix the Freshclam config to make it consistent

vi /etc/freshclam.conf

Code: Select all

DatabaseDirectory /var/lib/clamav
finally, force one last update to make sure everything is behaving

Code: Select all

# force unofficial-dbs refresh
rm /usr/unofficial-dbs/configs/last-mbl-update.txt
rm /usr/unofficial-dbs/configs/last-si-update.txt
/usr/bin/freshclam
/usr/local/bin/clamav-unofficial-sigs.sh
Voila! Malware Patrol enabled.

Unresolved Issues
  1. both /var/clamav and /var/lib/clamav exist. There should be only 1 directory; I think it should be /var/lib/clamav as that appear to be the default. Did the other directory come from unofficial-sigs.sh or was that an efa decision?
  2. which id? clam:clam or clamav:clamav. efa wants the former and clamav wants the latter.
    Moderator: The correct path is /var/clamav . EPEL update tries to configure clamd to /var/lib/clamav. Correct user is clamav. Again, same reason
  3. http://sanesecurity.com/usage/signatures/ has more signatures available than are being used. The list should be reviewed and the /usr/local/etc/clamav-unofficial-sigs.conf updated to use/exclude those databases
  4. https://www.securiteinfo.com/ has also gone to a subscription model. They do allow some updates, but the better updates require an account. On the plus side, it is updated hourly. The unofficial-sigs.sh script should be changed to handle these.
  5. note to self: list of other services that might ingegrate?

Re: [FRQ] Integration of Malware Patrol

Posted: 22 Apr 2015 22:23
by shawniverson

Re: [FRQ] Integration of Malware Patrol

Posted: 26 Apr 2015 16:47
by shawniverson
both /var/clamav and /var/lib/clamav exist. There should be only 1 directory; I think it should be /var/lib/clamav as that appear to be the default. Did the other directory come from unofficial-sigs.sh or was that an efa decision?
which id? clam:clam or clamav:clamav. efa wants the former and clamav wants the latter.
This appears to be self-inflicted.

Using EPEL, by chance?

EPEL uses /var/lib/clamav and clam user instead of /var/clamav and clamav user

:oops:

Re: [FRQ] Integration of Malware Patrol

Posted: 26 Apr 2015 16:52
by pdwalker
Stock Centos 6 and build from scratch.

Should i reset everything to use /var/clamav?

Re: [FRQ] Integration of Malware Patrol

Posted: 26 Apr 2015 16:55
by shawniverson
I am thinking so. I am working through this issue right now using epel...

Re: [FRQ] Integration of Malware Patrol

Posted: 26 Apr 2015 17:48
by shawniverson
More info on epel issue here....

viewtopic.php?f=5&p=3459#p3459

Re: [FRQ] Integration of Malware Patrol

Posted: 27 Apr 2015 01:44
by pdwalker
yup, that was exactly my problem.

continuing the conversation in the other, more relevant thread