[FRQ] Integration of Malware Patrol

Questions and answers about how to do stuff
Post Reply
Patschi
Posts: 6
Joined: 20 Apr 2015 19:44
Location: Lower Austria, Austria
Contact:

[FRQ] Integration of Malware Patrol

Post by Patschi » 20 Apr 2015 21:43

Hello,

I would wish me an integration of Malware Patrol in the great email filter appliance with SpamAssassin and ClamAV to beat the spam even better. What you think about this idea?

Regards,
Patrik
Last edited by Patschi on 21 Apr 2015 07:55, edited 1 time in total.

User avatar
shawniverson
Posts: 3005
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: [FR] Integration of Malware Patrol

Post by shawniverson » 20 Apr 2015 21:56

Sounds great, but it looks like it requires a subscription to use it, including the free version?
Version eFa 4.0.2 now available!

Patschi
Posts: 6
Joined: 20 Apr 2015 19:44
Location: Lower Austria, Austria
Contact:

Re: [FR] Integration of Malware Patrol

Post by Patschi » 20 Apr 2015 22:05

Yes, both requires a registration at this site. But the free version is really free, you just need to sign up and you get a access to the free version of this service (I am currently using that, but thinking about to switch to an payed subscription). After that you receive something like an API key where you can retrieve the according data via an URL like lists.malwarepatrol.net/cgi/getfile?receipt=API_KEY&product=8&list=spamassassin.

This information have to be downloaded in an specific interval, parsed and integrated with the services. Cybercizi for example provided a simple way how I can be integrated with postfix (but the malwarepatrol URL is not working anymore in this article): http://www.cyberciti.biz/tips/howto-set ... lware.html

User avatar
pdwalker
Posts: 1202
Joined: 18 Mar 2015 09:16

Re: [FR] Integration of Malware Patrol

Post by pdwalker » 21 Apr 2015 06:32

[edit: warning - stream of consciousness posting]

If you subscribe to the free version, you'll get access to to the block lists.

They provide preconfigured files for:
  • - ClamAV
    - Postfix MTA
    - SpamAssassin
(there are other files for other programs, but these are the three relevant to efa)

They also provide a script (https://www.malwarepatrol.net/MalwarePatrolDownload.sh - you may have to log in to get the script) to aid you in downloading the data. You only have to configure two things in the script before you run it.

As the free data is only updated every 2 to 3 days, don't abuse their hospitality and run it more often than that.

Having tried it, it now takes spamassassin lint from 5 seconds to over 735, so I won't be doing that check too often with the rules enabled.

---

Looking further, it appears that ClamAV used to use the Malware Patrol db. In fact, I can see in the log /var/log/clamav-unofficial-sigs.log that it tries to download a block list to: /usr/unofficial-dbs/mbl-dbs/mbl.ndb with the following content:
###################################################################
# WARNING: this block list was discontinued on Oct/07/2013
# according to our previous announcements.
# Users are advised to visit https://www.malwarepatrol.net/
# for information on how to continue using our data feed.
###################################################################
So, I guess that ClamAV should be the way to integrate the MalwarePatrol block list.

Other info:

cron job:
/etc/cron.d/clamav-unofficial-sigs-cron

script file (https://sourceforge.net/projects/unofficial-sigs/files/ - not updated since 2013-12-02)
/usr/local/bin/clamav-unofficial-sigs.sh

configuration file
/usr/local/etc/clamav-unofficial-sigs.conf

We should comment out lines 184-186 to avoid trying to download a list we cannot get until the "new" way of getting the data is working.

Also, we should set mbl_update_hours from 6 to 60.

---

The /usr/local/bin/clamav-unofficial-sigs.sh script will need updating to allow for the new way of downloading the MalwarePatrol block list. The relevant lines are between 1265 and 1380.

User avatar
pdwalker
Posts: 1202
Joined: 18 Mar 2015 09:16

Re: [FR] Integration of Malware Patrol

Post by pdwalker » 21 Apr 2015 06:48

solved.

writing up the details now

User avatar
pdwalker
Posts: 1202
Joined: 18 Mar 2015 09:16

[HOWTO] Enable Integration of Malware Patrol in efa 3.0.0.9

Post by pdwalker » 21 Apr 2015 08:51

ok, I have it working, and I fixed a minor problem with the ClamAV installation.

First, the minor problem

/etc/clamd.conf has the database directory set to:

Code: Select all

DatabaseDirectory /var/lib/clamav
/usr/local/etc/clamav-unofficial-sigs.conf has it set to:

Code: Select all

clam_dbs="/var/clamav"
Change the latter to match the former so the unofficial databases will be used, like so:

Code: Select all

clam_dbs="/var/lib/clamav"
(that's how it was configured in my installation - check yours to see what you have configured - the problem just may have been with me - do let me know, I'm curious)
Moderator: The correct path is /var/clamav . EPEL update tries to configure clamd to /var/lib/clamav.

Second, re-enable MailPatrol support with ClamAV

subscribe to the MailPatrol service
  1. go to https://www.malwarepatrol.net/lists.shtml and choose either the free or premium subscription
  2. you will get a receipt/password string.
  3. save this.
fix your /usr/local/etc/clamav-unofficial-sigs.conf configuration

diff clamav-unofficial-sigs.conf clamav-unofficial-sigs.conf.org
Moderator: The correct path is /var/clamav . EPEL update tries to configure clamd to /var/lib/clamav.

Code: Select all

53c53
< clam_dbs="/var/lib/clamav"
---
> clam_dbs="/var/clamav"
175c175
< si_update_hours="167"   # Most of these are hardly updated anymore.
---
> si_update_hours="4"   # Default is 4 hours (6 update checks daily).
188,189d187
< mbl_receipt="YOUR RECEIPT PASSWORD HERE"
<
198c196
< mbl_update_hours="60"   # Every 2.5 days.
---
> mbl_update_hours="6"   # Default is 6 hours (4 downloads daily).
Don't forget to update "YOUR RECEIPT PASSWORD HERE" with your proper receipt id for the Malware Patrol service

fix /usr/local/bin/clamav-unofficial-sigs.sh update script

diff clamav-unofficial-sigs.sh clamav-unofficial-sigs.sh.org

Code: Select all

19c19
< default_config="/usr/local/etc/clamav-unofficial-sigs.conf"
---
> default_config="/etc/clamav-unofficial-sigs.conf"
756c756
< mbl_url="lists.malwarepatrol.net"
---
> mbl_url="www.malwarepatrol.net"
1284d1283
<             # [Note: new format DB not available to free subscriptions]
1287c1286
<             # test -e $clam_dbs/$db_file -o -e $clam_dbs/$db_file-bak && rm -f -- "$clam_dbs/mbl.d*"
---
>             test -e $clam_dbs/$db_file -o -e $clam_dbs/$db_file-bak && rm -f -- "$clam_dbs/mbl.d*"
1294,1295c1293
<                --max-time "$curl_max_time" -o $mbl_dir/$db_file --compressed \
<                "https://$mbl_url/cgi/getfile?receipt=$mbl_receipt&product=8&list=clamav_basic"
---
>                --max-time "$curl_max_time" -o $mbl_dir/$db_file http://$mbl_url/cgi/submit?action=list_clamav_ext
test the command by running it and looking for errors

Code: Select all

/usr/local/bin/clamav-unofficial-sigs.sh
send yourself a bad message

Code: Select all

To: me
Subject: clamd test
Body:
this is an evil url:
http://update.ss838.com/
and send it to your efa protected account. You should see this message flag (make sure you use a current evil url)

Fix the Freshclam config to make it consistent

vi /etc/freshclam.conf

Code: Select all

DatabaseDirectory /var/lib/clamav
finally, force one last update to make sure everything is behaving

Code: Select all

# force unofficial-dbs refresh
rm /usr/unofficial-dbs/configs/last-mbl-update.txt
rm /usr/unofficial-dbs/configs/last-si-update.txt
/usr/bin/freshclam
/usr/local/bin/clamav-unofficial-sigs.sh
Voila! Malware Patrol enabled.

Unresolved Issues
  1. both /var/clamav and /var/lib/clamav exist. There should be only 1 directory; I think it should be /var/lib/clamav as that appear to be the default. Did the other directory come from unofficial-sigs.sh or was that an efa decision?
  2. which id? clam:clam or clamav:clamav. efa wants the former and clamav wants the latter.
    Moderator: The correct path is /var/clamav . EPEL update tries to configure clamd to /var/lib/clamav. Correct user is clamav. Again, same reason
  3. http://sanesecurity.com/usage/signatures/ has more signatures available than are being used. The list should be reviewed and the /usr/local/etc/clamav-unofficial-sigs.conf updated to use/exclude those databases
  4. https://www.securiteinfo.com/ has also gone to a subscription model. They do allow some updates, but the better updates require an account. On the plus side, it is updated hourly. The unofficial-sigs.sh script should be changed to handle these.
  5. note to self: list of other services that might ingegrate?


User avatar
shawniverson
Posts: 3005
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: [FRQ] Integration of Malware Patrol

Post by shawniverson » 26 Apr 2015 16:47

both /var/clamav and /var/lib/clamav exist. There should be only 1 directory; I think it should be /var/lib/clamav as that appear to be the default. Did the other directory come from unofficial-sigs.sh or was that an efa decision?
which id? clam:clam or clamav:clamav. efa wants the former and clamav wants the latter.
This appears to be self-inflicted.

Using EPEL, by chance?

EPEL uses /var/lib/clamav and clam user instead of /var/clamav and clamav user

:oops:
Version eFa 4.0.2 now available!

User avatar
pdwalker
Posts: 1202
Joined: 18 Mar 2015 09:16

Re: [FRQ] Integration of Malware Patrol

Post by pdwalker » 26 Apr 2015 16:52

Stock Centos 6 and build from scratch.

Should i reset everything to use /var/clamav?

User avatar
shawniverson
Posts: 3005
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: [FRQ] Integration of Malware Patrol

Post by shawniverson » 26 Apr 2015 16:55

I am thinking so. I am working through this issue right now using epel...
Version eFa 4.0.2 now available!

User avatar
shawniverson
Posts: 3005
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: [FRQ] Integration of Malware Patrol

Post by shawniverson » 26 Apr 2015 17:48

More info on epel issue here....

viewtopic.php?f=5&p=3459#p3459
Version eFa 4.0.2 now available!

User avatar
pdwalker
Posts: 1202
Joined: 18 Mar 2015 09:16

Re: [FRQ] Integration of Malware Patrol

Post by pdwalker » 27 Apr 2015 01:44

yup, that was exactly my problem.

continuing the conversation in the other, more relevant thread

Post Reply