Check Fisching from Phishtank

[nicola.piazzi]

I wrote this sample script that must be saved in /etc/mail/spamassassin/phishtank.sh and putted in crontab
It check every 2 hours if phistank file is changed and in that case it download and process to create /etc/mail/spamassassin/phishtank.cf
Then it restart mailscanner to get changes and send a mail to inform
Please copy code personalizing the BOLD text



#
# phishtank.sh
# ============
#
# Load Phishing url from https://www.phishtank.com/ and create rule to catch it in SpamAssassin
#
# Put this script into /etc/mail/spamassassin
# Add this line to crontab to check every 2 hours for a new file
# 01 */2 * * * /etc/mail/spamassassin/phishtank.sh > /dev/null
# Register for an application key to have grant to have unilimited downloads


# Export work foldera and changing directory
export F="/etc/mail/spamassassin"
cd $F

# Destination alert email
dest=yourmail@yourdomain.yourtld

# Create first time old status file
if [ ! -f $F/phishtank.status.o ]
then
> $F/phishtank.status.o
chmod 777 $F/phishtank.status.o
fi

# Get Online file info
# url + activation key for phishtank (Without this key, you will be limited to a few downloads per day.)
curl -sI "http://data.phishtank.com/data/[b]YOUR- ... ED-IN-SIDE[/b]/online-valid.csv" | grep Last-Modified > $F/phishtank.status
chmod 777 $F/phishtank.status

# Run pearl load script basing of diff of status files
diff $F/phishtank.status.o $F/phishtank.status > /dev/null
if [ $? -ne 0 ]
then
echo "Status Different . processing new PhishTank"
cat $F/phishtank.status > $F/phishtank.status.o

######################################################
### Load From phishtank.com and create cf file (START)
######################################################
script=$(cat <<'EOF'
use strict;
use warnings;
use Text::CSV;
use LWP::Simple;
use Data::Dumper;
my $SSCORE = "6.40"; # Score to give to a matching phishing
my $SFILEOUT = "phishtank.cf";
my $SFILECSV = "phishtank.csv";
# url + activation key for phishtank (Without this key, you will be limited to a few downloads per day.)
my $status = getstore("http://data.phishtank.com/data/[b]YOUR- ... ED-IN-SIDE[/b]/online-valid.csv", $SFILECSV);
if ( is_success($status) ){
print "File downloaded correctly\n";
} else {
print "Error downloading file: $status\n";
exit 0;
}
my $csv = Text::CSV->new();
open (my $csvfile, "<", $SFILECSV) or die $!;
# CSV parsing
my $sline="uri PHISHTANK /impossibletofindZ9z8";
my $item;
my $backslash = "\\/";
while (my $row = $csv->getline($csvfile)) {
if ($row) {
my @columns = @$row;
if ($columns[1] ne "url"){
$item = $columns[1];
$item =~ s/\./\\./g;
$item =~ s/\//$backslash/g;
$item =~ s/\#/\\#/g;
$item =~ s/\@/\\@/g;
$sline = $sline . "|" . $item;
}
} else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
exit 0;
}
}
close $csvfile;
print "Parse and push is correctly \n";
open (my $OUT, ">", $SFILEOUT) or die $!;
#print $OUT "###\n### Create file: " .gmtime()."\n### \n\n";
print $OUT $sline . "/is \n";
print $OUT "describe PHISHTANK Contain url listed in PhishTank.com (probable phishing) \n";
print $OUT "score PHISHTANK " . $SSCORE ." \n";
close $OUT;
print "Script successful";
EOF
)
perl -e "$script"
#####################################################
### Load From phishtank.com and create cf file (STOP)
#####################################################

# Restart MailScanner Service
sleep 5
service MailScanner restart

# Send email informing of new pattern loaded
sleep 20
echo -e "New PhishTank patterns loaded" | mail -r YOUR-MAIL-FROM -s "EFA - Antispam Mailserver Gateway" $dest
fi

[ovizii]

Can someone from the core team maybe clarify where this file originates from: /etc/MailScanner/phishing.bad.sites.conf - as far as I know, Mailscanner already downloads phishing signatures.

@Nicola:
what would be the main difference between your script and the official Mailscanner script: http://phishing.mailscanner.info/

[nicola.piazzi]

My script get the list from https://www.phishtank.com/ that is ONE online service
Script can be adapted to get data also from other services so if you find a valid one can tell me and i create a similar script for it

[pdwalker]

I've nothing to do with the development, but I can tell you that /etc/MailScanner/phishing.bad.sites.conf comes from the MailScanner webserver.

The script run is /usr/sbin/update_bad_phishing_sites and it is set to run every hour in /etc/cron.hourly/update_bad_phishing_sites

[ovizii]

@Nicola:

so what happens after the fiel is downloaded and saved into: /etc/mail/spamassassin/phishtank.cf
How does Mailscanner know about it and why is it being saved into that folder, I mean what is SA meant to do with it? (I assume Mailscanner needs to load it as MS is checking for phishing)

[tbuchanan]

I have been using this phishtank script for a while now. There was a download which contained a "|" and the parsing part of the script inserted another "|" which made an "OR" situation and tagged everything with that rule. I inserted <sed -i -r 's/\|\|/\|/g' /etc/mail/spamassassin/phishtank.cf> into the script to clear that up and it worked again successfully for a period of time. Now, I don't know whether my additional rule has caused another issue but for the last week or so, I've been getting an invalid regex error and I can't seem to find the problem since the string is so long. Is anyone else using this script as is, successfully? Or who might be able to tell me what to look for -- unescaped special characters, etc? Thanks.

[ovizii]

I've nothing to do with the development, but I can tell you that /etc/MailScanner/phishing.bad.sites.conf comes from the MailScanner webserver.

The script run is /usr/sbin/update_bad_phishing_sites and it is set to run every hour in /etc/cron.hourly/update_bad_phishing_sites

I just realized I have neither /etc/cron.hourly/update_bad_phishing_sites nor /usr/sbin/update_bad_phishing_sites on my EFA. Installed from the official appliance version 3.0.1.5

###edit###
turns out the files now have new names, its called: /usr/sbin/ms-update-bad-sites All good!

[ovizii]

@Nicola:

looking at: /etc/MailScanner/phishing.bad.sites.conf it says:

# This file is updated once per day from http://www.phishtank.com.

So I think there's no need for your script, right?

[pdwalker]

turns out the files now have new names, its called: /usr/sbin/ms-update-bad-sites All good!

yes, it appears the cron files were reorganized during one of the mailwatch upgrades.