Custom bad sites Phishtank.com

Questions and answers about how to do stuff
Post Reply
Alleyviper
Posts: 83
Joined: 16 Oct 2018 05:55
Location: Portugal

Custom bad sites Phishtank.com

Post by Alleyviper »

Hi there,

Hourly updated phishing sites from phishtank.com
Here I just added a cron for updating daily

a) Create an account at http://phishtank.com
b) create API Key by giving a APP name
c) This is the Database to be downloaded

Code: Select all

http://data.phishtank.com/data/online-valid.php_serialized

Now we need to populate the following file:

Code: Select all

/etc/Mailscanner/phishing.bad.sites.custom
d) To do so, first create a new cron file at:

Code: Select all

/etc/cron.daily/
Do

Code: Select all

nano /etc/cron.daily/<YourCronFileName.sh>
e) Add the following:

Code: Select all

#!/bin/bash                                                               
##############################################################            
# www.phishtank.com | Phishing Sites                               #                                   
##############################################################            
                                                                          
# Clear Screen                                                            
clear                                                                     
                                                                          
# Get into Mailscanner Folder                                             
cd /etc/MailScanner/                                                      
                                                                          
# Create phishing.bad.sites.custom Template                               
cp phishing.bad.sites.custom phishing.bad.sites.custom.template          
sed -i '/cp phishing.bad.sites.custom phishing.bad.sites.custom.template
                                                                          
# Delete Old phishing.bad.sites.custom and create new file from phishing.bad.sites.custom.template
rm -rf /etc/MailScanner/phishing.bad.sites.custom                         
cp phishing.bad.sites.custom.template phishing.bad.sites.custom

# Download database
wget http://data.phishtank.com/data/<YourApiKey>/online-valid.php_serialized

# Get only urls from database and remove www.phishtank.com urls | Get clean phishtank
cat online-valid.php_serialized | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | sort | uniq > phishtank
sed -i '/www.phishtank.com/d' phishtank

# Delete downloaded database from phishtank.com
rm -rf /etc/MailScanner/online-valid.php_serialized

# Update phishing.bad.sites.custom from phishtank
cat phishtank >> phishing.bad.sites.custom

# Delete phishtank
rm -rf /etc/MailScanner/phishtank

# Reload Mailscanner
/etc/init.d/mailscanner reload

# Cleanup
clear
echo "Finished"
f) Make it executable

Code: Select all

chmod +x /etc/cron.daily/<YourCronFileName.sh>
g) Check if it works:

Code: Select all

./etc/cron.daily/<YourCronFileName.sh>
h) Your phishing.bad.sites.custom should be now populated with urls being used in phishing.

Note: This is a rough script for getting it done

:dance:
thewomble
Posts: 50
Joined: 17 Jan 2017 12:52

Re: Custom bad sites Phishtank.com

Post by thewomble »

Alleyviper ok I have downloaded and it works, thanks.

But how does EFA use this data?
I went to www.phishtank.com and took the url at the top of the list added it into phishing.bad.sites.custom
saved it
reloaded MailScanner

sent myself an email externally with the phishing url

Could not see how it used this data at all from the score.
AWL scored it -9 and Bayes -1.9, only becuase it was on SpamHaus list it scored enough to be blocked.

Code: Select all

-9.34	AWL	Adjusted score from AWL reputation of From: address
-1.90	BAYES_00	Bayes spam probability is 0 to 1%
0.10	DKIM_SIGNED	Message has a DKIM or DK signature, not necessarily valid
-0.10	DKIM_VALID	Message has at least one valid DKIM or DK signature
-0.10	DKIM_VALID_AU	Message has a valid DKIM or DK signature from author's domain
-2.00	DMARC_PASS	DMARC:validation seems valid
-1.00	DMARC_SPF_PASS	DMARC:SPF validation passes
0.00	FREEMAIL_FROM	Sender email is commonly abused enduser mail provider
0.00	HTML_MESSAGE	HTML included in message
2.00	JA_URIBL_AB_SURBL	Contains an URL listed in the AB SURBL blocklist
-0.10	MAILMX_TLS	TLS:Message seemed to have been received by using TLS to Mailmx
3.00	PYZOR_CHECK	Listed in Pyzor (https://pyzor.readthedocs.io/en/latest/)
-0.00	RCVD_IN_DNSWL_NONE	Sender listed at http://www.dnswl.org/, no trust
-0.90	SENDERSCORE_87	SenderScore Reputation 87% (score.senderscore.com)
-1.00	SO_PUB_SNDR_DOMAIN_DKIM_60	Sender's domain DKIM is listed in reputation-sender-60.rbl.scrolloutf1.com
1.00	SO_PUB_URIBL_NS_40	URL's ns address is listed in reputation-ns-40.rbl.scrolloutf1.com
-0.00	SPF_HELO_PASS	SPF: HELO matches SPF record
-0.00	SPF_PASS	SPF: sender matches SPF record
1.25	URIBL_ABUSE_SURBL	Contains an URL listed in the ABUSE SURBL blocklist
2.50	URIBL_DBL_PHISH	Contains a Phishing URL listed in the Spamhaus DBL blocklist
15.00	URIBL_DBL_SPAMHAUS	URI's listed in Spamhaus DBL (dbl.spamhaus.org) DNSBL
1.62	URIBL_SBL	Contains an URL's NS IP listed in the Spamhaus SBL blocklist
0.10	URIBL_SBL_A	Contains URL's A record listed in the Spamhaus SBL blocklist
-1.10	WDKIM_PASS	DKIM:Successful verification. DKIM Signature exists, selector in signing domain exists, and the required headers are unmodified.
0.10	W_DKIMWL_Freemail	DKIMWL.org - Freemail
0.10	W_DKIMWL_Med	DKIMWL.org - Med scoring
henk
Posts: 517
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Custom bad sites Phishtank.com

Post by henk »

I disabled AWL en enabled TxRep. In the new efa4 TxRep is default, since AWL had some issues, just google on it.
I can confirm TxRep works :!:
“We are stuck with technology when what we really want is just stuff that works.” -Douglas Adams
Alleyviper
Posts: 83
Joined: 16 Oct 2018 05:55
Location: Portugal

Re: Custom bad sites Phishtank.com

Post by Alleyviper »

Hi thewomble,

The email will just be blocked. I see no log on EFA yet.

I've sent an e-mail from gmail to a domain protected by EFA and got a blocked sent message at gmail.

Code: Select all

Message blocked
Your message has been blocked by email@domain.tld. See details for ...
The details...

Code: Select all

Message rejected. See https://support.google.com/mail/answer/69585 for more information.
Alleyviper
Posts: 83
Joined: 16 Oct 2018 05:55
Location: Portugal

Re: Custom bad sites Phishtank.com

Post by Alleyviper »

Hi Henk,

Can you provide steps for:

I disabled AWL en enabled TxRep. In the new efa4 TxRep is default, since AWL had some issues, just google on it.
I can confirm TxRep works :!:
It is good for documentation and I'm just not an expert on this :D
henk
Posts: 517
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Custom bad sites Phishtank.com

Post by henk »

s good for documentation
Using eFa forum als documentation :whistle:
Just google why change from AWL to TxRep :shifty: https://wiki.apache.org/spamassassin/AwlWrongWay

Compair your files and make the changes. ( the # must match your config to :!: )
The configs:

Code: Select all

/etc/mail/spamassassin/v310.pre
# AWL - do auto-whitelist checks
#
#loadplugin Mail::SpamAssassin::Plugin::AWL

Code: Select all

/etc/mail/spamassassin/v341.pre
# TxRep - Reputation database that replaces AWL
loadplugin Mail::SpamAssassin::Plugin::TxRep

Code: Select all

/etc/MailScanner/spamassassin.conf
...
# now turned on by default
# use_auto_whitelist 0
#auto_whitelist_factory Mail::SpamAssassin::SQLBasedAddrList
txrep_factory Mail::SpamAssassin::SQLBasedAddrList
txrep_track_messages 0
user_awl_sql_override_username TxRep
user_awl_sql_table txrep
use_txrep 1
#End E.F.A. mods for MySQL

And check the SpamAssassin Rule Hits via Search and Reports after some time. The TXREP score will be visible as TXREP Score normalizing based on sender's reputation
Last edited by henk on 31 Jan 2019 22:42, edited 1 time in total.
“We are stuck with technology when what we really want is just stuff that works.” -Douglas Adams
Alleyviper
Posts: 83
Joined: 16 Oct 2018 05:55
Location: Portugal

Re: Custom bad sites Phishtank.com

Post by Alleyviper »

Hi Henk,

Thanks a lot for starts, I just got half way to it, that is why I've asked for help :)

Just to confirm.. at:

Code: Select all

/etc/MailScanner/spamassassin.conf

Code: Select all

#Begin E.F.A. mods for MySQL                                              
bayes_store_module              Mail::SpamAssassin::BayesStore::SQL       
bayes_sql_dsn                   DBI:mysql:sa_bayes:localhost              
bayes_sql_username              sa_user                                   
bayes_sql_password              <password>         
#auto_whitelist_factory         Mail::SpamAssassin::SQLBasedAddrList      
user_awl_dsn                    DBI:mysql:sa_bayes:localhost              
user_awl_sql_username           sa_user                                   
user_awl_sql_password           <password>           
bayes_sql_override_username     mailwatch                                 
txrep_factory                   Mail::SpamAssassin::SQLBasedAddrList      
txrep_track_messages            0                                         
user_awl_sql_override_username  TxRep                                     
user_awl_sql_table              txrep                                     
use_txrep                       1                                         
#End E.F.A. mods for MySQL
Is this ok?

Best regards.
Alleyviper
Posts: 83
Joined: 16 Oct 2018 05:55
Location: Portugal

Re: Custom bad sites Phishtank.com

Post by Alleyviper »

Hi Henk,

We have a bypass situation here. I've sent an URL from phishtank and it was correctly blocked.

Notice: This message is shown at gmail and no info on EFA.

Code: Select all

Message blocked Your message has been blocked by email@domain.tld. See the technical details below for more information. KNOW MORE
Nevertheless, if I use the same URL encoded in a Google short link the message passes clean.

I Will Research if there is a SpamAssassin plugin to check the url final destination and block if there is a match.

Do you know any existing plugin?

Here goes the scoring on the bad link Google shortned:

Code: Select all

:	
Score	Matching Rule	Description	 
4	requerido	 
-1.90	BAYES_00	Bayes spam probability is 0 to 1%
0.10	DKIM_SIGNED	Message has a DKIM or DK signature, not necessarily valid
-0.10	DKIM_VALID	Message has at least one valid DKIM or DK signature
-0.10	DKIM_VALID_AU	Message has a valid DKIM or DK signature from author's domain
0.00	FREEMAIL_FROM	Sender email is commonly abused enduser mail provider
0.00	HTML_MESSAGE	HTML included in message
0.00	KAM_SHORT	Use of a URL Shortener for very short URL
-0.00	RCVD_IN_DNSWL_NONE	Sender listed at http://www.dnswl.org/, no trust
The relevant Score is this:

Code: Select all

0.00	KAM_SHORT	Use of a URL Shortener for very short URL
Best regards.
benscha
Posts: 19
Joined: 23 Jan 2018 07:19

Re: Custom bad sites Phishtank.com

Post by benscha »

Hi Guys

i have extended the Script with some Blacklists.
  • ransomwaretracker.abuse.ch
  • malwaredomains.com

Code: Select all

#!/bin/bash                                                               
##############################################################            
# www.phishtank.com | Phishing Sites                               #                                   
##############################################################            
                                                                          
# Clear Screen                                                            
clear                                                                     
                                                                          
# Get into Mailscanner Folder                                             
cd /etc/MailScanner/                                                      
                                                                          
# Create phishing.bad.sites.custom Template                               
\cp -fR phishing.bad.sites.custom phishing.bad.sites.custom.template          
sed -i '/cp phishing.bad.sites.custom phishing.bad.sites.custom.template
                                                                          
# Delete Old phishing.bad.sites.custom and create new file from phishing.bad.sites.custom.template
rm -rf /etc/MailScanner/phishing.bad.sites.custom                         
cp phishing.bad.sites.custom.template phishing.bad.sites.custom

# Download BAD URLS
wget http://data.phishtank.com/data/3e71373433fe683add611bf2e3fd2efa34265c134c4b7bed2b5ab9e998af4855/online-valid.php_serialized -O /tmp/phishtank.txt
wget https://www.openphish.com/feed.txt -O /tmp/feed.txt
wget https://ransomwaretracker.abuse.ch/downloads/RW_URLBL.txt -O /tmp/rw_urlbl.txt
wget https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt -O /tmp/rw_dombl.txt
wget http://mirror2.malwaredomains.com/files/immortal_domains.txt -O /tmp/immortal_domains.txt


# Remove the first 7 Lines drom Blacklist Files
sed -i 1,7d /tmp/rw_urlbl.txt
sed -i 1,7d /tmp/rw_dombl.txt

# Get only urls from database and remove www.phishtank.com urls | Get clean phishtank
cat /tmp/phishtank.txt | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | sort | uniq > /tmp/phishtank
sed -i '/www.phishtank.com/d /tmp/phishtank

# Delete downloaded database from phishtank.com
rm -rf /tmp/phishtank.txt

# Update phishing.bad.sites.custom from phishtank
cat /tmp/phishtank >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/feed.txt >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/rw_urlbl.txt >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/rw_dombl.txt >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/immortal_domains.txt >> /etc/MailScanner/phishing.bad.sites.custom

# Delete phishtank
rm -rf /tmp/phishtank
rm -rf /etc/MailScanner/phishtank
rm -rf /tmp/feed.txt
rm -rf /tmp/rw_urlbl.txt
rm -rf /tmp/rw_dombl.txt
rm -rf /tmp/immortal_domains.txt


# Load Custom Bad URLS
cat /scripts/bad_mail_urls >>/etc/MailScanner/phishing.bad.sites.custom

# Reload Mailscanner
/etc/init.d/mailscanner reload


wc -l /etc/MailScanner/phishing.bad.sites.custom | mail -s "Phishing Sites" report.recipient@domain.com 
# Cleanup
clear
echo "Finished"
:violin: cheers
Last edited by benscha on 05 Mar 2019 07:40, edited 1 time in total.
always happy for any hints and tipps! :clap: | EFA 3.0.2.6
Alleyviper
Posts: 83
Joined: 16 Oct 2018 05:55
Location: Portugal

Re: Custom bad sites Phishtank.com

Post by Alleyviper »

Nice ;)
ovizii
Posts: 463
Joined: 11 May 2016 08:08

Re: Custom bad sites Phishtank.com

Post by ovizii »

I do think adding phishtank is unnecessary as after reading this old thread it seems it is or was already incorporated: viewtopic.php?f=14&t=1736

/etc/MailScanner/phishing.bad.sites.conf USED to include this line:
# This file is updated once per day from http://www.phishtank.com.
that line has disappeared so I guess we need to figure out if it still gets updated from phishtank...
ovizii
Posts: 463
Joined: 11 May 2016 08:08

Re: Custom bad sites Phishtank.com

Post by ovizii »

@alleyviper: you should add this to the github issue tracker as a feature suggestion for EFA v4 though => https://github.com/E-F-A/v4/issues
Woger
Posts: 67
Joined: 15 Mar 2017 10:54

Re: Custom bad sites Phishtank.com

Post by Woger »

Hi,
Thanks for the script but it gives an error on this line:
sed -i '/cp phishing.bad.sites.custom phishing.bad.sites.custom.template

because of the single '

But I can't figure out where the closing ' should be.

The phishing.bad.sites.custom seems ok.

Also, it's not working. And I think it's because spamassassin is not loading .custom files. I have also a phishing.bad.sites.conf which is generated by EFA itself.

Greetings,
Roger
cdburgess75
Posts: 49
Joined: 11 Jun 2014 21:43

Re: Custom bad sites Phishtank.com

Post by cdburgess75 »

Updated Script: (works for me)

#!/bin/bash
##############################################################
# Phishing Sites #
##############################################################

# Clear Screen
clear

# Get into Mailscanner Folder
cd /etc/MailScanner/

# Create phishing.bad.sites.custom Template
\cp -fR phishing.bad.sites.custom phishing.bad.sites.custom.template
#sed -i '/
cp phishing.bad.sites.custom phishing.bad.sites.custom.template

# Delete Old phishing.bad.sites.custom and create new file from phishing.bad.sites.custom.template
rm -rf /etc/MailScanner/phishing.bad.sites.custom
cp phishing.bad.sites.custom.template phishing.bad.sites.custom

# Download BAD URLS
wget http://data.phishtank.com/data/3e713734 ... serialized -O /tmp/phishtank.txt
wget https://www.openphish.com/feed.txt -O /tmp/feed.txt
wget https://ransomwaretracker.abuse.ch/down ... _URLBL.txt -O /tmp/rw_urlbl.txt
wget https://ransomwaretracker.abuse.ch/down ... _DOMBL.txt -O /tmp/rw_dombl.txt
#wget http://mirror2.malwaredomains.com/files ... omains.txt -O /tmp/immortal_domains.txt


# Remove the first 7 Lines drom Blacklist Files
sed -i 1,7d /tmp/rw_urlbl.txt
sed -i 1,7d /tmp/rw_dombl.txt

# Get only urls from database and remove www.phishtank.com urls | Get clean phishtank
cat /tmp/phishtank.txt | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | sort | uniq > /tmp/phishtank
sed -i '/www.phishtank.com/d' /tmp/phishtank

# Delete downloaded database from phishtank.com
rm -rf /tmp/phishtank.txt

# Update phishing.bad.sites.custom from phishtank
cat /tmp/phishtank >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/feed.txt >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/rw_urlbl.txt >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/rw_dombl.txt >> /etc/MailScanner/phishing.bad.sites.custom
#cat /tmp/immortal_domains.txt >> /etc/MailScanner/phishing.bad.sites.custom

rm -rf /tmp/phishtank.txt

# Update phishing.bad.sites.custom from phishtank
cat /tmp/phishtank >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/feed.txt >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/rw_urlbl.txt >> /etc/MailScanner/phishing.bad.sites.custom
cat /tmp/rw_dombl.txt >> /etc/MailScanner/phishing.bad.sites.custom
#cat /tmp/immortal_domains.txt >> /etc/MailScanner/phishing.bad.sites.custom

# Delete phishtank
rm -rf /tmp/phishtank
rm -rf /etc/MailScanner/phishtank
rm -rf /tmp/feed.txt
rm -rf /tmp/rw_urlbl.txt
rm -rf /tmp/rw_dombl.txt
rm -rf /tmp/immortal_domains.txt


# Load Custom Bad URLS
cat /scripts/bad_mail_urls >>/etc/MailScanner/phishing.bad.sites.custom

# Reload Mailscanner
/etc/init.d/mailscanner reload


wc -l /etc/MailScanner/phishing.bad.sites.custom | mail -s "Phishing Sites" email@yourself.com




rm -rf /etc/MailScanner/phishing.bad.sites.conf

sed -e 's/http[s]\?:\/\///g' /etc/MailScanner/phishing.bad.sites.custom > /etc/MailScanner/phishing.bad.sites.conf


echo "Finished"
Post Reply