URI DNS blocklist

General eFa discussion
Post Reply
smyers119
Posts: 108
Joined: 29 Nov 2019 11:36

URI DNS blocklist

Post by smyers119 »

Does eFa take advantage of

Code: Select all

uridnsbl_skip_domain
to keep down the number of uribl queries? If not can we add a feature request for this. Maybe even a mailwatch editable list for easy updates.
smyers119
Posts: 108
Joined: 29 Nov 2019 11:36

Re: URI DNS blocklist

Post by smyers119 »

I ended up using the script in this post, It's a better then a static list, works on 4.0.2 with little modification

viewtopic.php?f=14&t=3591&p=14150&hilit=uribl#p14150
jogomes
Posts: 21
Joined: 12 Oct 2016 15:59

Re: URI DNS blocklist

Post by jogomes »

smyers119 wrote: 17 Apr 2020 16:40 I ended up using the script in this post, It's a better then a static list, works on 4.0.2 with little modification

viewtopic.php?f=14&t=3591&p=14150&hilit=uribl#p14150
Hi, would be grateful, if you post the 'litle modification'.

Thank you and stay safe and well.
JG
smyers119
Posts: 108
Joined: 29 Nov 2019 11:36

Re: URI DNS blocklist

Post by smyers119 »

Here's a copy of the working script for eFa 4.0.2

Code: Select all

# uriskip.sh
# ----------
#
#Originial script by forum user:  nicola.piazzi
# This script is useful to create an exclusion file for uri search
# Uri search is very expensive and lots of external uribl services allow limited queries
# This script can be scheduled in cron and refresh a .cf file that exclude most used domains
# this limit dns queries number, this script check rbl before exclude
# Put it into /etc/eFa/ and put it in cron, the script restart mailscanner at the end
# 01 02 * * 0 /etc/eFa/uriskip.sh

# Parameters
ROOTPWD=$(grep MYSQLROOTPWD /etc/eFa/MySQL-Config | sed 's/.*://')
VCF=/etc/mail/spamassassin/uriskip.cf      # Exclusion file fullpath
VDIRECTIVE=uridnsbl_skip_domain            # Eclusion directive
VDOMINLINE=5                               # Domain to exclude on each line of file
VSCANLOGDAYS=2                             # Numbers of days old to scan maillog table headers
VEXCLUDENUM=200                            # Number of domains to exclude, they are sorted by maximum hits
# Uri rbl to check for and left part of return code
VURICHECKS="dbl.spamhaus.org 127.0.1\
            uribl.abuse.ro 127.0.0\
            hostkarma.junkemailfilter.com 127.0.0.2\
            hostkarma.junkemailfilter.com 127.0.0.4\
            mailsl.dnsbl.rjek.com 127.0.0.2\
            urlsl.dnsbl.rjek.com 127.0.0.2\
            fresh.spameatingmonkey.net 127.0.0.2\
            uribl.spfbl.net 127.0.0.2\
            uribl.spfbl.net 127.0.0.3\
            multi.surbl.org 127.0.0\
            uribl.swinog.ch 127.0.0.2\
            rhsbl.zapbl.net 127.0.0.2\
            multi.uribl.com 127.0.0.2\
            multi.uribl.com 127.0.0.4\
            multi.uribl.com 127.0.0.8\
            "
                                           # My domains that allow
VURIFIXED="your.static.domain\
           gmail.com\
           "
VPUTURIFIXED=s                             # Put VURIFIXED (s/n), usually they are included because are found
VRELOADMS=s                                # Restart MailScanner at the end
VRELOADMSCMD="service MailScanner restart" # Restart MailScanner command


# Date & Time
NOW=$(date +"%m-%d-%Y %r")
start=`date +%s.%N`


# Drop table commented (only for testing purpose)
#C="DROP TABLE uriskip ;"
#mysql -N -u root -p$ROOTPWD -D mailscanner -se "$C"


# Create Table uriskip if not exist
C="CREATE TABLE IF NOT EXISTS uriskip (ts DATETIME default null, \
  domain VARCHAR(63), domaincnt INTEGER, domainlisted INTEGER, PRIMARY KEY domain_key (domain) ) ENGINE = MyISAM; "
mysql -N -u root -p$ROOTPWD -D mailscanner -se "$C"


# Delete elements from table uriskip
C="DELETE FROM uriskip; "
mysql -N -u root -p$ROOTPWD -D mailscanner -se "$C"


# Cycle on headers of table mailscanner and insert into table uriskip counting occurrences
c=0
S="SELECT headers FROM maillog WHERE timestamp > DATE_SUB(CURDATE(), INTERVAL $VSCANLOGDAYS DAY); "
mysql -N -u root -p$ROOTPWD -D mailscanner -se "$S"  | ( while read -r headers
do
 ((c++))
 #echo $c
 #echo $headers
 echo $headers | while IFS='\ ;><'  read -ra linearray
 do
  for element in "${linearray[@]}"
  do :
   # Scan all elements
   #echo "> $element"
   # Search domain
   dmone=$(echo $element | grep -E -o "[\/|\.|\@][A-Za-z0-9.-]+\.[A-Za-z]{2,6}")
   # Go on only if have a value
   if [ -n "$dmone" ]; then
    # Clean first char from domain found
    dmone="${dmone:1}"
    # Extract second level domain
    dmtwo=$(echo $dmone | grep -E -o "[a-zA-Z0-9_-]+\.\w+$")
    # Go on only if have a value
    if [ -n "$dmtwo" ]; then
     # Lowercase dmtwo
     dmtwo=${dmtwo,,}
     #echo $dmtwo
     # Insert domain in table uriskip and if exist increment cnt
     S="INSERT INTO uriskip (domain, domaincnt, domainlisted) VALUES ('$dmtwo', 1, 0) ON DUPLICATE KEY UPDATE domaincnt = domaincnt + 1; "
     mysql -N -u root -p$ROOTPWD -D mailscanner -se "$S"
    fi
   fi
  done
 done
done
)


# Load URICHECKS arrays
declare -a vuriserver
declare -a vuricode
x=0;xx=0
for w in $VURICHECKS; do
 if [[ $((x % 2)) -eq 0 ]]; then
   vuriserver[xx]=$w
 else
   vuricode[xx]=$w; ((xx++))
 fi; ((x++))
done; ((xx--))
echo "List of uribl lists to check and part of return code to match"
for y in $(eval echo {0..$xx}); do
 echo $y ${vuriserver[y]}  echo ${vuricode[y]}
done
echo ""


# Cycle on uriskip table to query uribls and update hit flag, we do it only for entries that will be used
c=0;
S="SELECT domain, domainlisted FROM uriskip ORDER BY domaincnt DESC;  "
mysql -N -u root -p$ROOTPWD -D mailscanner -se "$S"  | ( while read -r domain domainlisted
do
 echo "";echo $c "Testing domain $domain"
 wlisted=0;x=0
 # Loop on uribl list to test given return code
 for y in $(eval echo {0..$xx}); do
  #echo $y ${vuriserver[y]}  echo ${vuricode[y]}
  #echo "Testing on $domain.${vuriserver[y]} ${vuricode[y]}"
  r=$(dig +short "$domain.${vuriserver[y]}"| grep ${vuricode[y]})
  if [ ! -z $r ] ; then
   ((wlisted++))
   echo $domain.${vuriserver[y]} $r
  fi
 done
 # Add counter if not listed and excit on VEXCLUDENUM reached
 if [ $wlisted -eq 0 ] ; then ((c++)); fi
 if [ $c -gt $VEXCLUDENUM ] ; then exit 1; fi
 # Insert / update domain in table uriskip
 U="UPDATE uriskip SET domainlisted = $wlisted WHERE domain = '$domain'; "
 mysql -N -u root -p$ROOTPWD -D mailscanner -se "$U"
 #sleep 1
done
)


# Initialize cf file
echo "# Exclusion file for uri dns check" > $VCF
echo "# Generated by $0 on $(date)" >> $VCF
echo "" >> $VCF
chmod 775 $VCF
chown postfix:postfix $VCF


# Cycle on mydomain, if selected
if [ $VPUTURIFIXED == "s" ]; then
 i=0;wrline="$VDIRECTIVE"
 for domain in $VURIFIXED;do
  #echo "$domain"
  ((i++))
  wrline="$wrline $domain"
  if [ $i -ge $VDOMINLINE ] ; then
   echo $wrline >> $VCF
   i=0;wrline="$VDIRECTIVE"
  fi
 done
 if [ $i -gt 0 ] ; then echo $wrline >> $VCF; fi
fi

# Cycle on uriskip and create cf exclusion file for domain with hit flag 0
i=0;wrline="$VDIRECTIVE"
S="SELECT domain FROM uriskip WHERE domainlisted = 0 ORDER BY domaincnt DESC LIMIT $VEXCLUDENUM; "
mysql -N -u root -p$ROOTPWD -D mailscanner -se "$S"  | ( while read -r domain
do
 #echo "$domain"
 ((i++))
 wrline="$wrline $domain"
 if [ $i -ge $VDOMINLINE ] ; then
  echo $wrline >> $VCF
  i=0;wrline="$VDIRECTIVE"
 fi
done
if [ $i -gt 0 ] ; then echo $wrline >> $VCF; fi
)


# Reload mailscanner if selected
if [ $VRELOADMS == "s" ] ; then
 $VRELOADMSCMD
fi


# Logging
end=`date +%s.%N`
runtime="$(bc <<<"$end-$start")"
echo "time elapsed: $runtime sec."
Last edited by smyers119 on 21 Apr 2020 02:39, edited 1 time in total.
jogomes
Posts: 21
Joined: 12 Oct 2016 15:59

Re: URI DNS blocklist

Post by jogomes »

smyers119 wrote: 17 Apr 2020 18:59 Here's a copy of the working script for eFa 4.0.2
Thank you so much.
Cheers.
JG
keysteal
Posts: 20
Joined: 10 Nov 2018 07:25

Re: URI DNS blocklist

Post by keysteal »

But are we sure that this script works?
smyers119
Posts: 108
Joined: 29 Nov 2019 11:36

Re: URI DNS blocklist

Post by smyers119 »

keysteal wrote: 09 Jun 2020 16:59 But are we sure that this script works?
Yes it does. Works great.
Post Reply