Uribl Blocked also with dns recursive

Report bugs and workarounds
Post Reply
nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Uribl Blocked also with dns recursive

Post by nicola.piazzi » 09 Apr 2019 15:14

0.00 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/Dns ... nsbl-block for more information.

Do you have an idea of this problem ?

henk
Posts: 395
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Uribl Blocked also with dns recursive

Post by henk » 09 Apr 2019 16:16

Hi nicola,

as you did not provide any info on your config, I can only assume your dns server is not using a public address.
viewtopic.php?t=2565
viewtopic.php?t=934
viewtopic.php?t=1820

On top of that, a working dns config is essential. It will lower your processing time for each message( since a lot of dns requests are needed)
viewtopic.php?t=2567

To use the efa unbound recursive dns, check your /etc/resolv.conf

Code: Select all

cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search example.lan.
nameserver 127.0.0.1

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 09 Apr 2019 16:19

Hi henk
As i wrote i configured recursive dns and my resolv.conf is nameserver 127.0.0.1

henk
Posts: 395
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Uribl Blocked also with dns recursive

Post by henk » 09 Apr 2019 16:34

You did read the mentioned posts in 3 minutes?

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 09 Apr 2019 16:35

no, sorry, Tomorrow i read, i checked only dns

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 10 Apr 2019 10:23

Hi
195.120.124.41 is server IP itself, sigh

[root@EFA41 spamassassin]# dig test.uribl.com.multi.uribl.com txt +short
"127.0.0.1 -> Query Refused. See http://uribl.com/refused.shtml for more information [Your DNS IP: 195.120.124.41]"

henk
Posts: 395
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Uribl Blocked also with dns recursive

Post by henk » 10 Apr 2019 14:35

1. Check against your dns server (dig dns.gruppocomet.it)

Code: Select all

host -t TXT 2.0.0.127.multi.uribl.com 195.120.124.2
2. Change forwarder to your dns server

Code: Select all

/etc/unbound/conf.d/forwarders.conf
forward-zone:
name: "."
forward-addr: 195.120.124.2
forward-first: yes

3. restart unbound

Code: Select all

service unbound restart
4. Check

Code: Select all

unbound-control list_forwards
5 final check

Code: Select all

host -t TXT 2.0.0.127.multi.uribl.com 127.0.0.1
6. Check caching

Code: Select all

unbound-control stats_noreset |grep total
Let me know the result ;)

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 10 Apr 2019 15:26

Using my dns go ok
one thing i dont understand
why efa ip have done too much query to uribl ?
now i expect that also my dns IP will be considered too high query

henk
Posts: 395
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Uribl Blocked also with dns recursive

Post by henk » 10 Apr 2019 17:41

do you use more than 100K queries a day ? viewtopic.php?f=5&t=934&p=11365#p11365

point 6 Check caching, will give some ideas about the number of efa-unbound cached queries.

rooter_c
Posts: 13
Joined: 05 Mar 2013 04:52

Re: Uribl Blocked also with dns recursive

Post by rooter_c » 11 Apr 2019 02:23

Hi. Whilst on this topic, I think I have my Unbound DNS set up correctly, i.e. recursive, no forwarding, but some domains are not being resolved. I get a "No servers could be reached" error when using dig commands. The common factor so far seems to be domains that use Office365 e.g xxx.protection.outlook.com in their MX record. Any clues? I have discovered I can set up forwarders for those domains with unbound-control forward_add but don't really want to have to manage this manually.

Thanks

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 12 Apr 2019 14:24

hi henk

now the opposite

with forward-addr: to my dns ceased to work :
2.0.0.127.multi.uribl.com descriptive text "127.0.0.1 -> Query Refused. See http://uribl.com/refused.shtml for more information [Your DNS IP: 195.120.124.2]

removed forward works now :
2.0.0.127.multi.uribl.com descriptive text "permanent testpoint"

henk
Posts: 395
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Uribl Blocked also with dns recursive

Post by henk » 12 Apr 2019 22:38

Quite sure the uribl will happen again soon on your efa public IP ...

how many mails ( dns queries) a day? ( as mentioned the 100 k limit)

Do you use efa only inbound?

How many scanners active (clamav / sophos / sophosavi / ...)

Also the number of dnbl lists that are configured will have an impact on this number.

So check your postfix main.cf rbl's and your spamassassin rbl's.

Each defined rblcheck will add a number of dns queries for each mail to be scanned.

Is recursion working as you think it works? Check the unbound cache stats (unbound-control stats_noreset |grep total)

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 17 Apr 2019 10:26

*** Quite sure the uribl will happen again soon on your efa public IP ...
Yes, this is my EFA IP because now is recursive without forwarders
[root@EFA41 ~]# host -tTXT 2.0.0.127.multi.uribl.com
2.0.0.127.multi.uribl.com descriptive text "127.0.0.1 -> Query Refused. See http://uribl.com/refused.shtml for more information [Your DNS IP: 195.120.124.41]

*** how many mails ( dns queries) a day? ( as mentioned the 100 k limit)
15 k mails inbound & outbound

*** Do you use efa only inbound?
No, efa receive my Exchange email but i think that with this directive it doesnt ask for outbound
internal_networks 10.1.0.0/16
trusted_networks 10.1.0.0/16

*** How many scanners active (clamav / sophos / sophosavi / ...)
clamd + sophossavi, but they query uribl ?

Also the number of dnbl lists that are configured will have an impact on this number.
*** But this error seems to be from uribl.com, what matter to uribl.com if i query another list ?

*** So check your postfix main.cf rbl's and your spamassassin rbl's.
I have a large number of rbl (http://multirbl.valli.org)

*** Each defined rblcheck will add a number of dns queries for each mail to be scanned.
Why if the message arrive from uribl.com, how is possible that they count other queryes ?

*** Is recursion working as you think it works? Check the unbound cache stats (unbound-control stats_noreset |grep total)
[root@EFA41 ~]# unbound-control stats_noreset |grep total
total.num.queries=215865
total.num.cachehits=80976
total.num.cachemiss=134889
total.num.prefetch=2307
total.num.recursivereplies=134889
total.requestlist.avg=30.1142
total.requestlist.max=287
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=1
total.requestlist.current.user=0
total.recursion.time.avg=0.305242
total.recursion.time.median=0.233437
mem.total.sbrk=18239488

henk
Posts: 395
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Uribl Blocked also with dns recursive

Post by henk » 17 Apr 2019 20:07

Before you can take action, you need to know the reason why you receive the URIBL_BLOCKED message.As there are many members processing huge amounts of mail, it could help others with the same issue.

***Just forgot to ask, do you use ipv6?

*** Do you use efa only inbound?
internal_networks and trusted_networks aren't there to avoid scanning (although there is an ALL_TRUSTED rule with a zero score), they are
there to help SA determine which relays are relevant to certain tests.

To be sure, take a look at 1 Inbound and 1 outbound message and see what checks are done, just for spamassassin.

Code: Select all

spamassassin 2>&1 -D -t msg | grep untrusted | less
You could also debug posfix if the above check reveals no clue.
Enable verbose logging: in /etc/postfix/master.cf

Code: Select all

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       -       -       -       smtpd -v

Code: Select all

service postfix restart

tail -f /var/log/mail.log
*** How many scanners
You are right, no impact on the uribl reject message. But as you already noticed, this will have an big impact on total scanning time per message. Due the heavy cpu load of sophos, you succesfully configured saphosavi.

** Also the number of dnbl lists
looking at the unbound stats, there is no way to see the multi.uribl.com queries only. The total shown is since the last start/reset of unbound for all queries.

*** So check your postfix main.cf rbl's and your spamassassin rbl's.
Just take a look at the following massive rblbl example. You can have multiple checks on a single rbl list ( like grey or black) aka 2 queries per message.

Postfix

smtpd_recipient_restrictions =
reject_invalid_hostname,
reject_unknown_recipient_domain,
reject_unauth_pipelining,
permit_mynetworks,
permit_sasl_authenticated,
reject_non_fqdn_recipient,
reject_unauth_destination,
reject_unverified_recipient,
reject_non_fqdn_sender,
reject_unknown_sender_domain,
reject_rbl_client access.redhawk.org,
reject_rbl_client all.spamrats.com,
reject_rbl_client b.barracudacentral.org,
reject_rbl_client bl.spamcannibal.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client blackholes.mail-abuse.org,
reject_rbl_client bogons.cymru.com,
reject_rbl_client cbl.abuseat.org,
reject_rbl_client cblless.anti-spam.org.cn,
reject_rbl_client combined.njabl.org,
reject_rbl_client csi.cloudmark.com,
reject_rbl_client db.wpbl.info,
reject_rbl_client dnsbl.dronebl.org,
reject_rbl_client dnsbl.inps.de,
reject_rbl_client dnsbl.njabl.org,
reject_rbl_client dnsbl.sorbs.net,
reject_rbl_client drone.abuse.ch,
reject_rbl_client dsn.rfc-ignorant.org,
reject_rbl_client httpbl.abuse.ch,
reject_rbl_client ix.dnsbl.manitu.net,
reject_rbl_client korea.services.net,
reject_rbl_client multi.surbl.org,
reject_rbl_client netblock.pedantic.org,
reject_rbl_client opm.tornevall.org,
reject_rbl_client pbl.spamhaus.org,
reject_rbl_client psbl.surriel.com,
reject_rbl_client query.senderbase.org,
reject_rbl_client rbl.efnetrbl.org,
reject_rbl_client rbl.interserver.net,
reject_rbl_client rbl.rbldns.ru,
reject_rbl_client rbl.spamlab.com,
reject_rbl_client rbl.suresupport.com,
reject_rbl_client rbl-plus.mail-abuse.org,
reject_rbl_client relays.mail-abuse.org,
reject_rbl_client sbl.spamhaus.org,
reject_rbl_client short.rbl.jp,
reject_rbl_client spam.dnsbl.sorbs.net,
reject_rbl_client spamguard.leadmon.net,
reject_rbl_client spamrbl.imp.ch,
reject_rbl_client tor.dan.me.uk,
reject_rbl_client ubl.unsubscore.com,
reject_rbl_client virbl.bit.nl,
reject_rbl_client virus.rbl.jp,
reject_rbl_client wormrbl.imp.ch,
reject_rbl_clientbl.spamhaus.org,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client dnsbl-1.uceprotect.net,
reject_rhsbl_sender dbl.spamhaus.org,
reject_rhsbl_helo dbl.spamhaus.org,
reject_rhsbl_client dbl.spamhaus.org,
reject_rhsbl_helo black.uribl.com,
reject_rhsbl_sender black.uribl.com,
reject_rhsbl_client black.uribl.com,
reject_rhsbl_helo multi.surbl.org,
reject_rhsbl_sender multi.surbl.org,
reject_rhsbl_client multi.surbl.org,
reject_rhsbl_helo multi.uribl.com,
reject_rhsbl_sender multi.uribl.com,
reject_rhsbl_client multi.uribl.com,
reject_rhsbl_helo rhsbl.ahbl.org,
reject_rhsbl_sender rhsbl.ahbl.org,
reject_rhsbl_client rhsbl.ahbl.org
check_sender_access hash:/etc/postfix/sender_access,
check_sender_access hash:/etc/postfix/whitelist,
check_client_access hash:/etc/postfix/rbl_override, permit

SpamAssassin

# Custom Rules
urirhssub URIBL_BLACK multi.uribl.com. A 2
body URIBL_BLACK eval:check_uridnsbl('URIBL_BLACK')
describe URIBL_BLACK Contains an URL listed in the URIBL blacklist
tflags URIBL_BLACK net
score URIBL_BLACK 3.0

urirhssub URIBL_GREY multi.uribl.com. A 4
body URIBL_GREY eval:check_uridnsbl('URIBL_GREY')
describe URIBL_GREY Contains an URL listed in the URIBL greylist
tflags URIBL_GREY net
score URIBL_GREY 0.25

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 18 Apr 2019 08:12

Now i made these modifications :
vi /etc/unbound/unbound.conf
# msg-cache-size: 4m
msg-cache-size: 64m
# rrset-cache-size: 4m
rrset-cache-size: 64m
# cache-min-ttl: 0
# cache-min-ttl: 0
cache-min-ttl: 7200
# cache-max-ttl: 86400
cache-max-ttl: 172800
# infra-host-ttl: 900
infra-host-ttl: 1800
# infra-cache-numhosts: 10000
infra-cache-numhosts: 20000
And I hope to make more hits

now are in about 1 hour
total.num.queries=195262
total.num.cachehits=106532
total.num.cachemiss=88730

2 problems :

(1)
seems that also internal messages query rbl & uribl also if received from trusted network, do you know how to exclude ?

(2)
i think that uribl do more quesry for each message that have lot of urls in it
can be useful to use lastexternal for rbl s

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 18 Apr 2019 10:23

command to see hit %
[root@EFA41 batch]# date;echo "100/$(unbound-control stats_noreset |grep total.num.queries| sed 's/.*=//')*$(unbound-control stats_noreset |grep total.num.cachehits| sed 's/.*=//')" | bc -l
Thu Apr 18 12:20:41 CEST 2019
55.07517104789340933180

55% after 1 hour of run, now i configured minimum TTL of 8 hours so i expect that i have a stable value after 7 hours from now and it mantain about that value forever

henk
Posts: 395
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Uribl Blocked also with dns recursive

Post by henk » 18 Apr 2019 16:53

***Do you use ipv6?

problem 1
"You should also consider adding uri skips on your company domains. Especially if your mail clients append footers with your company url in each email. Because our DNS cache TTL is so low, each email containing your company domain could generate one or more queries.

uridnsbl_skip_domain mydomain.com mydomain.net mydomain.org and take a look at ALL_TRUSTED rule

problem2: you could remove all rbl lists, and then add them 1 by one to see the impact of each rbl added. :whistle:

About changing unbound ttl values: viewtopic.php?t=3143

nicola.piazzi
Posts: 283
Joined: 23 Apr 2015 09:45

Re: Uribl Blocked also with dns recursive

Post by nicola.piazzi » 19 Apr 2019 16:31

Now it seems solved ,
some hour results :-)
thread0.num.queries=251792
thread0.num.cachehits=178785
thread0.num.cachemiss=73007

Post Reply