SQLGrey - Greylisting Improvements with SPF\DNSWL
SQLGrey - Greylisting Improvements with SPF\DNSWL
Hi Guys,
I just want to share something i implemented on my system with greylisting,
first, i most say that greylisting is great! and you all should use it because spammers, hackers etc.. rarely send the mails 2 times in X intervals
they have no real MTA doing the work for them.. and if they have one and your greylisting is set to higher waiting value,
30 minutes for instance.. and you're using RBLs queries.. you can trust that in those 30 minutes they will be listed and
when they try to come again, their window is closed!
but there is also the delays, greylisting may kill your enemies but it can also injure your allies,
and this may delay your win in battle.
until fully operated database of the dynamic white-listing by SQLGrey we will suffer delays,
the first idea here is to eliminate delays for those we can trust,
and also deny spoofers from getting in.
so, how to achieve this?
pretty simple... SPF!
since SPF becoming more and more usable by companies and others, i thought to myself..
if the sender-domain publish SPF and the IP that send the mail is authorized we know for sure that this
mail is not coming from spam-bot, open relay, etc., but from a legitimate mail system,
so why delay him?
and by saying this, we can have another advantage..
if the sender-domain using restrictive SPF (with the -all signed) it mean that ONLY the IPs/domain
represented in their SPF is allowed to send emails on-behalf of their domain,
so way not deny IPs that try to pretend as they come from that domain at the postfix\greylisting (MTA) level?
with simple DNS query i'm keeping the message out from hitting Greylisting database,
MailScanner rules.. SpamAssassin.. Virus Scan..
saving me processing throughput, CPU, Memory.. etc.
Another technic i implemented was the use of DNSWL.org,
i like to use it as another method to avoid greylisting, but you can disable it if you like.
its rule is that if the client IP is listed in DNSWL for that sender-domain
and it have low\med\high reputation we will allow him to pass greylisting,
if it has "no-trust" reputation we send him to the greylist database.
ok, first download the file and overwrite it,
be sure you are using sqlgrey 1.8.0 because i modified this file,
i'm not related to the developers team so if you're using other version you may want to consider..
check version:
[root@efa ~]# sqlgrey --version
sqlgrey 1.8.0
if it's 1.8.0 we good to go, else.. well you do it at your own risk.
run:
cd /usr/sbin
curl https://transfer.sh/ajZVN/sqlgrey > sqlgrey
now add to /etc/sqlgrey/sqlgrey.conf those lines:
vi /etc/sqlgrey/sqlgrey.conf
spf_autowhitelist = 1
dnswl_autowhitelist = 1
#dnswl_reject_unauthorized = 1
replace sqlgrey-check_policy_service order in /etc/postfix/main.cf
so you don't have those queries on your trusted network IPs,
take "check_policy_service inet:127.0.0.1:2501" out from smtpd_recipient_restrictions
and put it in smtpd_client_restrictions with permit_mynetworks like this:
smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, check_policy_service inet:127.0.0.1:2501
restart the services:
service sqlgrey restart
service postfix restart
you can disable those checks anytime, for now, leave dnswl_reject_unauthorized commented (with '#'),
its not RFC compliance, it means that if the IP is listed on DNSWL for a different
domain it will be rejected at the MTA level, its like an SPF perspective but the IT guys
not ready yet to take DNSWL seriously, so lots of mail servers have relaying servers
delivering the mails for them, if an IP is listed for the sender-domain, sqlgrey will let him pass greylisting..
if the IP is listed for other domain, it still ok but without SPF we greylist the suspect.
ok, the script calculation is this:
SPF Pass = immediately allow him to pass greylisting.
SPF Fail = reject message: "senderdomain.com using restrictive SPF policy and i don't see your name on it!"
SPF None,Neutral,SoftFail - Message will be greylisting.
DNSWL: only if SPF result is None,Neutral,SoftFail DNSWL will make it's query,
DNSWL - Pass - relaying IP matches the domain in DNSWL list and have low\med\high reputation = immediately allow him to pass greylisting.
DNSWL - Fail - relaying IP NOT matches the domain in DNSWL list = dnswl_reject_unauthorized action (greylist default).
now we can make reconnect_delay with high value like 30 or even 60 minutes,
and lowering the awl_age value for like 20 days, making greylisting more efficient..
you can trust that your greylisting database also be empty if you swim on the big pool.
check X-Greylist or /var/log/maillog results to know how sqlgrey deal with the incoming mails..
and by the way, it's highly recommended to run\configure unbound as a recursive and caching DNS that already shipped with the EFA
to minimize DNS queries.
Good luck,
Zohman.
I just want to share something i implemented on my system with greylisting,
first, i most say that greylisting is great! and you all should use it because spammers, hackers etc.. rarely send the mails 2 times in X intervals
they have no real MTA doing the work for them.. and if they have one and your greylisting is set to higher waiting value,
30 minutes for instance.. and you're using RBLs queries.. you can trust that in those 30 minutes they will be listed and
when they try to come again, their window is closed!
but there is also the delays, greylisting may kill your enemies but it can also injure your allies,
and this may delay your win in battle.
until fully operated database of the dynamic white-listing by SQLGrey we will suffer delays,
the first idea here is to eliminate delays for those we can trust,
and also deny spoofers from getting in.
so, how to achieve this?
pretty simple... SPF!
since SPF becoming more and more usable by companies and others, i thought to myself..
if the sender-domain publish SPF and the IP that send the mail is authorized we know for sure that this
mail is not coming from spam-bot, open relay, etc., but from a legitimate mail system,
so why delay him?
and by saying this, we can have another advantage..
if the sender-domain using restrictive SPF (with the -all signed) it mean that ONLY the IPs/domain
represented in their SPF is allowed to send emails on-behalf of their domain,
so way not deny IPs that try to pretend as they come from that domain at the postfix\greylisting (MTA) level?
with simple DNS query i'm keeping the message out from hitting Greylisting database,
MailScanner rules.. SpamAssassin.. Virus Scan..
saving me processing throughput, CPU, Memory.. etc.
Another technic i implemented was the use of DNSWL.org,
i like to use it as another method to avoid greylisting, but you can disable it if you like.
its rule is that if the client IP is listed in DNSWL for that sender-domain
and it have low\med\high reputation we will allow him to pass greylisting,
if it has "no-trust" reputation we send him to the greylist database.
ok, first download the file and overwrite it,
be sure you are using sqlgrey 1.8.0 because i modified this file,
i'm not related to the developers team so if you're using other version you may want to consider..
check version:
[root@efa ~]# sqlgrey --version
sqlgrey 1.8.0
if it's 1.8.0 we good to go, else.. well you do it at your own risk.
run:
cd /usr/sbin
curl https://transfer.sh/ajZVN/sqlgrey > sqlgrey
now add to /etc/sqlgrey/sqlgrey.conf those lines:
vi /etc/sqlgrey/sqlgrey.conf
spf_autowhitelist = 1
dnswl_autowhitelist = 1
#dnswl_reject_unauthorized = 1
replace sqlgrey-check_policy_service order in /etc/postfix/main.cf
so you don't have those queries on your trusted network IPs,
take "check_policy_service inet:127.0.0.1:2501" out from smtpd_recipient_restrictions
and put it in smtpd_client_restrictions with permit_mynetworks like this:
smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, check_policy_service inet:127.0.0.1:2501
restart the services:
service sqlgrey restart
service postfix restart
you can disable those checks anytime, for now, leave dnswl_reject_unauthorized commented (with '#'),
its not RFC compliance, it means that if the IP is listed on DNSWL for a different
domain it will be rejected at the MTA level, its like an SPF perspective but the IT guys
not ready yet to take DNSWL seriously, so lots of mail servers have relaying servers
delivering the mails for them, if an IP is listed for the sender-domain, sqlgrey will let him pass greylisting..
if the IP is listed for other domain, it still ok but without SPF we greylist the suspect.
ok, the script calculation is this:
SPF Pass = immediately allow him to pass greylisting.
SPF Fail = reject message: "senderdomain.com using restrictive SPF policy and i don't see your name on it!"
SPF None,Neutral,SoftFail - Message will be greylisting.
DNSWL: only if SPF result is None,Neutral,SoftFail DNSWL will make it's query,
DNSWL - Pass - relaying IP matches the domain in DNSWL list and have low\med\high reputation = immediately allow him to pass greylisting.
DNSWL - Fail - relaying IP NOT matches the domain in DNSWL list = dnswl_reject_unauthorized action (greylist default).
now we can make reconnect_delay with high value like 30 or even 60 minutes,
and lowering the awl_age value for like 20 days, making greylisting more efficient..
you can trust that your greylisting database also be empty if you swim on the big pool.
check X-Greylist or /var/log/maillog results to know how sqlgrey deal with the incoming mails..
and by the way, it's highly recommended to run\configure unbound as a recursive and caching DNS that already shipped with the EFA
to minimize DNS queries.
Good luck,
Zohman.
Last edited by zohman on 07 Oct 2015 19:01, edited 1 time in total.
- shawniverson
- Posts: 3650
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
Zohman,
I've completed the steps you mentioned above, except this one:
I've completed the steps you mentioned above, except this one:
I can't find anything that matches sqlgrey in this file.zohman wrote:replace sqlgrey-check_policy_service order in /etc/postfix/main.cf
so you don't have those queries on your trusted network IPs,
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
oh, sorry for the misunderstanding, i meant "check_policy_service inet:127.0.0.1:2501" (this is the sqlgrey line you find)heronimus wrote:I can't find anything that matches sqlgrey in this file.
you wont find sqlgrey-check_policy_service in the file..
just take "check_policy_service inet:127.0.0.1:2501" out from smtpd_recipient_restrictions
and put it in smtpd_client_restrictions with permit_mynetworks.
for example, if you have on main.cf:
Code: Select all
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient, reject_unknown_recipient_domain, check_policy_service inet:127.0.0.1:2501
smtpd_client_restrictions = permit_sasl_authenticated
Code: Select all
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient, reject_unknown_recipient_domain
smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, check_policy_service inet:127.0.0.1:2501
Zohman.
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
OK, i've already taken these steps. I will now restart the services and monitor how it works. Thanks!zohman wrote: you wont find sqlgrey-check_policy_service in the file..
just take "check_policy_service inet:127.0.0.1:2501" out from smtpd_recipient_restrictions
and put it in smtpd_client_restrictions with permit_mynetworks.
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
Great,heronimus wrote: OK, i've already taken these steps. I will now restart the services and monitor how it works. Thanks!
Zohman.
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
One day after i implemented the improvement, i've already 2 complaining people who can't reach us by mail. I examined their SPF record; both their domains were in error (sending on other IP address and SPF signed with -all ). So the rejection of this improvement works very well! To discover the SPF refuses, it would be nice to see them in a log file, or on the webpage... But thats a big next step maybe. Meanwhile i haven't found where that log file resides.
Vice versa, i tried to mail from a domain with a good SPF (google apps), but that message is still greylisted. This part of the improvement seems not to work yet.
Edit : The position of the ~all mechanism was not at the end of the SPF record. This seems to be important.
Regards, Heronimus
Vice versa, i tried to mail from a domain with a good SPF (google apps), but that message is still greylisted. This part of the improvement seems not to work yet.
Edit : The position of the ~all mechanism was not at the end of the SPF record. This seems to be important.
Regards, Heronimus
Last edited by heronimus on 14 Oct 2015 06:42, edited 2 times in total.
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
Does this prevent the improvement from being overwritten by a new release? Or should i backup something in case EFA auto-updates itself?
Regards,
Heronimus
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
Thanks for the feedbackheronimus wrote:One day after i implemented the improvement, i've already 2 complaining people who can't reach us by mail. I examined their SPF record; both their domains were in error (sending on other IP address and SPF signed with -all ). So the rejection of this improvement works very well! To discover the SPF refuses, it would be nice to see them in a log file, or on the webpage... But thats a big next step maybe. Meanwhile i haven't found where that log file resides.
Vice versa, i tried to mail from a domain with a good SPF (google apps), but that message is still greylisted. This part of the improvement seems not to work yet.
Edit : The position of the ~all mechanism was not at the end of the SPF record. This seems to be important.
Regards, Heronimus
for the logs,
you can see logs on maillog like this:
efa sqlgrey: spf: SPF record:v=spf1 a:mail.externaldomain.com ~all
efa sqlgrey: spf: Received-SPF: pass, 21.21.21.21(mail.externaldomain.com), user@externaldomain.com -> user@example.com
try to find some with this:
cat /var/log/maillog | grep "spf"
about google apps, if some domain using google apps \ office365 services he still need to use SPF on his domain letting us know who is in charge.
i most say that i saw not few IT guys that publish thier SPF imporoperly and this is worse then having SPF at all.
SPF syntex always should end with ~all or -all or +all , if it wont.. SPF result will be PermError,
and that result will make the domain greylisted on our script.
Zohman.
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
This is excellent work - I've been wondering how to make EFA more pro-active to learning regular traffic! As I am implementing it, however, I am confused by the following from the initial instructions:
Shortly following is a comment:
Should dnswl_reject_unauthorized be commented out as indicated, or not? I'm not completely understanding how the implementation of these checks is working. I still have lots to learn with EFA!
Thanks!
Code: Select all
#dnswl_reject_unauthorized = 1
Code: Select all
but don't comment out dnswl_reject_unauthorized yet
Should dnswl_reject_unauthorized be commented out as indicated, or not? I'm not completely understanding how the implementation of these checks is working. I still have lots to learn with EFA!
Thanks!
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
skoppes wrote:
Should dnswl_reject_unauthorized be commented out as indicated, or not? I'm not completely understanding how the implementation of these checks is working. I still have lots to learn with EFA!
Thanks!
well, don't comment out this if you not fully understand the consequences and how DNSWL queries working.
"it means that if the IP is listed on DNSWL for a different domain it will be REJECTED at the MTA level".
Zohman.
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
Zohman,
Thank you again for the very informative post, and for the attempted clarification. Unfortunately, I'm still confused. I understand 'comment out' to mean any line starting with # in this case. When I read the 'don't comment out this' I am confused, because it is already commented out in your example. I do not wish to reject the messages, but I'm not sure if I am supposed to leave the line commented out as your initial instructions suggest, or if I am supposed to remove the comment #.
I do not wish to reject these messages. I want them to pass to sqlgrey as usual.
My guess is that the following, with no comment #, activates the non RFC-compliant rejection, and is not what I want:
Sorry if I'm coming across as a newbie. I really am trying to learn this, without breaking more stuff along the way.
Thanks again!
Thank you again for the very informative post, and for the attempted clarification. Unfortunately, I'm still confused. I understand 'comment out' to mean any line starting with # in this case. When I read the 'don't comment out this' I am confused, because it is already commented out in your example. I do not wish to reject the messages, but I'm not sure if I am supposed to leave the line commented out as your initial instructions suggest, or if I am supposed to remove the comment #.
I do not wish to reject these messages. I want them to pass to sqlgrey as usual.
My guess is that the following, with no comment #, activates the non RFC-compliant rejection, and is not what I want:
Code: Select all
dnswl_reject_unauthorized = 1
Sorry if I'm coming across as a newbie. I really am trying to learn this, without breaking more stuff along the way.
Thanks again!
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
skoppes wrote:Zohman,
Thank you again for the very informative post, and for the attempted clarification. Unfortunately, I'm still confused. I understand 'comment out' to mean any line starting with # in this case. When I read the 'don't comment out this' I am confused, because it is already commented out in your example. I do not wish to reject the messages, but I'm not sure if I am supposed to leave the line commented out as your initial instructions suggest, or if I am supposed to remove the comment #.
you are absolotly right!
i wanted to say leave it out from the config file and i said comment-out,
post edited, Thanks, and sorry for the confusing.
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
Thank you very much for your clarification. I had it configured correctly from the get-go, but ended up with poor results.
Unfortunately, the spammers are using legitimate SPF records now. After enabling this, we were getting slammed with messages coming from spammers with valid SPF records for much of the day. Previously, they would hit the greylist and never come back for the same recipient, so the message never got to us.
After this experience, I'd be very careful setting up bypass-lists for sqlgrey!
Here's a link to what we experienced after setting this up:
viewtopic.php?f=5&t=1258
Edited to add link to post
Unfortunately, the spammers are using legitimate SPF records now. After enabling this, we were getting slammed with messages coming from spammers with valid SPF records for much of the day. Previously, they would hit the greylist and never come back for the same recipient, so the message never got to us.
After this experience, I'd be very careful setting up bypass-lists for sqlgrey!
Here's a link to what we experienced after setting this up:
viewtopic.php?f=5&t=1258
Edited to add link to post
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
hi,
the file you provide is not available anymore. can you please upload again?
or can you post your changes?
thank you!
the file you provide is not available anymore. can you please upload again?
or can you post your changes?
thank you!
Re: SQLGrey - Greylisting Improvements with SPF\DNSWL
I can't wait for version 3.1.0.0shawniverson wrote: ↑04 Oct 2015 08:14
Going to see about pulling this in
https://github.com/E-F-A/v3/issues/213
Meanwhile i lost the Greylist improvement because i reïnstalled EFA