efa-project.org

Hi Guys,

I just want to share something i implemented on my system with greylisting,

first, i most say that greylisting is great! and you all should use it because spammers, hackers etc.. rarely send the mails 2 times in X intervals
they have no real MTA doing the work for them.. and if they have one and your greylisting is set to higher waiting value,
30 minutes for instance.. and you're using RBLs queries.. you can trust that in those 30 minutes they will be listed and
when they try to come again, their window is closed!

but there is also the delays, greylisting may kill your enemies but it can also injure your allies,
and this may delay your win in battle.
until fully operated database of the dynamic white-listing by SQLGrey we will suffer delays,
the first idea here is to eliminate delays for those we can trust,
and also deny spoofers from getting in.

so, how to achieve this?
pretty simple... SPF!

since SPF becoming more and more usable by companies and others, i thought to myself..
if the sender-domain publish SPF and the IP that send the mail is authorized we know for sure that this
mail is not coming from spam-bot, open relay, etc., but from a legitimate mail system,
so why delay him?

and by saying this, we can have another advantage..
if the sender-domain using restrictive SPF (with the -all signed) it mean that ONLY the IPs/domain
represented in their SPF is allowed to send emails on-behalf of their domain,
so way not deny IPs that try to pretend as they come from that domain at the postfix\greylisting (MTA) level?

with simple DNS query i'm keeping the message out from hitting Greylisting database,
MailScanner rules.. SpamAssassin.. Virus Scan..
saving me processing throughput, CPU, Memory.. etc.

Another technic i implemented was the use of DNSWL.org,
i like to use it as another method to avoid greylisting, but you can disable it if you like.
its rule is that if the client IP is listed in DNSWL for that sender-domain
and it have low\med\high reputation we will allow him to pass greylisting,
if it has "no-trust" reputation we send him to the greylist database.

ok, first download the file and overwrite it,
be sure you are using sqlgrey 1.8.0 because i modified this file,
i'm not related to the developers team so if you're using other version you may want to consider..
check version:
[root@efa ~]# sqlgrey --version
sqlgrey 1.8.0
if it's 1.8.0 we good to go, else.. well you do it at your own risk.

run:

cd /usr/sbin
curl https://transfer.sh/ajZVN/sqlgrey > sqlgrey

now add to /etc/sqlgrey/sqlgrey.conf those lines:

vi /etc/sqlgrey/sqlgrey.conf

spf_autowhitelist = 1
dnswl_autowhitelist = 1
#dnswl_reject_unauthorized = 1

replace sqlgrey-check_policy_service order in /etc/postfix/main.cf
so you don't have those queries on your trusted network IPs,
take "check_policy_service inet:127.0.0.1:2501" out from smtpd_recipient_restrictions
and put it in smtpd_client_restrictions with permit_mynetworks like this:

smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, check_policy_service inet:127.0.0.1:2501

restart the services:
service sqlgrey restart
service postfix restart

you can disable those checks anytime, for now, leave dnswl_reject_unauthorized commented (with '#'),
its not RFC compliance, it means that if the IP is listed on DNSWL for a different
domain it will be rejected at the MTA level, its like an SPF perspective but the IT guys
not ready yet to take DNSWL seriously, so lots of mail servers have relaying servers
delivering the mails for them, if an IP is listed for the sender-domain, sqlgrey will let him pass greylisting..
if the IP is listed for other domain, it still ok but without SPF we greylist the suspect.

ok, the script calculation is this:

SPF Pass = immediately allow him to pass greylisting.
SPF Fail = reject message: "senderdomain.com using restrictive SPF policy and i don't see your name on it!"
SPF None,Neutral,SoftFail - Message will be greylisting.

DNSWL: only if SPF result is None,Neutral,SoftFail DNSWL will make it's query,
DNSWL - Pass - relaying IP matches the domain in DNSWL list and have low\med\high reputation = immediately allow him to pass greylisting.
DNSWL - Fail - relaying IP NOT matches the domain in DNSWL list = dnswl_reject_unauthorized action (greylist default).

now we can make reconnect_delay with high value like 30 or even 60 minutes,
and lowering the awl_age value for like 20 days, making greylisting more efficient..
you can trust that your greylisting database also be empty if you swim on the big pool.

check X-Greylist or /var/log/maillog results to know how sqlgrey deal with the incoming mails..
and by the way, it's highly recommended to run\configure unbound as a recursive and caching DNS that already shipped with the EFA
to minimize DNS queries.

Good luck,
Zohman.

Going to see about pulling this in

https://github.com/E-F-A/v3/issues/213

Zohman,

I've completed the steps you mentioned above, except this one:

zohman wrote:replace sqlgrey-check_policy_service order in /etc/postfix/main.cf
so you don't have those queries on your trusted network IPs,

I can't find anything that matches sqlgrey in this file.

heronimus wrote:I can't find anything that matches sqlgrey in this file.

oh, sorry for the misunderstanding, i meant "check_policy_service inet:127.0.0.1:2501" (this is the sqlgrey line you find)
you wont find sqlgrey-check_policy_service in the file..
just take "check_policy_service inet:127.0.0.1:2501" out from smtpd_recipient_restrictions
and put it in smtpd_client_restrictions with permit_mynetworks.

for example, if you have on main.cf: you change it to:
get the idea?

Zohman.

zohman wrote: you wont find sqlgrey-check_policy_service in the file..
just take "check_policy_service inet:127.0.0.1:2501" out from smtpd_recipient_restrictions
and put it in smtpd_client_restrictions with permit_mynetworks.

OK, i've already taken these steps. I will now restart the services and monitor how it works. Thanks!

heronimus wrote: OK, i've already taken these steps. I will now restart the services and monitor how it works. Thanks!

Great,
Zohman.

One day after i implemented the improvement, i've already 2 complaining people who can't reach us by mail. I examined their SPF record; both their domains were in error (sending on other IP address and SPF signed with -all ). So the rejection of this improvement works very well! To discover the SPF refuses, it would be nice to see them in a log file, or on the webpage... But thats a big next step maybe. Meanwhile i haven't found where that log file resides.

Vice versa, i tried to mail from a domain with a good SPF (google apps), but that message is still greylisted. This part of the improvement seems not to work yet.

Edit : The position of the ~all mechanism was not at the end of the SPF record. This seems to be important.

Regards, Heronimus

shawniverson wrote:
Going to see about pulling this in
https://github.com/E-F-A/v3/issues/213

Does this prevent the improvement from being overwritten by a new release? Or should i backup something in case EFA auto-updates itself?

Regards,
Heronimus

heronimus wrote:One day after i implemented the improvement, i've already 2 complaining people who can't reach us by mail. I examined their SPF record; both their domains were in error (sending on other IP address and SPF signed with -all ). So the rejection of this improvement works very well! To discover the SPF refuses, it would be nice to see them in a log file, or on the webpage... But thats a big next step maybe. Meanwhile i haven't found where that log file resides.

Vice versa, i tried to mail from a domain with a good SPF (google apps), but that message is still greylisted. This part of the improvement seems not to work yet.

Edit : The position of the ~all mechanism was not at the end of the SPF record. This seems to be important.

Regards, Heronimus

Thanks for the feedback

for the logs,
you can see logs on maillog like this:

efa sqlgrey: spf: SPF record:v=spf1 a:mail.externaldomain.com ~all
efa sqlgrey: spf: Received-SPF: pass, 21.21.21.21(mail.externaldomain.com), user@externaldomain.com -> user@example.com

try to find some with this:
cat /var/log/maillog | grep "spf"

about google apps, if some domain using google apps \ office365 services he still need to use SPF on his domain letting us know who is in charge.
i most say that i saw not few IT guys that publish thier SPF imporoperly and this is worse then having SPF at all.

SPF syntex always should end with ~all or -all or +all , if it wont.. SPF result will be PermError,
and that result will make the domain greylisted on our script.

Zohman.

This is excellent work - I've been wondering how to make EFA more pro-active to learning regular traffic! As I am implementing it, however, I am confused by the following from the initial instructions:
Shortly following is a comment:

Should dnswl_reject_unauthorized be commented out as indicated, or not? I'm not completely understanding how the implementation of these checks is working. I still have lots to learn with EFA!

Thanks!

skoppes wrote:
Should dnswl_reject_unauthorized be commented out as indicated, or not? I'm not completely understanding how the implementation of these checks is working. I still have lots to learn with EFA!

Thanks!

well, don't comment out this if you not fully understand the consequences and how DNSWL queries working.
"it means that if the IP is listed on DNSWL for a different domain it will be REJECTED at the MTA level".

Zohman.

Zohman,

Thank you again for the very informative post, and for the attempted clarification. Unfortunately, I'm still confused. I understand 'comment out' to mean any line starting with # in this case. When I read the 'don't comment out this' I am confused, because it is already commented out in your example. I do not wish to reject the messages, but I'm not sure if I am supposed to leave the line commented out as your initial instructions suggest, or if I am supposed to remove the comment #.

I do not wish to reject these messages. I want them to pass to sqlgrey as usual.

My guess is that the following, with no comment #, activates the non RFC-compliant rejection, and is not what I want:
Sorry if I'm coming across as a newbie. I really am trying to learn this, without breaking more stuff along the way.

Thanks again!

skoppes wrote:Zohman,

Thank you again for the very informative post, and for the attempted clarification. Unfortunately, I'm still confused. I understand 'comment out' to mean any line starting with # in this case. When I read the 'don't comment out this' I am confused, because it is already commented out in your example. I do not wish to reject the messages, but I'm not sure if I am supposed to leave the line commented out as your initial instructions suggest, or if I am supposed to remove the comment #.

you are absolotly right!
i wanted to say leave it out from the config file and i said comment-out,
post edited, Thanks, and sorry for the confusing.

Thank you very much for your clarification. I had it configured correctly from the get-go, but ended up with poor results.

Unfortunately, the spammers are using legitimate SPF records now. After enabling this, we were getting slammed with messages coming from spammers with valid SPF records for much of the day. Previously, they would hit the greylist and never come back for the same recipient, so the message never got to us.

After this experience, I'd be very careful setting up bypass-lists for sqlgrey!


Here's a link to what we experienced after setting this up:
viewtopic.php?f=5&t=1258

Edited to add link to post

hi,
the file you provide is not available anymore. can you please upload again?
or can you post your changes?

thank you!

shawniverson wrote: 04 Oct 2015 08:14

Going to see about pulling this in

https://github.com/E-F-A/v3/issues/213

I can't wait for version 3.1.0.0
Meanwhile i lost the Greylist improvement because i reïnstalled EFA