Regex, attachments and Spam

Questions and answers about how to do stuff
Post Reply
Rod-IT
Posts: 11
Joined: 11 Sep 2018 19:37

Regex, attachments and Spam

Post by Rod-IT »

Hi All,
my first post, but I have read many topics on here already.

I will break my questions down as there is 3 parts.

The easier of them I believe is this one;

If an attachment gets blocked, for example a password protected PDF within a zip, even with the release button enabled, the email is never released from quarantine - any pointers on why this is the case or how to make it release quarantined files when we specify them to be released? Side note, if we whitelist the sender address, would this also allow blocked attachments to come through?

Part 2 - I want to create a regular expression that if a specific subject is matched (outbound only), then a filter is applied, in our case an SMTP redirect to another gateway to apply encryption.

The regex I have is \b\w*abc encrypt\w*\b /gi (where 'abc encrypt' is the subject we want to match - case not relevant, nor should location within the subject be), however this does not seem to work, both Postfix and Mailscanner have been restarted since adding the rules, I have checked them via webmin and SSH and they seem good - online regex sites clarify this works, but it does not when sent via email.


abc encrypt this is a test - this will re-route

this is a test abc encrypt - does not trigger, it seems to only look if the keyword(s) are at the start of the subject.


Third - i have read many posts about spam and read that geo-blocking is not a good idea, it works but we should look at tuning the rules, the majority of what spam we do get seems to be Invoice or Payment related DOC files which then open a link within, i am getting to the point where DOC files may get blocked to combat this in a quick way, but are there any other tips for how to ensure this is tuned - i am forever marking as spam and the SA does seem to learn them, but we can get bombarded with quite a lot of these lately.

I also read where this could be related to DNS and RLBs, is there any way to check?

Just to add, i am no linux expert, i can find my way around and adjust settings etc. but may need a little extra guidance on specifics.


Thanks in advance for your help, guidance, pointers, and a great product.
Rod-IT
Posts: 11
Joined: 11 Sep 2018 19:37

Re: Regex, attachments and Spam

Post by Rod-IT »

No one has any ideas on any of the issues above?
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: Regex, attachments and Spam

Post by pdwalker »

Part 1: When you release a blocked message, do you see it in the message list? Does it show as blocked again? If so, is your locahost whitelisted?

Part 2: conditional sending based on subject to different gateways? I don't know if that is even possible. Where did you put your regex, expecting it to work?22
Rod-IT
Posts: 11
Joined: 11 Sep 2018 19:37

Re: Regex, attachments and Spam

Post by Rod-IT »

Apologies, I did not get notified of an update on this.

1. This has been resolved on another post by a colleague of mine.
2. This is currently working, but only if the content is the first words in the subject, I will try to get a screenshot if you want one, but this is configured in Webmin, it has a route to SMTP:IPofotherserver (I'm not in front of the screen at the moment)


That said, we are still getting a lot of spam, any tips?


Using Spamhaus and others seems to block legitimate emails, and manual checks suggest the IPs are not on the blocked lists, so I dont understand why they flag as they are.
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: Regex, attachments and Spam

Post by pdwalker »

1/ How did you resolve the problem? (Just in case someone else asks in the future)

2/ In order to work out why a message has been flagged as spam, you'll need to look at the spam report in the message details.

For example, this message is "maybe spam":
Screen Shot 2018-09-26 at 16.11.10.png
Screen Shot 2018-09-26 at 16.11.10.png (98.89 KiB) Viewed 7701 times
If I decided this message was spam/not spam, then I would flag it in spamassassin so the bayes filter would be updated (right now it shows at 20-40% which is neutral.

In this example, none of the other rules are worth me changing. (ignore the 3.0 score - that's a bonus added when another spam detection thinks the message is spam)

As you look at the scores - you can then decide what is normal, or not normal - and perhaps customize your scores appropriately. For example, I really boost the spam score when bayes says it is 99% likely spam.

Could you post a screenshot of the spam score for a message you think shouldn't be spam? Perhaps one of us may have further suggestions.
Rod-IT
Posts: 11
Joined: 11 Sep 2018 19:37

Re: Regex, attachments and Spam

Post by Rod-IT »

Apologies, I am not being notified on updates to this topic.

1. I only know at this point it was something in the php code that allowed for the files to be released (viewtopic.php?f=14&t=3174)

2. The spam we are getting can be in the - values or very low scoring, meaning they are legitimate, enabling RBLs seems to stop some of these, but also a lot of legitimate stuff. (I will try and get you some of the content you want though if I remember tomorrow to login and grab it (I'm at home now)).

FWIW we are marking spam and non-spam in the bayes filter, quite often in bulk.


I've checked my settings for notifications and enabled being notified again so hopefully I will see updates now, and will get the screenshots and examples as soon as I can - thanks for the help thus far.
Rod-IT
Posts: 11
Joined: 11 Sep 2018 19:37

Re: Regex, attachments and Spam

Post by Rod-IT »

Side point.

A lot of newsletter type stuff and community/forum emails seem to get stopped by Spamhaus (spamcop did the same), but if I lookup the IPs within a short period of time they show as not blocked, they are not on the list.


The below screenshot is for an email I should have had from another forum, which came in to me about 5 minutes ago, got stopped and is not showing as on the Spamhaus lists

Efa Spamhaus.JPG
Efa Spamhaus.JPG (26.1 KiB) Viewed 7662 times
Efa IP.JPG
Efa IP.JPG (21.77 KiB) Viewed 7662 times

Emails from MailChimp also tend to be stopped for the same reason and while I can release them and learn them as ham - they are already not hitting the spam score, they get stopped by and RBL filter.


Given they are not stopped for being spam, I am doubtful marking them as ham is doing anything to help.
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Regex, attachments and Spam

Post by shawniverson »

Are you using full recursion for your dns?

It seems that something is interfering with the rbl checks and triggering false positives.
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: Regex, attachments and Spam

Post by pdwalker »

Also, I'd still like to see a full spam report.

I also use the same block lists (with my own recursing DNS on EFA) and I don't get any problem from the mailing lists that come in.
Rod-IT
Posts: 11
Joined: 11 Sep 2018 19:37

Re: Regex, attachments and Spam

Post by Rod-IT »

I will check the DNS recursion setting on the work system tomorrow, and I will get the content for the spam too.

The box in my lab I installed, but the one at work I did not.

FWIW the notification for these updates was also stopped, claiming to be on the Spamhaus RBL list, though at home i am not using recursion, though i am happy to enable it if you believe this will help.
elfranko
Posts: 19
Joined: 03 Sep 2018 08:03

Re: Regex, attachments and Spam

Post by elfranko »

I am the installer of the box, and I can confirm Recursion is enabled.

Code: Select all

 cat /etc/EFA-Config
 returns
 RECURSION:ENABLED
Here is a set of headers from an email:

Code: Select all

X-Greylist: from auto-whitelisted by SQLgrey-1.8.0
Received: from h2701864.stratoserver.net (unknown [85.214.33.48])
     (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
     (No client certificate requested)
     by our-EFA (Postfix) with ESMTPS id 8A6F626E78
     for <aaaa.aaaaaaaaaaaa@bbbbbb.bbb.uk>; Wed, 3 Oct 2018 14:16:37 +0100 (BST)
Received: from sprynet.com (ec2-52-33-197-150.us-west-2.compute.amazonaws.com [52.33.197.150])
     by h2701864.stratoserver.net (Postfix) with ESMTPSA id EE90B66EDA00;
     Wed, 3 Oct 2018 14:14:05 +0200 (CEST)
Message-ID: <4F4AFCAFA4705E9D41440DB2669F6EEE@doomsdaywarriors.eu>
Reply-To: "Marital Dating" <dating@eslostenn.de>
From: "Marital Dating" <fls-atlan@doomsdaywarriors.eu>
Subject: Welcome to Private dating site
Date: Wed, 3 Oct 2018 12:13:55 +0000
MIME-Version: 1.0
Content-Type: multipart/alternative;
     boundary="----=_NextPart_000_067E_01D45B12.8E69E2F0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Windows Live Mail 15.4.3538.513
X-MimeOLE: Produced By Microsoft MimeOLE V15.4.3538.513
This particular one got stopped (looking at the subject I am happy it is rubbish)

Image


I will get my colleague to select a suitable email to show the headers / Spam report.

Cheers
ElFranko
Attachments
spamreport.JPG
spamreport.JPG (57.17 KiB) Viewed 7639 times
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: Regex, attachments and Spam

Post by pdwalker »

Your current example really looks like spam. Are you sure it is not spam?
Rod-IT
Posts: 11
Joined: 11 Sep 2018 19:37

Re: Regex, attachments and Spam

Post by Rod-IT »

I will get the non-spam that should be spam headers soon (sorry, busy with other tasks), but I wanted to point out I enabled recursion on my lab machine and now more emails get stopped by the RBL than before, but they are legitimate emails, newsletters, mailshots and even EFA forum notification emails for posts on here.

I am at the point in my lab where I am looking to disable RBL again as it's blocking more legitimate emails
User avatar
pdwalker
Posts: 1553
Joined: 18 Mar 2015 09:16

Re: Regex, attachments and Spam

Post by pdwalker »

Is the above example an example of a message you think should be considered non-spam?
Post Reply