Spam getting through even if it has been marked as spam

General eFa discussion
tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Spam getting through even if it has been marked as spam

Post by tentaclefi » 26 Feb 2020 00:50

Hi,

Lately we have received more spam than usual.

Some repeating spam emails have been marked as spam for 5-10 times but still copies of same email get through.

Where should I start to investigate what is causing this?

We are using EFA 4.0.2 and my gut feeling is that when I first installed the appliance it worked better than now.

Just my feeling though and nothing to back it up :)

Thank you for the great product and I hope I can make it work even better!

tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Re: Spam getting through even if it has been marked as spam

Post by tentaclefi » 29 Feb 2020 03:26

Seems that DCC has problems with SELinux:

Feb 29 05:11:05 mailscanner-in dccproc[109196]: open(/var/dcc/map): Permission denied

[root@mailscanner-in spamassassin]# ls -ltra /var/dcc/map
-rw-------. 1 postfix postfix 7700 Feb 29 04:39 /var/dcc/map

----
time->Sat Feb 29 05:11:05 2020
type=PROCTITLE msg=audit(1582945865.803:265716): proctitle=2F62696E2F64636370726F63002D43002D780030002D68002F7661722F646363002D52002D77007768697465636C6E74
type=SYSCALL msg=audit(1582945865.803:265716): arch=c000003e syscall=21 success=no exit=-13 a0=6beb8c a1=6 a2=6bee95 a3=7ffdf3451980 items=0 ppid=109191 pid=109196 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dccproc" exe="/usr/bin/dccproc" subj=system_u:system_r:dcc_client_t:s0 key=(null)
type=AVC msg=audit(1582945865.803:265716): avc: denied { dac_override } for pid=109196 comm="dccproc" capability=1 scontext=system_u:system_r:dcc_client_t:s0 tcontext=system_u:system_r:dcc_client_t:s0 tclass=capability permissive=0
----

There's also lots of ps getting denied:

----
time->Sat Feb 29 05:20:05 2020
type=PROCTITLE msg=audit(1582946405.802:265884): proctitle=7073006178
type=SYSCALL msg=audit(1582946405.802:265884): arch=c000003e syscall=4 success=no exit=-13 a0=2576310 a1=7fb2f8259ac0 a2=7fb2f8259ac0 a3=0 items=0 ppid=110458 pid=110459 auid=4294967295 uid=996 gid=993 euid=996 suid=996 fsuid=996 egid=993 sgid=993 fsgid=993 tty=(none) ses=4294967295 comm="ps" exe="/usr/bin/ps" subj=system_u:system_r:httpd_sys_script_t:s0 key=(null)
type=AVC msg=audit(1582946405.802:265884): avc: denied { getattr } for pid=110459 comm="ps" path="/proc/109198" dev="proc" ino=8743436 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:spamd_t:s0 tclass=dir permissive=0
----

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Spam getting through even if it has been marked as spam

Post by shawniverson » 29 Feb 2020 12:39

I'll add those to the eFa selinux policy.
Version eFa 4.0.2 now available!

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Spam getting through even if it has been marked as spam

Post by shawniverson » 29 Feb 2020 16:05

Wait, no.

Please check the label on /var/dcc/map. There's already a rule for dcc_client_map_t

Code: Select all

-rw-------. postfix postfix system_u:object_r:dcc_client_map_t:s0 /var/dcc/map
And spamd should not be running....
Version eFa 4.0.2 now available!

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 01 Mar 2020 15:05

We’re also getting very very much more spam mails in the last few days. SA learn doesn’t work at all. Every time the same spam mails are getting through EFA...

henk
Posts: 442
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Spam getting through even if it has been marked as spam

Post by henk » 01 Mar 2020 17:37

Hi Alexander,

can you exec

Code: Select all

ausearch -m 'AVC'
This wil show selinux issues, when present.

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 02 Mar 2020 07:27

Hi henk,

thank you!

When I execute the command I get the following output:
2020-03-02 08_27_31-mxadmin@mail_~.png
2020-03-02 08_27_31-mxadmin@mail_~.png (46.89 KiB) Viewed 646 times

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 03 Mar 2020 07:23

Can anyone help me please?
We are getting many many many spam which is not detected by EFA - lots of my colleagues are complaning about but I don't know anymore where to look...

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Spam getting through even if it has been marked as spam

Post by shawniverson » 03 Mar 2020 19:00

You can ignore the denials from ps.

Can you head over to the Tools section in MailWatch and run Spamassassin Lint and MailScanner Lint test please? Please attach the results.
Version eFa 4.0.2 now available!

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 03 Mar 2020 22:11

here are the screenshots. sorry for the bad quality due to upload size restrictions...
Attachments
Bildschirmfoto 2020-03-03 um 23.10.49.png
Bildschirmfoto 2020-03-03 um 23.10.49.png (1.72 MiB) Viewed 584 times
sa.jpg
sa.jpg (1.59 MiB) Viewed 584 times

tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Re: Spam getting through even if it has been marked as spam

Post by tentaclefi » 03 Mar 2020 22:40

Here's mine.

https://pasteboard.co/IXry4xE.png

https://pasteboard.co/IXrzWut.png

Spamassassin screenshot is missing the very last of it so here
once more as text.

https://pastebin.com/eRp6KjBv

User avatar
shawniverson
Posts: 3029
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Spam getting through even if it has been marked as spam

Post by shawniverson » 05 Mar 2020 01:02

Those lint tests actually look good. Does anybody have a spam report from an affected email they could share?
Version eFa 4.0.2 now available!

tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Re: Spam getting through even if it has been marked as spam

Post by tentaclefi » 05 Mar 2020 04:47

Hi,

After learning four similar emails came one more and here's the outcome.

The fifth and most recent one has different recipient but otherwise same email.
Attachments
Screenshot 2020-03-05 at 12.39.38.png
Screenshot 2020-03-05 at 12.39.38.png (190.69 KiB) Viewed 542 times
Screenshot 2020-03-05 at 12.40.01.png
Screenshot 2020-03-05 at 12.40.01.png (240.83 KiB) Viewed 542 times

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 05 Mar 2020 07:25

Here is my result and how the spam mail looks like. There are a lot of similar ones, also including images...
Attachments
IMG_6535.PNG
IMG_6535.PNG (964.69 KiB) Viewed 536 times
2020-03-05 08_22_03-RDP-Manager.png
2020-03-05 08_22_03-RDP-Manager.png (84.06 KiB) Viewed 536 times

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 05 Mar 2020 09:02

all these mails are spam mails and the SA score is very low and also negative...
Attachments
2020-03-05 10_00_33-RDP-Manager.png
2020-03-05 10_00_33-RDP-Manager.png (143.96 KiB) Viewed 535 times

tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Re: Spam getting through even if it has been marked as spam

Post by tentaclefi » 05 Mar 2020 12:09

Almost all of the spam we get is in Finnish and Alexanderbrix seems to get spam in German in turn.

But still I can't get it why learning same messages over and over won't work.

Interested to find out of course!

henk
Posts: 442
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Spam getting through even if it has been marked as spam

Post by henk » 05 Mar 2020 18:19

@tentaclefi
What about Shawn's remark about spamd?

There is no indication that Bayes is active in your message detail Spam Report
Did you modify efa? ( install packages or whatever modification)

henk
Posts: 442
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Spam getting through even if it has been marked as spam

Post by henk » 05 Mar 2020 18:22

alexanderbrix

Take a look at your last detail line: URIBL_Blocked Administrator notice
No recursion active and 2 internal dns servers 192.168.x.x

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 05 Mar 2020 21:12

Hi henk,

I just activated DNS recursion via putty in IP settings. Now there are no entries at "11) Primary DNS" and "12) Secondary DNS" anymore. Is this right then?
IPv6 settings in 7), 8) and 9) are empty since we don't use IPv6.

But there are still coming mails with "URIBL_BLOCKED - ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/Dns ... nsbl-block for more information."
Unfortunately I do not know what to do now :-(
Attachments
Bildschirmfoto 2020-03-05 um 22.09.05.png
Bildschirmfoto 2020-03-05 um 22.09.05.png (309.8 KiB) Viewed 499 times

tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Re: Spam getting through even if it has been marked as spam

Post by tentaclefi » 06 Mar 2020 01:05

henk wrote:
05 Mar 2020 18:19
@tentaclefi
What about Shawn's remark about spamd?

There is no indication that Bayes is active in your message detail Spam Report
Did you modify efa? ( install packages or whatever modification)
Yes I realised that it has started because I once restarted spamassassin by hand but rebooted afterwards after realising my error.

I haven't modified efa in any way.

There's something happening in Bayes database though:
Attachments
Screenshot 2020-03-06 at 9.09.33.png
Screenshot 2020-03-06 at 9.09.33.png (92.09 KiB) Viewed 488 times

alexanderbrix
Posts: 35
Joined: 20 Oct 2016 06:09

Re: Spam getting through even if it has been marked as spam

Post by alexanderbrix » 06 Mar 2020 08:56

Hello,

my problem is solved now.

I followed this instruction and it seems to work now :-)
https://spielwiese.la-evento.com/xelasb ... OCKED.html
Attachments
2020-03-06 09_55_39-RDP-Manager.png
2020-03-06 09_55_39-RDP-Manager.png (108.68 KiB) Viewed 481 times

tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Re: Spam getting through even if it has been marked as spam

Post by tentaclefi » 10 Mar 2020 09:00

How could I check why Bayes isn't used in filtering?

User avatar
pdwalker
Posts: 1212
Joined: 18 Mar 2015 09:16

Re: Spam getting through even if it has been marked as spam

Post by pdwalker » 12 Mar 2020 08:08

Can you run a test of a spammy message and send the results?

Use the following command

Code: Select all

spamassassin -D -t < [message] 
This will run the spam test against [message] and give you the gory results. There should be some information as to what the bayes classifier is doing.

[message] will be the name of a mail file, found in your (I'm still on v3 :shock:) /var/spool/MailScanner/quarantine/[date]/spam/ directory. just pick any of the files

For example, here are some of the results I get when I test a spammy message:

Code: Select all

Mar 12 16:03:25.232 [30820] dbg: plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC
...
Mar 12 16:03:25.459 [30820] dbg: config: fixed relative path: /var/lib/spamassassin/3.004001/updates_spamassassin_org/23_bayes.cf
Mar 12 16:03:25.459 [30820] dbg: config: using "/var/lib/spamassassin/3.004001/updates_spamassassin_org/23_bayes.cf" for included file
Mar 12 16:03:25.459 [30820] dbg: config: read file /var/lib/spamassassin/3.004001/updates_spamassassin_org/23_bayes.cf
...
Mar 12 16:03:27.667 [30820] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x4a1c5b0) implements 'learner_new', priority 0
Mar 12 16:03:27.667 [30820] dbg: plugin: Mail::SpamAssassin::Plugin::TxRep=HASH(0x4e193b8) implements 'learner_new', priority 0
Mar 12 16:03:27.667 [30820] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x4a1c5b0), bayes_store_module=Mail::SpamAssassin::BayesStore::SQL
Mar 12 16:03:27.705 [30820] dbg: bayes: using username: mailwatch
Mar 12 16:03:27.705 [30820] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::SQL=HASH(0x517e4b0)
Mar 12 16:03:27.705 [30820] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x4a1c5b0) implements 'learner_is_scan_available', priority 0
Mar 12 16:03:27.797 [30820] dbg: bayes: database connection established
Mar 12 16:03:27.798 [30820] dbg: bayes: found bayes db version 3
Mar 12 16:03:27.798 [30820] dbg: bayes: Using userid: 1
...
Mar 12 16:03:28.037 [30820] dbg: bayes: corpus size: nspam = 46728, nham = 203831
Mar 12 16:03:28.041 [30820] dbg: bayes: tokenized body: 415 tokens
Mar 12 16:03:28.042 [30820] dbg: bayes: tokenized uri: 75 tokens
Mar 12 16:03:28.042 [30820] dbg: bayes: tokenized invisible: 16 tokens
Mar 12 16:03:28.047 [30820] dbg: bayes: tokenized header: 217 tokens
Mar 12 16:03:28.050 [30820] dbg: bayes: tok_get_all: token count: 456
Mar 12 16:03:28.060 [30820] dbg: bayes: score = 0.999783199575169
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTCHAMMY is now ready, value: 20
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTCSPAMMY is now ready, value: 103
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTCLEARNED is now ready, value: 267
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTC is now ready, value: 456
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag HAMMYTOKENS is now ready, value: CODE(0x39a84b0)
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag SPAMMYTOKENS is now ready, value: CODE(0x579b128)
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag TOKENSUMMARY is now ready, value: CODE(0x4a5e728)
Mar 12 16:03:28.064 [30820] dbg: rules: ran eval rule BAYES_99 ======> got hit (1)
Mar 12 16:03:28.065 [30820] dbg: rules: ran eval rule BAYES_999 ======> got hit (1)
...
Mar 12 16:03:38.981 [30820] dbg: check: tests=BAYES_99,BAYES_999,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HELO_MISC_IP,HTML_MESSAGE,KAM_NUMSUBJECT,ML_SPAM_HEADER_YES,ML_SPF_PASS,MPART_ALT_DIFF,MXPF_TEST,RCVD_IN_PSBL,RCVD_IN_RP_RNBL,RDNS_NONE,SPF_FAIL
...
Mar 12 16:03:38.998 [30820] dbg: timing: total 14141 ms - init: 2972 (21.0%), b_tie_ro: 93 (0.7%), parse: 5 (0.0%), extract_message_metadata: 107 (0.8%), get_uri_detail_list: 3.2 (0.0%), tests_pri_-1000: 55 (0.4%), compile_gen: 516 (3.6%), compile_eval: 92 (0.7%), tests_pri_-950: 7 (0.0%), tests_pri_-900: 7 (0.1%), tests_pri_-90: 40 (0.3%), check_bayes: 28 (0.2%), b_tokenize: 12 (0.1%), b_tok_get_all: 7 (0.0%), b_comp_prob: 3.9 (0.0%), b_tok_touch_all: 1.42 (0.0%), b_finish: 1.66 (0.0%), tests_pri_0: 2271 (16.1%), check_spf: 25 (0.2%), dkim_load_modules: 23 (0.2%), check_dkim_signature: 474 (3.4%), tests_pri_10: 297 (2.1%), check_dcc: 286 (2.0%), tests_pri_20: 3919 (27.7%), check_razor2: 3900 (27.6%), tests_pri_30: 4024 (28.5%), check_pyzor: 4007 (28.3%), tests_pri_500: 368 (2.6%), tests_pri_1000: 20 (0.1%)
...
Content analysis details:   (11.5 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 4.0 BAYES_99               BODY: Bayes spam probability is 99 to 100%
                            [score: 0.9998]
 2.0 BAYES_999              BODY: Bayes spam probability is 99.9 to 100%
                            [score: 0.9998]
...

tentaclefi
Posts: 20
Joined: 12 Nov 2016 07:11

Re: Spam getting through even if it has been marked as spam

Post by tentaclefi » 12 Mar 2020 09:24

No mention of bayes in the output.

In the config files use_bayes is also commented out but it says it's on by default though.

-----

[root@mailscanner-in spamassassin]# grep bayes *
local.cf:# use_bayes 1
local.cf:# bayes_auto_learn 1
local.cf:# bayes_ignore_header X-Bogosity
local.cf:# bayes_ignore_header X-Spam-Flag
local.cf:# bayes_ignore_header X-Spam-Status
local.cf:# and a well-trained bayes DB can save running rules, too
MailScanner.conf:# use_bayes 0
MailScanner.conf:# will be created as /var/spool/spamassassin/bayes_msgcount, etc.
MailScanner.conf:# bayes_path should NOT be directory!
MailScanner.conf:# In this example, the trailing "bayes" will be the "bayes*" +
MailScanner.conf:# files in the directory "/etc/MailScanner/bayes/"
MailScanner.conf:#bayes_path /etc/MailScanner/bayes/bayes
MailScanner.conf:# bayes_file_mode 0770
MailScanner.conf:# To disable bayes autolearn
MailScanner.conf:# bayes_auto_learn 0
MailScanner.conf:# You will just end up with # MailScanner: big bayes_toks.new files
MailScanner.conf:#bayes_auto_expire 0
MailScanner.conf:bayes_store_module Mail::SpamAssassin::BayesStore::SQL
MailScanner.conf:bayes_sql_dsn DBI:mysql:sa_bayes:localhost
MailScanner.conf:bayes_sql_username sa_user
MailScanner.conf:bayes_sql_password XXXXX
MailScanner.conf:bayes_sql_override_username postfix
MailScanner.conf: user_awl_dsn DBI:mysql:sa_bayes:localhost
MailScanner.conf:bayes_auto_learn 1
MailScanner.conf:bayes_auto_learn_threshold_nonspam 0.1
MailScanner.conf:bayes_auto_learn_threshold_spam 6
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa-SpamCheck
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa-SpamScore
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa-Information
v320.pre:# and create a header containing ASN data for bayes tokenization.

henk
Posts: 442
Joined: 14 Dec 2015 22:16
Location: Netherlands
Contact:

Re: Spam getting through even if it has been marked as spam

Post by henk » 12 Mar 2020 11:33

Checking the logs in /var/log is the first step I take when there is an issue.
(Update the GEOPIP Db and Spamassasin Rule Descriptions via the Gui-> Tools and Links should work without errors.)
Bayed needs a minimal number of spam and ham (200) to be able to do the job. To check if spamassasin is using bayes ( your Gui bayes info looks fine)

Code: Select all

spamassassin -D --lint 2>&1 | grep bayes:
The numbers should match the Gui->Tools and Lint-> bayes database info.

Another way to check
Gui->Search and Reports-> SpamAssasin Rule Hits
Check the Rule Hits named BAYES_

To find spam mail, add a filter in Search and Reports where spamAssasin > 3 (or just where Is Spam > 0 )
In the details you can check the scores

Post Reply