Spam getting through even if it has been marked as spam

[tentaclefi]

Hi,

Lately we have received more spam than usual.

Some repeating spam emails have been marked as spam for 5-10 times but still copies of same email get through.

Where should I start to investigate what is causing this?

We are using EFA 4.0.2 and my gut feeling is that when I first installed the appliance it worked better than now.

Just my feeling though and nothing to back it up

Thank you for the great product and I hope I can make it work even better!

[tentaclefi]

Seems that DCC has problems with SELinux:

Feb 29 05:11:05 mailscanner-in dccproc[109196]: open(/var/dcc/map): Permission denied

[root@mailscanner-in spamassassin]# ls -ltra /var/dcc/map
-rw-------. 1 postfix postfix 7700 Feb 29 04:39 /var/dcc/map

----
time->Sat Feb 29 05:11:05 2020
type=PROCTITLE msg=audit(1582945865.803:265716): proctitle=2F62696E2F64636370726F63002D43002D780030002D68002F7661722F646363002D52002D77007768697465636C6E74
type=SYSCALL msg=audit(1582945865.803:265716): arch=c000003e syscall=21 success=no exit=-13 a0=6beb8c a1=6 a2=6bee95 a3=7ffdf3451980 items=0 ppid=109191 pid=109196 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dccproc" exe="/usr/bin/dccproc" subj=system_u:system_r:dcc_client_t:s0 key=(null)
type=AVC msg=audit(1582945865.803:265716): avc: denied { dac_override } for pid=109196 comm="dccproc" capability=1 scontext=system_u:system_r:dcc_client_t:s0 tcontext=system_u:system_r:dcc_client_t:s0 tclass=capability permissive=0
----

There's also lots of ps getting denied:

----
time->Sat Feb 29 05:20:05 2020
type=PROCTITLE msg=audit(1582946405.802:265884): proctitle=7073006178
type=SYSCALL msg=audit(1582946405.802:265884): arch=c000003e syscall=4 success=no exit=-13 a0=2576310 a1=7fb2f8259ac0 a2=7fb2f8259ac0 a3=0 items=0 ppid=110458 pid=110459 auid=4294967295 uid=996 gid=993 euid=996 suid=996 fsuid=996 egid=993 sgid=993 fsgid=993 tty=(none) ses=4294967295 comm="ps" exe="/usr/bin/ps" subj=system_u:system_r:httpd_sys_script_t:s0 key=(null)
type=AVC msg=audit(1582946405.802:265884): avc: denied { getattr } for pid=110459 comm="ps" path="/proc/109198" dev="proc" ino=8743436 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:spamd_t:s0 tclass=dir permissive=0
----

[shawniverson]

I'll add those to the eFa selinux policy.

[shawniverson]

Wait, no.

Please check the label on /var/dcc/map. There's already a rule for dcc_client_map_t

Code: Select all

-rw-------. postfix postfix system_u:object_r:dcc_client_map_t:s0 /var/dcc/map

And spamd should not be running....

[alexanderbrix]

We’re also getting very very much more spam mails in the last few days. SA learn doesn’t work at all. Every time the same spam mails are getting through EFA...

[henk]

Hi Alexander,

can you exec

Code: Select all

ausearch -m 'AVC'

This wil show selinux issues, when present.

[alexanderbrix]

Hi henk,

thank you!

When I execute the command I get the following output:

2020-03-02 08_27_31-mxadmin@mail_~.png (46.89 KiB) Viewed 27387 times

[alexanderbrix]

Can anyone help me please?
We are getting many many many spam which is not detected by EFA - lots of my colleagues are complaning about but I don't know anymore where to look...

[shawniverson]

You can ignore the denials from ps.

Can you head over to the Tools section in MailWatch and run Spamassassin Lint and MailScanner Lint test please? Please attach the results.

[alexanderbrix]

here are the screenshots. sorry for the bad quality due to upload size restrictions...

[tentaclefi]

Here's mine.

https://pasteboard.co/IXry4xE.png

https://pasteboard.co/IXrzWut.png

Spamassassin screenshot is missing the very last of it so here
once more as text.

https://pastebin.com/eRp6KjBv

[shawniverson]

Those lint tests actually look good. Does anybody have a spam report from an affected email they could share?

[tentaclefi]

Hi,

After learning four similar emails came one more and here's the outcome.

The fifth and most recent one has different recipient but otherwise same email.

[alexanderbrix]

Here is my result and how the spam mail looks like. There are a lot of similar ones, also including images...

[alexanderbrix]

all these mails are spam mails and the SA score is very low and also negative...

[tentaclefi]

Almost all of the spam we get is in Finnish and Alexanderbrix seems to get spam in German in turn.

But still I can't get it why learning same messages over and over won't work.

Interested to find out of course!

[henk]

@tentaclefi
What about Shawn's remark about spamd?

There is no indication that Bayes is active in your message detail Spam Report
Did you modify efa? ( install packages or whatever modification)

[henk]

alexanderbrix

Take a look at your last detail line: URIBL_Blocked Administrator notice
No recursion active and 2 internal dns servers 192.168.x.x

[alexanderbrix]

Hi henk,

I just activated DNS recursion via putty in IP settings. Now there are no entries at "11) Primary DNS" and "12) Secondary DNS" anymore. Is this right then?
IPv6 settings in 7), 8) and 9) are empty since we don't use IPv6.

But there are still coming mails with "URIBL_BLOCKED - ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/Dns ... nsbl-block for more information."
Unfortunately I do not know what to do now

[tentaclefi]

@tentaclefi
What about Shawn's remark about spamd?

There is no indication that Bayes is active in your message detail Spam Report
Did you modify efa? ( install packages or whatever modification)

Yes I realised that it has started because I once restarted spamassassin by hand but rebooted afterwards after realising my error.

I haven't modified efa in any way.

There's something happening in Bayes database though:

[alexanderbrix]

Hello,

my problem is solved now.

I followed this instruction and it seems to work now
https://spielwiese.la-evento.com/xelasb ... OCKED.html

[tentaclefi]

How could I check why Bayes isn't used in filtering?

[pdwalker]

Can you run a test of a spammy message and send the results?

Use the following command

Code: Select all

spamassassin -D -t < [message] 

This will run the spam test against [message] and give you the gory results. There should be some information as to what the bayes classifier is doing.

[message] will be the name of a mail file, found in your (I'm still on v3 ) /var/spool/MailScanner/quarantine/[date]/spam/ directory. just pick any of the files

For example, here are some of the results I get when I test a spammy message:

Code: Select all

Mar 12 16:03:25.232 [30820] dbg: plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC
...
Mar 12 16:03:25.459 [30820] dbg: config: fixed relative path: /var/lib/spamassassin/3.004001/updates_spamassassin_org/23_bayes.cf
Mar 12 16:03:25.459 [30820] dbg: config: using "/var/lib/spamassassin/3.004001/updates_spamassassin_org/23_bayes.cf" for included file
Mar 12 16:03:25.459 [30820] dbg: config: read file /var/lib/spamassassin/3.004001/updates_spamassassin_org/23_bayes.cf
...
Mar 12 16:03:27.667 [30820] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x4a1c5b0) implements 'learner_new', priority 0
Mar 12 16:03:27.667 [30820] dbg: plugin: Mail::SpamAssassin::Plugin::TxRep=HASH(0x4e193b8) implements 'learner_new', priority 0
Mar 12 16:03:27.667 [30820] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x4a1c5b0), bayes_store_module=Mail::SpamAssassin::BayesStore::SQL
Mar 12 16:03:27.705 [30820] dbg: bayes: using username: mailwatch
Mar 12 16:03:27.705 [30820] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::SQL=HASH(0x517e4b0)
Mar 12 16:03:27.705 [30820] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x4a1c5b0) implements 'learner_is_scan_available', priority 0
Mar 12 16:03:27.797 [30820] dbg: bayes: database connection established
Mar 12 16:03:27.798 [30820] dbg: bayes: found bayes db version 3
Mar 12 16:03:27.798 [30820] dbg: bayes: Using userid: 1
...
Mar 12 16:03:28.037 [30820] dbg: bayes: corpus size: nspam = 46728, nham = 203831
Mar 12 16:03:28.041 [30820] dbg: bayes: tokenized body: 415 tokens
Mar 12 16:03:28.042 [30820] dbg: bayes: tokenized uri: 75 tokens
Mar 12 16:03:28.042 [30820] dbg: bayes: tokenized invisible: 16 tokens
Mar 12 16:03:28.047 [30820] dbg: bayes: tokenized header: 217 tokens
Mar 12 16:03:28.050 [30820] dbg: bayes: tok_get_all: token count: 456
Mar 12 16:03:28.060 [30820] dbg: bayes: score = 0.999783199575169
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTCHAMMY is now ready, value: 20
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTCSPAMMY is now ready, value: 103
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTCLEARNED is now ready, value: 267
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag BAYESTC is now ready, value: 456
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag HAMMYTOKENS is now ready, value: CODE(0x39a84b0)
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag SPAMMYTOKENS is now ready, value: CODE(0x579b128)
Mar 12 16:03:28.063 [30820] dbg: check: tagrun - tag TOKENSUMMARY is now ready, value: CODE(0x4a5e728)
Mar 12 16:03:28.064 [30820] dbg: rules: ran eval rule BAYES_99 ======> got hit (1)
Mar 12 16:03:28.065 [30820] dbg: rules: ran eval rule BAYES_999 ======> got hit (1)
...
Mar 12 16:03:38.981 [30820] dbg: check: tests=BAYES_99,BAYES_999,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HELO_MISC_IP,HTML_MESSAGE,KAM_NUMSUBJECT,ML_SPAM_HEADER_YES,ML_SPF_PASS,MPART_ALT_DIFF,MXPF_TEST,RCVD_IN_PSBL,RCVD_IN_RP_RNBL,RDNS_NONE,SPF_FAIL
...
Mar 12 16:03:38.998 [30820] dbg: timing: total 14141 ms - init: 2972 (21.0%), b_tie_ro: 93 (0.7%), parse: 5 (0.0%), extract_message_metadata: 107 (0.8%), get_uri_detail_list: 3.2 (0.0%), tests_pri_-1000: 55 (0.4%), compile_gen: 516 (3.6%), compile_eval: 92 (0.7%), tests_pri_-950: 7 (0.0%), tests_pri_-900: 7 (0.1%), tests_pri_-90: 40 (0.3%), check_bayes: 28 (0.2%), b_tokenize: 12 (0.1%), b_tok_get_all: 7 (0.0%), b_comp_prob: 3.9 (0.0%), b_tok_touch_all: 1.42 (0.0%), b_finish: 1.66 (0.0%), tests_pri_0: 2271 (16.1%), check_spf: 25 (0.2%), dkim_load_modules: 23 (0.2%), check_dkim_signature: 474 (3.4%), tests_pri_10: 297 (2.1%), check_dcc: 286 (2.0%), tests_pri_20: 3919 (27.7%), check_razor2: 3900 (27.6%), tests_pri_30: 4024 (28.5%), check_pyzor: 4007 (28.3%), tests_pri_500: 368 (2.6%), tests_pri_1000: 20 (0.1%)
...
Content analysis details:   (11.5 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 4.0 BAYES_99               BODY: Bayes spam probability is 99 to 100%
                            [score: 0.9998]
 2.0 BAYES_999              BODY: Bayes spam probability is 99.9 to 100%
                            [score: 0.9998]
...

[tentaclefi]

No mention of bayes in the output.

In the config files use_bayes is also commented out but it says it's on by default though.

-----

[root@mailscanner-in spamassassin]# grep bayes *
local.cf:# use_bayes 1
local.cf:# bayes_auto_learn 1
local.cf:# bayes_ignore_header X-Bogosity
local.cf:# bayes_ignore_header X-Spam-Flag
local.cf:# bayes_ignore_header X-Spam-Status
local.cf:# and a well-trained bayes DB can save running rules, too
MailScanner.conf:# use_bayes 0
MailScanner.conf:# will be created as /var/spool/spamassassin/bayes_msgcount, etc.
MailScanner.conf:# bayes_path should NOT be directory!
MailScanner.conf:# In this example, the trailing "bayes" will be the "bayes*" +
MailScanner.conf:# files in the directory "/etc/MailScanner/bayes/"
MailScanner.conf:#bayes_path /etc/MailScanner/bayes/bayes
MailScanner.conf:# bayes_file_mode 0770
MailScanner.conf:# To disable bayes autolearn
MailScanner.conf:# bayes_auto_learn 0
MailScanner.conf:# You will just end up with # MailScanner: big bayes_toks.new files
MailScanner.conf:#bayes_auto_expire 0
MailScanner.conf:bayes_store_module Mail::SpamAssassin::BayesStore::SQL
MailScanner.conf:bayes_sql_dsn DBI:mysql:sa_bayes:localhost
MailScanner.conf:bayes_sql_username sa_user
MailScanner.conf:bayes_sql_password XXXXX
MailScanner.conf:bayes_sql_override_username postfix
MailScanner.conf: user_awl_dsn DBI:mysql:sa_bayes:localhost
MailScanner.conf:bayes_auto_learn 1
MailScanner.conf:bayes_auto_learn_threshold_nonspam 0.1
MailScanner.conf:bayes_auto_learn_threshold_spam 6
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa-SpamCheck
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa-SpamScore
MailScanner.conf:bayes_ignore_header X-xxx-MailScanner-eFa-Information
v320.pre:# and create a header containing ASN data for bayes tokenization.

[henk]

Checking the logs in /var/log is the first step I take when there is an issue.
(Update the GEOPIP Db and Spamassasin Rule Descriptions via the Gui-> Tools and Links should work without errors.)
Bayed needs a minimal number of spam and ham (200) to be able to do the job. To check if spamassasin is using bayes ( your Gui bayes info looks fine)

Code: Select all

spamassassin -D --lint 2>&1 | grep bayes:

The numbers should match the Gui->Tools and Lint-> bayes database info.

Another way to check
Gui->Search and Reports-> SpamAssasin Rule Hits
Check the Rule Hits named BAYES_

To find spam mail, add a filter in Search and Reports where spamAssasin > 3 (or just where Is Spam > 0 )
In the details you can check the scores