3.0.0.9 SA Rule Description problems in the MailWatch web interface [with solutions]

[pdwalker]

I've noticed a minor problem with my installation that's finally bothered me enough to get off my ass and investigate it.

Problem 1: the Spam Assassin Rule Hits report does not display the full table

Examining the report HTML shows the full table is output, but the problem is in the SA rule description; specifically:

Code: Select all

MANY_SPAN_IN_TEXT  Many <SPAN> tags embedded within text
STYLE_GIBBERISH    Nonsense in HTML <STYLE> tag

With the first entry, the <SPAN> is eaten by the browser resulting in the description looking like "Many tags embedded within text"

With the second, the table stops at the point this appears on the page.

Solution:
(a) change the descriptions, or
(b) html encode the <>'s that appear

Further Investigation:
change line 134 in rep_sa_rule_hits.php from:

Code: Select all

<TD>" . return_sa_rule_desc(strtoupper($key)) . "</TD>

to:

Code: Select all

<TD>" . htmlentities(return_sa_rule_desc(strtoupper($key))) . "</TD>

Changing the descriptions in local.cf also works, but that just avoids the problem.


Problem 2: the Spam Assassin Rule Hits report doesn't show all the descriptions

For example, in mine installation KAM_LAZY_DOMAIN_SECURITY shows no description. Running the "Update SpamAssassin Rule Descriptions" under Tools refreshes the description database, but KAM_LAZY_DOMAIN_SECURITY doesn't show up on that page. The rule does have a description defined in the KAM.cf

Workaround:
Add your own description lines into local.cf at the bottom of the file and reload the SpamAssassin Rule Description Update page.

Further Investigation:
I suspect the problem is in sa_rules_update.php, specifically line 66, but I've not yet had the chance to test to see if it actually goes wrong here.

[pdwalker]

*ding*

found the source for problem 2:

the command to find the descriptions is "grep -hr '^describe'"; however some of the descriptions do not start at the first character of the line, but start indented a few spaces in. Look at KAM_VERY_BLACK_DBL in /etc/mail/spamassassin/KAM.cf for an example.

Solution: change the grep command to allow for spaces preceding the describe keyword...

[pdwalker]

sa_rules_update.php, line 66
from:

Code: Select all

"grep -hr '^describe' " . SA_RULES_DIR . " /usr/share/spamassassin /usr/local/share/spamassassin /etc/MailScanner/spam.assassin.prefs.conf /opt/MailScanner/etc/spam.assassin.prefs.conf /usr/local/etc/mail/spamassassin /etc/mail/spamassassin /var/lib/spamassassin 2>/dev/null | sort | uniq",

to:

Code: Select all

"grep -hr -e '^[[:space:]]*describe' " . SA_RULES_DIR . " /usr/share/spamassassin /usr/local/share/spamassassin /etc/MailScanner/spam.assassin.prefs.conf /opt/MailScanner/etc/spam.assassin.prefs.conf /usr/local/etc/mail/spamassassin /etc/mail/spamassassin /var/lib/spamassassin 2>/dev/null | sort | uniq",

and line 73 from:

Code: Select all

preg_match("/^describe\s+(\S+)\s+(.+)$/", $line, $regs);

to:

Code: Select all

preg_match("/^[[:space:]]*describe\s+(\S+)\s+(.+)$/", $line, $regs);

That seems to solve most of my missing description errors. The ones that remain and that I've checked genuinely do not have descriptions that I can find, like HTTPS_HTTP_MISMATCH

[pdwalker]

Fixed in the mailwatch project.

[ovizii]

awesome, thanks for the fix!

[pdwalker]

there are still a few rules without descriptions that I've had to add to my local.cf.

For my installation, these are the ones I found missing, or could not find anything at all

Code: Select all

# missing descriptions?  text based on what I could determine from the rule itself
describe FAKE_REPLY_C   The mail's subject begins "Re: ", indicating a reply, but does not include the References: header.
describe HTTPS_HTTP_MISMATCH    Link shows https but href is http
describe RCVD_IN_BRBL_LASTEXT   The last external relay in the Received chain was listed in the DNSBL Barracuda Reputation Block List (BRBL).
describe STOX_REPLY_TYPE        The mail's content type is "text/plain" and has the "reply-type=original" attribute.
describe STOX_REPLY_TYPE_WITHOUT_QUOTES The mail's content type is "text/plain" and has the "reply-type=original" attribute and subject doesn't begin with re:, fw:, or the body doesn't contain quoting > characters
describe TVD_RCVD_SPACE_BRACKET "UNIX:" is part of the received line in the headers
describe T_FUZZY_SPRM *unsure* - something to do with replacing tags in the body
describe T_HK_NAME_FM_DR message from free email provider and message is "From:" a "Dr"
describe T_HK_NAME_FM_FROM message from free email provider and message is "From:" "from"
describe FILL_THIS_FORM_FRAUD_PHISH     possible phishing email looking with form for capturing information
describe FREEMAIL_DOC_PDF_BCC   MS document or PDF attachment, from freemail, all recipients hidden
describe FROM_MISSP_SPF_FAIL    runon From line(?) and failed spf check
describe FSL_HELO_BARE_IP_2     helo from bare ip address of untrusted external server
describe HK_LOTTO       Lottery spam
describe HK_NAME_DR     mail From: a doctor and not from a freemail account
describe HK_NAME_FM_MR_MRS      Mail from freemail account named Mr/Miss/Mrs/etc
describe HK_NAME_MR_MRS Mail from account named Mr/Miss/Mrs/etc
describe HK_SCAM_N2     Nigerian scam - next of kin
describe HK_SCAM_N8     Nigerian scam - compensation
describe HTML_TITLE_SUBJ_DIFF   html mail with differing subject and <title>
describe ImageCerberusPLG0      image Cerebrus check 0 - not image spam
describe ImageCerberusPLG1      image Cerebrus check 1 - possible image spam
describe ImageCerberusPLG2      image Cerebrus check 2 - possible image spam
describe ImageCerberusPLG3      image Cerebrus check 3 - likely image spam
describe ImageCerberusPLG4      image Cerebrus check 4 - likely image spam
describe ImageCerberusPLG5      image Cerebrus check 5 - image spam
describe LOTTERY_PH_004470      lottery spam
describe MSOE_MID_WRONG_CASE    fake MS Outlook Express, wrong case on Message-Id: header
describe REPLYTO_WITHOUT_TO_CC  Reply-To: header exists, but no one CC'd
describe TVD_PH_BODY_META       phishing email - you have some problem with your email account type messages
describe TVD_SPACE_RATIO        the ratio of spaces to non-spaces in each paragraph where there are lots of spaces means it is likely spam
describe TVD_VISIT_PHARMA       online pharmacy spam
describe T_HK_NAME_FM_MR_MRS    freemail account, message from Mr/Mrs/etc
describe T_HK_NAME_FROM mail from "from", spammy like
describe T_HK_SPAMMY_FILENAME   attached filename with spam like names
describe T_TVD_MIME_EPI mime message with invalid mime flags(?)
describe T_TVD_MIME_NO_HEADERS  mime message with missing mime headers
describe T_TVD_FW_GRAPHIC_ID2 mime message with a certain Content-Id type(?)

# unknown rules - old rules that no longer exist?
describe HK_SCAM_N1     *unknown, cannot find rule*
describe REPLYTO_EMPTY  *unknown, cannot find rule*
describe TVD_SPACE_ENCODED      *unknown, cannot find rule*
describe UPGRADE_MAILBOX *unknown, cannot find rule*
describe VERY_LONG_REPTO_SHORT_MSG *unknown, cannot find rule*
describe AXB_JT_FOLNO0  *unknown, cannot find rule*
describe BODY_SINGLE_URI        *unknown, cannot find rule*
describe DUP_SUSP_HDR   *unknown, cannot find rule*
describe FSL_HELO_FAKE  *unknown, cannot find rule*
describe GOOG_REDIR_HTML_ONLY   *unknown, cannot find rule*
describe HK_LOTTO_NAME  *unknown, cannot find rule*
describe HK_SCAM_N13    *unknown, cannot find rule*
describe HK_SCAM_N15    *unknown, cannot find rule*
describe TVD_SPACE_ENC_FM_MIME  *unknown, cannot find rule*
describe T_FILL_THIS_FORM_LONG  *unknown, cannot find rule*

[ovizii]

Thanks for sharing!

[shawniverson]

https://github.com/E-F-A/v3/issues/287