Could not open file

General eFa discussion
r00tsh3ll
Posts: 30
Joined: 14 Feb 2021 06:34

Re: Could not open file

Post by r00tsh3ll »

Doesn't work if the Message-Id header is the last header (exits loop)


Hello Shawnn,

I finally had some time to debug a bit the message-id extraction logic.

The first issue is that with the actual code, it will return an empty message-id if the Message-ID header is the last header. This happens more often that we could think.
This was due to the loop needing another run to the next line to identify that previous line was the end of the message-id.
To fix this I moved some stuff out of the loop.

I also replaced whites spaces in regex'es with '\s' in case it would be a problem.

Finally when removing the header name from the buffer string I used 'message-id:\s+' instead of 'message-id:\s' in case there are multiple spaces.
It seems this fixed part of the problem for unfolding as the buffer seems to contain double space at some point, like shown in the debug output:

Code: Select all

Nov 21 06:39:34 mx1 MailScanner[31696]: MailWatch: DEBUG step1: messageidbuffer: 'Message-ID: '
Nov 21 06:39:34 mx1 MailScanner[31696]: MailWatch: DEBUG step1-unfold: messageidbuffer: 'Message-ID:  <DBBPR09MB3048541501331ADC215B1708F39E9@DBBPR09MB3048.eurprd09.prod.outlook.com>'
Nov 21 06:39:34 mx1 MailScanner[31696]: MailWatch: DEBUG HEADER -message-id- : '<DBBPR09MB3048541501331ADC215B1708F39E9@DBBPR09MB3048.eurprd09.prod.outlook.com>'
I trim the final result to be sure there is not leading or trailing spaces.

Bonus: I also added a log warning in case of $messageid still being blank after all the processing, to catch other possible issues.

Here is the code:

Code: Select all

    # Message-ID
    my ($messageid, $inmessageid, $messageidbuffer);
    $messageid = "";
    $messageidbuffer = "";
    $inmessageid = 0;

    # Extract message id from header (unfold header if needed)
    foreach (@{$message->{headers}}) {
        if ( $_ =~ /^message-id:\s/i ) {
            # RFC 822 unfold message-id
            $messageidbuffer = $_;
            $inmessageid = 1;
            next;
        } elsif ($inmessageid) {
            if ($_ =~ /^\s/) {
                # In continuation line
                $messageidbuffer .= $_;
            } else {
                # End of message-id field
                last;
            }
        }
    }

    # Set the re-formatted message-id and trim it
    ($messageid = $messageidbuffer) =~ s/^message-id:\s+//i;
    $messageid =~ s/^\s+|\s+$//g;

    # Warn if Message-ID was not found
    if ($messageid eq "") {
      MailScanner::Log::WarnLog("MailWatch: Could not extract Message-ID for %s", $message->{id});
    }
Please take a look at it. It might be optimized, my perl is a bit rusty...

On my tests here it seems to have improved the message-id extraction success rate and the unfolding seems to work now.
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Could not open file

Post by shawniverson »

To answer your question about autoupdate, on CentOS7 yum-cron is in use. iirc, there's a setting /etc/yum.conf you can toggle to disable the autoupdate.
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Could not open file

Post by shawniverson »

Great work on the unfolding, I knew there were problems with the initial logic, I appreciate the help in debugging this. I will get this code into the test repo today and work on a strategy for dealing with multiple, separate recipients with the same message-id.
r00tsh3ll
Posts: 30
Joined: 14 Feb 2021 06:34

Re: Could not open file

Post by r00tsh3ll »

Hey Shawn,
No problem. Thanks to you for all the work!
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Could not open file

Post by shawniverson »

Got some sweet patches for MailWatch for testing. I'll get them in the repo. I *think* I have the multiple recipients issue with the relay data resolved as well.
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Could not open file

Post by shawniverson »

Got the repos updated. Please test when you get a chance. Includes the latest unfold logic and improved relay stat handling for multiple recipients with the same messsage-id.

Code: Select all

sudo yum clean all
sudo yum update
r00tsh3ll
Posts: 30
Joined: 14 Feb 2021 06:34

Re: Could not open file

Post by r00tsh3ll »

Hey, thanks.

I've updated the system and took a look at the changes and did some tests. It seems to handle multiple recipients flawlessly now. Great job :)

It seems eFa now also includes my pull requests for the disable backup thing, thanks!

I've checked a bit the maillog for Message-ID not extracted:

Code: Select all

Nov 22 03:53:25 mx1 MailScanner[161941]: MailWatch: Could not extract Message-ID for 4HyBfS2Mq7zMqYkQ
Nov 22 03:56:53 mx1 MailScanner[169937]: MailWatch: Could not extract Message-ID for 4HyBjl1Xg7zMqZdH
Nov 22 04:19:32 mx1 MailScanner[171863]: MailWatch: Could not extract Message-ID for 4HyCDT11pSzMqb16
There are a few reported, but they all really miss the Message-ID in their header, so I think we're not bad with the extraction routine.
I guess there is no relay tracking possible for these messages, but they are mostly spam anyway.

Thanks again!
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Could not open file

Post by shawniverson »

Glad to hear it! I will release this to the production repos. I'll let you know when that happens so you can switch back, if desired.
User avatar
shawniverson
Posts: 3644
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: Could not open file

Post by shawniverson »

Got the changes in production. Thanks for your help!
Post Reply