opendmarc.service failed - kills mailscanner?

Bugs in eFa 4
MattS
Posts: 20
Joined: 12 Dec 2017 14:00

Re: opendmarc.service failed - kills mailscanner?

Post by MattS »

Yeah, it didn't work for me either when I tried it yesterday.

My efa stops processing mail every 4 hours at 59 minutes past the hour without fail. I think it's the same email triggering the seg fault every time and suspect the email is stuck in the queue of the external backup smtp service we use, hence the 4hr retry cycle. Obviously can't block the IP of the "sender" at the firewall as it's coming from the legitimate IP address of our provider.

As it's happening with such predicatable regularity, I just scheduled a root cron job to restart the opendmarc service every 4 hours on the hour, which has at least meant I don't need to sit here all day monitoring it until a fixed version of the opendmarc package is released.
1an3
Posts: 24
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 »

bizzare, my opendmarc was crashing all over the weekend and always restarted itself.

There is another way to do it in opendamrc.conf, where you can configure restart auto restart behaviour, max attempts, rates, etc.
kandegama
Posts: 7
Joined: 28 Sep 2017 17:38

Re: opendmarc.service failed - kills mailscanner?

Post by kandegama »

I have a similar issue. please help with this.EFA use as the primary spam gateway. this happened after auto-update happen in DMRC RPMs
1an3
Posts: 24
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 »

Hi Shawn will this fix for opendmarc make it into the repo for yum to get hold of? Not sure I have it in me to compile a patch etc unless there are noddy instructions.
kandegama
Posts: 7
Joined: 28 Sep 2017 17:38

Re: opendmarc.service failed - kills mailscanner?

Post by kandegama »

Please can you help me how to deploy SRC package to my EFA enviorment
User avatar
shawniverson
Posts: 3647
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: opendmarc.service failed - kills mailscanner?

Post by shawniverson »

I am preparing to send an update out that includes this patch
kandegama
Posts: 7
Joined: 28 Sep 2017 17:38

Re: opendmarc.service failed - kills mailscanner?

Post by kandegama »

shawniverson wrote: 17 Jun 2021 18:39 I am preparing to send an update out that includes this patch
Thanks Shawn
kandegama
Posts: 7
Joined: 28 Sep 2017 17:38

Re: opendmarc.service failed - kills mailscanner?

Post by kandegama »

shawniverson wrote: 17 Jun 2021 18:39 I am preparing to send an update out that includes this patch
today morning updated following RPM's
Jun 18 06:10:11 Updated: libopendmarc.x86_64 1.4.1.1-1.eFa.1.el7
Jun 18 06:10:11 Updated: opendmarc.x86_64 1.4.1.1-1.eFa.1.el7
Jun 18 06:10:50 Updated: eFa.noarch 1:4.0.4-13.eFa.el7

but after that Mailwatch web console is hanging but there is no error in the mail log file.
kandegama
Posts: 7
Joined: 28 Sep 2017 17:38

Re: opendmarc.service failed - kills mailscanner?

Post by kandegama »

Gateway timeout error appear in web page
forhire
Posts: 30
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire »

I just ran yum updates and a new version was updated. The epel version has been replaced by an eFa4 rpm.
Updated libopendmarc-1.4.1-1.el7.x86_64 @epel
Update 1.4.1.1-1.eFa.1.el7.x86_64 @eFa4
Updated opendmarc-1.4.1-1.el7.x86_64 @epel
Update 1.4.1.1-1.eFa.1.el7.x86_64 @eFa4
bostjanc
Posts: 165
Joined: 01 Jun 2016 17:18

Re: opendmarc.service failed - kills mailscanner?

Post by bostjanc »

Forhire do you have the same issue as others? Gateway timeouts?
I havent upgraded yet on our EFAs
jamerson
Posts: 164
Joined: 19 Aug 2017 18:57
Location: kaaskop

Re: opendmarc.service failed - kills mailscanner?

Post by jamerson »

its appear to be a brut force or something similar to manipulate EFA.
what firewall are you using in front of the EFA? what ports are open from the WAN to the EFA? are you using some kind of ids/IPS?
Version eFa 4.0.0 RC1 now available in testing repo. Come join us in advancing eFa!
kandegama
Posts: 7
Joined: 28 Sep 2017 17:38

Re: opendmarc.service failed - kills mailscanner?

Post by kandegama »

Fortinet firewall and IPS and spam enable in firewall side
MattS
Posts: 20
Joined: 12 Dec 2017 14:00

Re: opendmarc.service failed - kills mailscanner?

Post by MattS »

I noticed the three new rpm's being available this morning but hung back from applying them pending any teething problems. However, my eFa instance appears to have automatically updated itself in the last hour. Luckily without any issue. I've even just seen the offending email that was causing our problem get delivered and processed in mailwatch before correctly being identified as black listed.

The fact eFa automatically installed the broken opendmarc package caused the problem in the first place but the automatic updating has obviously fixed the problem a week later. I'm in two minds as to whether to disable automatic system package updates, if there's an easy way to do that, in order to sanity check any future updates.
forhire
Posts: 30
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire »

bostjanc wrote: 18 Jun 2021 05:58 Forhire do you have the same issue as others? Gateway timeouts?
I havent upgraded yet on our EFAs
I haven't had any issues.
kandegama
Posts: 7
Joined: 28 Sep 2017 17:38

Re: opendmarc.service failed - kills mailscanner?

Post by kandegama »

forhire wrote: 18 Jun 2021 15:31
bostjanc wrote: 18 Jun 2021 05:58 Forhire do you have the same issue as others? Gateway timeouts?
I havent upgraded yet on our EFAs
I haven't had any issues.
Thanks for update
I had an issue. anyway after restart again still there is no issue. will update on if any problem arises.
aztek0
Posts: 3
Joined: 21 Dec 2021 20:49

Re: opendmarc.service failed - kills mailscanner?

Post by aztek0 »

This issue seems to have not gone away completely (either that or I am missing something).

Here are the versions I have:

MailWatch Version: 1.2.18
Operating System Version: CentOS Linux 7 (Core)
Postfix Version: 3.5.9
MailScanner Version: 5.4.4
ClamAV Version: 0.103.5
SpamAssassin Version: 3.4.6
PHP Version: 7.4.28
MySQL Version: 10.2.30-MariaDB

The system was updated with eFa.noarch 1:4.0.4-27.eFa.el7 not too long ago (February 26th) but as mentioned in this topic, only specific emails would trigger, in my case, the segfault:

Message log:

Code: Select all

Mar  7 15:07:50 fryssmtpout104 kernel: opendmarc[1772]: segfault at 0 ip 00007fa5003e65d4 sp 00007fa4fe9ad118 error 4 in libbsd.so.0.8.3[7fa5003d8000+14000]
Mar  7 15:07:50 fryssmtpout104 systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Mar  7 15:07:50 fryssmtpout104 systemd: Unit opendmarc.service entered failed state.
Mar  7 15:07:50 fryssmtpout104 systemd: opendmarc.service failed.
Mail log:

Code: Select all

Mar  7 15:07:49 fryssmtpout104 postfix/cleanup[32035]: 4KCDdK5GhYz5KSBZ: message-id=<1646662068582.15868@awi.de>
Mar  7 15:07:50 fryssmtpout104 opendkim[29273]: 4KCDdK5GhYz5KSBZ: mailouts.awi.de [134.1.2.99] not internal
Mar  7 15:07:50 fryssmtpout104 opendkim[29273]: 4KCDdK5GhYz5KSBZ: not authenticated
Mar  7 15:07:50 fryssmtpout104 opendkim[29273]: 4KCDdK5GhYz5KSBZ: DKIM verification successful
Mar  7 15:07:50 fryssmtpout104 opendmarc[8891]: 4KCDdK5GhYz5KSBZ: ignoring invalid ARC-Authentication-Results header "i=1;#012#011smtpd-out;#012#011none"
All three instances of the segfault today were caused by the same email address. This had not happened before since eFa was put in production a couple years back. I have to mention that even though several kernels have been installed, I have not gotten around to reboot the appliance in over 4 months, so it is still running on this kernel:

3.10.0-1160.45.1.el7.x86_64

These are the versions of opendmarc installed:

yum list installed | grep dmarc
libopendmarc.x86_64 1.4.1.1-3.el7 @epel
opendmarc.x86_64 1.4.1.1-3.el7 @epel

In the meantime I applied the very tiny band-aid of blocking the IP of that mail server, plus the regular-sized band-aid of restarting opendmarc on failure.

Any ideas?

Regards
A.
User avatar
shawniverson
Posts: 3647
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: opendmarc.service failed - kills mailscanner?

Post by shawniverson »

This kind've sounds like another potential DoS attack on opendmarc. Is there any other details your can share? Is this a spam email that is causing this?
asulkowski
Posts: 2
Joined: 10 Mar 2022 13:50

Re: opendmarc.service failed - kills mailscanner?

Post by asulkowski »

Have the same issue:

Feb 27 08:27:31 efa postfix/qmqpd[51847]: 4K5w835hTcz54b4W: client=localhost[127.0.0.1]
Feb 27 08:27:31 efa opendmarc[51850]: /etc/opendmarc.conf:
Feb 27 08:27:31 efa postfix/cleanup[50275]: 4K5w835hTcz54b4W: message-id=<9f8398f6629d4ca88f5b53faf3bd754c@ex13-1.dmawi.de>
Feb 27 08:27:31 efa opendkim[97825]: 4K5w835hTcz54b4W: DKIM verification successful
Feb 27 08:27:31 efa opendmarc[51850]: 4K5w835hTcz54b4W: ignoring invalid ARC-Authentication-Results header "i=1;#012#011smtpd-out;#012#011none"
Feb 27 08:27:31 efa postfix/cleanup[50275]: warning: milter inet:localhost:8893: can't read SMFIC_BODYEOB reply packet header: Success
Feb 27 08:27:31 efa postfix/cleanup[50275]: 4K5w835hTcz54b4W: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 4.7.1 Service unavailable - try again later; from=<xxxxx.xxxxx@awi.de> to=<xxx@xxx.xxx.xx> proto=QMQP

I resolved it by add host awi.de to /etc/opendmarc/ignore.hosts
aztek0
Posts: 3
Joined: 21 Dec 2021 20:49

Re: opendmarc.service failed - kills mailscanner?

Post by aztek0 »

It is interesting that the email that breaks opendmarc comes from the same domain in asulkowski's case as mine.

At this point in time, is the email saved somewhere? I would think the email has not even been accepted at all so no. But there are people here that know more than I do.

If the email is retrievable, then we could dissect the issue.

Is there extra debugging levels on opendmarc so we can have more verbosity? I am willing to remove the band-aid to see if we can get the info.

My opendmarc has not died since I blocked that domain...so early stages of DoS?

Regards,
A.
User avatar
shawniverson
Posts: 3647
Joined: 13 Jan 2014 23:30
Location: Indianapolis, Indiana USA
Contact:

Re: opendmarc.service failed - kills mailscanner?

Post by shawniverson »

So, I would love to get my hands on a sample.

I also notice that this is occurring in the post scanning phase.

There's no reason for opendmarc to verify during this phase. We should add 127.0.0.1 and ::1 to ignore.hosts
asulkowski
Posts: 2
Joined: 10 Mar 2022 13:50

Re: opendmarc.service failed - kills mailscanner?

Post by asulkowski »

aztek0 wrote: 11 Mar 2022 21:48 It is interesting that the email that breaks opendmarc comes from the same domain in asulkowski's case as mine.

At this point in time, is the email saved somewhere? I would think the email has not even been accepted at all so no. But there are people here that know more than I do.

If the email is retrievable, then we could dissect the issue.

Is there extra debugging levels on opendmarc so we can have more verbosity? I am willing to remove the band-aid to see if we can get the info.

My opendmarc has not died since I blocked that domain...so early stages of DoS?

Regards,
A.
Hi, nessesary this e-mail never arrive to our mailserver and eFa. But after when I myself fixed this issue this e-mail correctly income, so if still interested I can give you some e-mail headers, please give me feedback if need it.
efa@kalthof.net
Posts: 6
Joined: 16 May 2023 17:42

Re: opendmarc.service failed - kills mailscanner?

Post by efa@kalthof.net »

Hi there,

I still get the same error on eFa 4.0.4

As mentioned before the sender domain in my case is awi.de, too!

the mentioned bug fixes describe other errors.

Is there some weird bug, if the sender domain is awi.de?

Yours

Gerald
SteveC
Posts: 6
Joined: 16 May 2023 19:16

Re: opendmarc.service failed - kills mailscanner?

Post by SteveC »

Can confirm that it has happened to our eFa install as well.

Blocked the awi.de domain for now.

Steve

Edit: more debug info added

First crash was in libbsd.so.0.8.3 at offset E5D4 I don't have the disassembly for this.

Second crash was in libbsd.so.0.11.7 at offset CA84 with the disassembly below.

000000000000ca70 <strlcpy@@LIBBSD_0.0>:
ca70: 48 85 d2 test %rdx,%rdx
ca73: 74 3b je cab0 <strlcpy@@LIBBSD_0.0+0x40>
ca75: 48 8d 4c 17 ff lea -0x1(%rdi,%rdx,1),%rcx
ca7a: 48 89 f0 mov %rsi,%rax
ca7d: eb 14 jmp ca93 <strlcpy@@LIBBSD_0.0+0x23>
ca7f: 90 nop
ca80: 48 83 c0 01 add $0x1,%rax
ca84: 0f b6 50 ff movzbl -0x1(%rax),%edx
ca88: 48 83 c7 01 add $0x1,%rdi
ca8c: 84 d2 test %dl,%dl
ca8e: 88 57 ff mov %dl,-0x1(%rdi)
ca91: 74 15 je caa8 <strlcpy@@LIBBSD_0.0+0x38>
ca93: 48 39 cf cmp %rcx,%rdi
ca96: 75 e8 jne ca80 <strlcpy@@LIBBSD_0.0+0x10>
ca98: 48 83 c0 01 add $0x1,%rax
ca9c: c6 07 00 movb $0x0,(%rdi)
ca9f: 80 78 ff 00 cmpb $0x0,-0x1(%rax)
caa3: 75 13 jne cab8 <strlcpy@@LIBBSD_0.0+0x48>
caa5: 0f 1f 00 nopl (%rax)
caa8: 48 29 f0 sub %rsi,%rax
caab: 48 83 e8 01 sub $0x1,%rax
caaf: c3 retq
cab0: 48 89 f0 mov %rsi,%rax
cab3: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
cab8: 48 83 c0 01 add $0x1,%rax
cabc: 80 78 ff 00 cmpb $0x0,-0x1(%rax)
cac0: 75 f6 jne cab8 <strlcpy@@LIBBSD_0.0+0x48>
cac2: 48 29 f0 sub %rsi,%rax
cac5: 48 83 e8 01 sub $0x1,%rax
cac9: c3 retq
caca: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)

Edit2: Here's the segfault in case I didn't do the offset calculations correct, and to see the message generated.

May 16 11:39:52 XXX.XXX.XXX opendmarc[22710]: 4QLL5h57Fhz3wmX: ignoring invalid ARC-Authentication-Results header "i=1;
smtpd-out;
none"
May 16 11:39:52 XXX.XXX.XXX kernel: opendmarc[3717]: segfault at 0 ip 00007f26ae9cfa84 sp 00007f26a4fd4118 error 4 in libbsd.so.0.11.7[7f26ae9c3000+14000]
Post Reply