Page 1 of 3

opendmarc.service failed - kills mailscanner?

Posted: 10 Jun 2021 06:40
by 1an3
Hi
Overnight and a few times this morning my efa box has died - maillog shows 4.7.1 please try later to all connection attempts.
This is logged in /var/log/messages

Jun 9 23:37:51 Hyena kernel: opendmarc[23006]: segfault at 0 ip 00007f19a91b34a5 sp 00007f19a6c1c1a8 error 4 in libc-2.17.so[7f19a9072000+1c4000]
Jun 9 23:37:51 Hyena systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 9 23:37:51 Hyena systemd: Unit opendmarc.service entered failed state.
Jun 9 23:37:51 Hyena systemd: opendmarc.service failed.

If I start it, it works.

It appears that on my system opendmarc was updated recently

yum.log
Jun 07 05:51:13 Updated: libopendmarc.x86_64 1.4.1-1.el7
Jun 07 05:51:13 Updated: opendmarc.x86_64 1.4.1-1.el7

Please can someone point this novice at troubleshooting this?

Thanks

Re: opendmarc.service failed - kills mailscanner?

Posted: 10 Jun 2021 17:13
by forhire
I'm seeing the same issue. Started on the 7th for me.

Jun 7 07:11:37 smtp systemd: opendmarc.service: main process exited, code=killed, status=6/ABRT
Jun 7 07:11:37 smtp systemd: Unit opendmarc.service entered failed state.
Jun 7 07:11:37 smtp systemd: opendmarc.service failed.
Jun 9 08:57:35 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 9 08:57:35 smtp systemd: Unit opendmarc.service entered failed state.
Jun 9 08:57:35 smtp systemd: opendmarc.service failed.
Jun 9 20:23:21 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 9 20:23:21 smtp systemd: Unit opendmarc.service entered failed state.
Jun 9 20:23:21 smtp systemd: opendmarc.service failed.
Jun 10 01:35:48 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 10 01:35:48 smtp systemd: Unit opendmarc.service entered failed state.
Jun 10 01:35:48 smtp systemd: opendmarc.service failed.

I'm looking at this thread which suggests it might be due to dns recursion and/or DKIM.
viewtopic.php?t=3771

Maybe someone familiar with this error will chime in. ;)

Re: opendmarc.service failed - kills mailscanner?

Posted: 10 Jun 2021 18:13
by 1an3
I don’t *think* I have dns recursion enabled, as when I did I couldn’t lookup my relays/smarthost and I wasn’t clever enough to fix it. :)

Fingers crossed it’s been stable since 7.30 this morning, it initially died half 11 last night , then I fixed it at 5.15 and a couple of times afterwards.

Re: opendmarc.service failed - kills mailscanner?

Posted: 10 Jun 2021 18:41
by forhire
I toggled DKIM but haven't tried anything else. It's working right now so only time will tell.

Re: opendmarc.service failed - kills mailscanner?

Posted: 10 Jun 2021 20:21
by forhire
This is my last failure of opendmarc. The failure this morning was opendkim.
Jun 10 12:49:13 smtp kernel: opendmarc[21249]: segfault at 0 ip 00007f8add13f269 sp 00007f8ad8c551a8 error 4 in libc-2.17.so[7f8add0af000+1c4000]
Jun 10 12:49:13 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 10 12:49:13 smtp systemd: Unit opendmarc.service entered failed state.
Jun 10 12:49:13 smtp systemd: opendmarc.service failed.

I restarted opendmarc using:
sudo systemctl start opendmarc

Re: opendmarc.service failed - kills mailscanner?

Posted: 10 Jun 2021 20:30
by 1an3
Same - that’s my fix. It just feels strange that the only thing to have changed is that opendmarc has recently updated. Maybe now only a specific circumstance has found a bug. Like fastly :)

Re: opendmarc.service failed - kills mailscanner?

Posted: 11 Jun 2021 16:16
by forhire
It died again. I'm thinking about rolling the RPM back unless I can figure you what is causing 1.4.1-1 crash. 1.3.2-1 has been stable for months. I've spent some time reading through the release notes but nothing obvious is jumping out.

Packages Altered:
Updated libopendmarc-1.3.2-1.el7.x86_64 @epel
Update 1.4.1-1.el7.x86_64 @epel
Updated opendmarc-1.3.2-1.el7.x86_64 @epel
Update 1.4.1-1.el7.x86_64 @epel

Re: opendmarc.service failed - kills mailscanner?

Posted: 11 Jun 2021 16:33
by forhire
I enabled auto restart in /etc/opendmarc.conf. I'll check the logs later today and see if it restarts as it should.

AutoRestart true
AutoRestartCount 0
AutoRestartRate 10/1h

I've been monitoring my logs using this:
sudo grep opend /var/log/messages | grep 'failed state'

Re: opendmarc.service failed - kills mailscanner?

Posted: 11 Jun 2021 16:46
by 1an3
I have used systemctl to auto recover it

systemctl edit opendmarc then paste in:
[Service] Restart=always

Re: opendmarc.service failed - kills mailscanner?

Posted: 12 Jun 2021 02:52
by forhire
I just ran yum and installed a pile of updates. I was fully updated yesterday when I checked. This update includes kernel-3.10.0-1160.31.1.el7.x86_64 update which I suspect may solve problem. We'll see. ;)

Re: opendmarc.service failed - kills mailscanner?

Posted: 12 Jun 2021 12:54
by 1an3
I appear to be automatically yumming updates on (gulp) and it’s installed what I suspect you got. Seems to be working but I haven’t checked /log/messages to see if opendmarc is dying.

Re: opendmarc.service failed - kills mailscanner?

Posted: 12 Jun 2021 14:57
by forhire
No joy. It crashed overnight. The issue persists.

Re: opendmarc.service failed - kills mailscanner?

Posted: 12 Jun 2021 15:17
by 1an3
My Monday morning job (or tomorrow if I can’t sleep!) will be to check through the maillog to see if it’s a certain connection or action immediately before it dies. :x

Re: opendmarc.service failed - kills mailscanner?

Posted: 12 Jun 2021 20:15
by forhire
Just prior to opendmarc entering a failed state I'm seeing this error:
Jun 11 23:13:21 smtp postfix/cleanup[17792]: warning: milter inet:localhost:8893: can't read SMFIC_BODYEOB reply packet header: Success

And they have all been related to a spam email originating from:
client=os3-362-14218.vs.sakura.ne.jp[133.167.64.222]

Now I need to see if I can capture the message to examine it.

Re: opendmarc.service failed - kills mailscanner?

Posted: 14 Jun 2021 10:33
by 1an3
After trawling through logs from over the weekend, about a hundred instances of opendmarc failing, apart from the first couple, they all seem to immediately follow a connect from 1 ip, cortew3.mexicanafinanciero.com.mx[213.156.145.39], followed by an error in maillog

Code: Select all

can't read SMFIC_BODYEOB reply packet header: Success
followed by

Code: Select all

OpenDMARC Filter v1.4.1 starting
I've now blocked that IP in the boundary firewall and haven't seen a failure since (around 90mins).

Did you have any joy in capturing your suspected trigger?

Re: opendmarc.service failed - kills mailscanner?

Posted: 14 Jun 2021 14:31
by forhire
Unfortunately I haven't seen it since. I've been up for 48 hours.

This is being discussed over on opendmarc with a source patch but it hasn't been put in the rpm.
https://github.com/trusteddomainproject ... issues/179

Re: opendmarc.service failed - kills mailscanner?

Posted: 14 Jun 2021 22:33
by MattS
Just returned from 10 days leave to find we've got the same problem too. Started last Thursday evening by the sound of it. Hadn't even realised there was auto updating going on, other than for clamav.

Re: opendmarc.service failed - kills mailscanner?

Posted: 14 Jun 2021 23:03
by bostjanc
Hi.
Same issue here.

This was updated recently:
Jun 10 23:40:38 Updated: libopendmarc-1.4.1-1.el7.x86_64
Jun 10 23:40:38 Updated: opendmarc-1.4.1-1.el7.x86_64

I dont know if this is some cron job on EFA?

We had to block this IP's on firewall:
cortew3.mexicanafinanciero.com.mx[213.156.145.39]
client=os3-362-14218.vs.sakura.ne.jp[133.167.64.222]

But probably this will be only good for a short time, before those f*kers change their IP or use a different VPN provider.

any suggestions ?

Re: opendmarc.service failed - kills mailscanner?

Posted: 15 Jun 2021 02:17
by forhire
I attempted to build an updated RPM with the new patch but I must not have all my dependencies sorted as I'm seeing a LOT of errors during the build. Has anyone had success with the patch?

https://patch-diff.githubusercontent.co ... /178.patch

Re: opendmarc.service failed - kills mailscanner?

Posted: 15 Jun 2021 05:24
by 1an3
forhire wrote: 15 Jun 2021 02:17 I attempted to build an updated RPM with the new patch but I must not have all my dependencies sorted as I'm seeing a LOT of errors during the build. Has anyone had success with the patch?

https://patch-diff.githubusercontent.co ... /178.patch
Sorry this sort of thing is way beyond me!

Re: opendmarc.service failed - kills mailscanner?

Posted: 17 Jun 2021 08:26
by b19wll
This also started yesterday for me, I disabled DMARC and DKIM functionality. Mail started flowing again, I enabled DMARC/DKIM again and it ran for about 5 hours, then stopped at 19:16 last night and realised this morning when I was getting messages from my users. So I disabled it again and it all seems ok
Are there any know fixes yet?

Re: opendmarc.service failed - kills mailscanner?

Posted: 17 Jun 2021 08:29
by 1an3
Nothing hit the repos as far as I know.

There are a couple of workarounds in the previous posts ^
Set opendmarc to auo-restart either in its own conf file or with systemctl
work out what IP Address[es] cause the crash and block them with firewall.

Re: opendmarc.service failed - kills mailscanner?

Posted: 17 Jun 2021 08:51
by bostjanc
Hi
Can we get any instructions on how to achieve this on CentOS 7:
"Set opendmarc to auo-restart either in its own conf file or with systemctl"

From last conversation with Shawn the current status for EFA fix: Still working on it...technically opendmarc project hasn't released an update yet so need to integrate a patch and test

Re: opendmarc.service failed - kills mailscanner?

Posted: 17 Jun 2021 08:58
by 1an3
I did this, YMMV

Code: Select all

systemctl edit opendmarc
then paste in:

Code: Select all

[Service] Restart=always

Re: opendmarc.service failed - kills mailscanner?

Posted: 17 Jun 2021 10:08
by b19wll
I have done the below, but mail still seems to stop flowing after a period of time. Any other things I can try please?
1an3 wrote: 17 Jun 2021 08:58 I did this, YMMV

Code: Select all

systemctl edit opendmarc
then paste in:

Code: Select all

[Service] Restart=always