opendmarc.service failed - kills mailscanner?

Bugs in eFa 4
1an3
Posts: 17
Joined: 07 May 2021 13:05

opendmarc.service failed - kills mailscanner?

Post by 1an3 » 10 Jun 2021 06:40

Hi
Overnight and a few times this morning my efa box has died - maillog shows 4.7.1 please try later to all connection attempts.
This is logged in /var/log/messages

Jun 9 23:37:51 Hyena kernel: opendmarc[23006]: segfault at 0 ip 00007f19a91b34a5 sp 00007f19a6c1c1a8 error 4 in libc-2.17.so[7f19a9072000+1c4000]
Jun 9 23:37:51 Hyena systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 9 23:37:51 Hyena systemd: Unit opendmarc.service entered failed state.
Jun 9 23:37:51 Hyena systemd: opendmarc.service failed.

If I start it, it works.

It appears that on my system opendmarc was updated recently

yum.log
Jun 07 05:51:13 Updated: libopendmarc.x86_64 1.4.1-1.el7
Jun 07 05:51:13 Updated: opendmarc.x86_64 1.4.1-1.el7

Please can someone point this novice at troubleshooting this?

Thanks

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 10 Jun 2021 17:13

I'm seeing the same issue. Started on the 7th for me.

Jun 7 07:11:37 smtp systemd: opendmarc.service: main process exited, code=killed, status=6/ABRT
Jun 7 07:11:37 smtp systemd: Unit opendmarc.service entered failed state.
Jun 7 07:11:37 smtp systemd: opendmarc.service failed.
Jun 9 08:57:35 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 9 08:57:35 smtp systemd: Unit opendmarc.service entered failed state.
Jun 9 08:57:35 smtp systemd: opendmarc.service failed.
Jun 9 20:23:21 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 9 20:23:21 smtp systemd: Unit opendmarc.service entered failed state.
Jun 9 20:23:21 smtp systemd: opendmarc.service failed.
Jun 10 01:35:48 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 10 01:35:48 smtp systemd: Unit opendmarc.service entered failed state.
Jun 10 01:35:48 smtp systemd: opendmarc.service failed.

I'm looking at this thread which suggests it might be due to dns recursion and/or DKIM.
viewtopic.php?t=3771

Maybe someone familiar with this error will chime in. ;)

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 10 Jun 2021 18:13

I don’t *think* I have dns recursion enabled, as when I did I couldn’t lookup my relays/smarthost and I wasn’t clever enough to fix it. :)

Fingers crossed it’s been stable since 7.30 this morning, it initially died half 11 last night , then I fixed it at 5.15 and a couple of times afterwards.

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 10 Jun 2021 18:41

I toggled DKIM but haven't tried anything else. It's working right now so only time will tell.

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 10 Jun 2021 20:21

This is my last failure of opendmarc. The failure this morning was opendkim.
Jun 10 12:49:13 smtp kernel: opendmarc[21249]: segfault at 0 ip 00007f8add13f269 sp 00007f8ad8c551a8 error 4 in libc-2.17.so[7f8add0af000+1c4000]
Jun 10 12:49:13 smtp systemd: opendmarc.service: main process exited, code=killed, status=11/SEGV
Jun 10 12:49:13 smtp systemd: Unit opendmarc.service entered failed state.
Jun 10 12:49:13 smtp systemd: opendmarc.service failed.

I restarted opendmarc using:
sudo systemctl start opendmarc

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 10 Jun 2021 20:30

Same - that’s my fix. It just feels strange that the only thing to have changed is that opendmarc has recently updated. Maybe now only a specific circumstance has found a bug. Like fastly :)

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 11 Jun 2021 16:16

It died again. I'm thinking about rolling the RPM back unless I can figure you what is causing 1.4.1-1 crash. 1.3.2-1 has been stable for months. I've spent some time reading through the release notes but nothing obvious is jumping out.

Packages Altered:
Updated libopendmarc-1.3.2-1.el7.x86_64 @epel
Update 1.4.1-1.el7.x86_64 @epel
Updated opendmarc-1.3.2-1.el7.x86_64 @epel
Update 1.4.1-1.el7.x86_64 @epel

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 11 Jun 2021 16:33

I enabled auto restart in /etc/opendmarc.conf. I'll check the logs later today and see if it restarts as it should.

AutoRestart true
AutoRestartCount 0
AutoRestartRate 10/1h

I've been monitoring my logs using this:
sudo grep opend /var/log/messages | grep 'failed state'

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 11 Jun 2021 16:46

I have used systemctl to auto recover it

systemctl edit opendmarc then paste in:
[Service] Restart=always

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 12 Jun 2021 02:52

I just ran yum and installed a pile of updates. I was fully updated yesterday when I checked. This update includes kernel-3.10.0-1160.31.1.el7.x86_64 update which I suspect may solve problem. We'll see. ;)

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 12 Jun 2021 12:54

I appear to be automatically yumming updates on (gulp) and it’s installed what I suspect you got. Seems to be working but I haven’t checked /log/messages to see if opendmarc is dying.

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 12 Jun 2021 14:57

No joy. It crashed overnight. The issue persists.

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 12 Jun 2021 15:17

My Monday morning job (or tomorrow if I can’t sleep!) will be to check through the maillog to see if it’s a certain connection or action immediately before it dies. :x

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 12 Jun 2021 20:15

Just prior to opendmarc entering a failed state I'm seeing this error:
Jun 11 23:13:21 smtp postfix/cleanup[17792]: warning: milter inet:localhost:8893: can't read SMFIC_BODYEOB reply packet header: Success

And they have all been related to a spam email originating from:
client=os3-362-14218.vs.sakura.ne.jp[133.167.64.222]

Now I need to see if I can capture the message to examine it.

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 14 Jun 2021 10:33

After trawling through logs from over the weekend, about a hundred instances of opendmarc failing, apart from the first couple, they all seem to immediately follow a connect from 1 ip, cortew3.mexicanafinanciero.com.mx[213.156.145.39], followed by an error in maillog

Code: Select all

can't read SMFIC_BODYEOB reply packet header: Success
followed by

Code: Select all

OpenDMARC Filter v1.4.1 starting
I've now blocked that IP in the boundary firewall and haven't seen a failure since (around 90mins).

Did you have any joy in capturing your suspected trigger?

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 14 Jun 2021 14:31

Unfortunately I haven't seen it since. I've been up for 48 hours.

This is being discussed over on opendmarc with a source patch but it hasn't been put in the rpm.
https://github.com/trusteddomainproject ... issues/179

MattS
Posts: 18
Joined: 12 Dec 2017 14:00

Re: opendmarc.service failed - kills mailscanner?

Post by MattS » 14 Jun 2021 22:33

Just returned from 10 days leave to find we've got the same problem too. Started last Thursday evening by the sound of it. Hadn't even realised there was auto updating going on, other than for clamav.

bostjanc
Posts: 160
Joined: 01 Jun 2016 17:18

Re: opendmarc.service failed - kills mailscanner?

Post by bostjanc » 14 Jun 2021 23:03

Hi.
Same issue here.

This was updated recently:
Jun 10 23:40:38 Updated: libopendmarc-1.4.1-1.el7.x86_64
Jun 10 23:40:38 Updated: opendmarc-1.4.1-1.el7.x86_64

I dont know if this is some cron job on EFA?

We had to block this IP's on firewall:
cortew3.mexicanafinanciero.com.mx[213.156.145.39]
client=os3-362-14218.vs.sakura.ne.jp[133.167.64.222]

But probably this will be only good for a short time, before those f*kers change their IP or use a different VPN provider.

any suggestions ?

forhire
Posts: 12
Joined: 10 Jun 2021 16:54

Re: opendmarc.service failed - kills mailscanner?

Post by forhire » 15 Jun 2021 02:17

I attempted to build an updated RPM with the new patch but I must not have all my dependencies sorted as I'm seeing a LOT of errors during the build. Has anyone had success with the patch?

https://patch-diff.githubusercontent.co ... /178.patch

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 15 Jun 2021 05:24

forhire wrote:
15 Jun 2021 02:17
I attempted to build an updated RPM with the new patch but I must not have all my dependencies sorted as I'm seeing a LOT of errors during the build. Has anyone had success with the patch?

https://patch-diff.githubusercontent.co ... /178.patch
Sorry this sort of thing is way beyond me!

b19wll
Posts: 58
Joined: 22 Nov 2012 09:55

Re: opendmarc.service failed - kills mailscanner?

Post by b19wll » 17 Jun 2021 08:26

This also started yesterday for me, I disabled DMARC and DKIM functionality. Mail started flowing again, I enabled DMARC/DKIM again and it ran for about 5 hours, then stopped at 19:16 last night and realised this morning when I was getting messages from my users. So I disabled it again and it all seems ok
Are there any know fixes yet?

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 17 Jun 2021 08:29

Nothing hit the repos as far as I know.

There are a couple of workarounds in the previous posts ^
Set opendmarc to auo-restart either in its own conf file or with systemctl
work out what IP Address[es] cause the crash and block them with firewall.

bostjanc
Posts: 160
Joined: 01 Jun 2016 17:18

Re: opendmarc.service failed - kills mailscanner?

Post by bostjanc » 17 Jun 2021 08:51

Hi
Can we get any instructions on how to achieve this on CentOS 7:
"Set opendmarc to auo-restart either in its own conf file or with systemctl"

From last conversation with Shawn the current status for EFA fix: Still working on it...technically opendmarc project hasn't released an update yet so need to integrate a patch and test

1an3
Posts: 17
Joined: 07 May 2021 13:05

Re: opendmarc.service failed - kills mailscanner?

Post by 1an3 » 17 Jun 2021 08:58

I did this, YMMV

Code: Select all

systemctl edit opendmarc
then paste in:

Code: Select all

[Service] Restart=always

b19wll
Posts: 58
Joined: 22 Nov 2012 09:55

Re: opendmarc.service failed - kills mailscanner?

Post by b19wll » 17 Jun 2021 10:08

I have done the below, but mail still seems to stop flowing after a period of time. Any other things I can try please?
1an3 wrote:
17 Jun 2021 08:58
I did this, YMMV

Code: Select all

systemctl edit opendmarc
then paste in:

Code: Select all

[Service] Restart=always

Post Reply