Monitoring Software - delayed responses

Questions and answers about how to do stuff
Post Reply
paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Monitoring Software - delayed responses

Post by paulo88 » 06 Dec 2017 16:19

Hello,

I am currently in the process of setting up eFa in our environment.

It works great so far, but I have one issue regarding our external monitoring software.
After a few requests from our monitoring software it shows that the response times spikes for a period of time.

Could this be a feature of eFa or postfix, which delays the requests if to many invalid/incomplete connections are detected from an IP?
Our monitoring software only connects to the servers sends a helo, waits for the response and then closes the connection.

Hope someone can tell me if this is indeed a feature as the spikes seem to not be completely random.

Thanks in advance
Paul
Attachments
Capture.PNG
Capture.PNG (265 KiB) Viewed 3731 times

User avatar
pdwalker
Posts: 1299
Joined: 18 Mar 2015 09:16

Re: Monitoring Software - delayed responses

Post by pdwalker » 08 Dec 2017 07:30

How are you testing?

paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Re: Monitoring Software - delayed responses

Post by paulo88 » 12 Dec 2017 07:34

Hello,

the tool opens a SMTP connection on port 25, the delay occurs at this point, only after ~10 seconds I get the first answer from the server.

I checked the maillog and there I can only see the client after the 10 seconds.
Is there another log which tracks connections, that I could check?

Regards

User avatar
pdwalker
Posts: 1299
Joined: 18 Mar 2015 09:16

Re: Monitoring Software - delayed responses

Post by pdwalker » 12 Dec 2017 10:10

And where are you running the check from? Same machine? Remote machine?

If it is a remote machine, how many ip addresses are you connecting from?

User avatar
pdwalker
Posts: 1299
Joined: 18 Mar 2015 09:16

Re: Monitoring Software - delayed responses

Post by pdwalker » 12 Dec 2017 10:11

Also, can you log the full test session, or tell me exactly how you test port 25?

paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Re: Monitoring Software - delayed responses

Post by paulo88 » 12 Dec 2017 10:29

Hello,

check is run from an external machine on the internet.

I just connect to the 25 port using any SMTP client, eg, telnet using port 25 or putty using 25.
As soon as I click connect (or press Enter) I see a blank screen for about 10 seconds and then I get 220 response from postfix.

The problem occurs with different external clients.

User avatar
pdwalker
Posts: 1299
Joined: 18 Mar 2015 09:16

Re: Monitoring Software - delayed responses

Post by pdwalker » 13 Dec 2017 06:18

So, if it is the initial connection that is having an issue, I'd strongly consider there to be some kind of problem between the EFA box and the outside world.

So, I'd look at the route from the internet to the firewall, from the firewall to the EFA box (and every piece of network equipment in between). Is there an IP conflict on your network (not likely, the delays look too regular). Is the firewall doing something funny? Is your EFA instance a VM and the VM host is overloaded during those times?

At first, I suspected that it was the greylisting having an effect, but that wouldn't apply to making a direct connection.

Is there anything in your efa logs, either in your maillog or messages files that might show something funny?

Have you installed any additional software, or changed any security settings in your EFA instance?


In my opinion, there is nothing in a stock EFA instance that would cause this problem. My instance responds (almost) instantly no matter where I connect to it from, as I expect it too.

paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Re: Monitoring Software - delayed responses

Post by paulo88 » 13 Dec 2017 10:13

hello,

I was able to further test the behavior using verbose logging.

postfix can see the connection but for some reason does nothing for exactly 10 seconds, that is too long of a delay for the client and it closes the connection itself.

Here the entry from the session from the log:

Code: Select all

Dec 13 10:55:04 efa postfix/smtpd[12763]: connection established
Dec 13 10:55:04 efa postfix/smtpd[12763]: master_notify: status 0
Dec 13 10:55:04 efa postfix/smtpd[12763]: name_mask: resource
Dec 13 10:55:04 efa postfix/smtpd[12763]: name_mask: software
Dec 13 10:55:14 efa postfix/smtpd[12763]: connect from unknown[81.81.81.81]
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: unknown: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: 81.81.81.81: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: unknown: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: 81.81.81.81: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_hostname: debug_peer_list: unknown ~? 192.168.57.122
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_hostaddr: debug_peer_list: 81.81.81.81 ~? 192.168.57.122
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: unknown: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: 81.81.81.81: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: smtp_stream_setup: maxtime=300 enable_deadline=0
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_hostname: smtpd_client_event_limit_exceptions: unknown ~? 127.0.0.0/8
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_hostaddr: smtpd_client_event_limit_exceptions: 81.81.81.81 ~? 127.0.0.0/8
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: unknown: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: 81.81.81.81: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: send attr request = connect
Dec 13 10:55:14 efa postfix/smtpd[12763]: send attr ident = smtp:81.81.81.81
Dec 13 10:55:14 efa postfix/smtpd[12763]: private/anvil: wanted attribute: status
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute name: status
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute value: 0
Dec 13 10:55:14 efa postfix/smtpd[12763]: private/anvil: wanted attribute: count
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute name: count
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute value: 1
Dec 13 10:55:14 efa postfix/smtpd[12763]: private/anvil: wanted attribute: rate
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute name: rate
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute value: 1
Dec 13 10:55:14 efa postfix/smtpd[12763]: private/anvil: wanted attribute: (list terminator)
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute name: (end)
Dec 13 10:55:14 efa postfix/smtpd[12763]: > unknown[81.81.81.81]: 220 efa.rubicon.eu ESMTP Postfix
Dec 13 10:55:14 efa postfix/smtpd[12763]: xsasl_cyrus_server_create: SASL service=smtp, realm=(null)
Dec 13 10:55:14 efa postfix/smtpd[12763]: name_mask: noanonymous
Dec 13 10:55:14 efa postfix/smtpd[12763]: smtp_get: EOF
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_hostname: smtpd_client_event_limit_exceptions: unknown ~? 127.0.0.0/8
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_hostaddr: smtpd_client_event_limit_exceptions: 81.81.81.81 ~? 127.0.0.0/8
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: unknown: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: match_list_match: 81.81.81.81: no match
Dec 13 10:55:14 efa postfix/smtpd[12763]: send attr request = disconnect
Dec 13 10:55:14 efa postfix/smtpd[12763]: send attr ident = smtp:81.81.81.81
Dec 13 10:55:14 efa postfix/smtpd[12763]: private/anvil: wanted attribute: status
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute name: status
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute value: 0
Dec 13 10:55:14 efa postfix/smtpd[12763]: private/anvil: wanted attribute: (list terminator)
Dec 13 10:55:14 efa postfix/smtpd[12763]: input attribute name: (end)
Dec 13 10:55:14 efa postfix/smtpd[12763]: lost connection after CONNECT from unknown[81.81.81.81]
Dec 13 10:55:14 efa postfix/smtpd[12763]: disconnect from unknown[81.81.81.81] commands=0/0
Dec 13 10:55:14 efa postfix/smtpd[12763]: master_notify: status 1
Dec 13 10:55:14 efa postfix/smtpd[12763]: connection closed
As you can see the TCP connection itself is established (10:55:04), but there is no further action for 10 seconds after that (10:55:14).

I did not made any changes after deploying the VM aside from the initial setup.

What is interesting is that other SMTP connections to other Mailserver (Exchange) do not behave this way.
On the firewall, I also changed the portforwarding destination to another mailserver for testing and this did not show this delay.
Which leads me to the assumption that postfix is the cause.

Another thing I noticed is that the problem only occurs with clients connecting from the Internet. Clients connection internal IP addresses regardless if they are in the same subnet as the efa-Server do not show this problem. Connections that need to be routed also go through the firewall.

So it seems that the delay occurs on the smtpd-postfix service, but only if the client has a public IP address.

EDIT:
I should also mention, that the times when these delays occur are not the same for all clients.
What I mean is that Client1 can have the delay at this moment and Client2 can currently connect without delay. After a few minutes Client2 can connect without delay and Client2 has the delay. There are of course times when all Clients can instantly connect.

User avatar
pdwalker
Posts: 1299
Joined: 18 Mar 2015 09:16

Re: Monitoring Software - delayed responses

Post by pdwalker » 13 Dec 2017 20:08

What are your efa firewall settings?

paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Re: Monitoring Software - delayed responses

Post by paulo88 » 14 Dec 2017 07:11

Do you mean the my firewall in front of the efa or a service installed within the efa appliance?

In front of the efa appliance is a Fortigate Firewall with a simple "Virtual IP" (Portforwarding) setup. No Filters are applied, so traffic is not scanned or filtered.

If you mean a firewall in the efa appliance itself: I have to say that I did not know that there was one installed by default and I have not installed one. If you could tell me what firewall is installed I could check the settings.

Zwabber
Posts: 60
Joined: 14 Feb 2016 21:26

Re: Monitoring Software - delayed responses

Post by Zwabber » 14 Dec 2017 12:24

In the log there is a line who said: "private/anvil: wanted attribute: (list terminator)"
This looks like postfix is thinking it's under a smtp attack because of a wrong authentication/relay

paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Re: Monitoring Software - delayed responses

Post by paulo88 » 15 Dec 2017 11:01

I tried setting "smtpd_client_event_limit_exceptions" to the connecting client IPs so that the anvil ignores these IPs, but that did not help.

Is there another setting I could try.

User avatar
pdwalker
Posts: 1299
Joined: 18 Mar 2015 09:16

Re: Monitoring Software - delayed responses

Post by pdwalker » 18 Dec 2017 04:07

paulo88 wrote:
14 Dec 2017 07:11
Do you mean the my firewall in front of the efa or a service installed within the efa appliance?

In front of the efa appliance is a Fortigate Firewall with a simple "Virtual IP" (Portforwarding) setup. No Filters are applied, so traffic is not scanned or filtered.

If you mean a firewall in the efa appliance itself: I have to say that I did not know that there was one installed by default and I have not installed one. If you could tell me what firewall is installed I could check the settings.
I meant anything and everything.

So your fortigate firewall shouldn't be the issue, and your answer says you haven't changed anything on the efa firewall, but you can run an "iptables --list" to get a list of the EFA firewall rules running, just in case.

Also, is 81.81.81.81 the ip of your monitoring host?

paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Re: Monitoring Software - delayed responses

Post by paulo88 » 19 Dec 2017 07:50

OK, here the path from the Internet to efa:
Internet - Firewall - iptables - efa

Here the iptables output:

Code: Select all

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ndmp
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

paulo88
Posts: 12
Joined: 06 Dec 2017 16:06

Re: Monitoring Software - delayed responses

Post by paulo88 » 02 Jan 2018 12:54

After some tinkering I found out that the problem seems to only occur when I expose the eFa using one specific IP.
If I use any other IP in the same IP Range the problem does not appear.

All firewall and VM settings are the same. I only changed the external IP on the portforwarding, I was not able to find an exact reason for this behavior.
I'll just use another IP.

Thanks for the help.

User avatar
pdwalker
Posts: 1299
Joined: 18 Mar 2015 09:16

Re: Monitoring Software - delayed responses

Post by pdwalker » 04 Jan 2018 06:31

Weird.

Some kind of network issue.

Glad you found a solution, and thanks for updating us.

Post Reply