In my last post I wrote about a problem in my company with missing mails. We really had a hard time to debug this and only succeeded in analysing the network data stream.

Turning on a very high level of debuging for the Exchange SMTP server just showed, that we got contacted by Digi-Key’s mail servers. Everything looked ok, but nothing got delivered and no message in the logs anywhere. Even the message tracking didn’t show anything about those mails (I assume message tracking is written after SMTP finishes, but it didn’t).

Next step was to look at the network traffic. Here our network structure made it easy for us (even though it will become clear, that it is part of the problem), as it is easy to read everything that goes to our mail server on the server, that runs a PPTP daemon. What did we see?

Digi-Key tried to contact us using an MTU of 1400 bytes. But the PPTP link just had an MTU of 1396 bytes. So it did the right thing and sent back an ICMP error package asking for the server to lower it’s package size. But the server didn’t react and resent everything again with an MTU of 1400 bytes. Resulting again in an ICMP error package. And so on.

We discovered an PMTUD Black Hole.

Usually those black holes are result of badly configured firewalls that drop every ICMP packet sent to it. But it is bad to drop ICMP packets that are needed for Path MTU discovery!

The easy fix is to increase the MTU of our PPTP connection. Unfortunately we just found a very hackish way on the Linux side (here it is easy). On the Exchange side runs a Windows 2k3 Server with RRAS. There I didn’t find a way to increase the MTU of the PPTP connection. So it does work, but we are not sure if this is the real solution :(

The better solution would be to fix all firewalls on the net (which I think is impossible). I really wonder how often something similar happens. PPTP connections (or other VPN connections) aren’t so unusual, are they?

One great part of this story still needs to be told.

If you read the initial post an look at the first comment you’ll see, that after about 4 hours someone from Digi-Key commented!

Here I have to explain, why we didn’t contact Digi-Key at first. Or better: we did a bit (I asked someone at the support hotline and they did sent a test email, which didn’t arrive). The problem was that until then I could only tell: “Your mails don’t arrive”. Now I can tell more and I hope that the communication with Digi-Key helps to solve this problem for more people than us.

But this showed us again, why we like Digi-Key so much. (Besides ordering on a Friday at 19:00 and getting the parts on Monday at 10:00 from the USA to Germany.)

So where’s UPS? They have the same problem…