TLS Interception Side Effects

I ran into a tricky problem the other day at $WORK and thought it would be a good idea to share it here, if at least to make a note of it for later.

We had some devices on a client’s network that wouldn’t connect back to our servers anymore. As always in this case, nobody changed anything but it stopped working nonetheless.
Anyway, looking at our logs I noticed some weird HTTP 408 errors. First time I saw that so I went looking for an explanation on the Internets. Turns out it indicates that a client initiated a TCP connection to the server but didn’t send HTTP traffic before the server timeout kicked in.
On Apache, this timeout is set by default to 20s. I captured some traffic with tcpdump to check what was happening. Indeed the server was closing the TCP connection after 20s.
Now I should point out that our devices connect over HTTPS. I got the logs from one of those devices. They showed the connection failed because the device couldn’t negociate the TLS session. There was a server certificate error. Weird because we make sure to embed our TLS certs in the devices themselves.

After showing the customer we were clean, he started to investigate as well.
He came back to us explaining that our devices didn’t support his SSL inspection equipment. He had to exclude our devices traffic from it. They were now getting back online.

I knew SSL inspection (or TLS interception) was a thing but never bothered to research how it’s actually working. So now that I have, here is a quick explanation.

Our customer had to deploy a TLS certificate to every devices he wanted to inspect the traffic of. That TLS certificate has been generated by his SSL inspection equipment. It enables it to perform what is basically a man-in-the-middle attack on all corporate TLS traffic.

From the workstations with the extra certificate, the TLS negociation went like this:

But from our devices, here is where it went south:

In this case the device can’t validate the proxy certificate because it doesn’t trust it.

So that’s how it works. Fairly simple. In fact, so simple that it was an eye opener for me.
Whether it’s an extra CA certificate pushed to your browser by your IT or maliciously added to a browser CA bundle, there are different ways to perform TLS interception. And victims users are not always aware of that. The good news is that there are also ways to detect it.

From the client side:

• You can check the name of the CA that issued the certificate. You would need to know if it changed though.
• You can check some well known TLS certificate fingerprints (tool). That principle would also work if the attacker managed to get a CA to issue a certificate for the domain you’re visiting. Your browser would think it’s valid but the fingerprint would be different.

Web servers can also detect it by comparing User-Agent headers and the way the client performs TLS negocation (paper).

I have mixed feelings about this topic though.
On one hand, as we see unencrypted communications slowly disappear from the Internet, there are some legitimate use cases for being able to inspect TLS sessions. Malware detection being one.
On the other hand, I still feel this is an attack on privacy and we at least need to make it more obvious when TLS interception is happening.
This has to be built into web browsers like certificate validation aka “green lock” is now. It would make people aware that their traffic could be intercepted. Then they could decide whether the network they’re on is secure enough for their communications.