Discussion:
[squid-users] Strange delays (30 seconds) with TLS connections in WCCP/Transparent mode
Christophe Fillot
2017-01-23 11:57:32 UTC
Permalink
Hello all,

I have a strange problem where some TLS connections are delayed by 30
seconds when going through my transparent proxy with WCCP. This occurs
typically with sites behind Cloudflare (for example,
https://www.wireshark.org). No problem for Google websites for example.

I only want to log the SNI hostname, I do *not* want to
intercept/decrypt/re-encrypt connections with fake certificates.

Here is the setup:

- Linux Debian 7
- OpenSSL 1.0.1t
- Squid 3.5.23 (also tested Squid 4.0.17).

The Squid configure options are:

./configure --enable-delay-pools --with-large-files --enable-async-io
--enable-icmp --enable-kill-parent-hack
--enable-htpc --enable-forw-via-db --enable-cache-digests
--enable-dl-malloc --with-large-files
--enable-linux-netfilter --enable-ssl --enable-ssl-crtd --with-openssl

IPTables configuration (the routing device sending WCCP frames is a
Cisco ASA):

iptables -t nat -A PREROUTING -i wccp0 -p tcp -m tcp --dport 443 -j DNAT
--to-destination 195.83.155.53:3130

The output/public interface is eth0, the traffic returns to clients
through eth1 with the following:

iptables -t mangle -A OUTPUT -p tcp --sport 3130 -j MARK --set-mark 900
ip rule add fwmark 900 table 1

(The table 1 allows direct access to client networks with the
appropriate routes, this is needed because the return traffic must not
go through the ASA).

The SQUID configuration:

https_port 3130 intercept ssl-bump cert=/usr/local/squid/etc/proxyCA.pem

acl step1 at_step SslBump1
acl step2 at_step SslBump2
ssl_bump peek step1 all
ssl_bump splice step2 all

sslcrtd_program /usr/local/squid/libexec/ssl_crtd -s
/usr/local/squid/var/ssl_db -M 40MB
sslcrtd_children 5
wccp2_service dynamic 70 password=XXXXXXXXX
wccp2_service_info 70 protocol=tcp flags=src_ip_hash,dst_ip_hash
priority=240 ports=443

Even if I use only "ssl_bump splice all" the 30-second delay happens.

I have an example of captured traffic here, on the client and the
various network interfaces on the proxy server:
http://www.utc.fr/filex/get?k=6Dt169xGsHMswCKEF5L

As you can see in the captures, Squid returns the "Server Hello" 30
seconds (in cap_eth1.cap) after it has received it from the server (in
cap_eth0.cap).

This behavior is not systematic, sometimes the data is returned
immediately. What could cause this delay ? This looks like some timeout,
but for what reason ?

Thanks in advance for any suggestion !

Christophe
Christophe Fillot
2017-01-24 11:18:00 UTC
Permalink
Sorry for the noise, I was able to find the cause: we use "dstdomain"
ACLs and Squid does reverse lookups.

It seems that Cloudflare DNS servers do not respond to PTR requests, and
since Squid has the default "dns_timeout" value to 30 seconds...:

$ host www.wireshark.org
www.wireshark.org has address 104.25.219.21
www.wireshark.org has address 104.25.218.21
www.wireshark.org has IPv6 address 2400:cb00:2048:1::6819:da15
www.wireshark.org has IPv6 address 2400:cb00:2048:1::6819:db15

$ host 104.25.219.21
Host 21.219.25.104.in-addr.arpa not found: 2(SERVFAIL)

$ dig @arin.authdns.ripe.net. in ns 21.219.25.104.in-addr.arpa.
[...]
;; AUTHORITY SECTION:
25.104.in-addr.arpa. 86400 IN NS ns1.cloudflare.com.
25.104.in-addr.arpa. 86400 IN NS ns2.cloudflare.com.

$ dig @ns1.cloudflare.com. in ptr 21.219.25.104.in-addr.arpa.
[...]
;; connection timed out; no servers could be reached

Best regards,

Christophe

Loading...