[squid-users] Skype+intercept+ssl

Discussion:

[squid-users] Skype+intercept+ssl_bump

Evgeniy Kononov

2016-07-15 10:38:03 UTC

Hello!

Can you help me with correct settings for squid to use skype ?

My current config.

# squid -v
Squid Cache: Version 3.5.20
Service Name: squid
configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--verbose' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam,fake' '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP,eDirectory' '--enable-auth-negotiate=kerberos,wrapper' '--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group,LDAP_group,delayer,file_userip,SQL_session,unix_group,session,time_quota' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-storeio=aufs,diskd,ufs,rock' '--enable-wccpv2' '--enable-esi' '--enable-ssl-crtd' '--enable-icmp' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' '--with-included-ltdl' '--disable-arch-native' '--enable-ecap' '--without-nettle' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig' --enable-ltdl-convenience
#cat squid.conf
http_port 3128 options=NO_SSLv3:NO_SSLv2
http_port 192.168.10.240:3125 intercept options=NO_SSLv3:NO_SSLv2
https_port 192.168.10.240:3126 intercept ssl-bump options=ALL:NO_SSLv3:NO_SSLv2 connection-auth=off cert=/opt/squid_certs/squid.pem key=/opt/squid_certs/squid.pem dhparams=/opt/squid_certs/dhparam.pem cipher=HIGH:MEDIUM:RC4:3DES:
always_direct allow all
sslproxy_cert_error allow all
sslproxy_flags DONT_VERIFY_PEER
sslproxy_cafile /etc/pki/tls/certs/ca-bundle.crt
sslproxy_cipher HIGH:MEDIUM:RC4:3DES:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!SRP:!DSS
acl DiscoverSNIHost at_step SslBump1
acl NoSSLIntercept ssl::server_name_regex -i "/etc/squid/lists/url.nobump"
ssl_bump peek DiscoverSNIHost
ssl_bump splice NoSSLIntercept
ssl_bump bump all

#catÂ /etc/squid/lists/url.nobump
microsoft\.com
update\.microsoft\.com
update\.microsoft\.com\.akadns\.net
mobile\.pipe\.aria\.microsoft\.com
prd\.col\.aria.mobile\.skypedata\.akadns\.net
pipe\.skype\.com
pipe\.prd\.skypedata\.akadns\.net
api\.asm\.skype\.com
apps\.skype\.com
wildcard\.skype\.com\.edgekey\.net
e4593\.g\.akamaiedge\.net
\.skype\.com
\.skypeassets\.com
etag\.prod\.registrar\.skype\.com
prod\.registrar\.skype\.com
go\.trouter\.io
With this setup I have problem with group chats, calls and attachments in messages.
Attachments sended, but not delivered to respondent.
Unable to create group chats and if it created, what respondents do not see the chat or can not make calls.
I tried add IP regexp to access list, but after that all https traffic was spliced.
Skype work well when I change ssl_bump bump all to ssl_bump splice all
How can I exclude skype from SSL bumping ?
Thank you.

--
Evgeniy Kononov

Amos Jeffries

2016-07-18 07:27:22 UTC

Permalink

Post by Evgeniy Kononov
Hello!
Can you help me with correct settings for squid to use skype ?

FYI: there are currently no known "correct" setting for Skype when
SSL-Bump is involved.

There are settings known to work when Squid is setup as an explicit
proxy, and some which almost-always (but only 99.999%) working for Squid
intercepting port 80.

Intercepting port 443 and bumping the crypto has issues distinguishing
Skype-TLS from real TLS and HTTPS.

That said, I have been giving it some thought today and suspect that
since MS are apparently filtering Skype traffic through their own
machines these days we could maybe use the "dst" ACL reverse-DNS
behaviour to detect and splice that traffic.

If you want to experiment with that and have good results there are many
here who would like some good news on this.

Post by Evgeniy Kononov
With this setup I have problem with group chats, calls and attachments in messages.
Attachments sended, but not delivered to respondent.
Unable to create group chats and if it created, what respondents do not see the chat or can not make calls.
I tried add IP regexp to access list, but after that all https traffic was spliced.
Skype work well when I change ssl_bump bump all to ssl_bump splice all
How can I exclude skype from SSL bumping ?

The problem is with identifying it in fairly reliable way from all the
other traffic. That is where we are currently all stuck.

Yuri and Eliezer have been trying various things and talking about it
on-list in recent weeks/months. But so far no results I'm confident
about recommending.

Amos

Eliezer Croitoru

2016-07-18 17:52:38 UTC

Permalink

To clear out my idea,

I was thinking about an option to decide if to bump or not based on a SSL handshake test on the destination Service.
I do not know skype traffic that much but I do know that a PTR can be "faked" and have seen it couple times in the past.
I considered what to do and one of the options is to do the bump in two steps and to identify requests that was not supposed to be bumped.
It's a bit complicated since in the nature of the idea there would be at least one failure for the client attempt to reach a destination.
I do not like the idea and I know it's not a nice one but I think that if an admin can identify the goal and determine that he doesn't care about traffic
detained to a specific host for both filtering and caching then all traffic to these hosts can be tunneled or spliced.

The methods I have in mind are:
- Using firewall\kernel level of bumping exceptions
- Using some no-bump external_acl helper

I have a specific model for doing such a thing with Linux ipset and I only need couple domains to evaluate the concept.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: ***@ngtech.co.il

-----Original Message-----
From: squid-users [mailto:squid-users-***@lists.squid-cache.org] On Behalf Of Amos Jeffries
Sent: Monday, July 18, 2016 10:27 AM
To: squid-***@lists.squid-cache.org
Subject: Re: [squid-users] Skype+intercept+ssl_bump

Post by Evgeniy Kononov
Hello!
Can you help me with correct settings for squid to use skype ?

FYI: there are currently no known "correct" setting for Skype when SSL-Bump is involved.

There are settings known to work when Squid is setup as an explicit proxy, and some which almost-always (but only 99.999%) working for Squid intercepting port 80.

Intercepting port 443 and bumping the crypto has issues distinguishing Skype-TLS from real TLS and HTTPS.

That said, I have been giving it some thought today and suspect that since MS are apparently filtering Skype traffic through their own machines these days we could maybe use the "dst" ACL reverse-DNS behaviour to detect and splice that traffic.

If you want to experiment with that and have good results there are many here who would like some good news on this.

The problem is with identifying it in fairly reliable way from all the other traffic. That is where we are currently all stuck.

Yuri and Eliezer have been trying various things and talking about it on-list in recent weeks/months. But so far no results I'm confident about recommending.

Amos

Alex Rousskov

2016-07-18 23:03:24 UTC

Permalink

Post by Amos Jeffries

Post by Evgeniy Kononov
With this setup I have problem with group chats, calls and attachments in messages.

The problem is with identifying it in fairly reliable way from all the
other traffic. That is where we are currently all stuck.

I cannot offer a comprehensive solution for all Skype problems, but I
can share an in-progress triage that we are doing for one particular
problem related to Skype group chats. According to some of the logs I
have seen, group chat uses MSNP(?) messages instead of HTTP. Squid fails

Post by Amos Jeffries
RequestParser.cc(340) parse: Parse buf={length=68, data='CNT 1 CON 185
<connect><ver>2</ver><agent><os>Windows</os><osVer>'}
RequestParser.cc(228) parseHttpVersionField: invalid request-line: not HTTP

AFAICT, Squid then hits an on_unsupported_protocol bug: When deciding
whether to tunnel an intercepted unsupported protocol, Squid never
tunnels traffic on connections that have seen more than one HTTP request
already. The intent behind that check is noble (if a connection started
with a valid HTTP request, then it is probably an HTTP connection), but
the actual result is unfortunate:

1. Intercepted connections start with one or two fake SslBump CONNECT
requests that are counted as "seen HTTP requests".

2. The invalid HTTP request that we just failed to parse is also counted
as a "seen HTTP request".

In the particular case I have seen, once Squid bumps the Skype
connection and receives a non-HTTP MSNP request, the "seen requests"
counter probably reaches 2, and the on_unsupported_protocol option is
not checked.

I am trying to come up with a use case that would justify the current
request counting check. Would switching to a blind tunnel in the middle
of an intercepted connection expose Squid (or its users) to any
_additional_ risks compared to switching to a blind tunnel only when the
bumped connection starts with an invalid HTTP request? If not, it is
trivial to remove the check as the attached patch illustrates.

Thank you,

Alex.

Alex Rousskov

2016-07-30 19:21:22 UTC

Permalink

Post by Alex Rousskov

Post by Amos Jeffries

Post by Evgeniy Kononov
With this setup I have problem with group chats, calls and attachments in messages.

The problem is with identifying it in fairly reliable way from all the
other traffic. That is where we are currently all stuck.

AFAICT, Squid then hits an on_unsupported_protocol bug: When deciding
whether to tunnel an intercepted unsupported protocol, Squid never
tunnels traffic on connections that have seen more than one HTTP request
already. The intent behind that check is noble (if a connection started
with a valid HTTP request, then it is probably an HTTP connection), but
1. Intercepted connections start with one or two fake SslBump CONNECT
requests that are counted as "seen HTTP requests".
2. The invalid HTTP request that we just failed to parse is also counted
as a "seen HTTP request".
In the particular case I have seen, once Squid bumps the Skype
connection and receives a non-HTTP MSNP request, the "seen requests"
counter probably reaches 2, and the on_unsupported_protocol option is
not checked.
I am trying to come up with a use case that would justify the current
request counting check. Would switching to a blind tunnel in the middle
of an intercepted connection expose Squid (or its users) to any
_additional_ risks compared to switching to a blind tunnel only when the
bumped connection starts with an invalid HTTP request?

Update: The question still stands, but we now know more about what
happens if the on_unsupported_protocol bug (in code and/or
documentation, depending on how you look at it) discussed above is
fixed: Squid then starts tunneling traffic as it is told by the
on_unsupported_protocol directive, but forgets to use the existing
encrypted connection to the server and opens/uses a new Squid-to-server
unencrypted connection instead.

Thus, the patch I posted previously does not solve the known Skype
groups/MSNP problem -- it only exposes the next (and bigger!) obstacle
on the way to that solution.

We are working on supporting/fixing tunneling of bumped connections, but
feedback regarding request counting check question above is still welcomed.

Thank you,

Alex.

Marcus Kool

2016-08-27 17:20:48 UTC

Permalink

On 07/30/2016 04:21 PM, Alex Rousskov wrote:
*snip*

Post by Alex Rousskov
Update: The question still stands, but we now know more about what
happens if the on_unsupported_protocol bug (in code and/or
documentation, depending on how you look at it) discussed above is
fixed: Squid then starts tunneling traffic as it is told by the
on_unsupported_protocol directive, but forgets to use the existing
encrypted connection to the server and opens/uses a new Squid-to-server
unencrypted connection instead.
Thus, the patch I posted previously does not solve the known Skype
groups/MSNP problem -- it only exposes the next (and bigger!) obstacle
on the way to that solution.
We are working on supporting/fixing tunneling of bumped connections, but
feedback regarding request counting check question above is still welcomed.
Thank you,
Alex.

I am using squid-4.0.13-20160819-r14813 and have observed the following
with transparent intercept:
1) skype (on windows10) login fails, access.log contains
"CNT error:invalid-request HTTP/1.1" 400 3705 NONE:HIER_NONE -
2) whatsapp (on Android) fails, access.log contains
"NONE error:transaction-end-before-headers HTTP/0.0" 0 0 NONE:HIER_NONE -
"' error:invalid-request HTTP/1.1" 400 3705 NONE:HIER_NONE -
3) Samsung (monitoring?) app on my Samsung smartphone:
"CONNECT 54.76.6.24:80 HTTP/1.1" 403 3775 TCP_DENIED:HIER_NONE Host:%2054.76.6.24:80%0D%0A
"NONE error:invalid-request HTTP/1.1" 400 3705 NONE:HIER_NONE -

TCP_DENIED in 3) is OK since the app connects on port 80 and this port is
not in SSL_ports, but the error message "invalid-request" on the next line
is misleading.

If you need a cache.log with debug ALL,9 I can provide one.

The ssl-bump rules on my server are:
acl tls_s1_connect at_step SslBump1
acl tls_to_splice complex-acl-but-does-not-matter-what-it-has
ssl_bump peek tls_s1_connect
ssl_bump splice tls_to_splice
ssl_bump stare all
ssl_bump bump all

With best regards,

Marcus

Marcus Kool

2016-09-01 19:53:21 UTC

Permalink

Post by Marcus Kool
*snip*

Is there an expected date for a fix?
I volunteer for testing patches.

Marcus

Post by Marcus Kool

Post by Alex Rousskov
Thank you,
Alex.

I am using squid-4.0.13-20160819-r14813 and have observed the following
1) skype (on windows10) login fails, access.log contains
"CNT error:invalid-request HTTP/1.1" 400 3705 NONE:HIER_NONE -
2) whatsapp (on Android) fails, access.log contains
"NONE error:transaction-end-before-headers HTTP/0.0" 0 0 NONE:HIER_NONE -
"' error:invalid-request HTTP/1.1" 400 3705 NONE:HIER_NONE -
"CONNECT 54.76.6.24:80 HTTP/1.1" 403 3775 TCP_DENIED:HIER_NONE Host:%2054.76.6.24:80%0D%0A
"NONE error:invalid-request HTTP/1.1" 400 3705 NONE:HIER_NONE -
TCP_DENIED in 3) is OK since the app connects on port 80 and this port is
not in SSL_ports, but the error message "invalid-request" on the next line
is misleading.
If you need a cache.log with debug ALL,9 I can provide one.
acl tls_s1_connect at_step SslBump1
acl tls_to_splice complex-acl-but-does-not-matter-what-it-has
ssl_bump peek tls_s1_connect
ssl_bump splice tls_to_splice
ssl_bump stare all
ssl_bump bump all
With best regards,
Marcus

Alex Rousskov

2016-09-01 19:59:51 UTC

Permalink

Post by Marcus Kool

Post by Alex Rousskov
We are working on supporting/fixing tunneling of bumped connections

Is there an expected date for a fix?

I hesitate publishing private ETAs because priorities often change but
"in a few weeks" does not sound too outrageous to me right now.

Post by Marcus Kool
I volunteer for testing patches.

Thank you,

Alex.