[squid-users] Whitelist ONLY exception isn't working correctly

[Martin Hanson]

Hi,

I have a setup with a PF firewall that intercepts HTTP and HTTPS traffic and forwards that to Squid. Squid is setup to log all traffic and it uses a SSL bump for the HTTPS traffic.

In the setup I have a whitelist of domains that doesn't get logged, the rest of the traffic gets logged and redirected to SquidGuard for further blacklisting. All that works great.

What I cannot figure out is how to add a couple of local IP addresses that can ONLY access the whitelist (or possibly ANOTHER whitelist) and nothing else on the Internet.

The ACL for the "windows_boxes" is the one that is supposed to ONLY access the whitelisted domains. However, when I enter the URL https://www.mojang.com I get a "Access denied". However, if I add a NON-SSL domain to the whitelist, then those works.

This is my current squid.conf. I know I am overlooking something, but I cannot figure out what I am doing wrong.

<SNIP>
acl step1 at_step SslBump1
acl localnet src 192.168.1.0/24

# These boxes may ONLY access the whitelist.
acl windows_boxes src 192.168.1.201 192.168.1.202

acl whitelist ssl::server_name .mojang.com .minecraft.net d2pi0bc9ewx28h.cloudfront.net mcupdate.tumblr.com minecraft-textures-1196058387.us-east-1.elb.amazonaws.com .steampowered.com .steamcommunity.com .steamgames.com .steamusercontent.com .steamcontent.com .steamstatic.com .akamaihd.net .ubuntu.com

# We don't want the whitelist to be cached.
cache deny whitelist

# We want direct access on the whitelist.
always_direct allow whitelist

# Don't redirect to SquidGuard.
redirector_access deny whitelist

# We only redirect HTTP and HTTPS.
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# !!! THIS ISN'T WORKING !!! ubuntu.com, mojang.com still gets blocked on these boxes.
http_access deny windows_boxes !whitelist

http_access allow localhost
http_access allow localnet

http_access deny all

# We'll intercept trafic using PF.
http_port 127.0.0.1:3129 intercept
https_port 127.0.0.1:3130 intercept ssl-bump cert=/etc/squid/ssl_cert/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
sslproxy_cafile /usr/local/openssl/cabundle.file

# Become a TCP tunnel without decrypting proxied traffic for the whitelist.
ssl_bump splice whitelist
ssl_bump peek step1
ssl_bump bump all

# We want the query strings as well.
strip_query_terms off

# Leave coredumps in the first cache dir
coredump_dir /var/squid/cache

redirect_program /usr/local/bin/squidGuard -c /etc/squidguard/squidguard.conf
</SNIP>

Any feedback would be greatly appreciated! Thank you in advance!

Kindest regards.

[Alex Rousskov]

Post by Martin Hanson
# !!! THIS ISN'T WORKING !!! ubuntu.com, mojang.com still gets blocked on these boxes.
http_access deny windows_boxes !whitelist

...

Post by Martin Hanson
http_access deny all

You have no rules that allow windows_boxes to access whitelist servers,
and you have a "deny all" default. Consider this alternative:

http_access allow windows_boxes whitelist
http_access deny windows_boxes
...
http_access deny all


HTH,

Alex.

[Amos Jeffries]

Post by Alex Rousskov

Post by Martin Hanson
# !!! THIS ISN'T WORKING !!! ubuntu.com, mojang.com still gets blocked on these boxes.
http_access deny windows_boxes !whitelist

...

Post by Martin Hanson
http_access deny all

You have no rules that allow windows_boxes to access whitelist servers,

I don't think that is the problem. There is "allow localnet" which
includes those windows_boxes.

It is probably !whitelist matching true for the initial CONNECT before
ssl::server_name is available.

Amos

[Martin Hanson]

I also tried the above, but the result is the same.

[Amos Jeffries]

Post by Martin Hanson
This is my current squid.conf. I know I am overlooking something, but I cannot figure out what I am doing wrong.

The comments on this config tell a story of some misunderstandings ...

Post by Martin Hanson
<SNIP>
acl step1 at_step SslBump1
acl localnet src 192.168.1.0/24
# These boxes may ONLY access the whitelist.
acl windows_boxes src 192.168.1.201 192.168.1.202
acl whitelist ssl::server_name .mojang.com .minecraft.net d2pi0bc9ewx28h.cloudfront.net mcupdate.tumblr.com minecraft-textures-1196058387.us-east-1.elb.amazonaws.com .steampowered.com .steamcommunity.com .steamgames.com .steamusercontent.com .steamcontent.com .steamstatic.com .akamaihd.net .ubuntu.com
# We don't want the whitelist to be cached.
cache deny whitelist

"store_miss deny whitelist" is the current way to do this.

Post by Martin Hanson
# We want direct access on the whitelist.
always_direct allow whitelist

"DIRECT" access is always allowed, all this does is prevents cache_peer
being used. You have no cache_peer - so the above line is pointless.

Post by Martin Hanson
# Don't redirect to SquidGuard.
redirector_access deny whitelist

Er. The idea that SquidGuard is redirected *to* is wrong. SG is the
agent _doing_ the redirection.

What the above does is prevent SG having any part of the whitelist
transactions. And redirector_access was deprecated by url_rewrite_access
years ago.

Post by Martin Hanson
# We only redirect HTTP and HTTPS.
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager
# !!! THIS ISN'T WORKING !!! ubuntu.com, mojang.com still gets blocked on these boxes.
http_access deny windows_boxes !whitelist
http_access allow localhost
http_access allow localnet
http_access deny all
# We'll intercept trafic using PF.
http_port 127.0.0.1:3129 intercept
https_port 127.0.0.1:3130 intercept ssl-bump cert=/etc/squid/ssl_cert/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
sslproxy_cafile /usr/local/openssl/cabundle.file
# Become a TCP tunnel without decrypting proxied traffic for the whitelist.
ssl_bump splice whitelist
ssl_bump peek step1
ssl_bump bump all
# We want the query strings as well.
strip_query_terms off
# Leave coredumps in the first cache dir
coredump_dir /var/squid/cache
redirect_program /usr/local/bin/squidGuard -c /etc/squidguard/squidguard.conf

redirect_program is deprecated by url_rewrite_program.

Also, Squidguard is deprecated. Convert basic SG actions into squid.conf
actions where possible. And/or, use ufdbguard as a direct drop-in
replacement that can handle HTTPS and modern Squid helper protocols.

Amos

[Martin Hanson]

Post by Amos Jeffries

Post by Martin Hanson
This is my current squid.conf. I know I am overlooking something, but I cannot figure out what I am doing wrong.

The comments on this config tell a story of some misunderstandings ...

Thank you for all the valuable feedback. I originally set this up years ago. I have updated the different options.

However, I am still faced with the same problem.

If I add a HTTP domain to the whitelist, then the whitelist works for the HTTP domains ONLY, but the domains in the list which are HTTPS based are still blocked.

I have also changed the "localnet" in order to avoid any "clashes" between the boxes, but that doesn't change anything.

This is my updated configuration:

Sitting on one of the "windows_boxes" and going to https://www.ubuntu.com/ gives: "Access Denied" from Squid.

<SNIP>
max_filedesc 4096

acl step1 at_step SslBump1
#acl localnet src 192.168.1.0/24
acl localnet src 192.168.1.2-192.168.1.200

# These boxes may ONLY access the whitelist.
acl windows_boxes src 192.168.1.201 192.168.1.202

acl whitelist ssl::server_name .mojang.com .minecraft.net d2pi0bc9ewx28h.cloudfront.net mcupdate.tumblr.com minecraft-textures-1196058387.us-east-1.elb.amazonaws.com .steampowered.com .steamcommunity.com .steamgames.com .steamusercontent.com .steamcontent.com .steamstatic.com .akamaihd.net .launchpad.net .ubuntu.com

# We don't want these to be cached.
store_miss deny whitelist

# Don't let SquidGuard do anything with the whitelisted domains.
url_rewrite_access deny whitelist

# We only redirect HTTP and HTTPS.
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# THIS ISN'T WORKING!!!
# https://www.ubuntu.com is blocked with "Access Denied" from Squid.
http_access allow windows_boxes whitelist

http_access allow localhost
http_access allow localnet

http_access deny all

# We'll intercept trafic using PF from clan.
http_port 127.0.0.1:3129 intercept
https_port 127.0.0.1:3130 intercept ssl-bump cert=/etc/squid/ssl_cert/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB

sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
sslproxy_cafile /usr/local/openssl/cabundle.file

# Become a TCP tunnel without decrypting proxied traffic for the whitelist.
ssl_bump splice whitelist
ssl_bump peek step1
ssl_bump bump all

# We want the query strings as well.
strip_query_terms off

# Leave coredumps in the first cache dir
coredump_dir /var/squid/cache

redirect_program /usr/local/bin/squidGuard -c /etc/squidguard/squidguard.conf
</SNIP>

Thanks for all the help!

Kind regards.

[Alex Rousskov]

Post by Martin Hanson
# THIS ISN'T WORKING!!!
# https://www.ubuntu.com is blocked with "Access Denied" from Squid.
http_access allow windows_boxes whitelist

I suspect the request is blocked during SslBump step1 because there is
not enough information in the fake CONNECT request for ssl::server_name
to match ubuntu.com. Please keep in mind that ssl::server_name does not
do (reverse) DNS lookups, and the fake CONNECT request during step1 only
has an IP address, not a domain name.

One way to test this theory is to (temporary) http_access allow CONNECT
requests to (ubuntu) IP addresses. Does that get you to SslBump step2,
where the fake CONNECT usually gets a domain name?


HTH,

Alex.

[Amos Jeffries]

Post by Alex Rousskov

Post by Martin Hanson
# THIS ISN'T WORKING!!!
# https://www.ubuntu.com is blocked with "Access Denied" from Squid.
http_access allow windows_boxes whitelist

I suspect the request is blocked during SslBump step1 because there is
not enough information in the fake CONNECT request for ssl::server_name
to match ubuntu.com. Please keep in mind that ssl::server_name does not
do (reverse) DNS lookups, and the fake CONNECT request during step1 only
has an IP address, not a domain name.
One way to test this theory is to (temporary) http_access allow CONNECT
requests to (ubuntu) IP addresses. Does that get you to SslBump step2,
where the fake CONNECT usually gets a domain name?

Alex:
since you mentioned earlier that the SSL-Bump info based ACLs should
work in following transaction access controls should this work?

acl step1 at_step SslBump1
http_access allow CONNECT step1


Amos

[Alex Rousskov]

Post by Amos Jeffries
should this work?
acl step1 at_step SslBump1
http_access allow CONNECT step1

Yes, step1 should work in http_access IMO, but I do not know whether it
does. According to the latest Amish email, it does work, which is good news!


Cheers,

Alex.

[Martin Hanson]

Post by Alex Rousskov

Post by Martin Hanson
# THIS ISN'T WORKING!!!
# https://www.ubuntu.com is blocked with "Access Denied" from Squid.
http_access allow windows_boxes whitelist

I suspect the request is blocked during SslBump step1 because there is
not enough information in the fake CONNECT request for ssl::server_name
to match ubuntu.com. Please keep in mind that ssl::server_name does not
do (reverse) DNS lookups, and the fake CONNECT request during step1 only
has an IP address, not a domain name.
One way to test this theory is to (temporary) http_access allow CONNECT
requests to (ubuntu) IP addresses. Does that get you to SslBump step2,
where the fake CONNECT usually gets a domain name?

Hi Alex,

That makes sense and it seems you're right.

I tried adding (the IP's the box currently sees as ubuntu.com, checked the log that it didn't change):

acl ubuntu dst 91.189.89.103 91.189.89.110
http_access allow CONNECT ubuntu

Then it works!

How do I fix it then?

Kind regards.

[Martin Hanson]

I have enabled debugging and found something quite strange.

In order to better debug I have limited the whitelist to two domains, one HTTP and one with HTTPS:

acl whitelist ssl::server_name .ubuntu.com .sundkat.dk

When I go to http://www.sundkat.dk, which is a HTTP domain, I get the following:

2018/05/14 02:42:49.859 kid1| 85,2| src/client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.sundkat.dk/ is ALLOWED; last ACL checked: whitelist

But when I go to https://www.ubuntu.com, I get the following:

2018/05/14 02:43:44.262 kid1| 85,2| src/client_side_request.cc(745) clientAccessCheckDone: The request CONNECT 91.189.89.103:443 is DENIED; last ACL checked: all

It's like when the traffic is HTTP the whitelist is working, but when the traffic is HTTPS the whitelist isn't working. But this is ONLY for the "windows_boxes", for everything else it's working as it should.

I don't understand what's going on here.

I am re-posting my entire squid.conf here again just to keep things complete:

<SNIP>
debug_options ALL,2

max_filedesc 4096

acl step1 at_step SslBump1
acl localnet src 192.168.1.2-192.168.1.200

# These boxes may ONLY access the whitelist.
acl windows_boxes src 192.168.1.201 192.168.1.202

acl whitelist ssl::server_name .ubuntu.com .sundkat.dk

# We don't want these to be cached.
store_miss deny whitelist

# Don't let SquidGuard do anything with the whitelisted domains.
url_rewrite_access deny whitelist

# We only redirect HTTP and HTTPS.
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# THIS ISN'T WORKING!!!
http_access allow windows_boxes whitelist

http_access allow localhost
http_access allow localnet

http_access deny all

# We'll intercept trafic using PF from clan.
http_port 127.0.0.1:3129 intercept
https_port 127.0.0.1:3130 intercept ssl-bump cert=/etc/squid/ssl_cert/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB

sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
sslproxy_cafile /usr/local/openssl/cabundle.file

# Become a TCP tunnel without decrypting proxied traffic for the whitelist.
ssl_bump splice whitelist
ssl_bump peek step1
ssl_bump bump all

# We want the query strings as well.
strip_query_terms off

# Leave coredumps in the first cache dir
coredump_dir /var/squid/cache

redirect_program /usr/local/bin/squidGuard -c /etc/squidguard/squidguard.conf
</SNIP>

Kind regards.

[Amos Jeffries]

Post by Martin Hanson
I have enabled debugging and found something quite strange.
acl whitelist ssl::server_name .ubuntu.com .sundkat.dk
2018/05/14 02:42:49.859 kid1| 85,2| src/client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.sundkat.dk/ is ALLOWED; last ACL checked: whitelist
2018/05/14 02:43:44.262 kid1| 85,2| src/client_side_request.cc(745) clientAccessCheckDone: The request CONNECT 91.189.89.103:443 is DENIED; last ACL checked: all
It's like when the traffic is HTTP the whitelist is working, but when the traffic is HTTPS the whitelist isn't working.

Yes, that is exactly what is happening.


* When intercepting HTTP (port 80) traffic the protocol is HTTP. Squid
is receiving messages generated by the client *naming* the server it
wants to connect with, OR with just a raw-IP if client wants to do it
that way.

* When handling explicit proxy (port 3128) traffic the protocol is HTTP.
Squid is receiving CONNECT messages generated by the client again
*naming* the server it wants to connect with, OR with just a raw-IP if
client wants to do it that way.

* When intercepting HTTPS (port 443) traffic the protocol is initially
just TCP. Squid is receiving TCP SYN packet and fakes/generates a
CONNECT message to represent this opaque connection (ie. CONNECT to a
raw-IP).

If (and only if) a CONNECT is itself allowed into the proxy does
SSL-Bump begin for the TLS wrapped inside that message. That goes for
both types of CONNECT message - Squid or client generated.

It should be obvious from the above why you see different behaviour for
the two methods of using the proxy.

Post by Martin Hanson
But this is ONLY for the "windows_boxes", for everything else it's working as it should.
I don't understand what's going on here.

If the fake CONNECT with raw-IP at SSL-Bump step1 is not allowed to go
through the proxy then the TLS handshake cannot even start to happen. So
there will never be a ssl::server_name for the whitelist ACL to match.

Now that you have altered localnet to exclude the *.201 and *.202 IPs
the "allow localnet" is no longer permitting them to use the proxy.
AND the whitelist ACL is still not matching the raw-IPs which occur in
CONNECT messages. Which leaves Squid with "deny all".

Amos

[Martin Hanson]

Post by Amos Jeffries

Post by Martin Hanson
It's like when the traffic is HTTP the whitelist is working, but when the traffic is HTTPS the whitelist isn't working.

Yes, that is exactly what is happening.
* When intercepting HTTP (port 80) traffic the protocol is HTTP. Squid
is receiving messages generated by the client *naming* the server it
wants to connect with, OR with just a raw-IP if client wants to do it
that way.
* When handling explicit proxy (port 3128) traffic the protocol is HTTP.
Squid is receiving CONNECT messages generated by the client again
*naming* the server it wants to connect with, OR with just a raw-IP if
client wants to do it that way.
* When intercepting HTTPS (port 443) traffic the protocol is initially
just TCP. Squid is receiving TCP SYN packet and fakes/generates a
CONNECT message to represent this opaque connection (ie. CONNECT to a
raw-IP).
If (and only if) a CONNECT is itself allowed into the proxy does
SSL-Bump begin for the TLS wrapped inside that message. That goes for
both types of CONNECT message - Squid or client generated.
It should be obvious from the above why you see different behaviour for
the two methods of using the proxy.

Post by Martin Hanson
But this is ONLY for the "windows_boxes", for everything else it's working as it should.
I don't understand what's going on here.

If the fake CONNECT with raw-IP at SSL-Bump step1 is not allowed to go
through the proxy then the TLS handshake cannot even start to happen. So
there will never be a ssl::server_name for the whitelist ACL to match.
Now that you have altered localnet to exclude the *.201 and *.202 IPs
the "allow localnet" is no longer permitting them to use the proxy.
AND the whitelist ACL is still not matching the raw-IPs which occur in
CONNECT messages. Which leaves Squid with "deny all".
Amos

Thank you very very much Alex and Amos for all the help!

For future references, if anyone needs this, this is the working config:

<SNIP>
acl step1 at_step SslBump1

acl localnet src 192.168.1.0/24

# These boxes may ONLY access the whitelist.
acl windows_boxes src 192.168.1.201 192.168.1.202

acl whitelist ssl::server_name .mojang.com .minecraft.net d2pi0bc9ewx28h.cloudfront.net mcupdate.tumblr.com minecraft-textures-1196058387.us-east-1.elb.amazonaws.com .steampowered.com .steamcommunity.com .steamgames.com .steamusercontent.com .steamcontent.com .steamstatic.com .akamaihd.net .launchpad.net .ubuntu.com

# We don't want these to be cached.
store_miss deny whitelist

# Don't let SquidGuard do anything with the whitelisted domains.
url_rewrite_access deny whitelist

# We only redirect HTTP and HTTPS.
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

# We need this for the whitelist for the windows boxes because
# requests are blocked during SslBump step1 because there is not
# enough information in the fake CONNECT request for ssl::server_name
# to match domains in the whitelist.
http_access allow CONNECT step1

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# Windows boxes are only allowed access to the whitelist.
http_access allow windows_boxes whitelist
http_access deny windows_boxes

http_access allow localhost
http_access allow localnet

http_access deny all

http_port 127.0.0.1:3129 intercept
https_port 127.0.0.1:3130 intercept ssl-bump cert=/etc/squid/ssl_cert/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB

sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
sslproxy_cafile /usr/local/openssl/cabundle.file

# Become a TCP tunnel without decrypting proxied traffic for the whitelist.
ssl_bump splice whitelist
ssl_bump peek step1 all
ssl_bump bump all

# We want the query strings as well.
strip_query_terms off

# Leave coredumps in the first cache dir
coredump_dir /var/squid/cache

redirect_program /usr/local/bin/squidGuard -c /etc/squidguard/squidguard.conf
</SNIP>

Kind regards