[squid-users] Squid 4.3: SSL Bump fails to send client certificate

[Sid]

Hi,

I have following Squid version installed on CentOS 7:
[***@localhost ~]# squid -v
Squid Cache: Version 4.3
Service Name: squid

This binary uses OpenSSL 1.0.2k-fips 26 Jan 2017. For legal restrictions on
distribution see https://www.openssl.org/source/license.html

configure options: '--prefix=/usr' '--includedir=/usr/include'
'--datadir=/usr/share' '--bindir=/usr/sbin' '--libexecdir=/usr/lib/squid'
'--localstatedir=/var' '--sysconfdir=/etc/squid' '--with-openssl'
'--enable-ssl-crtd'

Squid.conf:
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
#acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN)
#acl localnet src 10.0.0.0/8 # RFC 1918 local private network
(LAN)
#acl localnet src 100.64.0.0/10 # RFC 6598 shared address space
(CGN)
#acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly
plugged) machines
#acl localnet src 172.16.0.0/12 # RFC 1918 local private network
(LAN)
#acl localnet src 10.133.64.0/22 # RFC 1918 local private
network (LAN)
acl localnet src 20.20.64.0/24
acl localnet src 192.168.1.0/24
#acl localnet src 10.133.65.0/22
#acl localnet src 10.133.66.0/22
#acl localnet src 10.133.67.0/22
#acl localnet src fc00::/7 # RFC 4193 local private network
range
#acl localnet src fe80::/10 # RFC 4291 link-local (directly
plugged) machines

acl SSL_ports port 443
acl SSL_ports port 8443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 8443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
#http_port 3128

http_port 3128 ssl-bump \
cert=/usr/local/squid/etc/ssl_cert/myCA.pem \
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB

# For squid 3.5.x
#sslcrtd_program /usr/local/squid/libexec/ssl_crtd -s /var/lib/ssl_db -M 4MB

# For squid 4.x
sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s
/var/lib/ssl_db -M 4MB

acl step1 at_step SslBump1

ssl_bump peek step1
ssl_bump bump all

tls_outgoing_options cafile=/usr/local/squid/etc/UCAppsCA.pem
sslproxy_foreign_intermediate_certs /usr/local/squid/etc/UCAppsCA.pem

# Uncomment and adjust the following to add a disk cache directory.
cache_effective_user squid
cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /usr/local/squid/var/cache/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Browser & HTTP UA Client connections are working with SSL bump properly; but
except for one connection.

Server sends certificate with SNI; but squid forwards it as IP Address to
client
1540529604.672 49 20.20.64.91 NONE/200 0 CONNECT 20.20.64.56:443 -
HIER_DIRECT/20.20.64.56 -

When I took wireshark on Squid; I can see Squid sends:

61 Alert (Level: Fatal, Description: Internal Error) to certificate sent by
Server

cache.log:
2018/10/30 12:55:47 kid1| ERROR: negotiating TLS on FD 21:
error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify
failed (1/-1/0)
2018/10/30 12:55:47 kid1| ERROR: negotiating TLS on FD 21:
error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify
failed (1/-1/0)

This is an internal server with its own CA certs.




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html

[Alex Rousskov]

Post by Sid
http_port 3128 ssl-bump \
cert=/usr/local/squid/etc/ssl_cert/myCA.pem \
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
ssl_bump peek step1
ssl_bump bump all
Browser & HTTP UA Client connections are working with SSL bump properly; but
except for one connection.
Server sends certificate with SNI; but squid forwards it as IP Address to
client

Two problems with that statement:

1. Servers never send SNI. Clients usually send SNI. Squid should
forward SNI it received from the client to the server, provided the
client actually sent SNI. Did your client send SNI?

2. Bugs notwithstanding, the implied order of events is not what
actually happens: Squid, as configured, does _not_ forward anything from
the server certificate to the client. Squid, as configured, generates a
certificate based on client-supplied information (not server-supplied
information). After sending that generated certificate to the client,
Squid establishes a TLS connection with the server.
For an accurate picture, in addition to Squid-server and server-Squid
traffic, look at what Squid has received from the client and what Squid
sent to the client, all in actual order.

Post by Sid
61 Alert (Level: Fatal, Description: Internal Error) to certificate sent by
Server
error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify
failed (1/-1/0)
error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify
failed (1/-1/0)
This is an internal server with its own CA certs.

Is your Squid configured to trust those internal CAs? If not, Squid
would not be able to validate the server certificate.


HTH,

Alex.

[Sid]

Thank you Alex for the reply.

Alex: 1. Servers never send SNI. Clients usually send SNI. Squid should
forward SNI it received from the client to the server, provided the client
actually sent SNI. Did your client send SNI?

Sid: I can see in Client Hello IP Address being sent by Client; so there is
no SNI from client itself.

Alex: 2. Bugs notwithstanding, the implied order of events is not what
actually happens: Squid, as configured, does _not_ forward anything from the
server certificate to the client. Squid, as configured, generates a
certificate based on client-supplied information (not server-supplied
information). After sending that generated certificate to the client, Squid
establishes a TLS connection with the server.

Sid: Thank you for explanation.

Alex: For an accurate picture, in addition to Squid-server and server-Squid
traffic, look at what Squid has received from the client and what Squid sent
to the client, all in actual order.

Sid: I took wireshark on Squid server (centOS 7); I took 2 wiresharks
between Client & Squid and then between Squid & Server. I can see client
being sent fake cert generated by Squid & client responds with "Client key
Exchange", "Change cipher spec", "Encrypted Handshake Message". But I can't
see actual client certificate sent to Squid. Is there a way to decypt in
Wireshark. In Wireshark between Squid & Server I can see Squid responding
with "61 Alert (Level: Fatal, Description: Internal Error)".

Alex: Is your Squid configured to trust those internal CAs? If not, Squid
would not be able to validate the server certificate.

Sid: I have added those chained certificates as following in squid.conf
tls_outgoing_options cafile=/usr/local/squid/etc/UCAppsCA.pem
sslproxy_foreign_intermediate_certs /usr/local/squid/etc/UCAppsCA.pem




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html

[Alex Rousskov]

Post by Sid
Sid: I took wireshark on Squid server (centOS 7); I took 2 wiresharks
between Client & Squid and then between Squid & Server. I can see client
being sent fake cert generated by Squid & client responds with "Client key
Exchange", "Change cipher spec", "Encrypted Handshake Message".

Sounds good. Does the generated fake certificate contain the right
origin server name?

Post by Sid
But I can't see actual client certificate sent to Squid.

Why do you expect the client to send a client certificate to Squid? In
most deployments, TLS servers do not request client certificates and,
hence, TLS clients do not send client certificates. IIRC, you did not
configure your Squid to request a client certificate from the client?

Or is there a terminology problem where "client certificate sent to
Squid" means something other than "an x509 certificate requested by a
TLS server and sent to that server by a TLS client during TLS
handshake"? Please note that Squid is a TLS server in this context.

Post by Sid
Is there a way to decypt in Wireshark.

Yes, there are several ways, including giving Wireshark your Squid's
private certificate key. Sorry, I do not have detailed instructions.
Please note that the encrypted part probably does not matter -- in most
cases prior to TLS v1.3, it is the plain text Hellos that are important
when it comes to bumping the connection.

Post by Sid
In Wireshark between Squid & Server I can see Squid responding
with "61 Alert (Level: Fatal, Description: Internal Error)".
Alex: Is your Squid configured to trust those internal CAs? If not, Squid
would not be able to validate the server certificate.
Sid: I have added those chained certificates as following in squid.conf
tls_outgoing_options cafile=/usr/local/squid/etc/UCAppsCA.pem
sslproxy_foreign_intermediate_certs /usr/local/squid/etc/UCAppsCA.pem

Perhaps the alert may not be related to certificate validation. If you
want to verify whether UCAppsCA.pem is enough to trust the origin
server, you can use "curl" or "openssl s_client" tools for a test. They
should fail to validate the server when not configured to use
UCAppsCA.pem and they should succeed otherwise.


HTH,

Alex.

[Sid]

Thank you Alex.

Sounds good. Does the generated fake certificate contain the right origin

server name?
Sid: Yes, It does contain correct IP Address in Server name sent by client.

Why do you expect the client to send a client certificate to Squid? In most

deployments, TLS servers do not request client certificates and, hence, TLS
clients do not send client certificates. IIRC, you did not configure your
Squid to request a client certificate from the client?

Or is there a terminology problem where "client certificate sent to

Squid" means something other than "an x509 certificate requested by a
TLS server and sent to that server by a TLS client during TLS
handshake"? Please note that Squid is a TLS server in this context.

Sid: Actually in my case Server is looking for a certificate to be sent by
client; it isn't a Web Server but SBC looking for a certificate sent by
a client to grant further voice & video call. How to configure Squid to get
this certificate from client for mutual authentication?

Perhaps the alert may not be related to certificate validation. If you want

to verify whether UCAppsCA.pem is enough to trust the origin server, you can
use "curl" or "openssl s_client" tools for a test. They should fail to
validate the server when not configured to use UCAppsCA.pem and they should
succeed otherwise.

Sid: I have tried following which shows "Verify return code: 0 (ok)":
openssl s_client -connect <Server FQDN>:443 -CAfile
/usr/local/squid/etc/UCAppsCA.pem






--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html

[Amos Jeffries]

Post by Sid
Thank you Alex.

Sounds good. Does the generated fake certificate contain the right origin

server name?
Sid: Yes, It does contain correct IP Address in Server name sent by client.

Alex asked about *name*. IP address is not part of the considerations
because using a raw-IP is not valid for SNI. Even though having one in
the cert "name" is valid it is not supposed to happen either.

Also by "right" he means that Squid is passing on either the *same* name
value from the client SNI (bumping at step 2) or from the real server
provided certificate (bumping at step 3).

Post by Sid

Why do you expect the client to send a client certificate to Squid? In most

deployments, TLS servers do not request client certificates and, hence, TLS
clients do not send client certificates. IIRC, you did not configure your
Squid to request a client certificate from the client?

Or is there a terminology problem where "client certificate sent to

Squid" means something other than "an x509 certificate requested by a
TLS server and sent to that server by a TLS client during TLS
handshake"? Please note that Squid is a TLS server in this context.
Sid: Actually in my case Server is looking for a certificate to be sent by
client; it isn't a Web Server but SBC looking for a certificate sent by
a client to grant further voice & video call. How to configure Squid to get
this certificate from client for mutual authentication?

Configure clientca= on the http(s)_port directive.
see <http://www.squid-cache.org/Doc/config/http_port/>

IIRC that should work when SSL-Bump functionality re-purposes the
cafile= option which was supposed to be the CA for client certificates.

Post by Sid

Perhaps the alert may not be related to certificate validation. If you want

to verify whether UCAppsCA.pem is enough to trust the origin server, you can
use "curl" or "openssl s_client" tools for a test. They should fail to
validate the server when not configured to use UCAppsCA.pem and they should
succeed otherwise.
openssl s_client -connect <Server FQDN>:443 -CAfile
/usr/local/squid/etc/UCAppsCA.pem

Amos

[Alex Rousskov]

This post might be inappropriate. Click to display it.

[Sid]

Thank you Amos and Alex for great help & support so far.

As per suggestions I have added lot more parameters in squid.conf for both
"http" & "tls_outgoing_options" directives:

http_port 3128 ssl-bump \
tls-cert=/usr/local/squid/etc/ssl_cert/myCA.pem \

cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!DH:!ADH
\
options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE \
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB \
tls-cafile=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
tls-dh=prime256v1:/usr/local/squid/etc/dhparam.pem \
tls-dh=secp384r1:/usr/local/squid/etc/dhparam.pem

tls_outgoing_options \
default-ca=off \
cafile=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE \

cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!DH:!ADH
\
flags=DONT_VERIFY_DOMAIN \
flags=DONT_VERIFY_PEERi \
min-version=1.2

Now, when I look into wireshark between Server <--> Squid; I no longer see
error: 61 Alert (Level: Fatal, Description: Internal Error) sent by Squid

Only issue is Squid sends:
2018-11-02_151705.jpg
<Loading Image...>

How to make Squid send certificate in it?






--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html

[Alex Rousskov]

Post by Sid
tls_outgoing_options \
default-ca=off \
cafile=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE \
http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377591/2018-11-02_151705.jpg
How to make Squid send certificate in it?

I believe I have answered that question in my previous email[1]. Your
current tls_outgoing_options appear to ignore my answer.

Please note that I have not tested whether tls_outgoing_options cert=...
key=.... mentioned in [1] works as it should in your environment, but
Squid documentation implies that it works. If it does not work, you may
want to file a bug report.

Alex.
[1]
http://lists.squid-cache.org/pipermail/squid-users/2018-November/019613.html