[squid-users] Is there any way to cache or forward https requests to an http proxy using Squid?

[Brett]

I currently have squid setup to use a self-signed certificate for MITM to
cache HTTPS requests. This works. If an item is not in the cache I want to
request from an online proxy like Crawlera. Unfortunately Crawlera only
offer an http endpoint. When I try to forward to this endpoint, everything
works for HTTP, but for HTTPS I received the error: Handshake with SSL
server failed: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol

I'm using squid 4.2. Is there a way I can configure squid so I can specify
it as a proxy for an https request and then have it act as a cache or
forward to an HTTP proxy (that supports CONNECT)? If at some point I'm
transmitting in plain text it doesn't matter at all for this application.

The following is my configuration for Squid:

http_port 3128 ssl-bump \
cert=/apps/server_crt.pem key=/apps/server_key.pem \
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /apps/squid/libexec/security_file_certgen -s
/apps/squid/var/lib/ssl_db -M 4MB sslcrtd_children 8 startup=1 idle=1
acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump bump all
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 1025-65535 # unregistered ports
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
coredump_dir /apps/squid/var/cache
maximum_object_size 10 GB
cache_dir ufs /apps/squid/var/cache/squid 100 16 256
cache_mem 256 MB
maximum_object_size_in_memory 512 KB
cache_replacement_policy heap LFUDA
range_offset_limit -1
quick_abort_min -1 KB
offline_mode on
http_access allow localnet
http_access allow localhost
http_access deny all
refresh_pattern . 525600 100% 525600 ignore-reload ignore-no-store
ignore-private ignore-auth ignore-must-revalidate store-stale

cache_peer proxy.crawlera.com parent 8010 0 ssl login=APIKEY:
never_direct allow all

Update


If I change the ssl_bump directives above to the following:

acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3

ssl_bump stare step2
ssl_bump bump step3

An HTTPS request will tunnel all the way through both proxies to the target
and correctly return the response to the caller, but it no longer has MITM
access at the Squid proxy to cache the results, so they CONNECT though to
Crawlera on subsequent requests for the same resource. HTTP on the other
hand will go through both proxies if it's not in the cache, otherwise it
does get returned from the cache.

This is still not the solution I'm looking for though, I would like to cache
HTTPS as well.



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html

[Alex Rousskov]

Post by Brett
I currently have squid setup to use a self-signed certificate for MITM to
cache HTTPS requests. This works. [...]
Is there a way I can configure squid so I can specify
it as a proxy for an https request and then have it act as a cache or
forward to an HTTP proxy (that supports CONNECT)?

AFAICT, you are asking about the missing "SslBump with cache_peer"
feature, which was covered in several recent threads, including this email:

http://lists.squid-cache.org/pipermail/squid-users/2018-July/018653.html

Post by Brett
ssl_bump peek step1
ssl_bump bump all

This configuration bumps everything at step2.

Post by Brett
ssl_bump stare step2
ssl_bump bump step3

This (misleading!) configuration should splice everything at step1. In
other words, it should be equivalent to this (clear) configuration:

ssl_bump splice all

or a disabled SslBump. According to your tests, that is exactly what
happens (and the lack of non-trivial SslBump involvement probably
explains why peering works in this corner case).


If you need more information about the equivalence of the last two
configurations, please consider studying the following wiki page and a
related recent email thread:

* https://wiki.squid-cache.org/Features/SslPeekAndSplice
*
http://lists.squid-cache.org/pipermail/squid-users/2018-September/019162.html


HTH,

Alex.

[Brett Anderson]

Thank you!

I reverted back to:

ssl_bump peek step1
ssl_bump bump all

And then based on that first link you sent me I rebuilt my Squid instance
from
https://github.com/measurement-factory/squid/tree/SQUID-360-peering-for-SslBump

Then tested and I think it's working now?

From my access log:
# testing https
# first request
1537477894.828 310 172.27.0.3 NONE/200 0 CONNECT foo.com:443 -
FIRSTUP_PARENT/64.58.117.175 -
1537477895.645 797 172.27.0.3 TCP_MISS/200 32374 GET
https://foo.com/js/bootstrap.min.js - FIRSTUP_PARENT/64.58.117.175
application/javascript
# second request
1537477899.009 336 172.27.0.3 NONE/200 0 CONNECT foo.com:443 -
FIRSTUP_PARENT/64.58.117.175 -
1537477899.019 0 172.27.0.3 TCP_MEM_HIT/200 32384 GET
https://foo.com/js/bootstrap.min.js - HIER_NONE/- application/javascript

# testing http
# first request
1537477956.088 1051 172.27.0.3 TCP_MISS/200 28203 GET
http://websites.web.com/ - FIRSTUP_PARENT/64.58.117.175 text/html
# second request
1537477957.888 2 172.27.0.3 TCP_MEM_HIT/200 28198 GET
http://websites.web.com/ - HIER_NONE/- text/html

Should I change anything else for more improvement? Should I build from the
master or a more recent branch of https://github.com/measurement-factory
<https://github.com/measurement-factory/squid/tree/SQUID-360-peering-for-SslBump>
?

Thanks again!
B.

On Thu, Sep 20, 2018 at 12:47 PM Alex Rousskov <

Post by Alex Rousskov

Post by Brett
I currently have squid setup to use a self-signed certificate for MITM to
cache HTTPS requests. This works. [...]
Is there a way I can configure squid so I can specify
it as a proxy for an https request and then have it act as a cache or
forward to an HTTP proxy (that supports CONNECT)?

AFAICT, you are asking about the missing "SslBump with cache_peer"
http://lists.squid-cache.org/pipermail/squid-users/2018-July/018653.html

Post by Brett
ssl_bump peek step1
ssl_bump bump all

This configuration bumps everything at step2.

Post by Brett
ssl_bump stare step2
ssl_bump bump step3

This (misleading!) configuration should splice everything at step1. In
ssl_bump splice all
or a disabled SslBump. According to your tests, that is exactly what
happens (and the lack of non-trivial SslBump involvement probably
explains why peering works in this corner case).
If you need more information about the equivalence of the last two
configurations, please consider studying the following wiki page and a
* https://wiki.squid-cache.org/Features/SslPeekAndSplice
*
http://lists.squid-cache.org/pipermail/squid-users/2018-September/019162.html
HTH,
Alex.

[Alex Rousskov]

Should I build from the master or a more recent branch?

IIRC, the unofficial branch you are using is the only branch containing
SslBump with cache_peer" feature today. We are working on submitting
that code for the official review. Please note that any unofficial code
comes with additional risks and is not eligible for the official Squid
Project support.

Alex.

Post by Brett
I currently have squid setup to use a self-signed certificate for

MITM to

Post by Brett
cache HTTPS requests. This works. [...]
Is there a way I can configure squid so I can specify
it as a proxy for an https request and then have it act as a cache or
forward to an HTTP proxy (that supports CONNECT)?

AFAICT, you are asking about the missing "SslBump with cache_peer"
http://lists.squid-cache.org/pipermail/squid-users/2018-July/018653.html

Post by Brett
ssl_bump peek step1
ssl_bump bump all

This configuration bumps everything at step2.

Post by Brett
ssl_bump stare step2
ssl_bump bump step3

This (misleading!) configuration should splice everything at step1. In
  ssl_bump splice all
or a disabled SslBump. According to your tests, that is exactly what
happens (and the lack of non-trivial SslBump involvement probably
explains why peering works in this corner case).
If you need more information about the equivalence of the last two
configurations, please consider studying the following wiki page and a
* https://wiki.squid-cache.org/Features/SslPeekAndSplice
*
http://lists.squid-cache.org/pipermail/squid-users/2018-September/019162.html
HTH,
Alex.

[Brett Anderson]

Thanks again Alex,

For anyone else trying to solve this issue, here's a repo I created which
sets everything up in Docker to allow ssl_bump and cache_peer to work.
https://github.com/brett--anderson/squid_proxy

On Fri, Sep 21, 2018 at 7:53 AM Alex Rousskov <

Post by Alex Rousskov

Should I build from the master or a more recent branch?

IIRC, the unofficial branch you are using is the only branch containing
SslBump with cache_peer" feature today. We are working on submitting
that code for the official review. Please note that any unofficial code
comes with additional risks and is not eligible for the official Squid
Project support.
Alex.

Post by Brett
I currently have squid setup to use a self-signed certificate for

MITM to

Post by Brett
cache HTTPS requests. This works. [...]
Is there a way I can configure squid so I can specify
it as a proxy for an https request and then have it act as a cache

or

Post by Brett
forward to an HTTP proxy (that supports CONNECT)?

AFAICT, you are asking about the missing "SslBump with cache_peer"

http://lists.squid-cache.org/pipermail/squid-users/2018-July/018653.html

Post by Brett
ssl_bump peek step1
ssl_bump bump all

This configuration bumps everything at step2.

Post by Brett
ssl_bump stare step2
ssl_bump bump step3

This (misleading!) configuration should splice everything at step1.

In

ssl_bump splice all
or a disabled SslBump. According to your tests, that is exactly what
happens (and the lack of non-trivial SslBump involvement probably
explains why peering works in this corner case).
If you need more information about the equivalence of the last two
configurations, please consider studying the following wiki page and

a

* https://wiki.squid-cache.org/Features/SslPeekAndSplice
*

http://lists.squid-cache.org/pipermail/squid-users/2018-September/019162.html

HTH,
Alex.