[squid-users] ICAP and HTTPS

Discussion:

Paul Carew

2015-10-06 16:14:10 UTC

Hi

Just a quick question regarding SSL bump and ICAP.

I have integrated Squid 3.5.9 with a commercial product that provides
an ICAP service. It works fine for HTTP.

Upon recieving an ICAP query for a blocked HTTP site the following
ICAP response is returned.

ICAP/1.0 200 OK
ISTAG: "PRODUCTNAME"
Attribute: Blocked Sites
Encapsulated: res-hdr=0, null-body=148

HTTP/1.0 302 Moved
Location: http://192.168.0.10/block?session=12345678
Pragma: no-cache
Cache-Control: no-cache

and the block page is correctly displayed in the users browser

However, when accessing a blocked site over HTTPS the following ICAP
response is received:

ICAP/1.0 200 OK
ISTAG: "PRODUCTNAME"
Attribute: Blocked Sites
Encapsulated: res-hdr=0, null-body=533

HTTP/1.0 403 Blocked
Content-Type: text/html
Pragma: no-cache
Cache-Control: no-cache
Location: http://192.168.0.10/block?session=12345678

<html>
<head>
<meta http-equiv="refresh"
content="0;url=http://192.168.0.10/block?session=12345678">
<title>Blocked</title>
</head>
<body>
<h4>You have been blocked.</h4>
<p>Click <a
href="http://192.168.0.10/block?session=12345678">here</a> for
details</p>
</body>
</html>

Chrome and IE just error upon receiving this response. In the case of
Chrome I get an ERR_TUNNEL_CONNECTION_FAILED error. I could be wrong
but I would imagine this error is by design, as Chrome will only
respond to a proxy authentication request or SSL handshake in response
to a HTTP CONNECT?

If that's correct, I was wondering if there is a way to get this to
work, with peek and splice possibly or any alternative method?

Thank you

Paul

Eliezer Croitoru

2015-10-06 16:57:21 UTC

Permalink

Hey Paul,

From what I have seen until now I believe that the ICAP service
response is for a CONNECT request.
From security reasons browsers are not allowing or rather then not
implanting support for a direct HTTP response to a CONNECT(tunnel) requests.
This is why you see this reaction from the browsers.
using peek and splice would not change the behavior of the browsers and
client and there for it will not change anything for this specific scenario.

However you can implement the inspection on the https level and not the
CONNECT level which would work.
If somehow you will find the right way to pass or to not pass CONNECT
requests to the ICAP service you might get to a situation which the
CONNECT by itself is allowed(and not inspected by the ICAP service) and
peek+splice would bump the connection and then the adaption(ICAP)
service will analyze the wrapped http level inside the CONNECT request.
This way the browser will receive the ICAP service response on the http
level and will show to the client the block page.

I know it is possible and Diladele solution does that already so there
is some light for you.

The issue is that I have not implemented it yet and there for cannot
give you instructions on how to implement it.

Eliezer

Post by Paul Carew
Hi
Just a quick question regarding SSL bump and ICAP.
I have integrated Squid 3.5.9 with a commercial product that provides
an ICAP service. It works fine for HTTP.
Upon recieving an ICAP query for a blocked HTTP site the following
ICAP response is returned.
ICAP/1.0 200 OK
ISTAG: "PRODUCTNAME"
Attribute: Blocked Sites
Encapsulated: res-hdr=0, null-body=148
HTTP/1.0 302 Moved
Location: http://192.168.0.10/block?session=12345678
Pragma: no-cache
Cache-Control: no-cache
and the block page is correctly displayed in the users browser
However, when accessing a blocked site over HTTPS the following ICAP
ICAP/1.0 200 OK
ISTAG: "PRODUCTNAME"
Attribute: Blocked Sites
Encapsulated: res-hdr=0, null-body=533
HTTP/1.0 403 Blocked
Content-Type: text/html
Pragma: no-cache
Cache-Control: no-cache
Location: http://192.168.0.10/block?session=12345678
<html>
<head>
<meta http-equiv="refresh"
content="0;url=http://192.168.0.10/block?session=12345678">
<title>Blocked</title>
</head>
<body>
<h4>You have been blocked.</h4>
<p>Click <a
href="http://192.168.0.10/block?session=12345678">here</a> for
details</p>
</body>
</html>
Chrome and IE just error upon receiving this response. In the case of
Chrome I get an ERR_TUNNEL_CONNECTION_FAILED error. I could be wrong
but I would imagine this error is by design, as Chrome will only
respond to a proxy authentication request or SSL handshake in response
to a HTTP CONNECT?
If that's correct, I was wondering if there is a way to get this to
work, with peek and splice possibly or any alternative method?
Thank you
Paul
_______________________________________________
squid-users mailing list
http://lists.squid-cache.org/listinfo/squid-users

Alex Rousskov

2015-10-06 17:05:11 UTC

Permalink

Post by Paul Carew
when accessing a blocked site over HTTPS the following ICAP
ICAP/1.0 200 OK
ISTAG: "PRODUCTNAME"
Attribute: Blocked Sites
Encapsulated: res-hdr=0, null-body=533
HTTP/1.0 403 Blocked
Content-Type: text/html
Pragma: no-cache
Cache-Control: no-cache
Location: http://192.168.0.10/block?session=12345678
<html>

...

Post by Paul Carew
</html>
Chrome and IE just error upon receiving this response. In the case of
Chrome I get an ERR_TUNNEL_CONNECTION_FAILED error. I could be wrong
but I would imagine this error is by design, as Chrome will only
respond to a proxy authentication request or SSL handshake in response
to a HTTP CONNECT?

Yes, this is by ["lazy"] browsers design (not specific to Chrome or IE).

Post by Paul Carew
If that's correct, I was wondering if there is a way to get this to
work, with peek and splice possibly or any alternative method?

Yes, you have a few options:

1. Do not block CONNECT. Block the HTTP request after CONNECT instead.
This does not require Squid modifications, but may require ICAP service
modifications, including keeping state between HTTP requests.

2. If you do not care about the actual error message displayed to the
user OR are OK with using [customizable] Squid error messages instead of
the ICAP-generated error messages, then you may:

2a. Teach Squid to treat certain ICAP responses as an instruction to
"block the virgin HTTP message". Squid eCAP client already supports that
(see 2d below). Squid ICAP client needs more work, including a decision
on how to define those "certain responses" in the ICAP context (should
probably be done via a new ACL-driven squid.conf directive).

2b. Teach the ICAP service to allow the CONNECT request but add an
extension HTTP header to it. Use adapted_http_access to block the
adapted CONNECT request.

2c. Teach Squid to use non-standard ICAP response headers as transaction
annotations (eCAP can do that already). Use adapted_http_access to block
the CONNECT transaction with your "blocking" annotations.

2d. Switch from ICAP to eCAP and use the existing
libecap::host::Xaction::blockVirgin() API. This option does not require
Squid development.

3. If you must use ICAP-generated error messages and cannot use option
#1 above, then you can do either #2a or #2d _and_ also teach Squid to
save and serve the custom error message returned by the adaptation
service. This option requires [very] difficult Squid development work
(in addition to easier development work required by #2a), but it is
possible.

All of the above options require bumping the connection. There is no
other way to serve an error message to the user (because of the browsers
design).

In the above text, "teach X to do Y" means "modify X code to
[optionally] do Y", including finding somebody who can perform those
source code modifications for you if needed.

I believe the above options are reasonable/valid, but I have not tested
any of them with a recent stock Squid. YMMV.

HTH,

Alex.

Paul Carew

2015-10-06 20:20:35 UTC

Permalink

Thanks Alex, Dieter & Eliezer

I've been trying to prevent the CONNECT request being processed by
ICAP and the following configuration in Squid 3.5.9 alongside a
standard SSL peek and splice config appears to work:

acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports

adaptation_access service_req deny CONNECT
adaptation_access service_req allow all

Although I haven't tested this thoroughly yet, it appears to allow the
initial tunnel creation but allow filtering of subsequent requests
within those tunnels. Although I guess this will only work for bumped
connections.

Thanks

Paul

On 6 October 2015 at 18:05, Alex Rousskov

Post by Alex Rousskov

...

Yes, this is by ["lazy"] browsers design (not specific to Chrome or IE).

Post by Paul Carew
If that's correct, I was wondering if there is a way to get this to
work, with peek and splice possibly or any alternative method?

1. Do not block CONNECT. Block the HTTP request after CONNECT instead.
This does not require Squid modifications, but may require ICAP service
modifications, including keeping state between HTTP requests.
2. If you do not care about the actual error message displayed to the
user OR are OK with using [customizable] Squid error messages instead of
2a. Teach Squid to treat certain ICAP responses as an instruction to
"block the virgin HTTP message". Squid eCAP client already supports that
(see 2d below). Squid ICAP client needs more work, including a decision
on how to define those "certain responses" in the ICAP context (should
probably be done via a new ACL-driven squid.conf directive).
2b. Teach the ICAP service to allow the CONNECT request but add an
extension HTTP header to it. Use adapted_http_access to block the
adapted CONNECT request.
2c. Teach Squid to use non-standard ICAP response headers as transaction
annotations (eCAP can do that already). Use adapted_http_access to block
the CONNECT transaction with your "blocking" annotations.
2d. Switch from ICAP to eCAP and use the existing
libecap::host::Xaction::blockVirgin() API. This option does not require
Squid development.
3. If you must use ICAP-generated error messages and cannot use option
#1 above, then you can do either #2a or #2d _and_ also teach Squid to
save and serve the custom error message returned by the adaptation
service. This option requires [very] difficult Squid development work
(in addition to easier development work required by #2a), but it is
possible.
All of the above options require bumping the connection. There is no
other way to serve an error message to the user (because of the browsers
design).
In the above text, "teach X to do Y" means "modify X code to
[optionally] do Y", including finding somebody who can perform those
source code modifications for you if needed.
I believe the above options are reasonable/valid, but I have not tested
any of them with a recent stock Squid. YMMV.
HTH,
Alex.

Rafael Akchurin

2015-10-06 21:05:10 UTC

Permalink

Hello Paul, Eliezer, Alex,

We (diladele ICAP) have an open bug /feature requests for this:
https://github.com/ra-at-diladele-com/qlproxy_external/issues/731
https://github.com/ra-at-diladele-com/qlproxy_external/issues/726

As Alex described most probably we will do the 2b approach - although the work has not started yet.
Hope to be able to have something by the end of this year!

P.S. The http://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html describes why the blocked message is not shown in simple words.

Best regards,
Rafael Akchurin
Diladele B.V.

-----Original Message-----
From: squid-users [mailto:squid-users-***@lists.squid-cache.org] On Behalf Of Paul Carew
Sent: Tuesday, October 6, 2015 10:21 PM
To: squid-***@lists.squid-cache.org
Subject: Re: [squid-users] ICAP and HTTPS

Thanks Alex, Dieter & Eliezer

I've been trying to prevent the CONNECT request being processed by ICAP and the following configuration in Squid 3.5.9 alongside a standard SSL peek and splice config appears to work:

acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports

adaptation_access service_req deny CONNECT adaptation_access service_req allow all

Although I haven't tested this thoroughly yet, it appears to allow the initial tunnel creation but allow filtering of subsequent requests within those tunnels. Although I guess this will only work for bumped connections.

Thanks

Paul

Post by Alex Rousskov

when accessing a blocked site over HTTPS the following ICAP response
ICAP/1.0 200 OK
ISTAG: "PRODUCTNAME"
Attribute: Blocked Sites
Encapsulated: res-hdr=0, null-body=533
HTTP/1.0 403 Blocked
Content-Type: text/html
Pragma: no-cache
Cache-Control: no-cache
Location: http://192.168.0.10/block?session=12345678
<html>

...

</html>
Chrome and IE just error upon receiving this response. In the case of
Chrome I get an ERR_TUNNEL_CONNECTION_FAILED error. I could be wrong
but I would imagine this error is by design, as Chrome will only
respond to a proxy authentication request or SSL handshake in
response to a HTTP CONNECT?

Yes, this is by ["lazy"] browsers design (not specific to Chrome or IE).

If that's correct, I was wondering if there is a way to get this to
work, with peek and splice possibly or any alternative method?

1. Do not block CONNECT. Block the HTTP request after CONNECT instead.
This does not require Squid modifications, but may require ICAP
service modifications, including keeping state between HTTP requests.
2. If you do not care about the actual error message displayed to the
user OR are OK with using [customizable] Squid error messages instead
2a. Teach Squid to treat certain ICAP responses as an instruction to
"block the virgin HTTP message". Squid eCAP client already supports
that (see 2d below). Squid ICAP client needs more work, including a
decision on how to define those "certain responses" in the ICAP
context (should probably be done via a new ACL-driven squid.conf directive).
2b. Teach the ICAP service to allow the CONNECT request but add an
extension HTTP header to it. Use adapted_http_access to block the
adapted CONNECT request.
2c. Teach Squid to use non-standard ICAP response headers as
transaction annotations (eCAP can do that already). Use
adapted_http_access to block the CONNECT transaction with your "blocking" annotations.
2d. Switch from ICAP to eCAP and use the existing
libecap::host::Xaction::blockVirgin() API. This option does not
require Squid development.
3. If you must use ICAP-generated error messages and cannot use option
#1 above, then you can do either #2a or #2d _and_ also teach Squid to
save and serve the custom error message returned by the adaptation
service. This option requires [very] difficult Squid development work
(in addition to easier development work required by #2a), but it is
possible.
All of the above options require bumping the connection. There is no
other way to serve an error message to the user (because of the
browsers design).
In the above text, "teach X to do Y" means "modify X code to
[optionally] do Y", including finding somebody who can perform those
source code modifications for you if needed.
I believe the above options are reasonable/valid, but I have not
tested any of them with a recent stock Squid. YMMV.
HTH,
Alex.

_______________________________________________
squid-users mailing list
squid-***@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Marcus Kool

2015-10-07 00:50:05 UTC

Permalink

Post by Rafael Akchurin
Hello Paul, Eliezer, Alex,
https://github.com/ra-at-diladele-com/qlproxy_external/issues/731
https://github.com/ra-at-diladele-com/qlproxy_external/issues/726
As Alex described most probably we will do the 2b approach - although the work has not started yet.
Hope to be able to have something by the end of this year!

The 2b) option a.k.a "simply always allow the CONNECT www.example.com and
later block GET https://www.example.com/index.html" _only_ works for
correctly SSL-bumped sites and does not work sites that do not use SSL+HTTP.

For Skype, SSH tunnels and other protocols that also use CONNECT
the ICAP server must block the CONNECT (if configured to do so).
There are even sites that use SSL+other protocol, so bumping such site may
initially seem OK since the SSL handshake was done without problems,
but since it is not followed by an HTTP protocol request, the ICAP server
will never see a followup HTTP GET/POST and hence will never be able to
block the site after it allowed the CONNECT.
So if the site must be blocked, the ICAP server must already decide what
to do when it receives a CONNECT request:
a) guess that the CONNECT is for SSL+HTTP, pass the CONNECT and wait for the followup GET/POST to be blocked, or
b) guess that the CONNECT is for a site that does not use SSL+HTTP and block the CONNECT.

Because of the above, I had a discussion a long time ago with the Squid developers
to extend Squid to send the (non-HTTP) content to the ICAP server in case
that the CONNECT tunnel does not have SSL+HTTP, but the implementation effort
was considered to be too much at that time.
If, however, this feature is implemented, the ICAP server no longer has to guess
and can always pass the CONNECT and decide later, knowing that it will either
receive an HTTP request or the non-HTTP content.

Best regards,
Marcus

Post by Rafael Akchurin
P.S. The http://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html describes why the blocked message is not shown in simple words.
Best regards,
Rafael Akchurin
Diladele B.V.
-----Original Message-----
Sent: Tuesday, October 6, 2015 10:21 PM
Subject: Re: [squid-users] ICAP and HTTPS
Thanks Alex, Dieter & Eliezer
acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports
adaptation_access service_req deny CONNECT adaptation_access service_req allow all
Although I haven't tested this thoroughly yet, it appears to allow the initial tunnel creation but allow filtering of subsequent requests within those tunnels. Although I guess this will only work for bumped connections.
Thanks
Paul

Post by Alex Rousskov

...

Yes, this is by ["lazy"] browsers design (not specific to Chrome or IE).

If that's correct, I was wondering if there is a way to get this to
work, with peek and splice possibly or any alternative method?

1. Do not block CONNECT. Block the HTTP request after CONNECT instead.
This does not require Squid modifications, but may require ICAP
service modifications, including keeping state between HTTP requests.
2. If you do not care about the actual error message displayed to the
user OR are OK with using [customizable] Squid error messages instead
2a. Teach Squid to treat certain ICAP responses as an instruction to
"block the virgin HTTP message". Squid eCAP client already supports
that (see 2d below). Squid ICAP client needs more work, including a
decision on how to define those "certain responses" in the ICAP
context (should probably be done via a new ACL-driven squid.conf directive).
2b. Teach the ICAP service to allow the CONNECT request but add an
extension HTTP header to it. Use adapted_http_access to block the
adapted CONNECT request.
2c. Teach Squid to use non-standard ICAP response headers as
transaction annotations (eCAP can do that already). Use
adapted_http_access to block the CONNECT transaction with your "blocking" annotations.
2d. Switch from ICAP to eCAP and use the existing
libecap::host::Xaction::blockVirgin() API. This option does not
require Squid development.
3. If you must use ICAP-generated error messages and cannot use option
#1 above, then you can do either #2a or #2d _and_ also teach Squid to
save and serve the custom error message returned by the adaptation
service. This option requires [very] difficult Squid development work
(in addition to easier development work required by #2a), but it is
possible.
All of the above options require bumping the connection. There is no
other way to serve an error message to the user (because of the
browsers design).
In the above text, "teach X to do Y" means "modify X code to
[optionally] do Y", including finding somebody who can perform those
source code modifications for you if needed.
I believe the above options are reasonable/valid, but I have not
tested any of them with a recent stock Squid. YMMV.
HTH,
Alex.

_______________________________________________
squid-users mailing list
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
http://lists.squid-cache.org/listinfo/squid-users

Alex Rousskov

2015-10-07 01:18:28 UTC

Permalink

Post by Marcus Kool
The 2b) option a.k.a "simply always allow the CONNECT www.example.com and
later block GET https://www.example.com/index.html" _only_ works for
correctly SSL-bumped sites and does not work sites that do not use SSL+HTTP.

If you want the user to see a non-browser error message, then _all_ of
those options require bumping, SSL, and HTTP (all three!). They only
differ in where the error message originates (Squid, ICAP, or eCAP) and
the associated development effort (if any).

Post by Marcus Kool
For Skype, SSH tunnels and other protocols that also use CONNECT
the ICAP server must block the CONNECT (if configured to do so).
There are even sites that use SSL+other protocol, so bumping such site may
initially seem OK since the SSL handshake was done without problems,
but since it is not followed by an HTTP protocol request, the ICAP server
will never see a followup HTTP GET/POST and hence will never be able to
block the site after it allowed the CONNECT.

A bumping Squid will essentially block the site in such cases though
(subject to on_unsupported_protocol controls in recent Squids).

Post by Marcus Kool
So if the site must be blocked, the ICAP server must already decide what
a) guess that the CONNECT is for SSL+HTTP, pass the CONNECT and wait for
the followup GET/POST to be blocked, or
b) guess that the CONNECT is for a site that does not use SSL+HTTP and block the CONNECT.

Recent Squids also have options for detecting non-HTTP and non-SSL
traffic (see the on_unsupported_protocol directive in squid.conf).

Post by Marcus Kool
Because of the above, I had a discussion a long time ago with the Squid developers
to extend Squid to send the (non-HTTP) content to the ICAP server in case
that the CONNECT tunnel does not have SSL+HTTP, but the implementation
effort was considered to be too much at that time.

What do you mean? Too much implementation effort to implement quickly?
... to implement for free? ... to implement in the next release?
Something else?

In general, improvements in Squid's non-SSL and non-HTTP content
handling should be welcomed and adaptation services may be used for that
IMO, especially when it comes to custom error generation.

Alex.

Continue reading on narkive:

Search results for '[squid-users] ICAP and HTTPS' (Questions and Answers)

replies

I thought politicians were supposed to GET jobs not LOSE them. WTF was that incredible fool Alexandria Ocasio Cortez thinking?

started 2019-02-22 17:39:55 UTC

politics & government

replies

Can the antibiotics, Sulprim & Oxytet, be used in conjuctin with each other?

started 2008-06-09 19:27:06 UTC

medicine

replies

What is ERP, EPR System, ERP Suite?