[squid-users] Squid 3.5.25 does not recognise ICAP 408 status code

[Arunabha Saha]

Squid 3.5.25 does not seem to recognise the 408 request timeout error code
from ICAP.
The more troublesome issue for me is the exception it generates and then
declares ICAP down after a certain number of such exceptions.

I don't want to disable the failure limit entirely given that we can often
have genuine failures that squid needs to detect.

What i'd like to see is squid should not throw an exception in this case.
The timeout is somewhat aggressive but works with an earlier version of
ICAP (0.1.x). The one i'm testing is 0.5.3.

Any suggestions? I tried looking at squid 3.5.26 through 3.5.28 but i
don't see any support for 408 timeout there (maybe i missed it?).

Debug Logs:

2018/10/29 20:20:25.656| 93,5| ModXact.cc(654) parseMore:
ICAP/1.0 408 Request timeout^M
Server: C-ICAP/0.5.3^M
Connection: close^M
ISTag: CI0001-XXXXXXXXX^M
^M

2018/10/29 20:20:25.656| 93,5| ModXact.cc(749) parseHeaders: parse ICAP
headers
2018/10/29 20:20:25.656| 93,5| ModXact.cc(1079) parseHead: have 98 head
bytes to parse; state: 0
2018/10/29 20:20:25.656| 93,5| ModXact.cc(1094) parseHead: parse success,
consume 98 bytes, return true
2018/10/29 20:20:25.656| 93,5| ModXact.cc(785) parseIcapHead: found
connection close
2018/10/29 20:20:25.656| 93,5| ModXact.cc(815) parseIcapHead: ICAP status
408
2018/10/29 20:20:25.656| 93,3| ../../../src/base/AsyncJobCalls.h(177) dial:
Adaptation::Icap::Xaction::noteCommRead threw exception: Unsupported ICAP
status code
2018/10/29 20:20:25.656| 11,5| HttpRequest.cc(472) detailError: current
error details: 35/396407234
2018/10/29 20:20:25.656| 93,4| Xaction.cc(514) setOutcome: ICAP_ERR_OTHER
2018/10/29 20:20:25.656| 93,4| ServiceRep.cc(80) noteFailure: failure 4
out of 10 allowed in 0sec [up,fail4]
...
2018/10/29 20:21:06.981| 93,5| ModXact.cc(815) parseIcapHead: ICAP status
408
2018/10/29 20:21:06.981| 93,3| ../../../src/base/AsyncJobCalls.h(177) dial:
Adaptation::Icap::Xaction::noteCommRead threw exception: Unsupported ICAP
status code
2018/10/29 20:21:06.981| 11,5| HttpRequest.cc(472) detailError: current
error details: 35/396407234
2018/10/29 20:21:06.981| 93,4| Xaction.cc(514) setOutcome: ICAP_ERR_OTHER
2018/10/29 20:21:06.981| 93,4| ServiceRep.cc(80) noteFailure: failure 11
out of 10 allowed in 0sec [up,fail11]
2018/10/29 20:21:06.981| suspending ICAP service for too many failures

[Alex Rousskov]

Post by Arunabha Saha
Squid 3.5.25 does not seem to recognise the 408 request timeout error
code from ICAP.

Squid effectively recognizes ICAP 408 response as an ICAP transaction
error response and blames the ICAP service for that error. That
(minimal) support can be improved, of course. See options #1 and #3 below.

As with any timeout, it is impossible to say in general which side of
the connection is at fault. This case has at least three sides: It could
be the HTTP agent, Squid, and/or the ICAP service. Did one of them stall
the transaction? Or was the ICAP service just too impatient? See option
#4 below.

Needless to say, treating all ICAP service timeouts as if nothing bad
happened would break some existing Squid deployments (while possibly
fixing yours). A proper general solution (option #3 below) would most
likely require making Squid behavior configurable.

Post by Arunabha Saha
The more troublesome issue for me is the exception it generates and then
declares ICAP down after a certain number of such exceptions.    
I don't want to disable the failure limit entirely given that we can
often have genuine failures that squid needs to detect.   
What i'd like to see is squid should not throw an exception in this case.

The "exception" is a minor low-level/technical detail. What you really
want to see is Squid blaming itself (rather than the ICAP service) for
the problem. Squid indeed lacks that kind of functionality, but it can
be added if really needed. See options #1 and #3 below.

Post by Arunabha Saha
The timeout is somewhat aggressive but works with an earlier
version of ICAP (0.1.x).  The one i'm testing is 0.5.3.

Please note that ICAP is a protocol, not a product/software name. It
probably does not matter what ICAP service you are using though.

Post by Arunabha Saha
Any suggestions?

I can suggest a few options, in no particular order:

1. Modify your Squid to treat 408 differently.
2. Modify your ICAP service to stop sending ICAP 408 responses to Squid.
3. Add proper ICAP timeout support feature to Squid.
4. Investigate why your ICAP service times out. If you are lucky, you
may be able to fix or work around the problem by adjusting Squid
and/or your ICAP service configuration.

For option #1, Adaptation::Icap::ModXact::parseIcapHead() may be a good
starting point.

For options #1 and #3, see also
https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F

In most cases, option #4 is the best first step but YMMV.


HTH,

Alex.

[Arunabha Saha]

Post by Alex Rousskov
As with any timeout, it is impossible to say in general which side of
the connection is at fault. This case has at least three sides: It could
be the HTTP agent, Squid, and/or the ICAP service. Did one of them stall
the transaction? Or was the ICAP service just too impatient? See option
#4 below.

I've tried to track this down. There are some persistent sockets used by
SaaS apps for APIs (otservice api from google sites) and sometimes the HTTP
response takes a long time to trickle in. I have seen upto 25 seconds for
the response body to trickle in after the response header. I don't know
yet if this is due to network delays but given that it happens only for
this particular uri I'm theorizing that this is how it works. I can
whitelist the one i am aware of that is causing this issue but again the
concern is what about others that might throw this exception. The
timeout at 10 seconds is somewhat aggressive so moving that up should help
but some code changes in either squid or icap as suggested look necessary.

Post by Alex Rousskov
Please note that ICAP is a protocol, not a product/software name. It
probably does not matter what ICAP service you are using though.

Correct. I was referring to the c-icap implementation.

Post by Alex Rousskov
Send squid-users mailing list submissions to
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.squid-cache.org/listinfo/squid-users
or, via email, send a message with subject or body 'help' to
You can reach the person managing the list at
When replying, please edit your Subject line so it is more specific
than "Re: Contents of squid-users digest..."
1.

Post by Arunabha Saha
Squid 3.5.25 does not seem to recognise the 408 request timeout error
code from ICAP.

Squid effectively recognizes ICAP 408 response as an ICAP transaction
error response and blames the ICAP service for that error. That
(minimal) support can be improved, of course. See options #1 and #3 below.
Needless to say, treating all ICAP service timeouts as if nothing bad
happened would break some existing Squid deployments (while possibly
fixing yours). A proper general solution (option #3 below) would most
likely require making Squid behavior configurable.

Post by Arunabha Saha
The more troublesome issue for me is the exception it generates and then
declares ICAP down after a certain number of such exceptions.
I don't want to disable the failure limit entirely given that we can
often have genuine failures that squid needs to detect.
What i'd like to see is squid should not throw an exception in this
case.

The "exception" is a minor low-level/technical detail. What you really
want to see is Squid blaming itself (rather than the ICAP service) for
the problem. Squid indeed lacks that kind of functionality, but it can
be added if really needed. See options #1 and #3 below.

Post by Arunabha Saha
The timeout is somewhat aggressive but works with an earlier
version of ICAP (0.1.x). The one i'm testing is 0.5.3.

Please note that ICAP is a protocol, not a product/software name. It
probably does not matter what ICAP service you are using though.

Post by Arunabha Saha
Any suggestions?

1. Modify your Squid to treat 408 differently.
2. Modify your ICAP service to stop sending ICAP 408 responses to Squid.
3. Add proper ICAP timeout support feature to Squid.
4. Investigate why your ICAP service times out. If you are lucky, you
may be able to fix or work around the problem by adjusting Squid
and/or your ICAP service configuration.
For option #1, Adaptation::Icap::ModXact::parseIcapHead() may be a good
starting point.
For options #1 and #3, see also
https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
In most cases, option #4 is the best first step but YMMV.
HTH,
Alex.
------------------------------
Message: 2
Date: Tue, 30 Oct 2018 23:59:18 -0500 (CDT)
Subject: Re: [squid-users] Squid 4.3: SSL Bump fails to send client
certificate
Content-Type: text/plain; charset=us-ascii
Thank you Alex for the reply.
Alex: 1. Servers never send SNI. Clients usually send SNI. Squid should
forward SNI it received from the client to the server, provided the client
actually sent SNI. Did your client send SNI?
Sid: I can see in Client Hello IP Address being sent by Client; so there is
no SNI from client itself.
Alex: 2. Bugs notwithstanding, the implied order of events is not what
actually happens: Squid, as configured, does _not_ forward anything from
the
server certificate to the client. Squid, as configured, generates a
certificate based on client-supplied information (not server-supplied
information). After sending that generated certificate to the client, Squid
establishes a TLS connection with the server.
Sid: Thank you for explanation.
Alex: For an accurate picture, in addition to Squid-server and server-Squid
traffic, look at what Squid has received from the client and what Squid
sent
to the client, all in actual order.
Sid: I took wireshark on Squid server (centOS 7); I took 2 wiresharks
between Client & Squid and then between Squid & Server. I can see client
being sent fake cert generated by Squid & client responds with "Client key
Exchange", "Change cipher spec", "Encrypted Handshake Message". But I can't
see actual client certificate sent to Squid. Is there a way to decypt in
Wireshark. In Wireshark between Squid & Server I can see Squid responding
with "61 Alert (Level: Fatal, Description: Internal Error)".
Alex: Is your Squid configured to trust those internal CAs? If not, Squid
would not be able to validate the server certificate.
Sid: I have added those chained certificates as following in squid.conf
tls_outgoing_options cafile=/usr/local/squid/etc/UCAppsCA.pem
sslproxy_foreign_intermediate_certs /usr/local/squid/etc/UCAppsCA.pem
--
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
------------------------------
Message: 3
Date: Wed, 31 Oct 2018 05:27:20 -0500 (CDT)
Subject: Re: [squid-users] redirect based on url (302)
Content-Type: text/plain; charset=us-ascii
Hi Amos,
Is there a git that I can use to push stuff up?
I think you need to split the string in an other way, look into this
#!/usr/bin/perl
use strict;
use warnings;
$|=1;
while (<>) {
my $string = $_;
print "Received '\$_' = ".$_."\n";
$string =~ m/^(\d+)\s(.*)$/;
print "After regexp '\$string' = ".$string."\n";
print "After regexp '\$1' = ".$1."\n";
print "After regexp '\$2' = ".$2."\n";
### Original split from sorce ###
### This doesn't split anything looks like elements of an array?
#my ($cid, $uid) = ($1, $2);
### Split the string ###
### Those two split based on one or more spaces
#my ($cid, $uid) = split(/\s+/ ,$_);
my ($cid, $uid) = split;
$cid =~ s/%(..)/pack("H*", $1)/ge;
$uid =~ s/%(..)/pack("H*", $1)/ge;
print "After split \$cid = ".$cid."\n";
print "After split \$uid = ".$uid."\n";
}
Received '$_' = 130.238.000.00 muse.jhu.edu -
After regexp '$string' = 130.238.000.00 muse.jhu.edu -
/Use of uninitialized value $1 in concatenation (.) or string at
./sed_test_reg.pl line 13, <> line 1.
After regexp '$1' =
Use of uninitialized value $2 in concatenation (.) or string at
./sed_test_reg.pl line 14, <> line 1.
After regexp '$2' = /
*After split $cid = 130.238.000.00
After split $uid = muse.jhu.edu*
Cheers
Fredrik
--
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
------------------------------
Subject: Digest Footer
_______________________________________________
squid-users mailing list
http://lists.squid-cache.org/listinfo/squid-users
------------------------------
End of squid-users Digest, Vol 50, Issue 75
*******************************************

[Alex Rousskov]

Post by Alex Rousskov
As with any timeout, it is impossible to say in general which side of
the connection is at fault. This case has at least three sides: It could
be the HTTP agent, Squid, and/or the ICAP service. Did one of them stall
the transaction? Or was the ICAP service just too impatient? See option
#4 below.

I've tried to track this down.   There are some  persistent sockets used
by SaaS apps for APIs (otservice api from google sites) and sometimes
the HTTP response takes a long time to trickle in.  I have seen upto 25
seconds for the response body to trickle in after the response header.

Glad you found the culprit!

 

The timeout at 10 seconds is somewhat aggressive so moving
that up should help but some code changes in either squid or icap as
suggested look necessary.

If you do not want to modify Squid, then the ICAP service should tell
Squid what to do with the timed out HTTP transaction (e.g., respond with
ICAP 204 or ICAP 200) instead of telling Squid that the service does not
know what to do (i.e. respond with ICAP 408). When the service does not
know what to do, Squid does not know what to do either, resulting in
transaction errors (that Squid blames the ICAP service for).

Please note that teaching Squid about the special meaning of ICAP 408
(Timeout) responses is not enough to address the problem -- Squid would
still need to know what to do with the HTTP transaction (e.g. block it,
pass through as is, or abort) and with the service (count a service
failure or ignore the timeout). Those decisions are likely to be
different for different admins/deployments.

Alex.

I was referring to the c-icap implementation.

Post by Alex Rousskov
Squid 3.5.25 does not seem to recognise the 408 request timeout error
code from ICAP.

Squid effectively recognizes ICAP 408 response as an ICAP transaction
error response and blames the ICAP service for that error. That
(minimal) support can be improved, of course. See options #1 and #3 below.
 
Needless to say, treating all ICAP service timeouts as if nothing bad
happened would break some existing Squid deployments (while possibly
fixing yours). A proper general solution (option #3 below) would most
likely require making Squid behavior configurable.

Post by Alex Rousskov
The more troublesome issue for me is the exception it generates

and then

Post by Alex Rousskov
declares ICAP down after a certain number of such exceptions.    
I don't want to disable the failure limit entirely given that we can
often have genuine failures that squid needs to detect.   
What i'd like to see is squid should not throw an exception in this
case.

The "exception" is a minor low-level/technical detail. What you really
want to see is Squid blaming itself (rather than the ICAP service) for
the problem. Squid indeed lacks that kind of functionality, but it can
be added if really needed. See options #1 and #3 below.

Post by Alex Rousskov
The timeout is somewhat aggressive but works with an earlier
version of ICAP (0.1.x).  The one i'm testing is 0.5.3.

Please note that ICAP is a protocol, not a product/software name. It
probably does not matter what ICAP service you are using though.

Post by Alex Rousskov
Any suggestions?

1. Modify your Squid to treat 408 differently.
2. Modify your ICAP service to stop sending ICAP 408 responses to Squid.
3. Add proper ICAP timeout support feature to Squid.
4. Investigate why your ICAP service times out. If you are lucky, you
   may be able to fix or work around the problem by adjusting Squid
   and/or your ICAP service configuration.
For option #1, Adaptation::Icap::ModXact::parseIcapHead() may be a good
starting point.
For options #1 and #3, see also
https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
In most cases, option #4 is the best first step but YMMV.
HTH,
Alex.