[squid-users] Need better debug_options values to track down cache MISSes

[Greg]

Hi all,

I'm new to squid administration, with basic *nix admin skills. My task now
is to replace an old Squid reverse proxy server (3.1.15 on an old Fedora)
with a new one (Squid 3.3.8 on Ubuntu 14 LTS) and ran into a problem.

I've spent 2 days tweaking-googling-debugging, now looking for some help...

Problem is, with the ~same configuration, the old Squid caches HTML pages
well and the new doesn't. This is a major concern, we're using Squid
exactly to do that and get the load off our appservers.

In this first mail, I'm only asking for a hint on what debug_options to
use. In previous list emails I've seen that some details help, so here they
are:
- Reverse proxy, port 80 only, one uplink straight to the app servers.
- Pretty standard configuration, standard refresh_patterns.
- Disk and memory cache, disk cache initialised with -z.
- Clocks are in sync (except for the old proxy server) using ntpd.
- Squid gives HITs for favicon.ico and RSS feeds only - these have
different headers and mimetypes compared to HTML files (and that's by
design).
- This new server seems to save accessed HTML files on disk but still gives
X-Cache-Lookup:MISS for all of them later.
- I've found the debug_options sections page but section names are
ambiguous for someone who's not a Squid programmer. I had to download the
source code, analysed it and tried "debug_options 11,1 22,6 85,3 88,5 33,1
31,5 90,5" which is too much and "debug_options 88,5 85,5 22,6 11,5 33,1"
which might be too little.
- Headers look okay (Cache-Control, Pragma, Vary), stale/fresh calculation
looks okay (pages are deemed fresh) in the debug log.
- The best I got so far is this: "client_side_reply.cc(1618)
identifyFoundObject: clientProcessRequest2: StoreEntry is NULL - MISS",
even for pages I see in the disk cache -- I'd like to know why does it
occur, probably need some more good debug_options. I don't understand the
code too well, don't know which direction to take.

Please suggest some more good debug_options to continue with.

Best regards,
Greg

PS. Also, I'll submit another mail with the details of the problem + the
config, hoping someone spots a problem right away.

[Pavel Kazlenka]

Hi Greg,

I use this combo to check if request is cached or no and why:
debug_options 20,9 27,9 31,9 70,9 82,9 22,9 84,9 90,9

Then in cache.log file you can search by 'YES' or 'NO' (uppercase) to
see if the content is cacheable (and cached) or not and the reason of
this decision. Hope this will help.

Best wishes,
Pavel

Post by Greg
Hi all,
I'm new to squid administration, with basic *nix admin skills. My task
now is to replace an old Squid reverse proxy server (3.1.15 on an old
Fedora) with a new one (Squid 3.3.8 on Ubuntu 14 LTS) and ran into a
problem.
I've spent 2 days tweaking-googling-debugging, now looking for some help...
Problem is, with the ~same configuration, the old Squid caches HTML
pages well and the new doesn't. This is a major concern, we're using
Squid exactly to do that and get the load off our appservers.
In this first mail, I'm only asking for a hint on what debug_options
to use. In previous list emails I've seen that some details help, so
- Reverse proxy, port 80 only, one uplink straight to the app servers.
- Pretty standard configuration, standard refresh_patterns.
- Disk and memory cache, disk cache initialised with -z.
- Clocks are in sync (except for the old proxy server) using ntpd.
- Squid gives HITs for favicon.ico and RSS feeds only - these have
different headers and mimetypes compared to HTML files (and that's by
design).
- This new server seems to save accessed HTML files on disk but still
gives X-Cache-Lookup:MISS for all of them later.
- I've found the debug_options sections page but section names are
ambiguous for someone who's not a Squid programmer. I had to download
the source code, analysed it and tried "debug_options 11,1 22,6 85,3
88,5 33,1 31,5 90,5" which is too much and "debug_options 88,5 85,5
22,6 11,5 33,1" which might be too little.
- Headers look okay (Cache-Control, Pragma, Vary), stale/fresh
calculation looks okay (pages are deemed fresh) in the debug log.
- The best I got so far is this: "client_side_reply.cc(1618)
identifyFoundObject: clientProcessRequest2: StoreEntry is NULL -
MISS", even for pages I see in the disk cache -- I'd like to know why
does it occur, probably need some more good debug_options. I don't
understand the code too well, don't know which direction to take.
Please suggest some more good debug_options to continue with.
Best regards,
Greg
PS. Also, I'll submit another mail with the details of the problem +
the config, hoping someone spots a problem right away.
_______________________________________________
squid-users mailing list
http://lists.squid-cache.org/listinfo/squid-users

[Greg]

Hi all,

so, there's my proxy problem I couldn't crack, even after spending 2+
days tweaking-googling-debugging. :(

The problem: my _new_ Squid installation (Ubuntu 14 LTS with Squid 3.3.8)
won't cache most pages the old Squid does (old Fedora with Squid 3.1.15).

Infos:
- Simple reverse proxy, port 80 only, one uplink straight to the app
servers. (The old one points to a load balancer. For testing purposes and
seeing the mail on "Thu, 4 Apr 2013 16:54:10 -0700", I'm pointing the new
one to a specific app server.)
- Pretty standard configuration, standard refresh_patterns. (I'd be
surprised if it was a config problem since ICOs are cached well.)
- I have disk caching on the new one, the disk cache has been initialised
with -z.
- The old one has a very relaxed iptables, the new one has ufw, later set
to allow port 80 and 22 only for incoming connections.
- Clocks are in sync using ntpd - the old one has +20min delay (ie. shows a
later time than now). Since it's the live one, it's working and it handles
a lot of load well, I haven't dared to correct it.
- The new Squid gives HITs for favicon.ico and RSS feeds only - these have
different headers and mimetypes compared to HTML files (and that's by
design).
- This new server seems to save accessed HTML files in the disk cache but
still gives X-Cache-Lookup:MISS for all of them later.
- I debugged using debug_options and logging: headers look okay
(Cache-Control, Pragma, Vary - pasted below), stale/fresh calculation looks
okay (pages are deemed fresh).
- The best I got so far is this: "client_side_reply.cc(1618)
identifyFoundObject: clientProcessRequest2: StoreEntry is NULL - MISS",
even for pages I see in the disk cache.
- I don't want to use overriding in refresh_patterns, because I don't
understand the problem yet, I don't know what problem would this hack hide
(and for how long).
- I don't want to move away from the current platform and version if
possible because this has LTS support and we'll have many proxy servers if
we finally figure out the right config.
- Strange thing, the old proxy seems to return stuff with HTTP/1.0 and the
new one with HTTP/1.1 in the same browser. No idea why.

What I've tried:
- Compared my squid.conf files - nothing special.
- Googling and mailing list archives - only found common MISS problems that
don't apply to my case.
- Have even experimented removing max-age from the headers sent by the
server and only featuring public, also set Last-Modified 10 minutes in the
past - no change.
- I've only moved one low-traffic live site to the new proxy so I can test
the old one and the new with the same app server engine.
- I know of Vary:Accept-Encoding issues with testing, so I test in the same
browser, clearing the cache before the second page access. With this, I see
HITs on the old server and MISS for the new.
- Heavy logging and debugging, checking out HTTP headers too - seems okay.
The same HTTP headers yield in HITs on the old server and MISS on the new
one.

I've collected some resources:
- Old server squid.conf - http://pastebin.com/h0C7t96n
- New server squid.conf - http://pastebin.com/KaHDVWYt
- Examples that gets cached on both servers - http://pastebin.com/Be4RqVLq
, http://pastebin.com/yVDeuyQp
- Example that gets cached on the old server but is a MISS on the new one -
http://pastebin.com/VvTU6ieR vs http://pastebin.com/ysKJwbmh
- Old server firewall settings - http://pastebin.com/v688dDMU
- New server firewall settings - http://pastebin.com/HyzGdjb4

Any tips, hints? This is driving me crazy!

Best regards,
Greg

[Yuri Voinov]

Post by Greg
Hi all,
so, there's my proxy problem I couldn't crack, even after spending
2+ days tweaking-googling-debugging. :(
The problem: my _new_ Squid installation (Ubuntu 14 LTS with Squid
3.3.8) won't cache most pages the old Squid does (old Fedora with
Squid 3.1.15).

Both versions are antique.

Man, you change one rancid meat to another rancid meat.

Just FYI - current Squid version at least 3.4.12. Oh, this branch is
already deprecated... shit, current version is 3.5.2!

This must be your starting point.

Post by Greg
Infos: - Simple reverse proxy, port 80 only, one uplink straight to
the app servers. (The old one points to a load balancer. For
testing purposes and seeing the mail on "Thu, 4 Apr 2013 16:54:10
-0700", I'm pointing the new one to a specific app server.) -
Pretty standard configuration, standard refresh_patterns. (I'd be
surprised if it was a config problem since ICOs are cached well.) -
I have disk caching on the new one, the disk cache has been
initialised with -z. - The old one has a very relaxed iptables, the
new one has ufw, later set to allow port 80 and 22 only for
incoming connections. - Clocks are in sync using ntpd - the old one
has +20min delay (ie. shows a later time than now). Since it's the
live one, it's working and it handles a lot of load well, I haven't
dared to correct it. - The new Squid gives HITs for favicon.ico and
RSS feeds only - these have different headers and mimetypes
compared to HTML files (and that's by design). - This new server
seems to save accessed HTML files in the disk cache but still gives
X-Cache-Lookup:MISS for all of them later. - I debugged using
debug_options and logging: headers look okay (Cache-Control,
Pragma, Vary - pasted below), stale/fresh calculation looks okay
clientProcessRequest2: StoreEntry is NULL - MISS", even for pages
I see in the disk cache. - I don't want to use overriding in
refresh_patterns, because I don't understand the problem yet, I
don't know what problem would this hack hide (and for how long). -
I don't want to move away from the current platform and version if
possible because this has LTS support and we'll have many proxy
servers if we finally figure out the right config. - Strange thing,
the old proxy seems to return stuff with HTTP/1.0 and the new one
with HTTP/1.1 in the same browser. No idea why.
What I've tried: - Compared my squid.conf files - nothing special.
- Googling and mailing list archives - only found common MISS
problems that don't apply to my case. - Have even experimented
removing max-age from the headers sent by the server and only
featuring public, also set Last-Modified 10 minutes in the past -
no change. - I've only moved one low-traffic live site to the new
proxy so I can test the old one and the new with the same app
server engine. - I know of Vary:Accept-Encoding issues with
testing, so I test in the same browser, clearing the cache before
the second page access. With this, I see HITs on the old server and
MISS for the new. - Heavy logging and debugging, checking out HTTP
headers too - seems okay. The same HTTP headers yield in HITs on
the old server and MISS on the new one.
I've collected some resources: - Old server squid.conf -
http://pastebin.com/h0C7t96n - New server squid.conf -
http://pastebin.com/KaHDVWYt - Examples that gets cached on both
servers - http://pastebin.com/Be4RqVLq ,
http://pastebin.com/yVDeuyQp - Example that gets cached on the old
server but is a MISS on the new one - http://pastebin.com/VvTU6ieR
vs http://pastebin.com/ysKJwbmh - Old server firewall settings -
http://pastebin.com/v688dDMU - New server firewall settings -
http://pastebin.com/HyzGdjb4
Any tips, hints? This is driving me crazy!
Best regards, Greg
_______________________________________________ squid-users mailing
http://lists.squid-cache.org/listinfo/squid-users

[Greg]

Subject: Re: [squid-users] Need better debug_options values to track
down cache MISSes
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Hi Greg,
debug_options 20,9 27,9 31,9 70,9 82,9 22,9 84,9 90,9
Then in cache.log file you can search by 'YES' or 'NO' (uppercase) to
see if the content is cacheable (and cached) or not and the reason of
this decision. Hope this will help.

Thanks, that was useful! I've tried that; the results:
- First access (at least from my computer in this session):
http://pastebin.com/EHNJ9N6F
- Second access (plus HTTP headers): http://pastebin.com/u8ZsHQfk

It's rather strange. Even though it's "cacheableReply: YES", it seems
that the system stores the page with key
C284A5F1C95D3B6C6D0F37A54517BEC9 but searches for it with
F588DBC3D17E4F32409C1AF22FC627FE in both test cases. (But in all
fairness, these might be different keys, I don't know the code too
well.)

Could you take a look at the debug log?

Best regards,
Greg