[squid-users] caching store increase then decrease during caching windows updates and all request are TCP

Discussion:

[squid-users] caching store increase then decrease during caching windows updates and all request are TCP_MISS ??!!!

--Ahmad--

2016-09-12 09:47:28 UTC

hi

i have caching store increase then decrease during caching windows updates and all request are TCP_MISS ??!!!

***@raspberrypi:~# du -sh /var/cache/squid/
558M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
561M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
564M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
647M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
650M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
652M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
604M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
605M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
605M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
606M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
606M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/
595M /var/cache/squid/
***@raspberrypi:~# du -sh /var/cache/squid/

here is sample of logs :

1473672787.641 444 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 872 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672799.084 11876 192.168.0.10 TCP_MISS/206 615396 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672808.553 8396 192.168.0.10 TCP_MISS/206 615396 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672811.054 24905 192.168.0.10 TCP_MISS/206 1017331 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672812.922 2687 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 1049501 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672817.014 325638 192.168.0.10 TCP_MISS/206 1049207 GET http://au.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2016/08/windows10.0-kb3176938-x64_6e080b0ebebeb8a463a297f5b14fcd03690eb1b8.psf - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672830.229 7673 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 615395 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672830.577 14270 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 1049513 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672834.348 332467 192.168.0.10 TCP_MISS/206 1049207 GET http://au.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2016/08/windows10.0-kb3176938-x64_6e080b0ebebeb8a463a297f5b14fcd03690eb1b8.psf - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672834.693 12139 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 1049501 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672839.837 19 192.168.0.10 TCP_MISS/200 875 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672839.874 21 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 874 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672839.876 22 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 874 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672839.879 23 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 874 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672839.884 25 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 874 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672839.970 90 192.168.0.10 TCP_MISS/206 81857 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672840.373 489 192.168.0.10 TCP_MISS/206 81857 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672840.374 471 192.168.0.10 TCP_MISS/206 81857 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672840.714 825 192.168.0.10 TCP_MISS/206 81857 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/74c45c94-e8f2-43c6-9c1a-0ef798b56116 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672840.768 24456 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 1049513 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672841.687 25 192.168.0.10 TCP_MISS/200 3818 GET http://dl.delivery.mp.microsoft.com/filestreamingservice//files/6bd00537-0160-46ff-b7c8-a58327e6722c/pieceshash - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672841.769 21 192.168.0.10 TCP_MISS/200 840 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/6bd00537-0160-46ff-b7c8-a58327e6722c? - ORIGINAL_DST/13.107.4.50 -
1473672841.791 37 192.168.0.10 TCP_MISS/200 840 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/6bd00537-0160-46ff-b7c8-a58327e6722c? - ORIGINAL_DST/13.107.4.50 -
1473672841.794 39 192.168.0.10 TCP_MISS/200 880 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/6bd00537-0160-46ff-b7c8-a58327e6722c? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672842.075 326 192.168.0.10 TCP_MISS/200 880 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/6bd00537-0160-46ff-b7c8-a58327e6722c? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672855.052 69572 192.168.0.10 TCP_MISS/206 311289 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ca1aed45-880f-4294-81ef-368554dcb282? - ORIGINAL_DST/13.107.4.50 -
1473672870.475 23892 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 1049513 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672877.996 53 192.168.0.10 TCP_MISS/200 299 POST http://webres1.bullguard.ctmail.com/SpamResolverNG/SpamResolverNG.dll? - ORIGINAL_DST/84.39.153.33 text/html
1473672881.817 35092 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 1049513 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.028 40 192.168.0.10 TCP_MISS/200 868 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.061 20 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 867 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.067 24 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 867 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.076 31 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 867 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.076 29 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 867 HEAD http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.113 32 192.168.0.10 TCP_MISS/206 15400 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.169 28 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 15399 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.507 362 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 15399 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672888.521 374 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 15399 GET http://dl.delivery.mp.microsoft.com/filestreamingservice/files/83501d77-70b0-4d63-8891-949938435c79 - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672889.692 20 192.168.0.10 TCP_MISS/200 833 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/b9956be9-38bc-41cd-8668-3691c7e0c235? - ORIGINAL_DST/13.107.4.50 -
1473672889.729 20 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 832 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/b9956be9-38bc-41cd-8668-3691c7e0c235? - ORIGINAL_DST/13.107.4.50 -
1473672889.732 21 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 832 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/b9956be9-38bc-41cd-8668-3691c7e0c235? - ORIGINAL_DST/13.107.4.50 -
1473672889.756 42 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 832 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/b9956be9-38bc-41cd-8668-3691c7e0c235? - ORIGINAL_DST/13.107.4.50 -
1473672890.050 335 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 832 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/b9956be9-38bc-41cd-8668-3691c7e0c235? - ORIGINAL_DST/13.107.4.50 -
1473672901.577 398176 192.168.0.10 TCP_MISS/206 1049207 GET http://au.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2016/08/windows10.0-kb3176938-x64_6e080b0ebebeb8a463a297f5b14fcd03690eb1b8.psf - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672913.571 408766 192.168.0.10 TCP_MISS/206 1049207 GET http://au.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2016/08/windows10.0-kb3176938-x64_6e080b0ebebeb8a463a297f5b14fcd03690eb1b8.psf - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672919.550 29816 192.168.0.10 TCP_MISS/206 437400 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/b9956be9-38bc-41cd-8668-3691c7e0c235? - ORIGINAL_DST/13.107.4.50 -
^C

here is config file :

#refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims
refresh_pattern ([^.]+.)?(download|(windows)?update).(microsoft.)?com/.*.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims
refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*.(pkg|dmg) 4320 100% 43200 reload-into-ims
range_offset_limit -1

quick_abort_min -1

http_port 3129 intercept

visible_hostname hey
cache_swap_low 90
cache_swap_high 95
fqdncache_size 65535
ipcache_size 65535
ipcache_low 98
ipcache_high 99
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
############
#cache_mem 512 MB
maximum_object_size 1 GB
################################################
maximum_object_size_in_memory 10 MB
#########
acl windowsupdate dstdomain fg.v4.download.windowsupdate.com
acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain au.download.windowsupdate.com
acl windowsupdate dstdomain ds.download.windowsupdate.com
acl windowsupdate dstdomain ctldl.windowsupdate.com
acl windowsupdate dstdomain .data.microsoft.com
acl windowsupdate dstdomain .l.windowsupdate.com
acl windowsupdate dstdomain .microsoft.com.akadns.net
acl windowsupdate dstdomain .deploy.akamaitechnologies.com
###################################################
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
####################################
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com.akadns.net/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i deploy.akamaitechnologies.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
########################################################
#refresh_pattern \^ftp: 1440 20% 10080
#refresh_pattern \^gopher: 1440 0% 1440
#refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
#refresh_pattern . 0 20% 4320
########################################################
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access allow CONNECT wuCONNECT localnet
http_access allow windowsupdate localnet
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 192.168.0.1:3128

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/cache/squid 20000 16 256
cache_dir aufs /var/cache/squid 10000 32 256

# Leave coredumps in the first cache dir
coredump_dir /var/cache/squid
##############

hope to help

cheers

Amos Jeffries

2016-09-12 14:13:05 UTC

Permalink

Post by --Ahmad--
hi
i have caching store increase then decrease during caching windows updates and all request are TCP_MISS ??!!!
558M /var/cache/squid/
561M /var/cache/squid/
564M /var/cache/squid/
647M /var/cache/squid/
650M /var/cache/squid/
652M /var/cache/squid/
604M /var/cache/squid/
605M /var/cache/squid/
605M /var/cache/squid/
606M /var/cache/squid/
606M /var/cache/squid/
595M /var/cache/squid/

What relevance is this?

HEAD request. None of the object got downloaded.

Post by --Ahmad--
1473672799.084 11876 192.168.0.10 TCP_MISS/206 615396 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672808.553 8396 192.168.0.10 TCP_MISS/206 615396 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672811.054 24905 192.168.0.10 TCP_MISS/206 1017331 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c? - ORIGINAL_DST/13.107.4.50 application/octet-stream

Some Range requests for URI with ?query parameters. You need to add
"strip_query_terms no" to your squid.conf. Then you will be able to see
if these are identical URI or not.

Right now all that can be said is the script on the server which is
producing objects is the same one for many of the requests. There is no
way to know if they are requesting the same object or different ones.

A request where the client required new content to be fetched.

The rest of your log shows a lot of repeats of the above things happening.

These refresh_pattern contain broken regex patterns.

1) Unless you explicitly use start/end anchors regex have implicit .*
sequences before and after the provided pattern.

That means "([^.]+.|)" and "([^.]+.)?" are just a very complex (and
slow to compare) ways to write .*

2) '.' is a reserved character. If you want it to match a '.' in the
domain name you need to write it as \.

Hint both above regex will match the non-Microsoft, non-Apple URI:
http://download.com/?pkg=caddy

Amos

--Ahmad--

2016-09-12 20:26:17 UTC

Permalink

amos thank you so much for reply

how can i correct the patterns below ?

im not sure if the other patterns are doing the game

but so far i donât have disk increasing at all

Post by Amos Jeffries

but i still donât have HITs

im not sure where should i tune ?

i also added stip query terms to off

not sure what should i do next âŠâŠ. all i need is windows updates go ok , Iâm not interested with other caching

hope to assist

cheers

Post by Amos Jeffries

What relevance is this?

Post by --Ahmad--
1473672787.641 444 192.168.0.10 TCP_CLIENT_REFRESH_MISS/200 872 HEAD http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? <http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56?> - ORIGINAL_DST/13.107.4.50 application/octet-stream

HEAD request. None of the object got downloaded.

Post by --Ahmad--
1473672799.084 11876 192.168.0.10 TCP_MISS/206 615396 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? <http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56?> - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672808.553 8396 192.168.0.10 TCP_MISS/206 615396 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? <http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56?> - ORIGINAL_DST/13.107.4.50 application/octet-stream
1473672811.054 24905 192.168.0.10 TCP_MISS/206 1017331 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c? <http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/1fc27b6e-d563-415b-b11a-9e7e6f1ad33c?> - ORIGINAL_DST/13.107.4.50 application/octet-stream

Some Range requests for URI with ?query parameters. You need to add
"strip_query_terms no" to your squid.conf. Then you will be able to see
if these are identical URI or not.
Right now all that can be said is the script on the server which is
producing objects is the same one for many of the requests. There is no
way to know if they are requesting the same object or different ones.

Post by --Ahmad--
1473672812.922 2687 192.168.0.10 TCP_CLIENT_REFRESH_MISS/206 1049501 GET http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56? <http://tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/ee82a2cb-f2cf-4cb3-907a-4f5496d2dc56?> - ORIGINAL_DST/13.107.4.50 application/octet-stream

A request where the client required new content to be fetched.
The rest of your log shows a lot of repeats of the above things happening.

These refresh_pattern contain broken regex patterns.
1) Unless you explicitly use start/end anchors regex have implicit .*
sequences before and after the provided pattern.
That means "([^.]+.|)" and "([^.]+.)?" are just a very complex (and
slow to compare) ways to write .*
2) '.' is a reserved character. If you want it to match a '.' in the
domain name you need to write it as \.
http://download.com/?pkg=caddy <http://download.com/?pkg=caddy>
Amos
_______________________________________________
squid-users mailing list
http://lists.squid-cache.org/listinfo/squid-users <http://lists.squid-cache.org/listinfo/squid-users>

Amos Jeffries

2016-09-13 03:04:42 UTC

Permalink

Post by --Ahmad--
amos thank you so much for reply
how can i correct the patterns below ?

Remove the ([^.]+.)? and ([^.]+.|) parts.

Replace the . where you want to match a '.' with \.

Post by --Ahmad--
im not sure if the other patterns are doing the game
but so far i don’t have disk increasing at all

but i still don’t have HITs

You have many CLIENT_REFRESH/20x. Which means cached content was found,
but the client required it to be refreshed. When revalidated the server
returned an entire new object.

With the above refresh_rattern, the client may have required the cached
data to be reloaded (Cache-Control:max-age=0) and the reload-into-ims
converted it to a CLIENT_REFRESH from a MISS.

Post by --Ahmad--
im not sure where should i tune ?

There is not much you can do when the server returns whole objects in
response to revalidation requests.

The client is requiring that only the latest copy be used, and the
server is insisting that the object has changed.

Since this is an OS update process it is exremely unsafe to make
assumptions yourself about anything being differently cacheable than
what the server is insisting. It is entirely possible that each response
is encrypted and hashed independently - so actually is different.

Post by --Ahmad--
i also added stip query terms to off
not sure what should i do next ……. all i need is windows updates go ok , I’m not interested with other caching

Eliezer Croitoru

2016-09-13 06:02:55 UTC

Permalink

Just as a side note to the server insisting on the full object:
Since we believe and estimate that SHA digests are considered collision safe on the 100+ MB files sizes for the next couple years it is safe to assume(from my point of view on things) that if the whole file was digested with a SHA1 and forward and the full object was found to match this same SHA1 (or other digest) it is safe to be cached.
However I must admit that if these is some communication channel over this url such as in the headers, it would be possible to assume that you cannot cache this content.
With all the above in mind I think that Caching Windows Updates in most cases might not be the right path.
Only if you can provide much beefier compute and storage resources then the original source it would be smart to cache these.

When an admin will try to cache with a much slower and less performing system then the original one it would be pretty simple to estimate that it will "slow down" things for the clients in many cases.
The practical example I can give is using a 5400RPM spinning disk for a cache which utilizes more then 100 IOPs from disk.

I believe that this should only be used when there are cases which you took anything you can into consideration and consulted any engineer which you should.

All The Bests,
Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: ***@ngtech.co.il

-----Original Message-----
From: squid-users [mailto:squid-users-***@lists.squid-cache.org] On Behalf Of Amos Jeffries
Sent: Tuesday, September 13, 2016 6:05 AM
To: --Ahmad--
Cc: squid-***@lists.squid-cache.org
Subject: Re: [squid-users] caching store increase then decrease during caching windows updates and all request are TCP_MISS ??!!!

Post by --Ahmad--
amos thank you so much for reply
how can i correct the patterns below ?

Remove the ([^.]+.)? and ([^.]+.|) parts.

Replace the . where you want to match a '.' with \.

Post by --Ahmad--
im not sure if the other patterns are doing the game
but so far i don’t have disk increasing at all

but i still don’t have HITs

Post by --Ahmad--
im not sure where should i tune ?

Post by --Ahmad--
i also added stip query terms to off
not sure what should i do next ……. all i need is windows updates go ok , I’m not interested with other caching

With query-terms in the access.log you can better see what requests
should be MISS and which ones have a chance of being HITs (or near-HITs).

Once you can tell which requests actually should be HIT-able you can
then look at the messages going through (in cache.log with debug_options
11,2) for those requests to see what else might be done.

Amos

_______________________________________________
squid-users mailing list
squid-***@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users