[squid-users] Squid issue of caching the m3u8 file

Discussion:

LIU Yaning

2017-06-05 11:54:56 UTC

Dear All,

I would like to cache the .m3u8 file to be able to provide offline caching
service by Squid. The played HLS video streaming is the link as below:
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8

However, the .m3u8 file is not be cached probably because it is mentioned
as a no-cache, no-store, max-age=0 in the "Cache-Control" in the HTTP
header.

HTTP/1.1 200 OK

Server: Apache

ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"

Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT

Expires: Fri, 02 Jun 2017 14:26:52 GMT

Cache-Control: max-age=0, no-cache, no-store

Pragma: no-cache

Date: Fri, 02 Jun 2017 14:26:52 GMT

Content-Length: 16046

Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug 2016
00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net

Content-Type: application/x-mpegURL

Access-Control-Allow-Origin: *

I added a new rule for .m3u8 file in squid.conf, however, it is still not
working.

refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
override-lastmod ignore-no-cache ignore-no-store

Does anyone know how to allow Squid caching the .m3u8 file? Thanks a lot in
advance.

Best Regards,
--
Yaning.

Amos Jeffries

2017-06-06 02:08:11 UTC

Permalink

Post by LIU Yaning
Dear All,
I would like to cache the .m3u8 file to be able to provide offline
caching service by Squid. The played HLS video streaming is the link
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8>
However, the .m3u8 file is not be cached probably because it is
mentioned as a no-cache, no-store, max-age=0 in the "Cache-Control" in
the HTTP header.

Nope. Only the CC:no-store is preventing caching. The other headers
simply put boundaries on what is to be done with the content in the
cache. In particular the "no-cache", max-age=0 and Expires values mean
it has to be revalidated (REFRESH in your access.log) before any future
uses - probably because of that Set-Cookie needing to be changed for
different end-users.

Post by LIU Yaning
HTTP/1.1 200 OK
Server: Apache
ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"
Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT
Expires: Fri, 02 Jun 2017 14:26:52 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 02 Jun 2017 14:26:52 GMT
Content-Length: 16046
Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug
2016 00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net
<http://qthttp.apple.com.edgesuite.net/>
Content-Type: application/x-mpegURL
Access-Control-Allow-Origin: *
I added a new rule for .m3u8 file in squid.conf, however, it is still
not working.
refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
override-lastmod ignore-no-cache ignore-no-store
Does anyone know how to allow Squid caching the .m3u8 file? Thanks a
lot in advance.

What makes you think it is not caching? The ignore-no-store alone should
be sufficient to allow current Squid versions to cache that object. You
could perhapse add "store-stale" option on that config line. Which
should make Squid cache object containing an Expires header with current
or past values. refresh_pattern settings do not affect that cacheable vs
non-cacheable decision. The no-cache header tells Squid the object needs
revalidating before every use. However, be aware the tool at redbot.org
tells me that this URL is badly broken in how it is using the ETag and
Vary headers - in a way which can break the revalidation when these
things are cached. Some of your clients may see very broken behaviour
accessing this object unless you follow the no-store requirement or the
server stops its broken ETag behaviour. PS. if you are using a Squid
version much older than 3.5.24 I recommend an upgrade. With an urgency
increasing the older your Squid is. Amos

LIU Yaning

2017-06-06 13:33:22 UTC

Permalink

Dear Amos,

Thanks a lot for your explanation and suggestion. I added the "store-stale"
to the refresh_pattern rule as:
refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire override-lastmod
ignore-no-cache ignore-no-store store-stale

However, I have checked the access.log, I am still getting TCP_Miss.
1496754869.963 13 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
application/x-mpegURL
1496754870.605 4 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
application/x-mpegURL
1496754871.194 15 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
application/x-mpegURL
1496754871.715 4 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
application/x-mpegURL

If I understand well, TCP_MISS/200 shows the content is not cached by
Squid. Could you please help me to see if anything I did wrong to make
.m3u8 not cached by Squid?

Date: Tue, 6 Jun 2017 14:08:11 +1200
From: Amos Jeffries <***@treenet.co.nz>
To: squid-***@lists.squid-cache.org
Subject: Re: [squid-users] Squid issue of caching the m3u8 file
Message-ID: <5af5e6e9-4880-f58e-106b-***@treenet.co.nz>
Content-Type: text/plain; charset=utf-8; format=flowed

--
Best Regards,
--
Yaning.

Amos Jeffries

2017-06-06 16:37:36 UTC

Permalink

Post by LIU Yaning
Dear Amos,
Thanks a lot for your explanation and suggestion. I added the
refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
override-lastmod ignore-no-cache ignore-no-store store-stale
However, I have checked the access.log, I am still getting TCP_Miss.
1496754869.963 13 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
1496754870.605 4 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
1496754871.194 15 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
1496754871.715 4 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
If I understand well, TCP_MISS/200 shows the content is not cached by
Squid. Could you please help me to see if anything I did wrong to make
.m3u8 not cached by Squid?

I'm out of ideas sorry.

FWIW: this is what I get from my test setup using your refresh_pattern:

1496766607.433 1284 ::1 TCP_MISS/200 16628 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 -
HIER_DIRECT/119.224.143.41 application/x-mpegURL
1496766610.254 902 ::1 TCP_REFRESH_UNMODIFIED/200 16499 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 -
HIER_DIRECT/119.224.143.41 application/x-mpegURL

I am currently testing Squid-4.0.20, but it should behave the same in
the recent few v3.5 releases.

Amos

joseph

2017-06-07 09:42:25 UTC

Permalink

is Set-Cookie: saved in cached file as well ?? amos

--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-issue-of-caching-the-m3u8-file-tp4682674p4682714.html
Sent from the Squid - Users mailing list archive at Nabble.com.

Amos Jeffries

2017-06-07 12:35:22 UTC

Permalink

Post by joseph
is Set-Cookie: saved in cached file as well ?? amos

Yes it is. The header on the cached object only gets removed on delivery
to a client.
Squid does not comply with the Cookie specifications in this regard.

With the extra wrinkle that the mandatory revalidation on these objects
may provide a new Set-Cookie header that gets added for delivery to the
new client after the old/cached header was removed. So you may still see
Set-Cookie on the revalidated HIT (aka REFRESH_UNMODIFIED). Adding a new
Set-Cookie is usually the purpose of the revalidation being required on
these sort of objects.

Amos

joseph

2017-06-07 13:00:37 UTC

Permalink

so if the server send same obj with new Cookie it will be miss since the
Cookie dose not match in cached obj

regarding the topping sorry if i ask it will clear all those question in
all the question has Ben asked before
example cached file header

link =
http://sa.bbc.co.uk/bbc/bbc/s?name=SET-COUNTER.page&ml_name=webmodule&ml_version=63
header

Server: nginx
Date: Thu, 04 May 2017 16:14:41 GMT
Content-Type: image/gif
Content-Length: 43
Connection: close
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
P3P: policyref="http://www.nedstat.com/w3c/p3p.xml", CP="NOI DSP COR NID PSA
ADM OUR IND NAV COM"
so all those should match right to have hit not just the link and some
of the header like Cache-Control or so

even cookie and p3p am i right here
or its just specific header should match ???

like his situation for miss not only the no-cache prevent HIT its cookie
as well
cookie on the link he provide change on every single click to that link

another question what about deleting cookie before save the content
1 since its same object they deliver so it will be hit dose it violate ??
2 or damage the clients web view ?? or anything bad

--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-issue-of-caching-the-m3u8-file-tp4682674p4682722.html
Sent from the Squid - Users mailing list archive at Nabble.com.

Amos Jeffries

2017-06-07 13:56:42 UTC

Permalink

Post by joseph
so if the server send same obj with new Cookie it will be miss since the
Cookie dose not match in cached obj

No, the Vary header does not say Cookie is part of the variance AFAICT.
Just URL plus Accept-Encoding.

The Cookie/Set-Cookie being accurate is just related to whether the
client browsing session is kept continuous or breaks. Depending on
whether this transaction is part of a purchase or something similar that
may be significant, or not.

At a guess since this is a media related object from Apple I suspect it
is linked to an iTunes account of some sort. So the Cookie might be
needed by the client for something. In that case getting a reply without
one (as a pure cache HIT would appear to the client) may have problems.

That is just speculation though to show that this traffic behaviour is
not completely unreasonable. Only the site and client software authors
actually know for sure what is intended to be going on and why.

So, to get back on topic. Yes Squid should be caching it. But don't
expect to see the letters "HIT" in the log anymore for this particular URL.

Amos

joseph

2017-06-07 14:04:36 UTC

Permalink

right

lets say i have obj with this header
if server send the same obj with different Set-Cookie value that will be
MISS i was refairing to this
not to the vary if it has cookie sorry if i did not explain it correctly

Server: nginx
Date: Mon, 22 May 2017 15:44:59 GMT
Content-Type: image/gif
Content-Length: 43
Expires: Fri, 20 Mar 2009 00:00:00 GMT
P3P: CP="CUR ADM OUR NOR STA NID"
Set-Cookie:
ljtrtb=eJyrrgUAAXUA%2BQ%3D%3D;Path=/;Domain=.lijit.com;Expires=Thu,
01-Jan-1970 00:00:00 GMT
Set-Cookie:
lijit_retarget=eJyrrgUAAXUA%2BQ%3D%3D;Path=/;Domain=.lijit.com;Expires=Thu,
01-Jan-1970 00:00:00 GMT
X-Sovrn-Pod: ap3iad3

--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-issue-of-caching-the-m3u8-file-tp4682674p4682724.html
Sent from the Squid - Users mailing list archive at Nabble.com.

Amos Jeffries

2017-06-07 15:11:50 UTC

Permalink

Post by joseph
right
lets say i have obj with this header
if server send the same obj with different Set-Cookie value that will be
MISS i was refairing to this
not to the vary if it has cookie sorry if i did not explain it correctly

Cookie(s) have nothing to do with MISS unless it is listed in Vary.

The fact that the object is coming from a server without involving any
object in the proxies own cache is what makes a transaction be
classified as MISS.

By comparison; things that involve both an object in the proxies cache
and a server are REFRESH, and things that _only_ involve the proxy cache
are HIT.

Amos