Squid issue of caching the m3u8 file

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Squid issue of caching the m3u8 file

LIU Yaning
Dear All,

I would like to cache the .m3u8 file to be able to provide offline caching service by Squid. The played HLS video streaming is the link as below:

However, the .m3u8 file is not be cached probably because it is mentioned as a no-cache, no-store, max-age=0 in the "Cache-Control" in the HTTP header.

HTTP/1.1 200 OK

Server: Apache

ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"

Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT

Expires: Fri, 02 Jun 2017 14:26:52 GMT

Cache-Control: max-age=0, no-cache, no-store

Pragma: no-cache

Date: Fri, 02 Jun 2017 14:26:52 GMT

Content-Length: 16046

Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug 2016 00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net

Content-Type: application/x-mpegURL

Access-Control-Allow-Origin: *


I added a new rule for .m3u8 file in squid.conf, however, it is still not working.

refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire override-lastmod ignore-no-cache ignore-no-store


Does anyone know how to allow Squid caching the .m3u8 file? Thanks a lot in advance.

Best Regards,
--
Yaning.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

Amos Jeffries
Administrator
On 05/06/17 23:54, LIU Yaning wrote:

> Dear All,
>
> I would like to cache the .m3u8 file to be able to provide offline
> caching service by Squid. The played HLS video streaming is the link
> as below:
> http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
> <http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8>
> However, the .m3u8 file is not be cached probably because it is
> mentioned as a no-cache, no-store, max-age=0 in the "Cache-Control" in
> the HTTP header.
Nope. Only the CC:no-store is preventing caching. The other headers
simply put boundaries on what is to be done with the content in the
cache. In particular the "no-cache", max-age=0 and Expires values mean
it has to be revalidated (REFRESH in your access.log) before any future
uses - probably because of that Set-Cookie needing to be changed for
different end-users.

>
> HTTP/1.1 200 OK
>
> Server: Apache
>
> ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"
>
> Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT
>
> Expires: Fri, 02 Jun 2017 14:26:52 GMT
>
> Cache-Control: max-age=0, no-cache, no-store
>
> Pragma: no-cache
>
> Date: Fri, 02 Jun 2017 14:26:52 GMT
>
> Content-Length: 16046
>
> Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug
> 2016 00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net
> <http://qthttp.apple.com.edgesuite.net/>
>
> Content-Type: application/x-mpegURL
>
> Access-Control-Allow-Origin: *
>
> I added a new rule for .m3u8 file in squid.conf, however, it is still
> not working.
>
> refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
> override-lastmod ignore-no-cache ignore-no-store
>
> Does anyone know how to allow Squid caching the .m3u8 file? Thanks a
> lot in advance.
What makes you think it is not caching? The ignore-no-store alone should
be sufficient to allow current Squid versions to cache that object. You
could perhapse add "store-stale" option on that config line. Which
should make Squid cache object containing an Expires header with current
or past values. refresh_pattern settings do not affect that cacheable vs
non-cacheable decision. The no-cache header tells Squid the object needs
revalidating before every use. However, be aware the tool at redbot.org
tells me that this URL is badly broken in how it is using the ETag and
Vary headers - in a way which can break the revalidation when these
things are cached. Some of your clients may see very broken behaviour
accessing this object unless you follow the no-store requirement or the
server stops its broken ETag behaviour. PS. if you are using a Squid
version much older than 3.5.24 I recommend an upgrade. With an urgency
increasing the older your Squid is. Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

LIU Yaning
In reply to this post by LIU Yaning
Dear Amos,

Thanks a lot for your explanation and suggestion. I added the "store-stale" to the refresh_pattern rule as:
refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire override-lastmod ignore-no-cache ignore-no-store store-stale

However, I have checked the access.log, I am still getting TCP_Miss. 
1496754869.963 13 192.168.0.100 TCP_MISS/200 16636 GET http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 - HIER_DIRECT/95.101.182.201 application/x-mpegURL
1496754870.605 4 192.168.0.100 TCP_MISS/200 16636 GET http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 - HIER_DIRECT/95.101.182.201 application/x-mpegURL
1496754871.194 15 192.168.0.100 TCP_MISS/200 16636 GET http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 - HIER_DIRECT/95.101.182.201 application/x-mpegURL
1496754871.715 4 192.168.0.100 TCP_MISS/200 16636 GET http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 - HIER_DIRECT/95.101.182.201 application/x-mpegURL

If I understand well, TCP_MISS/200 shows the content is not cached by Squid. Could you please help me to see if anything I did wrong to make .m3u8 not cached by Squid?


Date: Tue, 6 Jun 2017 14:08:11 +1200
From: Amos Jeffries <[hidden email]>
To: [hidden email]
Subject: Re: [squid-users] Squid issue of caching the m3u8 file
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=utf-8; format=flowed

On 05/06/17 23:54, LIU Yaning wrote:
> Dear All,
>
> I would like to cache the .m3u8 file to be able to provide offline
> caching service by Squid. The played HLS video streaming is the link
> as below:
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
> <http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8>
> However, the .m3u8 file is not be cached probably because it is
> mentioned as a no-cache, no-store, max-age=0 in the "Cache-Control" in
> the HTTP header.
Nope. Only the CC:no-store is preventing caching. The other headers
simply put boundaries on what is to be done with the content in the
cache. In particular the "no-cache", max-age=0 and Expires values mean
it has to be revalidated (REFRESH in your access.log) before any future
uses - probably because of that Set-Cookie needing to be changed for
different end-users.
>
> HTTP/1.1 200 OK
>
> Server: Apache
>
> ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"
>
> Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT
>
> Expires: Fri, 02 Jun 2017 14:26:52 GMT
>
> Cache-Control: max-age=0, no-cache, no-store
>
> Pragma: no-cache
>
> Date: Fri, 02 Jun 2017 14:26:52 GMT
>
> Content-Length: 16046
>
> Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug
> 2016 00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net
> <http://qthttp.apple.com.edgesuite.net/>
>
> Content-Type: application/x-mpegURL
>
> Access-Control-Allow-Origin: *
>
> I added a new rule for .m3u8 file in squid.conf, however, it is still
> not working.
>
> refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
> override-lastmod ignore-no-cache ignore-no-store
>
> Does anyone know how to allow Squid caching the .m3u8 file? Thanks a
> lot in advance.
What makes you think it is not caching? The ignore-no-store alone should
be sufficient to allow current Squid versions to cache that object. You
could perhapse add "store-stale" option on that config line. Which
should make Squid cache object containing an Expires header with current
or past values. refresh_pattern settings do not affect that cacheable vs
non-cacheable decision. The no-cache header tells Squid the object needs
revalidating before every use. However, be aware the tool at redbot.org
tells me that this URL is badly broken in how it is using the ETag and
Vary headers - in a way which can break the revalidation when these
things are cached. Some of your clients may see very broken behaviour
accessing this object unless you follow the no-store requirement or the
server stops its broken ETag behaviour. PS. if you are using a Squid
version much older than 3.5.24 I recommend an upgrade. With an urgency
increasing the older your Squid is. Amos



--
Best Regards,
--
Yaning.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

Amos Jeffries
Administrator
On 07/06/17 01:33, LIU Yaning wrote:

> Dear Amos,
>
> Thanks a lot for your explanation and suggestion. I added the
> "store-stale" to the refresh_pattern rule as:
> refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
> override-lastmod ignore-no-cache ignore-no-store store-stale
>
> However, I have checked the access.log, I am still getting TCP_Miss.
> 1496754869.963 13 192.168.0.100 TCP_MISS/200 16636 GET
> http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
> <http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
> HIER_DIRECT/95.101.182.201
> <http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
> 1496754870.605 4 192.168.0.100 TCP_MISS/200 16636 GET
> http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
> <http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
> HIER_DIRECT/95.101.182.201
> <http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
> 1496754871.194 15 192.168.0.100 TCP_MISS/200 16636 GET
> http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
> <http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
> HIER_DIRECT/95.101.182.201
> <http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
> 1496754871.715 4 192.168.0.100 TCP_MISS/200 16636 GET
> http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
> <http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg> -
> HIER_DIRECT/95.101.182.201
> <http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q> application/x-mpegURL
>
> If I understand well, TCP_MISS/200 shows the content is not cached by
> Squid. Could you please help me to see if anything I did wrong to make
> .m3u8 not cached by Squid?

I'm out of ideas sorry.


FWIW: this is what I get from my test setup using your refresh_pattern:

1496766607.433   1284 ::1 TCP_MISS/200 16628 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 -
HIER_DIRECT/119.224.143.41 application/x-mpegURL
1496766610.254    902 ::1 TCP_REFRESH_UNMODIFIED/200 16499 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 -
HIER_DIRECT/119.224.143.41 application/x-mpegURL

I am currently testing Squid-4.0.20, but it should behave the same in
the recent few v3.5 releases.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

gutemberg.aci
This post has NOT been accepted by the mailing list yet.
Https youtube cache does not work with version higher than 3.5.20

I'm trying to pass this on, no one else understands.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

joseph
In reply to this post by Amos Jeffries
is Set-Cookie:  saved in cached file as well ?? amos
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

Amos Jeffries
Administrator
On 07/06/17 21:42, joseph wrote:
> is Set-Cookie:  saved in cached file as well ?? amos

Yes it is. The header on the cached object only gets removed on delivery
to a client.
Squid does not comply with the Cookie specifications in this regard.

With the extra wrinkle that the mandatory revalidation on these objects
may provide a new Set-Cookie header that gets added for delivery to the
new client after the old/cached header was removed. So you may still see
Set-Cookie on the revalidated HIT (aka REFRESH_UNMODIFIED). Adding a new
Set-Cookie is usually the purpose of the revalidation being required on
these sort of objects.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

joseph
so if the  server send   same obj  with new Cookie it will be miss  since the Cookie dose not match in cached obj

regarding the topping sorry if i ask it will clear  all those question  in all the question has Ben asked before
example cached file header

link   =  http://sa.bbc.co.uk/bbc/bbc/s?name=SET-COUNTER.page&ml_name=webmodule&ml_version=63
header

Server: nginx
Date: Thu, 04 May 2017 16:14:41 GMT
Content-Type: image/gif
Content-Length: 43
Connection: close
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
P3P: policyref="http://www.nedstat.com/w3c/p3p.xml", CP="NOI DSP COR NID PSA ADM OUR IND NAV COM"
so all those  should match right  to have hit  not  just the link and  some of the header  like Cache-Control or so

even cookie and   p3p   am i right here
or  its just specific header should match ???

like his situation  for miss not only  the no-cache prevent HIT  its  cookie as well
cookie on the link he provide  change on every single  click to that link

another question  what about  deleting  cookie  before save the content
1 since  its same object they deliver so it will be  hit  dose it violate ??
2 or  damage the clients  web view ?? or anything bad



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

Amos Jeffries
Administrator
On 08/06/17 01:00, joseph wrote:
> so if the  server send   same obj  with new Cookie it will be miss  since the
> Cookie dose not match in cached obj

No, the Vary header does not say Cookie is part of the variance AFAICT.
Just URL plus Accept-Encoding.

The Cookie/Set-Cookie being accurate is just related to whether the
client browsing session is kept continuous or breaks. Depending on
whether this transaction is part of a purchase or something similar that
may be significant, or not.

At a guess since this is a media related object from Apple I suspect it
is linked to an iTunes account of some sort. So the Cookie might be
needed by the client for something. In that case getting a reply without
one (as a pure cache HIT would appear to the client) may have problems.

That is just speculation though to show that this traffic behaviour is
not completely unreasonable. Only the site and client software authors
actually know for sure what is intended to be going on and why.


So, to get back on topic. Yes Squid should be caching it. But don't
expect to see the letters "HIT" in the log anymore for this particular URL.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

joseph
right

lets say i have obj  with this header
if server  send the same  obj  with different Set-Cookie value that will be MISS i was refairing to this
not to the vary if it has cookie sorry if i did not explain it correctly

Server: nginx
Date: Mon, 22 May 2017 15:44:59 GMT
Content-Type: image/gif
Content-Length: 43
Expires: Fri, 20 Mar 2009 00:00:00 GMT
P3P: CP="CUR ADM OUR NOR STA NID"
Set-Cookie: ljtrtb=eJyrrgUAAXUA%2BQ%3D%3D;Path=/;Domain=.lijit.com;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: lijit_retarget=eJyrrgUAAXUA%2BQ%3D%3D;Path=/;Domain=.lijit.com;Expires=Thu, 01-Jan-1970 00:00:00 GMT
X-Sovrn-Pod: ap3iad3
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid issue of caching the m3u8 file

Amos Jeffries
Administrator
On 08/06/17 02:04, joseph wrote:
> right
>
> lets say i have obj  with this header
> if server  send the same  obj  with different Set-Cookie value that will be
> MISS i was refairing to this
> not to the vary if it has cookie sorry if i did not explain it correctly

Cookie(s) have nothing to do with MISS unless it is listed in Vary.

The fact that the object is coming from a server without involving any
object in the proxies own cache is what makes a transaction be
classified as MISS.

By comparison; things that involve both an object in the proxies cache
and a server are REFRESH, and things that _only_ involve the proxy cache
are HIT.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...