Quantcast

How to make squid ignore max-age?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to make squid ignore max-age?

Yang Zhang
We're using squid (3.0.STABLE19-1ubuntu0.1) as an application cache
and we're trying to cache everything, for a long time, but max-age=0
is throwing off squid.

I have this refresh_pattern in squid.conf:

$ grep ^refresh_pattern /etc/squid3/squid.conf
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern . 525600 100% 525600 override-expire ignore-private
ignore-no-cache ignore-no-store

I have the following response prepared to be sent from a dummy web server:

$ cat response
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Expires: 0
Pragma: no-cache
Cache-Control: no-cache, must-revalidate, max-age=0
Content-Type: text/html;charset=UTF-8
Date: Mon, 18 Apr 2011 10:06:26 GMT
Connection: close

<!DOCTYPE html>
<html lang="en">
</html>

$ while true; do nc -l 8765 < response; done

This response prevents anything from being cached; all requests for
http://localhost:8765 result in TCP_MISS.  I believe it's because of
the max-age=0 - once I remove that, this request is cached fine.  Yet
I had set override-expire in the refresh_pattern.

(Note that I don't think it's a must-revalidate/ignore-revalidate (new
in v3.1) problem - I can leave that in fine.)

Any hints on how to make squid ignore the max-age=0?  I've pored over
the refresh_pattern documentation a bunch of times but still can't see
what's wrong.  Thanks in advance for any help.

--
Yang Zhang
http://yz.mit.edu/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to make squid ignore max-age?

Amos Jeffries-2
 On Mon, 18 Apr 2011 20:07:33 -0700, Yang Zhang wrote:
> We're using squid (3.0.STABLE19-1ubuntu0.1) as an application cache
> and we're trying to cache everything, for a long time, but max-age=0
> is throwing off squid.

 max-age=0 is called "reload" in HTTP terminology.

>
> I have this refresh_pattern in squid.conf:
>
> $ grep ^refresh_pattern /etc/squid3/squid.conf
> refresh_pattern ^ftp:           1440    20%     10080
> refresh_pattern ^gopher:        1440    0%      1440
> refresh_pattern . 525600 100% 525600 override-expire ignore-private
> ignore-no-cache ignore-no-store
>

 To quote Frankenstein
  "Why? whyyyy?"


> I have the following response prepared to be sent from a dummy web
> server:
>
> $ cat response
> HTTP/1.1 200 OK
> Server: Apache-Coyote/1.1
> Expires: 0
> Pragma: no-cache
> Cache-Control: no-cache, must-revalidate, max-age=0
> Content-Type: text/html;charset=UTF-8
> Date: Mon, 18 Apr 2011 10:06:26 GMT
> Connection: close

 Those headers say two things:
  * that cached content MUST NOT be supplied to the client. EVER.
  * that any existing cached content also MUST be checked for updates on
 the next request.

 Ignoring this requirement *from the server*. Again... why?


 I can only imagine that you have fallen for the myth that "application
 caches" must always provide HITs. This is false and following it will
 break things for you in potentially nasty ways.

 A cachable application will not send max-age=0 or "Expires: 0", both of
 which mean the object was obsolete the second it was produced.

 Squid operating as a reverse-proxy will cache the replies and keep
 updating fresh copies as long as the server is available to present new
 ones. If the server dies an 5xx start coming back, the stale versions
 will start to be used. How long for and whether "must-revalidate"
 produces 4xx/5xx to the client depends on particular squid versions.
  I recommend you grab the newest release you can for the best HTTP/1.1
 caching behaviour. there are still things we are fixing but work is
 ongoing so the newer the release the better things work. I maintain a
 PPA for Ubuntu (https://launchpad.net/~yadi/+archive/ppa) with newer
 versions.

 <snip>
>
> This response prevents anything from being cached; all requests for
> http://localhost:8765 result in TCP_MISS.  I believe it's because of
> the max-age=0 - once I remove that, this request is cached fine.  Yet
> I had set override-expire in the refresh_pattern.
> (Note that I don't think it's a must-revalidate/ignore-revalidate
> (new
> in v3.1) problem - I can leave that in fine.)

 However "We're using squid (3.0.STABLE19-1ubuntu0.1)" ... not a v3.1
 release.

 "ignore-revalidate" blocks the *client* from requesting a revalidation
 (max-age=0). But still obeys the server controls.

>
> Any hints on how to make squid ignore the max-age=0?  I've pored over
> the refresh_pattern documentation a bunch of times but still can't
> see
> what's wrong.  Thanks in advance for any help.

 If you must;
  * "ignore-reload" to ignore the max-age=0 from either client or server
 IIRC.
  * "reload-into-ims" to convert a client max-age=0 ("force a new copy
 or die") into a soft IMS request ("new content or cached copy").

 'must-revalidate' will still have affect until you are using a 3.1
 release with the "ignore-must-revalidate". The result of revalidate
 should only be 304 (TCP_IMS_*) or 200 (TCP_REFRESH_UPDATE), not a
 "TCP_MISS" since the content being validated *is* in the cache.

 Amos
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to make squid ignore max-age?

Yang Zhang
On Mon, Apr 18, 2011 at 9:08 PM, Amos Jeffries <[hidden email]> wrote:

> On Mon, 18 Apr 2011 20:07:33 -0700, Yang Zhang wrote:
>>
>> We're using squid (3.0.STABLE19-1ubuntu0.1) as an application cache
>> and we're trying to cache everything, for a long time, but max-age=0
>> is throwing off squid.
>
> max-age=0 is called "reload" in HTTP terminology.
>
>>
>> I have this refresh_pattern in squid.conf:
>>
>> $ grep ^refresh_pattern /etc/squid3/squid.conf
>> refresh_pattern ^ftp:           1440    20%     10080
>> refresh_pattern ^gopher:        1440    0%      1440
>> refresh_pattern . 525600 100% 525600 override-expire ignore-private
>> ignore-no-cache ignore-no-store
>>
>
> To quote Frankenstein
>  "Why? whyyyy?"
>
>
>> I have the following response prepared to be sent from a dummy web server:
>>
>> $ cat response
>> HTTP/1.1 200 OK
>> Server: Apache-Coyote/1.1
>> Expires: 0
>> Pragma: no-cache
>> Cache-Control: no-cache, must-revalidate, max-age=0
>> Content-Type: text/html;charset=UTF-8
>> Date: Mon, 18 Apr 2011 10:06:26 GMT
>> Connection: close
>
> Those headers say two things:
>  * that cached content MUST NOT be supplied to the client. EVER.
>  * that any existing cached content also MUST be checked for updates on the
> next request.
>
> Ignoring this requirement *from the server*. Again... why?
>
>
> I can only imagine that you have fallen for the myth that "application
> caches" must always provide HITs. This is false and following it will break
> things for you in potentially nasty ways.
>
> A cachable application will not send max-age=0 or "Expires: 0", both of
> which mean the object was obsolete the second it was produced.
>
> Squid operating as a reverse-proxy will cache the replies and keep updating
> fresh copies as long as the server is available to present new ones. If the
> server dies an 5xx start coming back, the stale versions will start to be
> used. How long for and whether "must-revalidate" produces 4xx/5xx to the
> client depends on particular squid versions.
>  I recommend you grab the newest release you can for the best HTTP/1.1
> caching behaviour. there are still things we are fixing but work is ongoing
> so the newer the release the better things work. I maintain a PPA for Ubuntu
> (https://launchpad.net/~yadi/+archive/ppa) with newer versions.
>
> <snip>
>>
>> This response prevents anything from being cached; all requests for
>> http://localhost:8765 result in TCP_MISS.  I believe it's because of
>> the max-age=0 - once I remove that, this request is cached fine.  Yet
>> I had set override-expire in the refresh_pattern.
>> (Note that I don't think it's a must-revalidate/ignore-revalidate (new
>> in v3.1) problem - I can leave that in fine.)
>
> However "We're using squid (3.0.STABLE19-1ubuntu0.1)" ... not a v3.1
> release.
>
> "ignore-revalidate" blocks the *client* from requesting a revalidation
> (max-age=0). But still obeys the server controls.
>
>>
>> Any hints on how to make squid ignore the max-age=0?  I've pored over
>> the refresh_pattern documentation a bunch of times but still can't see
>> what's wrong.  Thanks in advance for any help.
>
> If you must;
>  * "ignore-reload" to ignore the max-age=0 from either client or server
> IIRC.

Just tried this out - it doesn't work.  The comments in the squid.conf
also say that it's for max-age=0 from the client (no mention of
server).

>  * "reload-into-ims" to convert a client max-age=0 ("force a new copy or
> die") into a soft IMS request ("new content or cached copy").

The client is not issuing any max-age=0.  Tried adding this in anyway
- didn't work.

>
> 'must-revalidate' will still have affect until you are using a 3.1 release
> with the "ignore-must-revalidate". The result of revalidate should only be
> 304 (TCP_IMS_*) or 200 (TCP_REFRESH_UPDATE), not a "TCP_MISS" since the
> content being validated *is* in the cache.

My point earlier is that the must-revalidate from the server doesn't
seem be what's preventing the response from getting cached....

>
> Amos
>



--
Yang Zhang
http://yz.mit.edu/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to make squid ignore max-age?

Yang Zhang
On Mon, Apr 18, 2011 at 9:18 PM, Yang Zhang <[hidden email]> wrote:

> On Mon, Apr 18, 2011 at 9:08 PM, Amos Jeffries <[hidden email]> wrote:
>> On Mon, 18 Apr 2011 20:07:33 -0700, Yang Zhang wrote:
>>>
>>> We're using squid (3.0.STABLE19-1ubuntu0.1) as an application cache
>>> and we're trying to cache everything, for a long time, but max-age=0
>>> is throwing off squid.
>>
>> max-age=0 is called "reload" in HTTP terminology.
>>
>>>
>>> I have this refresh_pattern in squid.conf:
>>>
>>> $ grep ^refresh_pattern /etc/squid3/squid.conf
>>> refresh_pattern ^ftp:           1440    20%     10080
>>> refresh_pattern ^gopher:        1440    0%      1440
>>> refresh_pattern . 525600 100% 525600 override-expire ignore-private
>>> ignore-no-cache ignore-no-store
>>>
>>
>> To quote Frankenstein
>>  "Why? whyyyy?"
>>
>>
>>> I have the following response prepared to be sent from a dummy web server:
>>>
>>> $ cat response
>>> HTTP/1.1 200 OK
>>> Server: Apache-Coyote/1.1
>>> Expires: 0
>>> Pragma: no-cache
>>> Cache-Control: no-cache, must-revalidate, max-age=0
>>> Content-Type: text/html;charset=UTF-8
>>> Date: Mon, 18 Apr 2011 10:06:26 GMT
>>> Connection: close
>>
>> Those headers say two things:
>>  * that cached content MUST NOT be supplied to the client. EVER.
>>  * that any existing cached content also MUST be checked for updates on the
>> next request.
>>
>> Ignoring this requirement *from the server*. Again... why?
>>
>>
>> I can only imagine that you have fallen for the myth that "application
>> caches" must always provide HITs. This is false and following it will break
>> things for you in potentially nasty ways.
>>
>> A cachable application will not send max-age=0 or "Expires: 0", both of
>> which mean the object was obsolete the second it was produced.
>>
>> Squid operating as a reverse-proxy will cache the replies and keep updating
>> fresh copies as long as the server is available to present new ones. If the
>> server dies an 5xx start coming back, the stale versions will start to be
>> used. How long for and whether "must-revalidate" produces 4xx/5xx to the
>> client depends on particular squid versions.
>>  I recommend you grab the newest release you can for the best HTTP/1.1
>> caching behaviour. there are still things we are fixing but work is ongoing
>> so the newer the release the better things work. I maintain a PPA for Ubuntu
>> (https://launchpad.net/~yadi/+archive/ppa) with newer versions.
>>
>> <snip>
>>>
>>> This response prevents anything from being cached; all requests for
>>> http://localhost:8765 result in TCP_MISS.  I believe it's because of
>>> the max-age=0 - once I remove that, this request is cached fine.  Yet
>>> I had set override-expire in the refresh_pattern.
>>> (Note that I don't think it's a must-revalidate/ignore-revalidate (new
>>> in v3.1) problem - I can leave that in fine.)
>>
>> However "We're using squid (3.0.STABLE19-1ubuntu0.1)" ... not a v3.1
>> release.
>>
>> "ignore-revalidate" blocks the *client* from requesting a revalidation
>> (max-age=0). But still obeys the server controls.
>>
>>>
>>> Any hints on how to make squid ignore the max-age=0?  I've pored over
>>> the refresh_pattern documentation a bunch of times but still can't see
>>> what's wrong.  Thanks in advance for any help.
>>
>> If you must;
>>  * "ignore-reload" to ignore the max-age=0 from either client or server
>> IIRC.
>
> Just tried this out - it doesn't work.  The comments in the squid.conf
> also say that it's for max-age=0 from the client (no mention of
> server).
>
>>  * "reload-into-ims" to convert a client max-age=0 ("force a new copy or
>> die") into a soft IMS request ("new content or cached copy").
>
> The client is not issuing any max-age=0.  Tried adding this in anyway
> - didn't work.
>
>>
>> 'must-revalidate' will still have affect until you are using a 3.1 release
>> with the "ignore-must-revalidate". The result of revalidate should only be
>> 304 (TCP_IMS_*) or 200 (TCP_REFRESH_UPDATE), not a "TCP_MISS" since the
>> content being validated *is* in the cache.
>
> My point earlier is that the must-revalidate from the server doesn't
> seem be what's preventing the response from getting cached....

Also, FWIW, these -max-age=0 responses are making it *into* the cache
(verified by grepping /var/spool/squid3), for whatever reason (not
sure why Squid is deciding to keep them if only to later not use
them).

--
Yang Zhang
http://yz.mit.edu/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to make squid ignore max-age?

Yang Zhang
On Tue, Apr 19, 2011 at 8:03 PM, Yang Zhang <[hidden email]> wrote:

> On Mon, Apr 18, 2011 at 9:18 PM, Yang Zhang <[hidden email]> wrote:
>> On Mon, Apr 18, 2011 at 9:08 PM, Amos Jeffries <[hidden email]> wrote:
>>> On Mon, 18 Apr 2011 20:07:33 -0700, Yang Zhang wrote:
>>>>
>>>> We're using squid (3.0.STABLE19-1ubuntu0.1) as an application cache
>>>> and we're trying to cache everything, for a long time, but max-age=0
>>>> is throwing off squid.
>>>
>>> max-age=0 is called "reload" in HTTP terminology.
>>>
>>>>
>>>> I have this refresh_pattern in squid.conf:
>>>>
>>>> $ grep ^refresh_pattern /etc/squid3/squid.conf
>>>> refresh_pattern ^ftp:           1440    20%     10080
>>>> refresh_pattern ^gopher:        1440    0%      1440
>>>> refresh_pattern . 525600 100% 525600 override-expire ignore-private
>>>> ignore-no-cache ignore-no-store
>>>>
>>>
>>> To quote Frankenstein
>>>  "Why? whyyyy?"
>>>
>>>
>>>> I have the following response prepared to be sent from a dummy web server:
>>>>
>>>> $ cat response
>>>> HTTP/1.1 200 OK
>>>> Server: Apache-Coyote/1.1
>>>> Expires: 0
>>>> Pragma: no-cache
>>>> Cache-Control: no-cache, must-revalidate, max-age=0
>>>> Content-Type: text/html;charset=UTF-8
>>>> Date: Mon, 18 Apr 2011 10:06:26 GMT
>>>> Connection: close
>>>
>>> Those headers say two things:
>>>  * that cached content MUST NOT be supplied to the client. EVER.
>>>  * that any existing cached content also MUST be checked for updates on the
>>> next request.
>>>
>>> Ignoring this requirement *from the server*. Again... why?
>>>
>>>
>>> I can only imagine that you have fallen for the myth that "application
>>> caches" must always provide HITs. This is false and following it will break
>>> things for you in potentially nasty ways.
>>>
>>> A cachable application will not send max-age=0 or "Expires: 0", both of
>>> which mean the object was obsolete the second it was produced.
>>>
>>> Squid operating as a reverse-proxy will cache the replies and keep updating
>>> fresh copies as long as the server is available to present new ones. If the
>>> server dies an 5xx start coming back, the stale versions will start to be
>>> used. How long for and whether "must-revalidate" produces 4xx/5xx to the
>>> client depends on particular squid versions.
>>>  I recommend you grab the newest release you can for the best HTTP/1.1
>>> caching behaviour. there are still things we are fixing but work is ongoing
>>> so the newer the release the better things work. I maintain a PPA for Ubuntu
>>> (https://launchpad.net/~yadi/+archive/ppa) with newer versions.
>>>
>>> <snip>
>>>>
>>>> This response prevents anything from being cached; all requests for
>>>> http://localhost:8765 result in TCP_MISS.  I believe it's because of
>>>> the max-age=0 - once I remove that, this request is cached fine.  Yet
>>>> I had set override-expire in the refresh_pattern.
>>>> (Note that I don't think it's a must-revalidate/ignore-revalidate (new
>>>> in v3.1) problem - I can leave that in fine.)
>>>
>>> However "We're using squid (3.0.STABLE19-1ubuntu0.1)" ... not a v3.1
>>> release.
>>>
>>> "ignore-revalidate" blocks the *client* from requesting a revalidation
>>> (max-age=0). But still obeys the server controls.
>>>
>>>>
>>>> Any hints on how to make squid ignore the max-age=0?  I've pored over
>>>> the refresh_pattern documentation a bunch of times but still can't see
>>>> what's wrong.  Thanks in advance for any help.
>>>
>>> If you must;
>>>  * "ignore-reload" to ignore the max-age=0 from either client or server
>>> IIRC.
>>
>> Just tried this out - it doesn't work.  The comments in the squid.conf
>> also say that it's for max-age=0 from the client (no mention of
>> server).
>>
>>>  * "reload-into-ims" to convert a client max-age=0 ("force a new copy or
>>> die") into a soft IMS request ("new content or cached copy").
>>
>> The client is not issuing any max-age=0.  Tried adding this in anyway
>> - didn't work.
>>
>>>
>>> 'must-revalidate' will still have affect until you are using a 3.1 release
>>> with the "ignore-must-revalidate". The result of revalidate should only be
>>> 304 (TCP_IMS_*) or 200 (TCP_REFRESH_UPDATE), not a "TCP_MISS" since the
>>> content being validated *is* in the cache.
>>
>> My point earlier is that the must-revalidate from the server doesn't
>> seem be what's preventing the response from getting cached....
>
> Also, FWIW, these -max-age=0 responses are making it *into* the cache
> (verified by grepping /var/spool/squid3), for whatever reason (not
> sure why Squid is deciding to keep them if only to later not use
> them).
>
> --
> Yang Zhang
> http://yz.mit.edu/
>

Update: using Squid 3.1.12.1 and ignore-must-revalidate fixed this.
Weird since the original problem only arose from the presence of
max-age=0; the revalidate itself wasn't preventing caching.

--
Yang Zhang
http://yz.mit.edu/
Loading...