squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Miroslaw Malinowski
Hi,

Probably, me missing on something silly or it can't be done but I don't know why but squid won't return the cached version even when I turn all override options ON in refresh_pattern. It's an API call where we call many of the same requests and by knowing it we would like to stop those calls to go out if it's already been sent once.
With debug, I can see the rule is matched and the cache is fresh but still in access.log is TCP_REFRESH_MODIFIED

squid conf:
refresh_pattern -i <URL> 4320 80% 129600 override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

curl headers:
curl --insecure --verbose --request GET --url 'URL' >/dev/null
* TCP_NODELAY set
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
 CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [6 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1956 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

> GET URL HTTP/1.1
> Host: URL
> User-Agent: curl/7.68.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [217 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [217 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Content-Type: application/json
< X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
< Date: Wed, 24 Mar 2021 15:04:34 GMT
< Server: Google Frontend
< Content-Length: 7950
< X-Cache: MISS from server
< X-Cache-Lookup: HIT from server
< Via: 1.1 server (squid/4.14)
< Connection: keep-alive

access log:
243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL - ORIGINAL_DST/IP application/json

cache log:
2021-03-24T15:04:34squid.710 kid1| 11,3| http.cc(982) haveParsedReplyHeaders: decided: cache positively and share because refresh check returned cacheable; HTTP status 200 e:=p2V/0x34868914670*3 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(470) refreshCheck: returning FRESH_MIN_RULE 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(455) refreshCheck: Object isn't stale.. 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(327) refreshCheck: Staleness = -1 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(199) refreshStaleness: FRESH: age (60 sec) is less than configured minimum (259200 sec) 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(166) refreshStaleness: No explicit expiry given, using heuristics to determine freshness 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(307) refreshCheck: entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(305) refreshCheck: check_time: Wed, 24 Mar 2021 15:05:34 GMT 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(303) refreshCheck: age: 60 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(301) refreshCheck: Matched 'URL 259200 80%% 7776000' 
2021-03-24T15:04:34squid.710 kid1| 22,3| refresh.cc(279) refreshCheck: checking freshness of URI: https://URL

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Alex Rousskov
On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:

> Probably, me missing on something silly or it can't be done but I don't
> know why but squid won't return the cached version even when I turn all
> override options ON in refresh_pattern.

AFAICT, no configuration options that can disable revalidation of
Cache-Control:no-cache responses. refresh_pattern does not have an
(equivalent of) "ignore-no-cache-in-responses" option.

IIRC, older Squids were violating an HTTP MUST by forgetting to
revalidate Cache-Control:no-cache responses, but that was fixed in [1].
Your Squid version has that fix.

[1]
https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa


> With debug, I can see the rule is matched and the cache is fresh but
> still in access.log is TCP_REFRESH_MODIFIED

> 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982)
> haveParsedReplyHeaders: decided: cache positively and share because

FYI: You are looking at cache.log lines logged _after_ Squid has already
decided to refresh the cached version. If you want to analyze why Squid
decided to refresh the cached version, you should look _before_ Squid
logged the request to the server (and before any FwdState.cc lines). I
have not checked the details, but I bet that your Squid revalidates
because of Cache-Control:no-cache in the response. Look for "YES: Must
revalidate stale object".


HTH,

Alex.

> squid conf:
> refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
> override-expire ignore-reload ignore-no-store ignore-private store-stale
>
> curl headers:
> curl --insecure --verbose --request GET --url 'URL' >/dev/null
> * TCP_NODELAY set
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> *   CAfile: /etc/ssl/certs/ca-certificates.crt
>  CApath: /etc/ssl/certs
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [512 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [122 bytes data]
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> { [6 bytes data]
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> { [1956 bytes data]
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> { [78 bytes data]
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> { [52 bytes data]
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> } [52 bytes data]
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>
>> GET URL HTTP/1.1
>> Host: URL
>> User-Agent: curl/7.68.0
>> Accept: */*
>>
> { [5 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * old SSL session ID is stale, removing
> { [5 bytes data]
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Cache-Control: no-cache
> < Content-Type: application/json
> < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
> < Date: Wed, 24 Mar 2021 15:04:34 GMT
> < Server: Google Frontend
> < Content-Length: 7950
> < X-Cache: MISS from server
> < X-Cache-Lookup: HIT from server
> < Via: 1.1 server (squid/4.14)
> < Connection: keep-alive
>
> access log:
> 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
> ORIGINAL_DST/IP application/json
>
> cache log:
> 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982)
> haveParsedReplyHeaders: decided: cache positively and share because
> refresh check returned cacheable; HTTP status 200 e:=p2V/0x34868914670*3  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(470) refreshCheck:
> returning FRESH_MIN_RULE  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(455) refreshCheck:
> Object isn't stale..  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(327) refreshCheck:
> Staleness = -1  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(199)
> refreshStaleness: FRESH: age (60 sec) is less than configured minimum
> (259200 sec)  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(166)
> refreshStaleness: No explicit expiry given, using heuristics to
> determine freshness  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(307) refreshCheck:
> entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(305) refreshCheck:
> check_time: Wed, 24 Mar 2021 15:05:34 GMT  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(303) refreshCheck:
> age: 60  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(301) refreshCheck:
> Matched 'URL 259200 80%% 7776000'  
> 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(279) refreshCheck:
> checking freshness of URI: https://URL <https://URL>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Miroslaw Malinowski
Hi,

You've right yes it's revalidating as API server I'm requesting data is setting Cache-Control: no-cache. My question is how I can force squid to cache and not validate as I know it's safe to do so. As I've explained earlier we are making the same request and receiving the same response from 100+ server so as to reduce number of requests to the external server we would like squid to cache the response and issue a cached version.

2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck: YES: Must revalidate stale object (origin set no-cache or private)

Mirek

On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov <[hidden email]> wrote:
On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:

> Probably, me missing on something silly or it can't be done but I don't
> know why but squid won't return the cached version even when I turn all
> override options ON in refresh_pattern.

AFAICT, no configuration options that can disable revalidation of
Cache-Control:no-cache responses. refresh_pattern does not have an
(equivalent of) "ignore-no-cache-in-responses" option.

IIRC, older Squids were violating an HTTP MUST by forgetting to
revalidate Cache-Control:no-cache responses, but that was fixed in [1].
Your Squid version has that fix.

[1]
https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa


> With debug, I can see the rule is matched and the cache is fresh but
> still in access.log is TCP_REFRESH_MODIFIED

> 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
> haveParsedReplyHeaders: decided: cache positively and share because

FYI: You are looking at cache.log lines logged _after_ Squid has already
decided to refresh the cached version. If you want to analyze why Squid
decided to refresh the cached version, you should look _before_ Squid
logged the request to the server (and before any FwdState.cc lines). I
have not checked the details, but I bet that your Squid revalidates
because of Cache-Control:no-cache in the response. Look for "YES: Must
revalidate stale object".


HTH,

Alex.

> squid conf:
> refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
> override-expire ignore-reload ignore-no-store ignore-private store-stale
>
> curl headers:
> curl --insecure --verbose --request GET --url 'URL' >/dev/null
> * TCP_NODELAY set
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> *   CAfile: /etc/ssl/certs/ca-certificates.crt
>  CApath: /etc/ssl/certs
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [512 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [122 bytes data]
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> { [6 bytes data]
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> { [1956 bytes data]
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> { [78 bytes data]
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> { [52 bytes data]
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> } [52 bytes data]
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>
>> GET URL HTTP/1.1
>> Host: URL
>> User-Agent: curl/7.68.0
>> Accept: */*
>>
> { [5 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * old SSL session ID is stale, removing
> { [5 bytes data]
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Cache-Control: no-cache
> < Content-Type: application/json
> < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
> < Date: Wed, 24 Mar 2021 15:04:34 GMT
> < Server: Google Frontend
> < Content-Length: 7950
> < X-Cache: MISS from server
> < X-Cache-Lookup: HIT from server
> < Via: 1.1 server (squid/4.14)
> < Connection: keep-alive
>
> access log:
> 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
> ORIGINAL_DST/IP application/json
>
> cache log:
> 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
> haveParsedReplyHeaders: decided: cache positively and share because
> refresh check returned cacheable; HTTP status 200 e:=p2V/0x34868914670*3       
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(470) refreshCheck:
> returning FRESH_MIN_RULE       
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(455) refreshCheck:
> Object isn't stale..   
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(327) refreshCheck:
> Staleness = -1         
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(199)
> refreshStaleness: FRESH: age (60 sec) is less than configured minimum
> (259200 sec)   
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(166)
> refreshStaleness: No explicit expiry given, using heuristics to
> determine freshness    
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(307) refreshCheck:
> entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT        
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(305) refreshCheck:
> check_time: Wed, 24 Mar 2021 15:05:34 GMT      
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(303) refreshCheck:
> age: 60        
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(301) refreshCheck:
> Matched 'URL 259200 80%% 7776000'      
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(279) refreshCheck:
> checking freshness of URI: https://URL <https://URL>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Miroslaw Malinowski
I've probably replied to quickly thinking there is a way to do it. So looking at the code and reading carefully your response, you're saying there is no way you can do it with squid.

Mirek

On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski <[hidden email]> wrote:
Hi,

You've right yes it's revalidating as API server I'm requesting data is setting Cache-Control: no-cache. My question is how I can force squid to cache and not validate as I know it's safe to do so. As I've explained earlier we are making the same request and receiving the same response from 100+ server so as to reduce number of requests to the external server we would like squid to cache the response and issue a cached version.

2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck: YES: Must revalidate stale object (origin set no-cache or private)

Mirek

On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov <[hidden email]> wrote:
On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:

> Probably, me missing on something silly or it can't be done but I don't
> know why but squid won't return the cached version even when I turn all
> override options ON in refresh_pattern.

AFAICT, no configuration options that can disable revalidation of
Cache-Control:no-cache responses. refresh_pattern does not have an
(equivalent of) "ignore-no-cache-in-responses" option.

IIRC, older Squids were violating an HTTP MUST by forgetting to
revalidate Cache-Control:no-cache responses, but that was fixed in [1].
Your Squid version has that fix.

[1]
https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa


> With debug, I can see the rule is matched and the cache is fresh but
> still in access.log is TCP_REFRESH_MODIFIED

> 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
> haveParsedReplyHeaders: decided: cache positively and share because

FYI: You are looking at cache.log lines logged _after_ Squid has already
decided to refresh the cached version. If you want to analyze why Squid
decided to refresh the cached version, you should look _before_ Squid
logged the request to the server (and before any FwdState.cc lines). I
have not checked the details, but I bet that your Squid revalidates
because of Cache-Control:no-cache in the response. Look for "YES: Must
revalidate stale object".


HTH,

Alex.

> squid conf:
> refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
> override-expire ignore-reload ignore-no-store ignore-private store-stale
>
> curl headers:
> curl --insecure --verbose --request GET --url 'URL' >/dev/null
> * TCP_NODELAY set
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> *   CAfile: /etc/ssl/certs/ca-certificates.crt
>  CApath: /etc/ssl/certs
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [512 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [122 bytes data]
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> { [6 bytes data]
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> { [1956 bytes data]
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> { [78 bytes data]
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> { [52 bytes data]
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> } [52 bytes data]
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>
>> GET URL HTTP/1.1
>> Host: URL
>> User-Agent: curl/7.68.0
>> Accept: */*
>>
> { [5 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * old SSL session ID is stale, removing
> { [5 bytes data]
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Cache-Control: no-cache
> < Content-Type: application/json
> < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
> < Date: Wed, 24 Mar 2021 15:04:34 GMT
> < Server: Google Frontend
> < Content-Length: 7950
> < X-Cache: MISS from server
> < X-Cache-Lookup: HIT from server
> < Via: 1.1 server (squid/4.14)
> < Connection: keep-alive
>
> access log:
> 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
> ORIGINAL_DST/IP application/json
>
> cache log:
> 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
> haveParsedReplyHeaders: decided: cache positively and share because
> refresh check returned cacheable; HTTP status 200 e:=p2V/0x34868914670*3       
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(470) refreshCheck:
> returning FRESH_MIN_RULE       
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(455) refreshCheck:
> Object isn't stale..   
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(327) refreshCheck:
> Staleness = -1         
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(199)
> refreshStaleness: FRESH: age (60 sec) is less than configured minimum
> (259200 sec)   
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(166)
> refreshStaleness: No explicit expiry given, using heuristics to
> determine freshness    
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(307) refreshCheck:
> entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT        
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(305) refreshCheck:
> check_time: Wed, 24 Mar 2021 15:05:34 GMT      
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(303) refreshCheck:
> age: 60        
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(301) refreshCheck:
> Matched 'URL 259200 80%% 7776000'      
> 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(279) refreshCheck:
> checking freshness of URI: https://URL <https://URL>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Alex Rousskov
On 3/24/21 2:49 PM, Miroslaw Malinowski wrote:

> looking at the code and reading carefully your response, you're saying
> there is no way you can do it with squid.

With Squid, your options include:

1. Squid source code changes. Should not be too difficult and, IMO, a
high-quality implementation would deserve official acceptance because it
is a generally useful feature in line with existing control knobs.
https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F

2. An adaptation service that removes Cache-Control:no-cache from the
response before Squid processes it:
https://wiki.squid-cache.org/SquidFaq/ContentAdaptation


HTH,

Alex.

> On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote:
>
>     Hi,
>
>     You've right yes it's revalidating as API server I'm requesting data
>     is setting Cache-Control: no-cache. My question is how I can force
>     squid to cache and not validate as I know it's safe to do so. As
>     I've explained earlier we are making the same request and receiving
>     the same response from 100+ server so as to reduce number of
>     requests to the external server we would like squid to cache the
>     response and issue a cached version.
>
>     2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck:
>     YES: Must revalidate stale object (origin set no-cache or private)
>
>     Mirek
>
>     On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov
>     <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:
>
>         > Probably, me missing on something silly or it can't be done
>         but I don't
>         > know why but squid won't return the cached version even when I
>         turn all
>         > override options ON in refresh_pattern.
>
>         AFAICT, no configuration options that can disable revalidation of
>         Cache-Control:no-cache responses. refresh_pattern does not have an
>         (equivalent of) "ignore-no-cache-in-responses" option.
>
>         IIRC, older Squids were violating an HTTP MUST by forgetting to
>         revalidate Cache-Control:no-cache responses, but that was fixed
>         in [1].
>         Your Squid version has that fix.
>
>         [1]
>         https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
>         <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>
>
>
>         > With debug, I can see the rule is matched and the cache is
>         fresh but
>         > still in access.log is TCP_REFRESH_MODIFIED
>
>         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>         > haveParsedReplyHeaders: decided: cache positively and share
>         because
>
>         FYI: You are looking at cache.log lines logged _after_ Squid has
>         already
>         decided to refresh the cached version. If you want to analyze
>         why Squid
>         decided to refresh the cached version, you should look _before_
>         Squid
>         logged the request to the server (and before any FwdState.cc
>         lines). I
>         have not checked the details, but I bet that your Squid revalidates
>         because of Cache-Control:no-cache in the response. Look for
>         "YES: Must
>         revalidate stale object".
>
>
>         HTH,
>
>         Alex.
>
>         > squid conf:
>         > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
>         > override-expire ignore-reload ignore-no-store ignore-private
>         store-stale
>         >
>         > curl headers:
>         > curl --insecure --verbose --request GET --url 'URL' >/dev/null
>         > * TCP_NODELAY set
>         > * ALPN, offering h2
>         > * ALPN, offering http/1.1
>         > * successfully set certificate verify locations:
>         > *   CAfile: /etc/ssl/certs/ca-certificates.crt
>         >  CApath: /etc/ssl/certs
>         > } [5 bytes data]
>         > * TLSv1.3 (OUT), TLS handshake, Client hello (1):
>         > } [512 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Server hello (2):
>         > { [122 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
>         > { [6 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Certificate (11):
>         > { [1956 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, CERT verify (15):
>         > { [78 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Finished (20):
>         > { [52 bytes data]
>         > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
>         > } [1 bytes data]
>         > * TLSv1.3 (OUT), TLS handshake, Finished (20):
>         > } [52 bytes data]
>         > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>         >
>         >> GET URL HTTP/1.1
>         >> Host: URL
>         >> User-Agent: curl/7.68.0
>         >> Accept: */*
>         >>
>         > { [5 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>         > { [217 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>         > { [217 bytes data]
>         > * old SSL session ID is stale, removing
>         > { [5 bytes data]
>         > * Mark bundle as not supporting multiuse
>         > < HTTP/1.1 200 OK
>         > < Cache-Control: no-cache
>         > < Content-Type: application/json
>         > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
>         > < Date: Wed, 24 Mar 2021 15:04:34 GMT
>         > < Server: Google Frontend
>         > < Content-Length: 7950
>         > < X-Cache: MISS from server
>         > < X-Cache-Lookup: HIT from server
>         > < Via: 1.1 server (squid/4.14)
>         > < Connection: keep-alive
>         >
>         > access log:
>         > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
>         > ORIGINAL_DST/IP application/json
>         >
>         > cache log:
>         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>         > haveParsedReplyHeaders: decided: cache positively and share
>         because
>         > refresh check returned cacheable; HTTP status 200
>         e:=p2V/0x34868914670*3       
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(470)
>         refreshCheck:
>         > returning FRESH_MIN_RULE       
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(455)
>         refreshCheck:
>         > Object isn't stale..   
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(327)
>         refreshCheck:
>         > Staleness = -1         
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(199)
>         > refreshStaleness: FRESH: age (60 sec) is less than configured
>         minimum
>         > (259200 sec)   
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(166)
>         > refreshStaleness: No explicit expiry given, using heuristics to
>         > determine freshness    
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(307)
>         refreshCheck:
>         > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT        
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(305)
>         refreshCheck:
>         > check_time: Wed, 24 Mar 2021 15:05:34 GMT      
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(303)
>         refreshCheck:
>         > age: 60        
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(301)
>         refreshCheck:
>         > Matched 'URL 259200 80%% 7776000'      
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(279)
>         refreshCheck:
>         > checking freshness of URI: https://URL <https://URL>
>         <https://URL <https://URL>>
>         >
>         >
>         > _______________________________________________
>         > squid-users mailing list
>         > [hidden email]
>         <mailto:[hidden email]>
>         > http://lists.squid-cache.org/listinfo/squid-users
>         <http://lists.squid-cache.org/listinfo/squid-users>
>         >
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Miroslaw Malinowski
I thought about upper service but as is not required at the moment, introducing extra hop just to remove the header looks a bit like a hammer approach. I'll look into how easily I can amend the code as the other option is to introduce a proxy like a feature to the application, so either way, it is a code change. The only problem here is that it's an OPNSense squid service so I have to compile from source on BSD and then keep adding in manually each time they do the update.

Mirek

On Wed, Mar 24, 2021 at 7:11 PM Alex Rousskov <[hidden email]> wrote:
On 3/24/21 2:49 PM, Miroslaw Malinowski wrote:

> looking at the code and reading carefully your response, you're saying
> there is no way you can do it with squid.

With Squid, your options include:

1. Squid source code changes. Should not be too difficult and, IMO, a
high-quality implementation would deserve official acceptance because it
is a generally useful feature in line with existing control knobs.
https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F

2. An adaptation service that removes Cache-Control:no-cache from the
response before Squid processes it:
https://wiki.squid-cache.org/SquidFaq/ContentAdaptation


HTH,

Alex.

> On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote:
>
>     Hi,
>
>     You've right yes it's revalidating as API server I'm requesting data
>     is setting Cache-Control: no-cache. My question is how I can force
>     squid to cache and not validate as I know it's safe to do so. As
>     I've explained earlier we are making the same request and receiving
>     the same response from 100+ server so as to reduce number of
>     requests to the external server we would like squid to cache the
>     response and issue a cached version.
>
>     2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck:
>     YES: Must revalidate stale object (origin set no-cache or private)
>
>     Mirek
>
>     On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov
>     <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:
>
>         > Probably, me missing on something silly or it can't be done
>         but I don't
>         > know why but squid won't return the cached version even when I
>         turn all
>         > override options ON in refresh_pattern.
>
>         AFAICT, no configuration options that can disable revalidation of
>         Cache-Control:no-cache responses. refresh_pattern does not have an
>         (equivalent of) "ignore-no-cache-in-responses" option.
>
>         IIRC, older Squids were violating an HTTP MUST by forgetting to
>         revalidate Cache-Control:no-cache responses, but that was fixed
>         in [1].
>         Your Squid version has that fix.
>
>         [1]
>         https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
>         <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>
>
>
>         > With debug, I can see the rule is matched and the cache is
>         fresh but
>         > still in access.log is TCP_REFRESH_MODIFIED
>
>         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>         > haveParsedReplyHeaders: decided: cache positively and share
>         because
>
>         FYI: You are looking at cache.log lines logged _after_ Squid has
>         already
>         decided to refresh the cached version. If you want to analyze
>         why Squid
>         decided to refresh the cached version, you should look _before_
>         Squid
>         logged the request to the server (and before any FwdState.cc
>         lines). I
>         have not checked the details, but I bet that your Squid revalidates
>         because of Cache-Control:no-cache in the response. Look for
>         "YES: Must
>         revalidate stale object".
>
>
>         HTH,
>
>         Alex.
>
>         > squid conf:
>         > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
>         > override-expire ignore-reload ignore-no-store ignore-private
>         store-stale
>         >
>         > curl headers:
>         > curl --insecure --verbose --request GET --url 'URL' >/dev/null
>         > * TCP_NODELAY set
>         > * ALPN, offering h2
>         > * ALPN, offering http/1.1
>         > * successfully set certificate verify locations:
>         > *   CAfile: /etc/ssl/certs/ca-certificates.crt
>         >  CApath: /etc/ssl/certs
>         > } [5 bytes data]
>         > * TLSv1.3 (OUT), TLS handshake, Client hello (1):
>         > } [512 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Server hello (2):
>         > { [122 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
>         > { [6 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Certificate (11):
>         > { [1956 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, CERT verify (15):
>         > { [78 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Finished (20):
>         > { [52 bytes data]
>         > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
>         > } [1 bytes data]
>         > * TLSv1.3 (OUT), TLS handshake, Finished (20):
>         > } [52 bytes data]
>         > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>         >
>         >> GET URL HTTP/1.1
>         >> Host: URL
>         >> User-Agent: curl/7.68.0
>         >> Accept: */*
>         >>
>         > { [5 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>         > { [217 bytes data]
>         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>         > { [217 bytes data]
>         > * old SSL session ID is stale, removing
>         > { [5 bytes data]
>         > * Mark bundle as not supporting multiuse
>         > < HTTP/1.1 200 OK
>         > < Cache-Control: no-cache
>         > < Content-Type: application/json
>         > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
>         > < Date: Wed, 24 Mar 2021 15:04:34 GMT
>         > < Server: Google Frontend
>         > < Content-Length: 7950
>         > < X-Cache: MISS from server
>         > < X-Cache-Lookup: HIT from server
>         > < Via: 1.1 server (squid/4.14)
>         > < Connection: keep-alive
>         >
>         > access log:
>         > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
>         > ORIGINAL_DST/IP application/json
>         >
>         > cache log:
>         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>         > haveParsedReplyHeaders: decided: cache positively and share
>         because
>         > refresh check returned cacheable; HTTP status 200
>         e:=p2V/0x34868914670*3       
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(470)
>         refreshCheck:
>         > returning FRESH_MIN_RULE       
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(455)
>         refreshCheck:
>         > Object isn't stale..   
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(327)
>         refreshCheck:
>         > Staleness = -1         
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(199)
>         > refreshStaleness: FRESH: age (60 sec) is less than configured
>         minimum
>         > (259200 sec)   
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(166)
>         > refreshStaleness: No explicit expiry given, using heuristics to
>         > determine freshness    
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(307)
>         refreshCheck:
>         > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT        
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(305)
>         refreshCheck:
>         > check_time: Wed, 24 Mar 2021 15:05:34 GMT      
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(303)
>         refreshCheck:
>         > age: 60        
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(301)
>         refreshCheck:
>         > Matched 'URL 259200 80%% 7776000'      
>         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3| refresh.cc(279)
>         refreshCheck:
>         > checking freshness of URI: https://URL <https://URL>
>         <https://URL <https://URL>>
>         >
>         >
>         > _______________________________________________
>         > squid-users mailing list
>         > [hidden email]
>         <mailto:[hidden email]>
>         > http://lists.squid-cache.org/listinfo/squid-users
>         <http://lists.squid-cache.org/listinfo/squid-users>
>         >
>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Alex Rousskov
On 3/24/21 3:34 PM, Miroslaw Malinowski wrote:
> I thought about upper service but as is not required at the moment,
> introducing extra hop just to remove the header looks a bit like a
> hammer approach. I'll look into how easily I can amend the code as the
> other option is to introduce a proxy like a feature to the application,
> so either way, it is a code change. The only problem here is that it's
> an OPNSense squid service so I have to compile from source on BSD and
> then keep adding in manually each time they do the update.

At the risk of stating the obvious: If your feature is officially
accepted into Squid sources, then you would not have to keep adding it
manually (once the changes reach your Squid packaging source).

Alex.


> On Wed, Mar 24, 2021 at 7:11 PM Alex Rousskov wrote:
>
>     On 3/24/21 2:49 PM, Miroslaw Malinowski wrote:
>
>     > looking at the code and reading carefully your response, you're saying
>     > there is no way you can do it with squid.
>
>     With Squid, your options include:
>
>     1. Squid source code changes. Should not be too difficult and, IMO, a
>     high-quality implementation would deserve official acceptance because it
>     is a generally useful feature in line with existing control knobs.
>     https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
>
>     2. An adaptation service that removes Cache-Control:no-cache from the
>     response before Squid processes it:
>     https://wiki.squid-cache.org/SquidFaq/ContentAdaptation
>
>
>     HTH,
>
>     Alex.
>
>     > On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote:
>     >
>     >     Hi,
>     >
>     >     You've right yes it's revalidating as API server I'm
>     requesting data
>     >     is setting Cache-Control: no-cache. My question is how I can force
>     >     squid to cache and not validate as I know it's safe to do so. As
>     >     I've explained earlier we are making the same request and
>     receiving
>     >     the same response from 100+ server so as to reduce number of
>     >     requests to the external server we would like squid to cache the
>     >     response and issue a cached version.
>     >
>     >     2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck:
>     >     YES: Must revalidate stale object (origin set no-cache or private)
>     >
>     >     Mirek
>     >
>     >     On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov
>     >     <[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >
>     >         On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:
>     >
>     >         > Probably, me missing on something silly or it can't be done
>     >         but I don't
>     >         > know why but squid won't return the cached version even
>     when I
>     >         turn all
>     >         > override options ON in refresh_pattern.
>     >
>     >         AFAICT, no configuration options that can disable
>     revalidation of
>     >         Cache-Control:no-cache responses. refresh_pattern does not
>     have an
>     >         (equivalent of) "ignore-no-cache-in-responses" option.
>     >
>     >         IIRC, older Squids were violating an HTTP MUST by
>     forgetting to
>     >         revalidate Cache-Control:no-cache responses, but that was
>     fixed
>     >         in [1].
>     >         Your Squid version has that fix.
>     >
>     >         [1]
>     >       
>      https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
>     <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>
>     >       
>      <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
>     <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>>
>     >
>     >
>     >         > With debug, I can see the rule is matched and the cache is
>     >         fresh but
>     >         > still in access.log is TCP_REFRESH_MODIFIED
>     >
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>     >         > haveParsedReplyHeaders: decided: cache positively and share
>     >         because
>     >
>     >         FYI: You are looking at cache.log lines logged _after_
>     Squid has
>     >         already
>     >         decided to refresh the cached version. If you want to analyze
>     >         why Squid
>     >         decided to refresh the cached version, you should look
>     _before_
>     >         Squid
>     >         logged the request to the server (and before any FwdState.cc
>     >         lines). I
>     >         have not checked the details, but I bet that your Squid
>     revalidates
>     >         because of Cache-Control:no-cache in the response. Look for
>     >         "YES: Must
>     >         revalidate stale object".
>     >
>     >
>     >         HTH,
>     >
>     >         Alex.
>     >
>     >         > squid conf:
>     >         > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
>     >         > override-expire ignore-reload ignore-no-store ignore-private
>     >         store-stale
>     >         >
>     >         > curl headers:
>     >         > curl --insecure --verbose --request GET --url 'URL'
>     >/dev/null
>     >         > * TCP_NODELAY set
>     >         > * ALPN, offering h2
>     >         > * ALPN, offering http/1.1
>     >         > * successfully set certificate verify locations:
>     >         > *   CAfile: /etc/ssl/certs/ca-certificates.crt
>     >         >  CApath: /etc/ssl/certs
>     >         > } [5 bytes data]
>     >         > * TLSv1.3 (OUT), TLS handshake, Client hello (1):
>     >         > } [512 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Server hello (2):
>     >         > { [122 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
>     >         > { [6 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Certificate (11):
>     >         > { [1956 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, CERT verify (15):
>     >         > { [78 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Finished (20):
>     >         > { [52 bytes data]
>     >         > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
>     >         > } [1 bytes data]
>     >         > * TLSv1.3 (OUT), TLS handshake, Finished (20):
>     >         > } [52 bytes data]
>     >         > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>     >         >
>     >         >> GET URL HTTP/1.1
>     >         >> Host: URL
>     >         >> User-Agent: curl/7.68.0
>     >         >> Accept: */*
>     >         >>
>     >         > { [5 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>     >         > { [217 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>     >         > { [217 bytes data]
>     >         > * old SSL session ID is stale, removing
>     >         > { [5 bytes data]
>     >         > * Mark bundle as not supporting multiuse
>     >         > < HTTP/1.1 200 OK
>     >         > < Cache-Control: no-cache
>     >         > < Content-Type: application/json
>     >         > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
>     >         > < Date: Wed, 24 Mar 2021 15:04:34 GMT
>     >         > < Server: Google Frontend
>     >         > < Content-Length: 7950
>     >         > < X-Cache: MISS from server
>     >         > < X-Cache-Lookup: HIT from server
>     >         > < Via: 1.1 server (squid/4.14)
>     >         > < Connection: keep-alive
>     >         >
>     >         > access log:
>     >         > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
>     >         > ORIGINAL_DST/IP application/json
>     >         >
>     >         > cache log:
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>     >         > haveParsedReplyHeaders: decided: cache positively and share
>     >         because
>     >         > refresh check returned cacheable; HTTP status 200
>     >         e:=p2V/0x34868914670*3       
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(470)
>     >         refreshCheck:
>     >         > returning FRESH_MIN_RULE       
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(455)
>     >         refreshCheck:
>     >         > Object isn't stale..   
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(327)
>     >         refreshCheck:
>     >         > Staleness = -1         
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(199)
>     >         > refreshStaleness: FRESH: age (60 sec) is less than
>     configured
>     >         minimum
>     >         > (259200 sec)   
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(166)
>     >         > refreshStaleness: No explicit expiry given, using
>     heuristics to
>     >         > determine freshness    
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(307)
>     >         refreshCheck:
>     >         > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT        
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(305)
>     >         refreshCheck:
>     >         > check_time: Wed, 24 Mar 2021 15:05:34 GMT      
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(303)
>     >         refreshCheck:
>     >         > age: 60        
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(301)
>     >         refreshCheck:
>     >         > Matched 'URL 259200 80%% 7776000'      
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(279)
>     >         refreshCheck:
>     >         > checking freshness of URI: https://URL <https://URL>
>     <https://URL <https://URL>>
>     >         <https://URL <https://URL> <https://URL <https://URL>>>
>     >         >
>     >         >
>     >         > _______________________________________________
>     >         > squid-users mailing list
>     >         > [hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >         > http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
>     >         <http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>>
>     >         >
>     >
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

Eliezer Croitoru-3
Hey Mirek,

This is not the first time this issue rises.
There are risks in implementing any solution for this *issue*.

I have implemented YouTube caching in the past using couple twisted techniques while leaving squid un-touched.
The desire to caching sometimes can overcome couple very big risks to the integrity if the date/content.
It is possible to use an ICAP service with a 206 response instead of 204 or 200 however I believe
that you wouldn’t need to cache any POST requests so a simple ICAP service would be sufficient.
I believe that It is preferred to leave squid sources un touched for such a purpose.
An example for such a twist is at:
* https://github.com/elico/squid-helpers/tree/master/squid_helpers/youtubetwist
* https://wiki.squid-cache.org/ConfigExamples/DynamicContent/Coordinator?highlight=%28cache_peer%29#Implementing_ICAP_solution
* https://ieeexplore.ieee.org/abstract/document/9072556

I wrote a public example of an ICAP server that was used to  prove vulnerabilities in HTTP which is now used in the prove of HTTPS vulnerabilities.
Take a peek at:
* https://github.com/elico/bgu-icap-example

It's written in GoLang and works under pretty heavy loads.

Let me know if you need more help,
Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]
Zoom: Coming soon


-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Alex Rousskov
Sent: Friday, March 26, 2021 10:36 PM
To: [hidden email]
Subject: Re: [squid-users] squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale

On 3/24/21 3:34 PM, Miroslaw Malinowski wrote:
> I thought about upper service but as is not required at the moment,
> introducing extra hop just to remove the header looks a bit like a
> hammer approach. I'll look into how easily I can amend the code as the
> other option is to introduce a proxy like a feature to the application,
> so either way, it is a code change. The only problem here is that it's
> an OPNSense squid service so I have to compile from source on BSD and
> then keep adding in manually each time they do the update.

At the risk of stating the obvious: If your feature is officially
accepted into Squid sources, then you would not have to keep adding it
manually (once the changes reach your Squid packaging source).

Alex.


> On Wed, Mar 24, 2021 at 7:11 PM Alex Rousskov wrote:
>
>     On 3/24/21 2:49 PM, Miroslaw Malinowski wrote:
>
>     > looking at the code and reading carefully your response, you're saying
>     > there is no way you can do it with squid.
>
>     With Squid, your options include:
>
>     1. Squid source code changes. Should not be too difficult and, IMO, a
>     high-quality implementation would deserve official acceptance because it
>     is a generally useful feature in line with existing control knobs.
>     https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
>
>     2. An adaptation service that removes Cache-Control:no-cache from the
>     response before Squid processes it:
>     https://wiki.squid-cache.org/SquidFaq/ContentAdaptation
>
>
>     HTH,
>
>     Alex.
>
>     > On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote:
>     >
>     >     Hi,
>     >
>     >     You've right yes it's revalidating as API server I'm
>     requesting data
>     >     is setting Cache-Control: no-cache. My question is how I can force
>     >     squid to cache and not validate as I know it's safe to do so. As
>     >     I've explained earlier we are making the same request and
>     receiving
>     >     the same response from 100+ server so as to reduce number of
>     >     requests to the external server we would like squid to cache the
>     >     response and issue a cached version.
>     >
>     >     2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck:
>     >     YES: Must revalidate stale object (origin set no-cache or private)
>     >
>     >     Mirek
>     >
>     >     On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov
>     >     <[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >
>     >         On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:
>     >
>     >         > Probably, me missing on something silly or it can't be done
>     >         but I don't
>     >         > know why but squid won't return the cached version even
>     when I
>     >         turn all
>     >         > override options ON in refresh_pattern.
>     >
>     >         AFAICT, no configuration options that can disable
>     revalidation of
>     >         Cache-Control:no-cache responses. refresh_pattern does not
>     have an
>     >         (equivalent of) "ignore-no-cache-in-responses" option.
>     >
>     >         IIRC, older Squids were violating an HTTP MUST by
>     forgetting to
>     >         revalidate Cache-Control:no-cache responses, but that was
>     fixed
>     >         in [1].
>     >         Your Squid version has that fix.
>     >
>     >         [1]
>     >      
>      https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
>     <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>
>     >      
>      <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
>     <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>>
>     >
>     >
>     >         > With debug, I can see the rule is matched and the cache is
>     >         fresh but
>     >         > still in access.log is TCP_REFRESH_MODIFIED
>     >
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>     >         > haveParsedReplyHeaders: decided: cache positively and share
>     >         because
>     >
>     >         FYI: You are looking at cache.log lines logged _after_
>     Squid has
>     >         already
>     >         decided to refresh the cached version. If you want to analyze
>     >         why Squid
>     >         decided to refresh the cached version, you should look
>     _before_
>     >         Squid
>     >         logged the request to the server (and before any FwdState.cc
>     >         lines). I
>     >         have not checked the details, but I bet that your Squid
>     revalidates
>     >         because of Cache-Control:no-cache in the response. Look for
>     >         "YES: Must
>     >         revalidate stale object".
>     >
>     >
>     >         HTH,
>     >
>     >         Alex.
>     >
>     >         > squid conf:
>     >         > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
>     >         > override-expire ignore-reload ignore-no-store ignore-private
>     >         store-stale
>     >         >
>     >         > curl headers:
>     >         > curl --insecure --verbose --request GET --url 'URL'
>     >/dev/null
>     >         > * TCP_NODELAY set
>     >         > * ALPN, offering h2
>     >         > * ALPN, offering http/1.1
>     >         > * successfully set certificate verify locations:
>     >         > *   CAfile: /etc/ssl/certs/ca-certificates.crt
>     >         >  CApath: /etc/ssl/certs
>     >         > } [5 bytes data]
>     >         > * TLSv1.3 (OUT), TLS handshake, Client hello (1):
>     >         > } [512 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Server hello (2):
>     >         > { [122 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
>     >         > { [6 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Certificate (11):
>     >         > { [1956 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, CERT verify (15):
>     >         > { [78 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Finished (20):
>     >         > { [52 bytes data]
>     >         > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
>     >         > } [1 bytes data]
>     >         > * TLSv1.3 (OUT), TLS handshake, Finished (20):
>     >         > } [52 bytes data]
>     >         > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>     >         >
>     >         >> GET URL HTTP/1.1
>     >         >> Host: URL
>     >         >> User-Agent: curl/7.68.0
>     >         >> Accept: */*
>     >         >>
>     >         > { [5 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>     >         > { [217 bytes data]
>     >         > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>     >         > { [217 bytes data]
>     >         > * old SSL session ID is stale, removing
>     >         > { [5 bytes data]
>     >         > * Mark bundle as not supporting multiuse
>     >         > < HTTP/1.1 200 OK
>     >         > < Cache-Control: no-cache
>     >         > < Content-Type: application/json
>     >         > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
>     >         > < Date: Wed, 24 Mar 2021 15:04:34 GMT
>     >         > < Server: Google Frontend
>     >         > < Content-Length: 7950
>     >         > < X-Cache: MISS from server
>     >         > < X-Cache-Lookup: HIT from server
>     >         > < Via: 1.1 server (squid/4.14)
>     >         > < Connection: keep-alive
>     >         >
>     >         > access log:
>     >         > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
>     >         > ORIGINAL_DST/IP application/json
>     >         >
>     >         > cache log:
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 11,3| http.cc(982)
>     >         > haveParsedReplyHeaders: decided: cache positively and share
>     >         because
>     >         > refresh check returned cacheable; HTTP status 200
>     >         e:=p2V/0x34868914670*3      
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(470)
>     >         refreshCheck:
>     >         > returning FRESH_MIN_RULE      
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(455)
>     >         refreshCheck:
>     >         > Object isn't stale..  
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(327)
>     >         refreshCheck:
>     >         > Staleness = -1        
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(199)
>     >         > refreshStaleness: FRESH: age (60 sec) is less than
>     configured
>     >         minimum
>     >         > (259200 sec)  
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(166)
>     >         > refreshStaleness: No explicit expiry given, using
>     heuristics to
>     >         > determine freshness    
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(307)
>     >         refreshCheck:
>     >         > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT        
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(305)
>     >         refreshCheck:
>     >         > check_time: Wed, 24 Mar 2021 15:05:34 GMT      
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(303)
>     >         refreshCheck:
>     >         > age: 60        
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(301)
>     >         refreshCheck:
>     >         > Matched 'URL 259200 80%% 7776000'      
>     >         > 2021-03-24T15:04:34   squid   .710 kid1| 22,3|
>     refresh.cc(279)
>     >         refreshCheck:
>     >         > checking freshness of URI: https://URL <https://URL>
>     <https://URL <https://URL>>
>     >         <https://URL <https://URL> <https://URL <https://URL>>>
>     >         >
>     >         >
>     >         > _______________________________________________
>     >         > squid-users mailing list
>     >         > [hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >         > http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
>     >         <http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>>
>     >         >
>     >
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users