squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
Locked
squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
 
|
Hi, Probably, me missing on something silly or it can't be done but I don't know why but squid won't
return the cached version even when I turn all override options ON in
refresh_pattern. It's an API call where we call many of the same
requests and by knowing it we would like to stop those calls to go out
if it's already been sent once. With debug, I can see the rule is matched and the cache is fresh but still in access.log is TCP_REFRESH_MODIFIED squid conf: refresh_pattern
-i <URL> 4320 80% 129600 override-lastmod override-expire
ignore-reload ignore-no-store ignore-private store-stale curl headers: curl --insecure --verbose --request GET --url 'URL' >/dev/null
* TCP_NODELAY set * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [1956 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [78 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [52 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [52 bytes data] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 > GET URL HTTP/1.1 > Host: URL > User-Agent: curl/7.68.0 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [217 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [217 bytes data] * old SSL session ID is stale, removing { [5 bytes data] * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Cache-Control: no-cache < Content-Type: application/json < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0 < Date: Wed, 24 Mar 2021 15:04:34 GMT < Server: Google Frontend < Content-Length: 7950 < X-Cache: MISS from server < X-Cache-Lookup: HIT from server < Via: 1.1 server (squid/4.14) < Connection: keep-alive access log: 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
ORIGINAL_DST/IP application/json cache log:
_______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
|
|
On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:
> Probably, me missing on something silly or it can't be done but I don't > know why but squid won't return the cached version even when I turn all > override options ON in refresh_pattern. AFAICT, no configuration options that can disable revalidation of Cache-Control:no-cache responses. refresh_pattern does not have an (equivalent of) "ignore-no-cache-in-responses" option. IIRC, older Squids were violating an HTTP MUST by forgetting to revalidate Cache-Control:no-cache responses, but that was fixed in [1]. Your Squid version has that fix. [1] https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa > With debug, I can see the rule is matched and the cache is fresh but > still in access.log is TCP_REFRESH_MODIFIED > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > haveParsedReplyHeaders: decided: cache positively and share because FYI: You are looking at cache.log lines logged _after_ Squid has already decided to refresh the cached version. If you want to analyze why Squid decided to refresh the cached version, you should look _before_ Squid logged the request to the server (and before any FwdState.cc lines). I have not checked the details, but I bet that your Squid revalidates because of Cache-Control:no-cache in the response. Look for "YES: Must revalidate stale object". HTH, Alex. > squid conf: > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod > override-expire ignore-reload ignore-no-store ignore-private store-stale > > curl headers: > curl --insecure --verbose --request GET --url 'URL' >/dev/null > * TCP_NODELAY set > * ALPN, offering h2 > * ALPN, offering http/1.1 > * successfully set certificate verify locations: > * CAfile: /etc/ssl/certs/ca-certificates.crt > CApath: /etc/ssl/certs > } [5 bytes data] > * TLSv1.3 (OUT), TLS handshake, Client hello (1): > } [512 bytes data] > * TLSv1.3 (IN), TLS handshake, Server hello (2): > { [122 bytes data] > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): > { [6 bytes data] > * TLSv1.3 (IN), TLS handshake, Certificate (11): > { [1956 bytes data] > * TLSv1.3 (IN), TLS handshake, CERT verify (15): > { [78 bytes data] > * TLSv1.3 (IN), TLS handshake, Finished (20): > { [52 bytes data] > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): > } [1 bytes data] > * TLSv1.3 (OUT), TLS handshake, Finished (20): > } [52 bytes data] > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 > >> GET URL HTTP/1.1 >> Host: URL >> User-Agent: curl/7.68.0 >> Accept: */* >> > { [5 bytes data] > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > { [217 bytes data] > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > { [217 bytes data] > * old SSL session ID is stale, removing > { [5 bytes data] > * Mark bundle as not supporting multiuse > < HTTP/1.1 200 OK > < Cache-Control: no-cache > < Content-Type: application/json > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0 > < Date: Wed, 24 Mar 2021 15:04:34 GMT > < Server: Google Frontend > < Content-Length: 7950 > < X-Cache: MISS from server > < X-Cache-Lookup: HIT from server > < Via: 1.1 server (squid/4.14) > < Connection: keep-alive > > access log: > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL - > ORIGINAL_DST/IP application/json > > cache log: > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > haveParsedReplyHeaders: decided: cache positively and share because > refresh check returned cacheable; HTTP status 200 e:=p2V/0x34868914670*3 > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(470) refreshCheck: > returning FRESH_MIN_RULE > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(455) refreshCheck: > Object isn't stale.. > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(327) refreshCheck: > Staleness = -1 > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(199) > refreshStaleness: FRESH: age (60 sec) is less than configured minimum > (259200 sec) > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(166) > refreshStaleness: No explicit expiry given, using heuristics to > determine freshness > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(307) refreshCheck: > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(305) refreshCheck: > check_time: Wed, 24 Mar 2021 15:05:34 GMT > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(303) refreshCheck: > age: 60 > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(301) refreshCheck: > Matched 'URL 259200 80%% 7776000' > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(279) refreshCheck: > checking freshness of URI: https://URL <https://URL> > > > _______________________________________________ > squid-users mailing list > [hidden email] > http://lists.squid-cache.org/listinfo/squid-users > _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
|
|
Hi, You've right yes it's revalidating as API server I'm requesting data is setting Cache-Control: no-cache. My question is how I can force squid to cache and not validate as I know it's safe to do so. As I've explained earlier we are making the same request and receiving the same response from 100+ server so as to reduce number of requests to the external server we would like squid to cache the response and issue a cached version. 2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck: YES: Must revalidate stale object (origin set no-cache or private) Mirek On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov <[hidden email]> wrote: On 3/24/21 12:48 PM, Miroslaw Malinowski wrote: _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
|
|
I've probably replied to quickly thinking there is a way to do it. So looking at the code and reading carefully your response, you're saying there is no way you can do it with squid. Mirek On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski <[hidden email]> wrote:
_______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
|
|
On 3/24/21 2:49 PM, Miroslaw Malinowski wrote:
> looking at the code and reading carefully your response, you're saying > there is no way you can do it with squid. With Squid, your options include: 1. Squid source code changes. Should not be too difficult and, IMO, a high-quality implementation would deserve official acceptance because it is a generally useful feature in line with existing control knobs. https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F 2. An adaptation service that removes Cache-Control:no-cache from the response before Squid processes it: https://wiki.squid-cache.org/SquidFaq/ContentAdaptation HTH, Alex. > On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote: > > Hi, > > You've right yes it's revalidating as API server I'm requesting data > is setting Cache-Control: no-cache. My question is how I can force > squid to cache and not validate as I know it's safe to do so. As > I've explained earlier we are making the same request and receiving > the same response from 100+ server so as to reduce number of > requests to the external server we would like squid to cache the > response and issue a cached version. > > 2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck: > YES: Must revalidate stale object (origin set no-cache or private) > > Mirek > > On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov > <[hidden email] > <mailto:[hidden email]>> wrote: > > On 3/24/21 12:48 PM, Miroslaw Malinowski wrote: > > > Probably, me missing on something silly or it can't be done > but I don't > > know why but squid won't return the cached version even when I > turn all > > override options ON in refresh_pattern. > > AFAICT, no configuration options that can disable revalidation of > Cache-Control:no-cache responses. refresh_pattern does not have an > (equivalent of) "ignore-no-cache-in-responses" option. > > IIRC, older Squids were violating an HTTP MUST by forgetting to > revalidate Cache-Control:no-cache responses, but that was fixed > in [1]. > Your Squid version has that fix. > > [1] > https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa > <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa> > > > > With debug, I can see the rule is matched and the cache is > fresh but > > still in access.log is TCP_REFRESH_MODIFIED > > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > > haveParsedReplyHeaders: decided: cache positively and share > because > > FYI: You are looking at cache.log lines logged _after_ Squid has > already > decided to refresh the cached version. If you want to analyze > why Squid > decided to refresh the cached version, you should look _before_ > Squid > logged the request to the server (and before any FwdState.cc > lines). I > have not checked the details, but I bet that your Squid revalidates > because of Cache-Control:no-cache in the response. Look for > "YES: Must > revalidate stale object". > > > HTH, > > Alex. > > > squid conf: > > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod > > override-expire ignore-reload ignore-no-store ignore-private > store-stale > > > > curl headers: > > curl --insecure --verbose --request GET --url 'URL' >/dev/null > > * TCP_NODELAY set > > * ALPN, offering h2 > > * ALPN, offering http/1.1 > > * successfully set certificate verify locations: > > * CAfile: /etc/ssl/certs/ca-certificates.crt > > CApath: /etc/ssl/certs > > } [5 bytes data] > > * TLSv1.3 (OUT), TLS handshake, Client hello (1): > > } [512 bytes data] > > * TLSv1.3 (IN), TLS handshake, Server hello (2): > > { [122 bytes data] > > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): > > { [6 bytes data] > > * TLSv1.3 (IN), TLS handshake, Certificate (11): > > { [1956 bytes data] > > * TLSv1.3 (IN), TLS handshake, CERT verify (15): > > { [78 bytes data] > > * TLSv1.3 (IN), TLS handshake, Finished (20): > > { [52 bytes data] > > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): > > } [1 bytes data] > > * TLSv1.3 (OUT), TLS handshake, Finished (20): > > } [52 bytes data] > > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 > > > >> GET URL HTTP/1.1 > >> Host: URL > >> User-Agent: curl/7.68.0 > >> Accept: */* > >> > > { [5 bytes data] > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > > { [217 bytes data] > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > > { [217 bytes data] > > * old SSL session ID is stale, removing > > { [5 bytes data] > > * Mark bundle as not supporting multiuse > > < HTTP/1.1 200 OK > > < Cache-Control: no-cache > > < Content-Type: application/json > > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0 > > < Date: Wed, 24 Mar 2021 15:04:34 GMT > > < Server: Google Frontend > > < Content-Length: 7950 > > < X-Cache: MISS from server > > < X-Cache-Lookup: HIT from server > > < Via: 1.1 server (squid/4.14) > > < Connection: keep-alive > > > > access log: > > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL - > > ORIGINAL_DST/IP application/json > > > > cache log: > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > > haveParsedReplyHeaders: decided: cache positively and share > because > > refresh check returned cacheable; HTTP status 200 > e:=p2V/0x34868914670*3 > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(470) > refreshCheck: > > returning FRESH_MIN_RULE > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(455) > refreshCheck: > > Object isn't stale.. > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(327) > refreshCheck: > > Staleness = -1 > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(199) > > refreshStaleness: FRESH: age (60 sec) is less than configured > minimum > > (259200 sec) > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(166) > > refreshStaleness: No explicit expiry given, using heuristics to > > determine freshness > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(307) > refreshCheck: > > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(305) > refreshCheck: > > check_time: Wed, 24 Mar 2021 15:05:34 GMT > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(303) > refreshCheck: > > age: 60 > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(301) > refreshCheck: > > Matched 'URL 259200 80%% 7776000' > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| refresh.cc(279) > refreshCheck: > > checking freshness of URI: https://URL <https://URL> > <https://URL <https://URL>> > > > > > > _______________________________________________ > > squid-users mailing list > > [hidden email] > <mailto:[hidden email]> > > http://lists.squid-cache.org/listinfo/squid-users > <http://lists.squid-cache.org/listinfo/squid-users> > > > _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
|
|
I thought about upper service but as is not required at the moment, introducing extra hop just to remove the header looks a bit like a hammer approach. I'll look into how easily I can amend the code as the other option is to introduce a proxy like a feature to the application, so either way, it is a code change. The only problem here is that it's an OPNSense squid service so I have to compile from source on BSD and then keep adding in manually each time they do the update. Mirek On Wed, Mar 24, 2021 at 7:11 PM Alex Rousskov <[hidden email]> wrote: On 3/24/21 2:49 PM, Miroslaw Malinowski wrote: _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
|
|
On 3/24/21 3:34 PM, Miroslaw Malinowski wrote:
> I thought about upper service but as is not required at the moment, > introducing extra hop just to remove the header looks a bit like a > hammer approach. I'll look into how easily I can amend the code as the > other option is to introduce a proxy like a feature to the application, > so either way, it is a code change. The only problem here is that it's > an OPNSense squid service so I have to compile from source on BSD and > then keep adding in manually each time they do the update. At the risk of stating the obvious: If your feature is officially accepted into Squid sources, then you would not have to keep adding it manually (once the changes reach your Squid packaging source). Alex. > On Wed, Mar 24, 2021 at 7:11 PM Alex Rousskov wrote: > > On 3/24/21 2:49 PM, Miroslaw Malinowski wrote: > > > looking at the code and reading carefully your response, you're saying > > there is no way you can do it with squid. > > With Squid, your options include: > > 1. Squid source code changes. Should not be too difficult and, IMO, a > high-quality implementation would deserve official acceptance because it > is a generally useful feature in line with existing control knobs. > https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F > > 2. An adaptation service that removes Cache-Control:no-cache from the > response before Squid processes it: > https://wiki.squid-cache.org/SquidFaq/ContentAdaptation > > > HTH, > > Alex. > > > On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote: > > > > Hi, > > > > You've right yes it's revalidating as API server I'm > requesting data > > is setting Cache-Control: no-cache. My question is how I can force > > squid to cache and not validate as I know it's safe to do so. As > > I've explained earlier we are making the same request and > receiving > > the same response from 100+ server so as to reduce number of > > requests to the external server we would like squid to cache the > > response and issue a cached version. > > > > 2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck: > > YES: Must revalidate stale object (origin set no-cache or private) > > > > Mirek > > > > On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov > > <[hidden email] > <mailto:[hidden email]> > > <mailto:[hidden email] > <mailto:[hidden email]>>> wrote: > > > > On 3/24/21 12:48 PM, Miroslaw Malinowski wrote: > > > > > Probably, me missing on something silly or it can't be done > > but I don't > > > know why but squid won't return the cached version even > when I > > turn all > > > override options ON in refresh_pattern. > > > > AFAICT, no configuration options that can disable > revalidation of > > Cache-Control:no-cache responses. refresh_pattern does not > have an > > (equivalent of) "ignore-no-cache-in-responses" option. > > > > IIRC, older Squids were violating an HTTP MUST by > forgetting to > > revalidate Cache-Control:no-cache responses, but that was > fixed > > in [1]. > > Your Squid version has that fix. > > > > [1] > > > https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa > <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa> > > > <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa > <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>> > > > > > > > With debug, I can see the rule is matched and the cache is > > fresh but > > > still in access.log is TCP_REFRESH_MODIFIED > > > > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > > > haveParsedReplyHeaders: decided: cache positively and share > > because > > > > FYI: You are looking at cache.log lines logged _after_ > Squid has > > already > > decided to refresh the cached version. If you want to analyze > > why Squid > > decided to refresh the cached version, you should look > _before_ > > Squid > > logged the request to the server (and before any FwdState.cc > > lines). I > > have not checked the details, but I bet that your Squid > revalidates > > because of Cache-Control:no-cache in the response. Look for > > "YES: Must > > revalidate stale object". > > > > > > HTH, > > > > Alex. > > > > > squid conf: > > > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod > > > override-expire ignore-reload ignore-no-store ignore-private > > store-stale > > > > > > curl headers: > > > curl --insecure --verbose --request GET --url 'URL' > >/dev/null > > > * TCP_NODELAY set > > > * ALPN, offering h2 > > > * ALPN, offering http/1.1 > > > * successfully set certificate verify locations: > > > * CAfile: /etc/ssl/certs/ca-certificates.crt > > > CApath: /etc/ssl/certs > > > } [5 bytes data] > > > * TLSv1.3 (OUT), TLS handshake, Client hello (1): > > > } [512 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Server hello (2): > > > { [122 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): > > > { [6 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Certificate (11): > > > { [1956 bytes data] > > > * TLSv1.3 (IN), TLS handshake, CERT verify (15): > > > { [78 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Finished (20): > > > { [52 bytes data] > > > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): > > > } [1 bytes data] > > > * TLSv1.3 (OUT), TLS handshake, Finished (20): > > > } [52 bytes data] > > > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 > > > > > >> GET URL HTTP/1.1 > > >> Host: URL > > >> User-Agent: curl/7.68.0 > > >> Accept: */* > > >> > > > { [5 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > > > { [217 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > > > { [217 bytes data] > > > * old SSL session ID is stale, removing > > > { [5 bytes data] > > > * Mark bundle as not supporting multiuse > > > < HTTP/1.1 200 OK > > > < Cache-Control: no-cache > > > < Content-Type: application/json > > > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0 > > > < Date: Wed, 24 Mar 2021 15:04:34 GMT > > > < Server: Google Frontend > > > < Content-Length: 7950 > > > < X-Cache: MISS from server > > > < X-Cache-Lookup: HIT from server > > > < Via: 1.1 server (squid/4.14) > > > < Connection: keep-alive > > > > > > access log: > > > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL - > > > ORIGINAL_DST/IP application/json > > > > > > cache log: > > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > > > haveParsedReplyHeaders: decided: cache positively and share > > because > > > refresh check returned cacheable; HTTP status 200 > > e:=p2V/0x34868914670*3 > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(470) > > refreshCheck: > > > returning FRESH_MIN_RULE > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(455) > > refreshCheck: > > > Object isn't stale.. > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(327) > > refreshCheck: > > > Staleness = -1 > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(199) > > > refreshStaleness: FRESH: age (60 sec) is less than > configured > > minimum > > > (259200 sec) > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(166) > > > refreshStaleness: No explicit expiry given, using > heuristics to > > > determine freshness > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(307) > > refreshCheck: > > > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(305) > > refreshCheck: > > > check_time: Wed, 24 Mar 2021 15:05:34 GMT > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(303) > > refreshCheck: > > > age: 60 > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(301) > > refreshCheck: > > > Matched 'URL 259200 80%% 7776000' > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(279) > > refreshCheck: > > > checking freshness of URI: https://URL <https://URL> > <https://URL <https://URL>> > > <https://URL <https://URL> <https://URL <https://URL>>> > > > > > > > > > _______________________________________________ > > > squid-users mailing list > > > [hidden email] > <mailto:[hidden email]> > > <mailto:[hidden email] > <mailto:[hidden email]>> > > > http://lists.squid-cache.org/listinfo/squid-users > <http://lists.squid-cache.org/listinfo/squid-users> > > <http://lists.squid-cache.org/listinfo/squid-users > <http://lists.squid-cache.org/listinfo/squid-users>> > > > > > > _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
|
|
Hey Mirek,
This is not the first time this issue rises. There are risks in implementing any solution for this *issue*. I have implemented YouTube caching in the past using couple twisted techniques while leaving squid un-touched. The desire to caching sometimes can overcome couple very big risks to the integrity if the date/content. It is possible to use an ICAP service with a 206 response instead of 204 or 200 however I believe that you wouldn’t need to cache any POST requests so a simple ICAP service would be sufficient. I believe that It is preferred to leave squid sources un touched for such a purpose. An example for such a twist is at: * https://github.com/elico/squid-helpers/tree/master/squid_helpers/youtubetwist * https://wiki.squid-cache.org/ConfigExamples/DynamicContent/Coordinator?highlight=%28cache_peer%29#Implementing_ICAP_solution * https://ieeexplore.ieee.org/abstract/document/9072556 I wrote a public example of an ICAP server that was used to prove vulnerabilities in HTTP which is now used in the prove of HTTPS vulnerabilities. Take a peek at: * https://github.com/elico/bgu-icap-example It's written in GoLang and works under pretty heavy loads. Let me know if you need more help, Eliezer ---- Eliezer Croitoru Tech Support Mobile: +972-5-28704261 Email: [hidden email] Zoom: Coming soon -----Original Message----- From: squid-users <[hidden email]> On Behalf Of Alex Rousskov Sent: Friday, March 26, 2021 10:36 PM To: [hidden email] Subject: Re: [squid-users] squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale On 3/24/21 3:34 PM, Miroslaw Malinowski wrote: > I thought about upper service but as is not required at the moment, > introducing extra hop just to remove the header looks a bit like a > hammer approach. I'll look into how easily I can amend the code as the > other option is to introduce a proxy like a feature to the application, > so either way, it is a code change. The only problem here is that it's > an OPNSense squid service so I have to compile from source on BSD and > then keep adding in manually each time they do the update. At the risk of stating the obvious: If your feature is officially accepted into Squid sources, then you would not have to keep adding it manually (once the changes reach your Squid packaging source). Alex. > On Wed, Mar 24, 2021 at 7:11 PM Alex Rousskov wrote: > > On 3/24/21 2:49 PM, Miroslaw Malinowski wrote: > > > looking at the code and reading carefully your response, you're saying > > there is no way you can do it with squid. > > With Squid, your options include: > > 1. Squid source code changes. Should not be too difficult and, IMO, a > high-quality implementation would deserve official acceptance because it > is a generally useful feature in line with existing control knobs. > https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F > > 2. An adaptation service that removes Cache-Control:no-cache from the > response before Squid processes it: > https://wiki.squid-cache.org/SquidFaq/ContentAdaptation > > > HTH, > > Alex. > > > On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote: > > > > Hi, > > > > You've right yes it's revalidating as API server I'm > requesting data > > is setting Cache-Control: no-cache. My question is how I can force > > squid to cache and not validate as I know it's safe to do so. As > > I've explained earlier we are making the same request and > receiving > > the same response from 100+ server so as to reduce number of > > requests to the external server we would like squid to cache the > > response and issue a cached version. > > > > 2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck: > > YES: Must revalidate stale object (origin set no-cache or private) > > > > Mirek > > > > On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov > > <[hidden email] > <mailto:[hidden email]> > > <mailto:[hidden email] > <mailto:[hidden email]>>> wrote: > > > > On 3/24/21 12:48 PM, Miroslaw Malinowski wrote: > > > > > Probably, me missing on something silly or it can't be done > > but I don't > > > know why but squid won't return the cached version even > when I > > turn all > > > override options ON in refresh_pattern. > > > > AFAICT, no configuration options that can disable > revalidation of > > Cache-Control:no-cache responses. refresh_pattern does not > have an > > (equivalent of) "ignore-no-cache-in-responses" option. > > > > IIRC, older Squids were violating an HTTP MUST by > forgetting to > > revalidate Cache-Control:no-cache responses, but that was > fixed > > in [1]. > > Your Squid version has that fix. > > > > [1] > > > https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa > <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa> > > > <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa > <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>> > > > > > > > With debug, I can see the rule is matched and the cache is > > fresh but > > > still in access.log is TCP_REFRESH_MODIFIED > > > > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > > > haveParsedReplyHeaders: decided: cache positively and share > > because > > > > FYI: You are looking at cache.log lines logged _after_ > Squid has > > already > > decided to refresh the cached version. If you want to analyze > > why Squid > > decided to refresh the cached version, you should look > _before_ > > Squid > > logged the request to the server (and before any FwdState.cc > > lines). I > > have not checked the details, but I bet that your Squid > revalidates > > because of Cache-Control:no-cache in the response. Look for > > "YES: Must > > revalidate stale object". > > > > > > HTH, > > > > Alex. > > > > > squid conf: > > > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod > > > override-expire ignore-reload ignore-no-store ignore-private > > store-stale > > > > > > curl headers: > > > curl --insecure --verbose --request GET --url 'URL' > >/dev/null > > > * TCP_NODELAY set > > > * ALPN, offering h2 > > > * ALPN, offering http/1.1 > > > * successfully set certificate verify locations: > > > * CAfile: /etc/ssl/certs/ca-certificates.crt > > > CApath: /etc/ssl/certs > > > } [5 bytes data] > > > * TLSv1.3 (OUT), TLS handshake, Client hello (1): > > > } [512 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Server hello (2): > > > { [122 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): > > > { [6 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Certificate (11): > > > { [1956 bytes data] > > > * TLSv1.3 (IN), TLS handshake, CERT verify (15): > > > { [78 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Finished (20): > > > { [52 bytes data] > > > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): > > > } [1 bytes data] > > > * TLSv1.3 (OUT), TLS handshake, Finished (20): > > > } [52 bytes data] > > > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 > > > > > >> GET URL HTTP/1.1 > > >> Host: URL > > >> User-Agent: curl/7.68.0 > > >> Accept: */* > > >> > > > { [5 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > > > { [217 bytes data] > > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): > > > { [217 bytes data] > > > * old SSL session ID is stale, removing > > > { [5 bytes data] > > > * Mark bundle as not supporting multiuse > > > < HTTP/1.1 200 OK > > > < Cache-Control: no-cache > > > < Content-Type: application/json > > > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0 > > > < Date: Wed, 24 Mar 2021 15:04:34 GMT > > > < Server: Google Frontend > > > < Content-Length: 7950 > > > < X-Cache: MISS from server > > > < X-Cache-Lookup: HIT from server > > > < Via: 1.1 server (squid/4.14) > > > < Connection: keep-alive > > > > > > access log: > > > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL - > > > ORIGINAL_DST/IP application/json > > > > > > cache log: > > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982) > > > haveParsedReplyHeaders: decided: cache positively and share > > because > > > refresh check returned cacheable; HTTP status 200 > > e:=p2V/0x34868914670*3 > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(470) > > refreshCheck: > > > returning FRESH_MIN_RULE > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(455) > > refreshCheck: > > > Object isn't stale.. > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(327) > > refreshCheck: > > > Staleness = -1 > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(199) > > > refreshStaleness: FRESH: age (60 sec) is less than > configured > > minimum > > > (259200 sec) > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(166) > > > refreshStaleness: No explicit expiry given, using > heuristics to > > > determine freshness > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(307) > > refreshCheck: > > > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(305) > > refreshCheck: > > > check_time: Wed, 24 Mar 2021 15:05:34 GMT > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(303) > > refreshCheck: > > > age: 60 > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(301) > > refreshCheck: > > > Matched 'URL 259200 80%% 7776000' > > > 2021-03-24T15:04:34 squid .710 kid1| 22,3| > refresh.cc(279) > > refreshCheck: > > > checking freshness of URI: https://URL <https://URL> > <https://URL <https://URL>> > > <https://URL <https://URL> <https://URL <https://URL>>> > > > > > > > > > _______________________________________________ > > > squid-users mailing list > > > [hidden email] > <mailto:[hidden email]> > > <mailto:[hidden email] > <mailto:[hidden email]>> > > > http://lists.squid-cache.org/listinfo/squid-users > <http://lists.squid-cache.org/listinfo/squid-users> > > <http://lists.squid-cache.org/listinfo/squid-users > <http://lists.squid-cache.org/listinfo/squid-users>> > > > > > > _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
«
Return to Squid - Users
|
1 view|%1 views
| Free forum by Nabble | Edit this page |