TCP_REFRESH_UNMODIFIEDs are really MISSes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

TCP_REFRESH_UNMODIFIEDs are really MISSes

Victor Sudakov
Dear Colleagues,

When updating several almost identical FreeBSD hosts via a
squid-3.5.27 proxy, I expect to see lots of HITs, because the patches
on the update server should be identical too.

However, I see lots of TCP_REFRESH_UNMODIFIED lines, and only
occasional HIT statuses. Please see the log excerpt at the end of the
message, and the example exchange between client and squid.

 From the squid documentation I figure out that a
TCP_REFRESH_UNMODIFIED status means that there was an IMS request
involved. But I don't understand who made this IMS request: client to
squid or squid to the webserver.

The "fetch" HTTP client the freebsd-update utility uses is rather dumb
and does not send IMS requests (also see below). Could someone please
explain what is happening here?

The HTTP replies from the squid cache to the HTTP client are
"X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080", so these
TCP_REFRESH_UNMODIFIED statuses seem to be really MISSes and I'm not
saving traffic on these updates.


1507630624.673    389 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 572 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/82a25f03517d9d65b3c0af22f77ed9cbe94d09f1af6f4cd835a93b5f191122dc-63836fc812fc7b03a2ad1d2df9c59dde3178e1eaf4d0e32342a8a04a7121ac6d - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630625.067    394 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 574 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/584713682ac3cc86fd2bfb7edbb9030aa48efbf027465ebcce705864310140af-0981c929f128ad067d3c8c577f9c84302523e24548b561edceee28a3357a5007 - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630625.515    448 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630625.914    399 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 532 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/aac782643cbfaad218bf72f037624af304d353f2a12fb6a5eeb117cdd6c9bc27-2f6df820f68859d2e86a8f45777a81a7186477048893456f891fd91dae19af3b - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630625.948     33 212.73.124.12 TCP_HIT/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_NONE/- application/octet-stream
1507630626.354    406 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 531 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/58764b03f019f1aa9e28e24ddb9e74abf323cfc3984bd899accf7b15bb2a6462-6d53fae808e6d79c6c251f6b1fe5e26fbd84e191f84900bcf01931d49855249f - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630626.391     36 212.73.124.12 TCP_HIT/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_NONE/- application/octet-stream
1507630626.391      0 212.73.124.12 TCP_MEM_HIT/200 527 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1af4cec823d8b1f42c3b8a58ec0da40bfc2f97efacd92e7758096eba081e08db-2ac8890502bc272d65f2b7dd73fa7def11fa4d3feb2005b82b58ba43747b57e4 - HIER_NONE/- application/octet-stream
1507630626.785    393 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 697 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/6c0eae2981826e2033797003f53aedba9adb7e317c936f6923912e5df2627216-341717a9aa21a95b7f9f83e29ac3de1cac6b9d58700ece767e4ab7cd028105c9 - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630627.184    399 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 1001 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/6f486e6baa2d057fa16d3fa43e666917ab9d045de2e59a7d05ace758c7d03373-653636793fd95f406da85e1375acd38f31b6c39c404b18cd8e0a641a2deb6065 - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630627.577    392 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 674 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/3c78173d804b5299f24199a125b3c52145c7e7680c97debb051d823526ae0579-595c12cd31ef1c4d084c308b7278d28bbb844873da246d88fb02db90f90b88b9 - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630627.967    389 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 1193 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/cc4d2e70a2c12900362eba48bae5fa8388bfd8ed9975246578d5c68fd00fb9b3-74473c002b6e269ddb7b3c385cfff0f43b4fcbf1dcba7f36dac7c7918b19f8a9 - HIER_DIRECT/198.148.79.66 application/octet-stream
1507630628.357    391 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 29745 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/2f0861149594653c2b91faec675aef008728c5ba13b352ca3877714a40bffbd6-e6b495c0d23d7eb5fe43bb07e5a8a71f3bad37844a0e8a121ef2178e5fd6ca4a - HIER_DIRECT/198.148.79.66 application/octet-stream


GET http://update.FreeBSD.org/to-10.4-RELEASE/i386/bp/4b363ee1c1e681fe5a67dd7201062ee1977f1437ce9de7c7bc9c9b84839f10df-6dc4edca0bcab3f4b0ae59d1a2714aeb0df22fce043c8897079ab9914fdafbf4 HTTP/1.1
Host: update.FreeBSD.org
User-Agent: freebsd-update (upgrade, 10.3-RELEASE-p20)
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Wed, 11 Oct 2017 01:38:16 GMT
Server: Apache/2.2.16 (FreeBSD) mod_ssl/2.2.16 OpenSSL/0.9.8n DAV/2
Last-Modified: Sun, 01 Oct 2017 19:32:05 GMT
ETag: "1f0c0de-951-55a81501e5740"
Accept-Ranges: bytes
Content-Length: 2385
Content-Type: text/plain
X-Cache: MISS from proxy2.sibptus.ru
X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080
Via: 1.1 proxy2.sibptus.ru (squid/3.5.27)
Connection: keep-alive

BSDIFF40.......

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP_REFRESH_UNMODIFIEDs are really MISSes

Alex Rousskov
On 10/10/2017 07:50 PM, Victor Sudakov wrote:

> When updating several almost identical FreeBSD hosts via a
> squid-3.5.27 proxy, I expect to see lots of HITs, because the patches
> on the update server should be identical too.
>
> However, I see lots of TCP_REFRESH_UNMODIFIED lines, and only
> occasional HIT statuses.


There are many kinds of "hits":

* If your primary concern is data transfer/bandwidth usage, then you
should focus on so called byte hits. TCP_REFRESH_UNMODIFIED is
essentially one of those because the origin server sends a tiny
headers-only response to Squid.

* If your primary concern is latency, then TCP_REFRESH_UNMODIFIED should
be treated as a miss because Squid did talk to the origin server.

You may find more relevant information and discussions if you search
around for pages containing phrases like "byte hit ratio" and "document
hit ratio".


> TCP_REFRESH_UNMODIFIED status means that there was an IMS request
> involved. But I don't understand who made this IMS request: client to
> squid or squid to the webserver.

If my quick reading of the code is correct, then the
TCP_REFRESH_UNMODIFIED status itself does not distinguish those three
cases: It could be Squid; it could be client; and it could be both. The
only thing we can say based on the status alone is that Squid decided
that the cached content was stale. Many factors can contribute to that
decision.


> The "fetch" HTTP client the freebsd-update utility uses is rather dumb
> and does not send IMS requests (also see below). Could someone please
> explain what is happening here?

If the client is not sending any conditional headers, then Squid decides
that the previously cached response has expired and needs to be revalidated.


> The HTTP replies from the squid cache to the HTTP client are
> "X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080", so these
> TCP_REFRESH_UNMODIFIED statuses seem to be really MISSes and I'm not
> saving traffic on these updates.

According to your pasted log and HTTP messages, the GET request URL that
resulted an "X-Cache-Lookup: MISS" response header does not match any of
the logged URLs; even the host part of the URL does not match. To be
sure that you are comparing apples to apples, you would need to log the
X-Cache-Lookup response header to access.log (or enable ALL,2 debugging
to see headers and/or collect packet traces).


HTH,

Alex.


> 1507630624.673    389 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 572 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/82a25f03517d9d65b3c0af22f77ed9cbe94d09f1af6f4cd835a93b5f191122dc-63836fc812fc7b03a2ad1d2df9c59dde3178e1eaf4d0e32342a8a04a7121ac6d - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.067    394 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 574 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/584713682ac3cc86fd2bfb7edbb9030aa48efbf027465ebcce705864310140af-0981c929f128ad067d3c8c577f9c84302523e24548b561edceee28a3357a5007 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.515    448 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.914    399 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 532 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/aac782643cbfaad218bf72f037624af304d353f2a12fb6a5eeb117cdd6c9bc27-2f6df820f68859d2e86a8f45777a81a7186477048893456f891fd91dae19af3b - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.948     33 212.73.124.12 TCP_HIT/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_NONE/- application/octet-stream
> 1507630626.354    406 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 531 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/58764b03f019f1aa9e28e24ddb9e74abf323cfc3984bd899accf7b15bb2a6462-6d53fae808e6d79c6c251f6b1fe5e26fbd84e191f84900bcf01931d49855249f - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630626.391     36 212.73.124.12 TCP_HIT/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_NONE/- application/octet-stream
> 1507630626.391      0 212.73.124.12 TCP_MEM_HIT/200 527 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1af4cec823d8b1f42c3b8a58ec0da40bfc2f97efacd92e7758096eba081e08db-2ac8890502bc272d65f2b7dd73fa7def11fa4d3feb2005b82b58ba43747b57e4 - HIER_NONE/- application/octet-stream
> 1507630626.785    393 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 697 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/6c0eae2981826e2033797003f53aedba9adb7e317c936f6923912e5df2627216-341717a9aa21a95b7f9f83e29ac3de1cac6b9d58700ece767e4ab7cd028105c9 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630627.184    399 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 1001 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/6f486e6baa2d057fa16d3fa43e666917ab9d045de2e59a7d05ace758c7d03373-653636793fd95f406da85e1375acd38f31b6c39c404b18cd8e0a641a2deb6065 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630627.577    392 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 674 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/3c78173d804b5299f24199a125b3c52145c7e7680c97debb051d823526ae0579-595c12cd31ef1c4d084c308b7278d28bbb844873da246d88fb02db90f90b88b9 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630627.967    389 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 1193 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/cc4d2e70a2c12900362eba48bae5fa8388bfd8ed9975246578d5c68fd00fb9b3-74473c002b6e269ddb7b3c385cfff0f43b4fcbf1dcba7f36dac7c7918b19f8a9 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630628.357    391 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 29745 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/2f0861149594653c2b91faec675aef008728c5ba13b352ca3877714a40bffbd6-e6b495c0d23d7eb5fe43bb07e5a8a71f3bad37844a0e8a121ef2178e5fd6ca4a - HIER_DIRECT/198.148.79.66 application/octet-stream
>
>
> GET http://update.FreeBSD.org/to-10.4-RELEASE/i386/bp/4b363ee1c1e681fe5a67dd7201062ee1977f1437ce9de7c7bc9c9b84839f10df-6dc4edca0bcab3f4b0ae59d1a2714aeb0df22fce043c8897079ab9914fdafbf4 HTTP/1.1
> Host: update.FreeBSD.org
> User-Agent: freebsd-update (upgrade, 10.3-RELEASE-p20)
> Connection: Keep-Alive
>
> HTTP/1.1 200 OK
> Date: Wed, 11 Oct 2017 01:38:16 GMT
> Server: Apache/2.2.16 (FreeBSD) mod_ssl/2.2.16 OpenSSL/0.9.8n DAV/2
> Last-Modified: Sun, 01 Oct 2017 19:32:05 GMT
> ETag: "1f0c0de-951-55a81501e5740"
> Accept-Ranges: bytes
> Content-Length: 2385
> Content-Type: text/plain
> X-Cache: MISS from proxy2.sibptus.ru
> X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080
> Via: 1.1 proxy2.sibptus.ru (squid/3.5.27)
> Connection: keep-alive
>
> BSDIFF40.......
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP_REFRESH_UNMODIFIEDs are really MISSes

Victor Sudakov
Alex Rousskov wrote:

> On 10/10/2017 07:50 PM, Victor Sudakov wrote:
>
> > When updating several almost identical FreeBSD hosts via a
> > squid-3.5.27 proxy, I expect to see lots of HITs, because the patches
> > on the update server should be identical too.
> >
> > However, I see lots of TCP_REFRESH_UNMODIFIED lines, and only
> > occasional HIT statuses.
>
>
> There are many kinds of "hits":
>
> * If your primary concern is data transfer/bandwidth usage, then you
> should focus on so called byte hits. TCP_REFRESH_UNMODIFIED is
> essentially one of those because the origin server sends a tiny
> headers-only response to Squid.

My primary concern is bandwidth usage. You are saying that the
TCP_REFRESH_UNMODIFIED transaction means that the bulk of the data is
not transferred from Webserver to Squid, only the headers are fetched?

If you look at this line:

1507630625.515    448 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_DIRECT/198.148.79.66 application/octet-stream

There cannot be 980593 bytes of headers? What kind of gigantic header
does it make?

[dd]

>
> > The "fetch" HTTP client the freebsd-update utility uses is rather dumb
> > and does not send IMS requests (also see below). Could someone please
> > explain what is happening here?
>
> If the client is not sending any conditional headers, then Squid decides
> that the previously cached response has expired and needs to be revalidated.
>
>
> > The HTTP replies from the squid cache to the HTTP client are
> > "X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080", so these
> > TCP_REFRESH_UNMODIFIED statuses seem to be really MISSes and I'm not
> > saving traffic on these updates.
>
> According to your pasted log and HTTP messages, the GET request URL that
> resulted an "X-Cache-Lookup: MISS" response header does not match any of
> the logged URLs; even the host part of the URL does not match.

Of course, there are just unrelated examples. There are thousands of
fetches from update*.freebsd.org

> To be sure that you are comparing apples to apples, you would need
> to log the X-Cache-Lookup response header to access.log (or enable
> ALL,2 debugging to see headers and/or collect packet traces).

I don't think the freebsd-update cares about the X-Cache-Lookup
response, I'm just trying to save bandwidth and don't quite like the
result.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP_REFRESH_UNMODIFIEDs are really MISSes

Amos Jeffries
Administrator
On 11/10/17 23:31, Victor Sudakov wrote:

> Alex Rousskov wrote:
>> On 10/10/2017 07:50 PM, Victor Sudakov wrote:
>>
>>> When updating several almost identical FreeBSD hosts via a
>>> squid-3.5.27 proxy, I expect to see lots of HITs, because the patches
>>> on the update server should be identical too.
>>>
>>> However, I see lots of TCP_REFRESH_UNMODIFIED lines, and only
>>> occasional HIT statuses.
>>
>>
>> There are many kinds of "hits":
>>
>> * If your primary concern is data transfer/bandwidth usage, then you
>> should focus on so called byte hits. TCP_REFRESH_UNMODIFIED is
>> essentially one of those because the origin server sends a tiny
>> headers-only response to Squid.
>
> My primary concern is bandwidth usage. You are saying that the
> TCP_REFRESH_UNMODIFIED transaction means that the bulk of the data is
> not transferred from Webserver to Squid, only the headers are fetched?

Yes.

>
> If you look at this line:
>
> 1507630625.515    448 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_DIRECT/198.148.79.66 application/octet-stream
>
> There cannot be 980593 bytes of headers? What kind of gigantic header
> does it make?
>

That is header only between Squid and server (REFRESH_UNMODIFIED), with
full object delivered to the client (final status "200").

...
>>
>> According to your pasted log and HTTP messages, the GET request URL that
>> resulted an "X-Cache-Lookup: MISS" response header does not match any of
>> the logged URLs; even the host part of the URL does not match.
>
> Of course, there are just unrelated examples. There are thousands of
> fetches from update*.freebsd.org

So you just posted them to waste everybody on this lists bandwidth?
Thanks, and kind of ironic given your "problem".

>
>> To be sure that you are comparing apples to apples, you would need
>> to log the X-Cache-Lookup response header to access.log (or enable
>> ALL,2 debugging to see headers and/or collect packet traces).
>
> I don't think the freebsd-update cares about the X-Cache-Lookup
> response, I'm just trying to save bandwidth and don't quite like the
> result.
>

The X-Cache* headers are debug output from Squid. Of course the updater
software does not care about it ... you do, sort of anyway since it
shows whether server bandwidth was involved. They do not show REFRESH,
so MISS means a server was contacted for _something_ and HIT means local
cache only was used.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP_REFRESH_UNMODIFIEDs are really MISSes

Alex Rousskov
On 10/11/2017 04:52 AM, Amos Jeffries wrote:
> On 11/10/17 23:31, Victor Sudakov wrote:
>> Alex Rousskov wrote:
>>> To be sure that you are comparing apples to apples, you would need
>>> to log the X-Cache-Lookup response header to access.log (or enable
>>> ALL,2 debugging to see headers and/or collect packet traces).

>> I don't think the freebsd-update cares about the X-Cache-Lookup
>> response, I'm just trying to save bandwidth and don't quite like the
>> result.

> The X-Cache* headers are debug output from Squid. [...]
> MISS means a server was contacted for _something_ and HIT means local
> cache only was used.


Here is a more accurate description of the X-Cache and X-Cache-Lookup
header fields:

X-Cache-Lookup:MISS means that Squid did not find the requested object
in its cache after parsing the client request, and X-Cache-Lookup:HIT
means that it did. In either case, the X-Cache-Lookup header field does
not tell us whether the server was contacted afterwards, although:

* X-Cache-Lookup:MISS implies that the server contact is very likely
(but various errors and other corner cases may prevent any server
contact) and

* X-Cache-Lookup:HIT implies that the server contact is less likely (but
client revalidation requests, stale cached responses, and other cases
make such contact possible).

* X-Cache-Lookup:NONE means that there was no cache lookup at all.
(Listed here for completeness sake)


There is also X-Cache header field, but its value is just a mapping of
the final(?) status code to HIT or MISS categories. In Squid v3.5, the
HIT category is assigned to TCP_HIT, TCP_IMS_HIT, TCP_INM_HIT,
TCP_REFRESH_FAIL_OLD, TCP_REFRESH_UNMODIFIED, TCP_NEGATIVE_HIT,
TCP_MEM_HIT, and TCP_OFFLINE_HIT statuses. The rest are categorized as
MISSes. The X-Cache header field does not tell us whether the server was
contacted.



To avoid misunderstanding, I have provided this information to help you
triage why Squid revalidates so much in your use case, and not to
justify Squid behavior or blame the freebsd-update client. You now know
what the logged status and the header values mean. And you now know that
you have to look at the same transaction to analyze the combination of
logged values and headers. If you want to know _why_ Squid does what it
does, the information Amos and I have provided may help you dig deeper.
Once you know "why", you may be able to adjust Squid configuration (or
something else!) to reduce excessive revalidations.


HTH,

Alex.
P.S. If the above is not documented on Squid wiki, please consider
documenting it. Thank you in advance.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP_REFRESH_UNMODIFIEDs are really MISSes

Victor Sudakov
In reply to this post by Amos Jeffries
Amos Jeffries wrote:

> > Alex Rousskov wrote:
> >> On 10/10/2017 07:50 PM, Victor Sudakov wrote:
> >>
> >>> When updating several almost identical FreeBSD hosts via a
> >>> squid-3.5.27 proxy, I expect to see lots of HITs, because the patches
> >>> on the update server should be identical too.
> >>>
> >>> However, I see lots of TCP_REFRESH_UNMODIFIED lines, and only
> >>> occasional HIT statuses.
> >>
> >>
> >> There are many kinds of "hits":
> >>
> >> * If your primary concern is data transfer/bandwidth usage, then you
> >> should focus on so called byte hits. TCP_REFRESH_UNMODIFIED is
> >> essentially one of those because the origin server sends a tiny
> >> headers-only response to Squid.
> >
> > My primary concern is bandwidth usage. You are saying that the
> > TCP_REFRESH_UNMODIFIED transaction means that the bulk of the data is
> > not transferred from Webserver to Squid, only the headers are fetched?
>
> Yes.
>
> >
> > If you look at this line:
> >
> > 1507630625.515    448 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_DIRECT/198.148.79.66 application/octet-stream
> >
> > There cannot be 980593 bytes of headers? What kind of gigantic header
> > does it make?
> >
>
> That is header only between Squid and server (REFRESH_UNMODIFIED), with
> full object delivered to the client (final status "200").

I see now, the 980593 bytes in the log is between Squid and client, not between Squid
and Webserver.

> >>
> >> According to your pasted log and HTTP messages, the GET request URL that
> >> resulted an "X-Cache-Lookup: MISS" response header does not match any of
> >> the logged URLs; even the host part of the URL does not match.
> >
> > Of course, there are just unrelated examples. There are thousands of
> > fetches from update*.freebsd.org
>
> So you just posted them to waste everybody on this lists bandwidth?

Hmm, well, no. Consider them two separate examples: a) a sample log
output and b) a sample HTTP session. Both were necessary for
illustrating the question. I never thought someone would try to
juxtapose them.

> Thanks, and kind of ironic given your "problem".

About 30 lines of logs and Wireshark output surely wasted the lists
bandwidth. Sorry about that.

>
> >
> >> To be sure that you are comparing apples to apples, you would need
> >> to log the X-Cache-Lookup response header to access.log (or enable
> >> ALL,2 debugging to see headers and/or collect packet traces).
> >
> > I don't think the freebsd-update cares about the X-Cache-Lookup
> > response, I'm just trying to save bandwidth and don't quite like the
> > result.
> >
>
> The X-Cache* headers are debug output from Squid. Of course the updater
> software does not care about it ... you do, sort of anyway since it
> shows whether server bandwidth was involved. They do not show REFRESH,
> so MISS means a server was contacted for _something_ and HIT means local
> cache only was used.

Thank you for clarification. I will not fear the
TCP_REFRESH_UNMODIFIEDs any more.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP_REFRESH_UNMODIFIEDs are really MISSes

Victor Sudakov
In reply to this post by Alex Rousskov
Alex Rousskov wrote:

> On 10/11/2017 04:52 AM, Amos Jeffries wrote:
> > On 11/10/17 23:31, Victor Sudakov wrote:
> >> Alex Rousskov wrote:
> >>> To be sure that you are comparing apples to apples, you would need
> >>> to log the X-Cache-Lookup response header to access.log (or enable
> >>> ALL,2 debugging to see headers and/or collect packet traces).
>
> >> I don't think the freebsd-update cares about the X-Cache-Lookup
> >> response, I'm just trying to save bandwidth and don't quite like the
> >> result.
>
> > The X-Cache* headers are debug output from Squid. [...]
> > MISS means a server was contacted for _something_ and HIT means local
> > cache only was used.
>
>
> Here is a more accurate description of the X-Cache and X-Cache-Lookup
> header fields:
>
> X-Cache-Lookup:MISS means that Squid did not find the requested object
> in its cache after parsing the client request, and X-Cache-Lookup:HIT
> means that it did. In either case, the X-Cache-Lookup header field does
> not tell us whether the server was contacted afterwards, although:
>
> * X-Cache-Lookup:MISS implies that the server contact is very likely
> (but various errors and other corner cases may prevent any server
> contact) and
>
> * X-Cache-Lookup:HIT implies that the server contact is less likely (but
> client revalidation requests, stale cached responses, and other cases
> make such contact possible).
>
> * X-Cache-Lookup:NONE means that there was no cache lookup at all.
> (Listed here for completeness sake)
>
>
> There is also X-Cache header field, but its value is just a mapping of
> the final(?) status code to HIT or MISS categories. In Squid v3.5, the
> HIT category is assigned to TCP_HIT, TCP_IMS_HIT, TCP_INM_HIT,
> TCP_REFRESH_FAIL_OLD, TCP_REFRESH_UNMODIFIED, TCP_NEGATIVE_HIT,
> TCP_MEM_HIT, and TCP_OFFLINE_HIT statuses. The rest are categorized as
> MISSes. The X-Cache header field does not tell us whether the server was
> contacted.

Alex, thanks for the details.

>
>
>
> To avoid misunderstanding, I have provided this information to help you
> triage why Squid revalidates so much in your use case, and not to
> justify Squid behavior or blame the freebsd-update client. You now know
> what the logged status and the header values mean. And you now know that
> you have to look at the same transaction to analyze the combination of
> logged values and headers. If you want to know _why_ Squid does what it
> does, the information Amos and I have provided may help you dig deeper.
> Once you know "why", you may be able to adjust Squid configuration (or
> something else!) to reduce excessive revalidations.

Now that I know that the revalidations (TCP_REFRESH_UNMODIFIEDs) don't
fetch the actual bodies of the patches, I'm fine with them.

> P.S. If the above is not documented on Squid wiki, please consider
> documenting it. Thank you in advance.

You suggest I do it?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP_REFRESH_UNMODIFIEDs are really MISSes

Alex Rousskov
On 10/12/2017 04:17 AM, Victor Sudakov wrote:
> Alex Rousskov wrote:
>> P.S. If the above is not documented on Squid wiki, please consider
>> documenting it. Thank you in advance.

> You suggest I do it?

I suggest that you should at least consider doing it. In a nutshell,
there is nobody better than you who is likely to do it in the
foreseeable future and who can dedicate enough time/effort to do it
well. Despite our prayers, there is no magic fairy that updates the wiki
whenever some beans of wisdom are spilled on the mailing list.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users