Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

duanyao
Hi,

When I test squid with rock storage, I have never got a TCP_MEM_HIT
(just TCP_HIT) in access.log. I do get TCP_MEM_HIT with ufs storage.

Is this normal, or something is mis-configured?


I run squid locally, and run `curl -v -x localhost:3128
http://localhost:8001/1000 > /dev/null` two times successively, and
expect at least the second one being TCP_MEM_HIT, but it is always TCP_HIT.


Attachment "squid.conf" is my configuration file; and "cache.rock.log"
is the part of cache.log during 2 successive accesses to a same URL;
"anysize-server.js" is the HTTP server at port 8001 for the test, and
you can run it by `node anysize-server.js`.


I tested squid 3.5.27 and 4.0.21, both compiled and run on Linux x86_64:
`./configure --with-default-user=duanyao
--prefix=/home/duanyao/project/proxy/squid-root/`.


Thanks.


Duan, Yao


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

squid.conf (1K) Download Attachment
cache.rock.log (21K) Download Attachment
anysize-server.js (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

Alex Rousskov
On 10/16/2017 10:30 PM, duanyao wrote:

> When I test squid with rock storage, I have never got a TCP_MEM_HIT
> (just TCP_HIT) in access.log.
>
> Is this normal, or something is mis-configured?

Neither :-). You are testing with a Vary-controlled response. The shared
memory cache does not support caching of Vary-controlled responses yet.
I failed to find a dedicated feature request for implementing this, but
the problem has been known for several years [1,2,3].

[1]
http://bazaar.launchpad.net/~measurement-factory/squid/bag5s/revision/12743
[2] https://bugs.squid-cache.org/show_bug.cgi?id=3806#c9
[3]
https://github.com/squid-cache/squid/commit/b8a899c01051c27f2d46f5b9fcbcd27622f388ff


> I do get TCP_MEM_HIT with ufs storage.

To be more precise, you are getting TCP_MEM_HIT with a non-shared memory
cache. This is not a Rock problem; it is a shared memory cache problem.
Switching to UFS appears to help because it disables SMP and shared
memory caching in your specific Squid configuration.


Please note that your cache.log does not contain the lines relevant to
storing the response in the memory cache so there may be other,
additional reasons why your test response does not get memory-cached. If
you continue testing and need to report more details, please start from
a clean cache and post the log containing the very first (cache miss)
transaction as well as the subsequent (hit) transactions. Compress the
logs as needed.


Thank you,

Alex.



> I run squid locally, and run `curl -v -x localhost:3128
> http://localhost:8001/1000 > /dev/null` two times successively, and
> expect at least the second one being TCP_MEM_HIT, but it is always TCP_HIT.
>
>
> Attachment "squid.conf" is my configuration file; and "cache.rock.log"
> is the part of cache.log during 2 successive accesses to a same URL;
> "anysize-server.js" is the HTTP server at port 8001 for the test, and
> you can run it by `node anysize-server.js`.
>
>
> I tested squid 3.5.27 and 4.0.21, both compiled and run on Linux x86_64:
> `./configure --with-default-user=duanyao
> --prefix=/home/duanyao/project/proxy/squid-root/`.
>
>
> Thanks.
>
>
> Duan, Yao
>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

duanyao
在 2017/10/18 上午12:23, Alex Rousskov 写道:

> On 10/16/2017 10:30 PM, duanyao wrote:
>
>> When I test squid with rock storage, I have never got a TCP_MEM_HIT
>> (just TCP_HIT) in access.log.
>>
>> Is this normal, or something is mis-configured?
> Neither :-). You are testing with a Vary-controlled response. The shared
> memory cache does not support caching of Vary-controlled responses yet.
> I failed to find a dedicated feature request for implementing this, but
> the problem has been known for several years [1,2,3].
>
> [1]
> http://bazaar.launchpad.net/~measurement-factory/squid/bag5s/revision/12743
> [2] https://bugs.squid-cache.org/show_bug.cgi?id=3806#c9
> [3]
> https://github.com/squid-cache/squid/commit/b8a899c01051c27f2d46f5b9fcbcd27622f388ff
Thanks!
Indeed, by testing with a fresh rock database, I got
   2017/10/18 11:39:15.283 kid1| 20,5| MemStore.cc(623) shouldCache:
Vary not yet supported: accept-encoding
in cache.log ; and if "vary" header was not sent from the server, I
could get TCP_MEM_HIT.

>
>> I do get TCP_MEM_HIT with ufs storage.
> To be more precise, you are getting TCP_MEM_HIT with a non-shared memory
> cache. This is not a Rock problem; it is a shared memory cache problem.
> Switching to UFS appears to help because it disables SMP and shared
> memory caching in your specific Squid configuration.
>
>
> Please note that your cache.log does not contain the lines relevant to
> storing the response in the memory cache so there may be other,
> additional reasons why your test response does not get memory-cached. If
> you continue testing and need to report more details, please start from
> a clean cache and post the log containing the very first (cache miss)
> transaction as well as the subsequent (hit) transactions. Compress the
> logs as needed.
>
>
> Thank you,
>
> Alex.
>
>
>
>> I run squid locally, and run `curl -v -x localhost:3128
>> http://localhost:8001/1000 > /dev/null` two times successively, and
>> expect at least the second one being TCP_MEM_HIT, but it is always TCP_HIT.
>>
>>
>> Attachment "squid.conf" is my configuration file; and "cache.rock.log"
>> is the part of cache.log during 2 successive accesses to a same URL;
>> "anysize-server.js" is the HTTP server at port 8001 for the test, and
>> you can run it by `node anysize-server.js`.
>>
>>
>> I tested squid 3.5.27 and 4.0.21, both compiled and run on Linux x86_64:
>> `./configure --with-default-user=duanyao
>> --prefix=/home/duanyao/project/proxy/squid-root/`.
>>
>>
>> Thanks.
>>
>>
>> Duan, Yao
>>
>>
>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

duanyao
In reply to this post by Alex Rousskov
在 2017/10/18 上午12:23, Alex Rousskov 写道:

> On 10/16/2017 10:30 PM, duanyao wrote:
>
>> When I test squid with rock storage, I have never got a TCP_MEM_HIT
>> (just TCP_HIT) in access.log.
>>
>> Is this normal, or something is mis-configured?
> Neither :-). You are testing with a Vary-controlled response. The shared
> memory cache does not support caching of Vary-controlled responses yet.
> I failed to find a dedicated feature request for implementing this, but
> the problem has been known for several years [1,2,3].
>
> [1]
> http://bazaar.launchpad.net/~measurement-factory/squid/bag5s/revision/12743
> [2] https://bugs.squid-cache.org/show_bug.cgi?id=3806#c9
> [3]
> https://github.com/squid-cache/squid/commit/b8a899c01051c27f2d46f5b9fcbcd27622f388ff
>

I guess this issue will unlikely be fixed any time soon. So is there any
workaround? Something in my mind:
1) Configure squid to ignore "Vary: Accept-Encoding" and assume all
clients can handle any Content-Encoding from original servers (this make
sense in my use case).
2) Use non-shared memory cache with Rock storage.

>> I do get TCP_MEM_HIT with ufs storage.
> To be more precise, you are getting TCP_MEM_HIT with a non-shared memory
> cache. This is not a Rock problem; it is a shared memory cache problem.
> Switching to UFS appears to help because it disables SMP and shared
> memory caching in your specific Squid configuration.
>
>
> Please note that your cache.log does not contain the lines relevant to
> storing the response in the memory cache so there may be other,
> additional reasons why your test response does not get memory-cached. If
> you continue testing and need to report more details, please start from
> a clean cache and post the log containing the very first (cache miss)
> transaction as well as the subsequent (hit) transactions. Compress the
> logs as needed.
>
>
> Thank you,
>
> Alex.
>
>
>
>> I run squid locally, and run `curl -v -x localhost:3128
>> http://localhost:8001/1000 > /dev/null` two times successively, and
>> expect at least the second one being TCP_MEM_HIT, but it is always TCP_HIT.
>>
>>
>> Attachment "squid.conf" is my configuration file; and "cache.rock.log"
>> is the part of cache.log during 2 successive accesses to a same URL;
>> "anysize-server.js" is the HTTP server at port 8001 for the test, and
>> you can run it by `node anysize-server.js`.
>>
>>
>> I tested squid 3.5.27 and 4.0.21, both compiled and run on Linux x86_64:
>> `./configure --with-default-user=duanyao
>> --prefix=/home/duanyao/project/proxy/squid-root/`.
>>
>>
>> Thanks.
>>
>>
>> Duan, Yao
>>
>>
>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

Alex Rousskov
On 10/17/2017 11:48 PM, duanyao wrote:
> 在 2017/10/18 上午12:23, Alex Rousskov 写道:

>> The shared
>> memory cache does not support caching of Vary-controlled responses yet.

> I guess this issue will unlikely be fixed any time soon.

The timing of adding Vary caching support for the shared memory cache is
largely controlled by you and other Squid users:

http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F


> So is there any workaround? Something in my mind:

> 1) Configure squid to ignore "Vary: Accept-Encoding" and assume all
> clients can handle any Content-Encoding from original servers (this make
> sense in my use case).

Vary is not limited to Content-Encoding. I do not think it is possible
to configure Squid to ignore Vary (reply_header_access does not work for
that!), and even if it is possible, it will certainly break web clients
in all but the most restrictive/limited environments.

You can modify Squid code to ignore Vary, of course. Sorry, I do not
have a ready-to-use instructions for doing that. Again, disabling Vary
is a bad idea unless proxied traffic does not use Vary at all.


> 2) Use non-shared memory cache with Rock storage.

That is always an option, but you should not use SMP then. If you want
to use rock cache_dirs without SMP, start Squid with -N.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

duanyao
在 2017/10/18 下午11:17, Alex Rousskov 写道:

> On 10/17/2017 11:48 PM, duanyao wrote:
>> 在 2017/10/18 上午12:23, Alex Rousskov 写道:
>>> The shared
>>> memory cache does not support caching of Vary-controlled responses yet.
>> I guess this issue will unlikely be fixed any time soon.
> The timing of adding Vary caching support for the shared memory cache is
> largely controlled by you and other Squid users:
>
> http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
>

Fair enough, I'll consider it. Maybe I can implement ignoring "Vary:
Accept-Encoding" as an option, it looks relatively simple.

>> So is there any workaround? Something in my mind:
>> 1) Configure squid to ignore "Vary: Accept-Encoding" and assume all
>> clients can handle any Content-Encoding from original servers (this make
>> sense in my use case).
> Vary is not limited to Content-Encoding. I do not think it is possible
> to configure Squid to ignore Vary (reply_header_access does not work for
> that!), and even if it is possible, it will certainly break web clients
> in all but the most restrictive/limited environments.
>
> You can modify Squid code to ignore Vary, of course. Sorry, I do not
> have a ready-to-use instructions for doing that. Again, disabling Vary
> is a bad idea unless proxied traffic does not use Vary at all.
>

I mean just ignore "Vary: Accept-Encoding" and nothing more —— if the
original server sends "Vary: User-Agent" or "Vary: Cookie", don't ignore
them.
To my knowledge, Accept-Encoding and Content-Encoding are only used for
compressing,
and almost all decent HTTP clients can handle compressed content, so the
risk looks low enough.

>> 2) Use non-shared memory cache with Rock storage.
> That is always an option, but you should not use SMP then. If you want
> to use rock cache_dirs without SMP, start Squid with -N.

So it's not possible to make each worker process to own a non-shared
memory cache?
Well, the main point to use Rock is SMP support.

>
> Alex.



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

Alex Rousskov
On 10/18/2017 09:37 PM, duanyao wrote:

> Maybe I can implement ignoring "Vary: Accept-Encoding" as an option,
> it looks relatively simple.

It is probably simple indeed. One caveat: If you want your changes to be
officially accepted, then you may want to discuss your idea on squid-dev
first. I suspect that a long-term implementation that the Project can
accept may involve quite a bit more than an option to ignore
Vary:Accept-Encoding.


>>> 2) Use non-shared memory cache with Rock storage.

>> That is always an option, but you should not use SMP then. If you want
>> to use rock cache_dirs without SMP, start Squid with -N.

> So it's not possible to make each worker process to own a non-shared
> memory cache?

It is possible (see memory_cache_shared), but:

* you must not combine SMP Squid, rock, and non-shared memory cache and,

* IMO, you should not combine SMP Squid and non-shared memory cache
  (and/or any other SMP-unaware cache such as ufs).

If you want more information on this confusing subject, please search
the mailing list archives. The topic of combining SMP-unaware caches and
SMP has been discussed extensively.


Thank you,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

duanyao
在 2017/10/19 上午11:59, Alex Rousskov 写道:

> On 10/18/2017 09:37 PM, duanyao wrote:
>
>> Maybe I can implement ignoring "Vary: Accept-Encoding" as an option,
>> it looks relatively simple.
> It is probably simple indeed. One caveat: If you want your changes to be
> officially accepted, then you may want to discuss your idea on squid-dev
> first. I suspect that a long-term implementation that the Project can
> accept may involve quite a bit more than an option to ignore
> Vary:Accept-Encoding.
>

I see, thanks.

>>>> 2) Use non-shared memory cache with Rock storage.
>>> That is always an option, but you should not use SMP then. If you want
>>> to use rock cache_dirs without SMP, start Squid with -N.
>> So it's not possible to make each worker process to own a non-shared
>> memory cache?
> It is possible (see memory_cache_shared), but:
>
> * you must not combine SMP Squid, rock, and non-shared memory cache and,
>
> * IMO, you should not combine SMP Squid and non-shared memory cache
>    (and/or any other SMP-unaware cache such as ufs).
>
> If you want more information on this confusing subject, please search
> the mailing list archives. The topic of combining SMP-unaware caches and
> SMP has been discussed extensively.

So this option is out because I still need SMP.

>
> Thank you,
>
> Alex.



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

Eliezer Croitoru
In reply to this post by duanyao
Hey,

It's not clear to me if the issue you are trying to resolve is for a specific service or squid-wide.
Your tests are done on localhost so it might reflect other services as well but still I want to be sure I understand what you are aiming for.
If you are indeed aiming a specific service then I believe that your setup+squid.conf maybe not what you need.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of duanyao
Sent: Tuesday, October 17, 2017 07:31
To: [hidden email]
Subject: [squid-users] Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

Hi,

When I test squid with rock storage, I have never got a TCP_MEM_HIT
(just TCP_HIT) in access.log. I do get TCP_MEM_HIT with ufs storage.

Is this normal, or something is mis-configured?


I run squid locally, and run `curl -v -x localhost:3128
http://localhost:8001/1000 > /dev/null` two times successively, and
expect at least the second one being TCP_MEM_HIT, but it is always TCP_HIT.


Attachment "squid.conf" is my configuration file; and "cache.rock.log"
is the part of cache.log during 2 successive accesses to a same URL;
"anysize-server.js" is the HTTP server at port 8001 for the test, and
you can run it by `node anysize-server.js`.


I tested squid 3.5.27 and 4.0.21, both compiled and run on Linux x86_64:
`./configure --with-default-user=duanyao
--prefix=/home/duanyao/project/proxy/squid-root/`.


Thanks.


Duan, Yao


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage

duanyao
在 2017/10/19 下午6:55, Eliezer Croitoru 写道:
> Hey,
>
> It's not clear to me if the issue you are trying to resolve is for a specific service or squid-wide.

I'm not sure what do "a specific service" and "squid-wide" mean.
I think Alex Rousskov already solved my issue in this thread: shared
memory cache can't hold responses with a "Vary" header.
Thank you anyway.

> Your tests are done on localhost so it might reflect other services as well but still I want to be sure I understand what you are aiming for.

My goal was figuring out why Squid with Rock always send TCP_HIT instead
of TCP_MEM_HIT for some responses (turned out they have "Vary" header).

> If you are indeed aiming a specific service then I believe that your setup+squid.conf maybe not what you need.
>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: [hidden email]
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of duanyao
> Sent: Tuesday, October 17, 2017 07:31
> To: [hidden email]
> Subject: [squid-users] Unable to get TCP_MEM_HIT (just TCP_HIT) with rock storage
>
> Hi,
>
> When I test squid with rock storage, I have never got a TCP_MEM_HIT
> (just TCP_HIT) in access.log. I do get TCP_MEM_HIT with ufs storage.
>
> Is this normal, or something is mis-configured?
>
>
> I run squid locally, and run `curl -v -x localhost:3128
> http://localhost:8001/1000 > /dev/null` two times successively, and
> expect at least the second one being TCP_MEM_HIT, but it is always TCP_HIT.
>
>
> Attachment "squid.conf" is my configuration file; and "cache.rock.log"
> is the part of cache.log during 2 successive accesses to a same URL;
> "anysize-server.js" is the HTTP server at port 8001 for the test, and
> you can run it by `node anysize-server.js`.
>
>
> I tested squid 3.5.27 and 4.0.21, both compiled and run on Linux x86_64:
> `./configure --with-default-user=duanyao
> --prefix=/home/duanyao/project/proxy/squid-root/`.
>
>
> Thanks.
>
>
> Duan, Yao
>
>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users