Squid 3.4 very high cpu - strace.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid 3.4 very high cpu - strace.

Alex Crow
Hi Amos, all,

I have set up a test box with latest 3.4.5 nightly. I get 95-100% cpu
even with one client accessing the cache. I've attached a compressed
strace of the child process in case anything is evident from that.
Please tell me what else I might need to do to help resolve this issue.

I'm hoping this will help get to the bottom of why a number of people
are having this issue on 3.4.x.

Any help much appreciated as always.

Alex

strace.txt.bz2 (22K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Alex Crow
I think I've just found something. I had this set:

memory_replacement_policy heap GDSF

replacing this with:

memory_replacement_policy lru

got rid of the high CPU in 3.4 (works ok in 3,3).

I will try heap LRU.

Cheers

Alex

On 20/05/14 19:54, Alex Crow wrote:

> Hi Amos, all,
>
> I have set up a test box with latest 3.4.5 nightly. I get 95-100% cpu
> even with one client accessing the cache. I've attached a compressed
> strace of the child process in case anything is evident from that.
> Please tell me what else I might need to do to help resolve this issue.
>
> I'm hoping this will help get to the bottom of why a number of people
> are having this issue on 3.4.x.
>
> Any help much appreciated as always.
>
> Alex

Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Alex Crow
Wrong on my part again.

Changing the memory_replacement_policy still got to 100% cpu after
Shift-reload in Thunderbird a few times - even disabling cache_mem
entirely did not eliminate it. 3.3 never gets about about 67% load no
matter how many time the page is reloaded.

So again hope the the trace shows something up.

Cheers

Alex



On 20/05/14 20:04, Alex Crow wrote:

> I think I've just found something. I had this set:
>
> memory_replacement_policy heap GDSF
>
> replacing this with:
>
> memory_replacement_policy lru
>
> got rid of the high CPU in 3.4 (works ok in 3,3).
>
> I will try heap LRU.
>
> Cheers
>
> Alex
>
> On 20/05/14 19:54, Alex Crow wrote:
>> Hi Amos, all,
>>
>> I have set up a test box with latest 3.4.5 nightly. I get 95-100% cpu
>> even with one client accessing the cache. I've attached a compressed
>> strace of the child process in case anything is evident from that.
>> Please tell me what else I might need to do to help resolve this issue.
>>
>> I'm hoping this will help get to the bottom of why a number of people
>> are having this issue on 3.4.x.
>>
>> Any help much appreciated as always.
>>
>> Alex
>

Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Amos Jeffries
Administrator
On 21/05/2014 8:11 a.m., Alex Crow wrote:
> Wrong on my part again.
>
> Changing the memory_replacement_policy still got to 100% cpu after
> Shift-reload in Thunderbird a few times - even disabling cache_mem
> entirely did not eliminate it. 3.3 never gets about about 67% load no
> matter how many time the page is reloaded.

Thunderbird, are these troubles all coming from  HTML emails?

Does using AUFS instead of diskd cache types help? there are a lot of
calls in that trace polling the diskd helpers.

Amos

Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Alex Crow

> Thunderbird, are these troubles all coming from  HTML emails?
>

I meant Firefox, sorry - I was writing the email in Thunderbird so typed
that in instead. Not quite 40 yet but already losing it!

> Does using AUFS instead of diskd cache types help? there are a lot of
> calls in that trace polling the diskd helpers.

I've not tried it but I'll have a go.

Cheers

Alex

Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Alex Crow
In reply to this post by Amos Jeffries

On 21/05/14 08:30, Amos Jeffries wrote:

> On 21/05/2014 8:11 a.m., Alex Crow wrote:
>> Wrong on my part again.
>>
>> Changing the memory_replacement_policy still got to 100% cpu after
>> Shift-reload in Thunderbird a few times - even disabling cache_mem
>> entirely did not eliminate it. 3.3 never gets about about 67% load no
>> matter how many time the page is reloaded.
> Thunderbird, are these troubles all coming from  HTML emails?
>
> Does using AUFS instead of diskd cache types help? there are a lot of
> calls in that trace polling the diskd helpers.
>
> Amos
>

Hi Amos,

aufs is no better - in fact it seems to build up CPU much faster than
diskd on just a couple of page reloads.

Alex
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Eliezer Croitoru
In reply to this post by Alex Crow
OK and what about the squid.conf?

I have not seen one until now and it seems to me kind of important...

I do know about systems that do not get 100% cpu and it's weird that we
have couple guys having the issue while others do not.

Thanks,
Eliezer
On 05/20/2014 09:54 PM, Alex Crow wrote:

> Hi Amos, all,
>
> I have set up a test box with latest 3.4.5 nightly. I get 95-100% cpu
> even with one client accessing the cache. I've attached a compressed
> strace of the child process in case anything is evident from that.
> Please tell me what else I might need to do to help resolve this issue.
>
> I'm hoping this will help get to the bottom of why a number of people
> are having this issue on 3.4.x.
>
> Any help much appreciated as always.
>
> Alex

Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Alex Crow

On 19/06/14 18:57, Eliezer Croitoru wrote:
> OK and what about the squid.conf?
>
> I have not seen one until now and it seems to me kind of important...
>
> I do know about systems that do not get 100% cpu and it's weird that
> we have couple guys having the issue while others do not.
>
> Thanks,
> Eliezer

FYI, config attached.

The same config works without CPU spikes in 3.3.

Alex



squid.conf (15K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Eliezer Croitoru
On 06/20/2014 03:32 PM, Alex Crow wrote:

>
> On 19/06/14 18:57, Eliezer Croitoru wrote:
>> OK and what about the squid.conf?
>>
>> I have not seen one until now and it seems to me kind of important...
>>
>> I do know about systems that do not get 100% cpu and it's weird that
>> we have couple guys having the issue while others do not.
>>
>> Thanks,
>> Eliezer
>
> FYI, config attached.
>
> The same config works without CPU spikes in 3.3.
>
> Alex
>
>
OK after reading the config file it seems like there are couple things
that we\you should be aware of when looking at the issue:
1. External helpers code was changed from 3.3 to 3.4 (one way)
2. you are using delay_pools.
3. you are using ntlm authentication.

In the past there was suspect which said that the new helpers related
code might cause an issue like that but yet to be verified.
(this needs testing and idea on how to show and proof that this is
either a real suspect or a bogus one)

About ntlm auth.. There is sure some overhead related to using ntlm and
cpu usage due to couple layers one on top of the other and it was
proofed that there is a difference between using ntlm and not using ntlm
at all.
It dosn't proof what in ntlm is causing the issue and I am not sure it
will be fixed due to the basic fact that ntlm maintenance stopped at
200X 3 or 6 and which I am not sure about the accurate date yet.

The only options I see is doing two things:
Remove the ntlm and group external helpers related acls for a testing
period to verify that only when these works\runs the high cpu usage is
there and while the delay_pools are still intact the system runs fine.
This will narrow down the issues from 3 to 2 "ideal" suspects.

There is also another suspect which is over-usage of squid ACLs to block
or allow domains\regex\etc but it can be verified that these are not an
issue by removing the external_acl and ntlm helpers and test how squid
behave.

** Another tiny detail would be: what bandwidth is this server pushing?
How many MBps or Mbps(MBps = mbps/8)?

I know that it can be painful to run these tests but if you have the
option to verify the issue it will narrow the issue down pretty fast.

Also I am almost sure that this thread should be summarized into either
a bug report or first a thread in squid-dev list so you would get better
help and directions from the developers.

Thanks,
Eliezer
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

FredB
In reply to this post by Alex Crow

> FYI, config attached.
>
> The same config works without CPU spikes in 3.3.
>
> Alex
>
>
>


Can you try with delay_access 1 allow xxxx !CONNECT  (for each rule)
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

FredB

>
> Can you try with delay_access 1 allow xxxx !CONNECT  (for each rule)
>

I forgot http://bugs.squid-cache.org/show_bug.cgi?id=2907
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Alex Crow
In reply to this post by Eliezer Croitoru

On 20/06/14 14:28, Eliezer Croitoru wrote:

>
>>
>>
> OK after reading the config file it seems like there are couple things
> that we\you should be aware of when looking at the issue:
> 1. External helpers code was changed from 3.3 to 3.4 (one way)
> 2. you are using delay_pools.
> 3. you are using ntlm authentication.
>
> In the past there was suspect which said that the new helpers related
> code might cause an issue like that but yet to be verified.
> (this needs testing and idea on how to show and proof that this is
> either a real suspect or a bogus one)
>
> About ntlm auth.. There is sure some overhead related to using ntlm
> and cpu usage due to couple layers one on top of the other and it was
> proofed that there is a difference between using ntlm and not using
> ntlm at all.
> It dosn't proof what in ntlm is causing the issue and I am not sure it
> will be fixed due to the basic fact that ntlm maintenance stopped at
> 200X 3 or 6 and which I am not sure about the accurate date yet.
>
> The only options I see is doing two things:
> Remove the ntlm and group external helpers related acls for a testing
> period to verify that only when these works\runs the high cpu usage is
> there and while the delay_pools are still intact the system runs fine.
> This will narrow down the issues from 3 to 2 "ideal" suspects.
>
> There is also another suspect which is over-usage of squid ACLs to
> block or allow domains\regex\etc but it can be verified that these are
> not an issue by removing the external_acl and ntlm helpers and test
> how squid behave.
>
> ** Another tiny detail would be: what bandwidth is this server
> pushing? How many MBps or Mbps(MBps = mbps/8)?
>
> I know that it can be painful to run these tests but if you have the
> option to verify the issue it will narrow the issue down pretty fast.
>
> Also I am almost sure that this thread should be summarized into
> either a bug report or first a thread in squid-dev list so you would
> get better help and directions from the developers.
>
> Thanks,
> Eliezer

Hi,

The first thing I'm going to try is disabling delay pools for CONNECT,
then after that for all requests.

As disabling NTLM will leave us more open than I'd like that would be
the following step.

Cheers

Alex
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Amos Jeffries
Administrator
In reply to this post by Alex Crow
On 21/06/2014 12:32 a.m., Alex Crow wrote:

>
> On 19/06/14 18:57, Eliezer Croitoru wrote:
>> OK and what about the squid.conf?
>>
>> I have not seen one until now and it seems to me kind of important...
>>
>> I do know about systems that do not get 100% cpu and it's weird that
>> we have couple guys having the issue while others do not.
>>
>> Thanks,
>> Eliezer
>
> FYI, config attached.
>
> The same config works without CPU spikes in 3.3.
>

Another experiment is to try purging and rebuilding the ssl_crtd helper
cache.


Your config file has some nits (may not be relevant to the problem though):

 * Try switching the order of "manager localhost" so localhost is tested
first. Manager has become a regex ACL.

 * hierarchy_stoplist can be removed completely. It is serving no
purpose in your config.

Amos

Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 very high cpu - strace.

Alex Crow

> Another experiment is to try purging and rebuilding the ssl_crtd helper
> cache.

Hi Amos,

We do the above on every squid restart anyway (via a wrapper script).



>
> Your config file has some nits (may not be relevant to the problem though):
>
>   * Try switching the order of "manager localhost" so localhost is tested
> first. Manager has become a regex ACL.
>
>   * hierarchy_stoplist can be removed completely. It is serving no
> purpose in your config.

Yeah, I know! This config has pretty much just been tweaked from an
original one that's about 11 years old. I'm still really keen to figure
out why we can't really proceed to 3.4 and then hopefully get it fixed,
Management have already asked to get in a reseller to look at
Bluecoat/Barracuda/Websense etc so I'll try my best to get a good number
of users on each config change I do to diagnose the problem.

Thanks for your time.

Cheers,

Alex