High utilization of CPU squid-3.5.23,squid-3.5.24

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

High utilization of CPU squid-3.5.23,squid-3.5.24

Vitaly Lavrov
Periodically squid begins to linearly increase the use of the CPU.
Sometimes this process reaches 100%. At random moment of time the CPU usage is reduced to 5-15%,
and in the presence of client requests can again start linearly increasing use of CPU.

In the protocols are no error messages.

CPU consumption does not correlate with the number of requests and traffic.

The increase CPU consumption from 0 to 60% occurs in about 4-5 hours, and to 100% for 6-8 hours.

A typical graph of CPU usage can be viewed on http://devel.aanet.ru/tmp/squid-cpu-x.png

With the "perf record -p` pgrep -f squid-1` - sleep 30" I have received the following information:

At 100% CPU load most of the time took 3 calls

  49.15% squid squid [.] MemObject :: dump
  25.11% squid squid [.] Mem_hdr :: freeDataUpto
  20.03% squid squid [.] Mem_hdr :: copy

When loading CPU 30-60% most of the time took 3 calls

  37.26% squid squid [.] Mem_node :: dataRange
  22.61% squid squid [.] Mem_hdr :: NodeCompare
  17.31% squid squid [.] Mem_hdr :: freeDataUpto

What is it ? Is it possible to somehow fix it?

System: slackware64 14.2

sslbump not used. http only.

Part of config:

memory_pools off
memory_pools_limit 512 MB
cache_mem 768 MB
maximum_object_size_in_memory 64 KB
cache_dir ufs           /cache/sq_c1 16312 16 256
cache_dir ufs           /cache/sq_c2 16312 16 256
cache_dir ufs           /cache/sq_c3 16312 16 256
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High utilization of CPU squid-3.5.23, squid-3.5.24

Yuri Voinov
It seems as IO bottleneck at first look.

02.02.2017 2:55, Vitaly Lavrov пишет:

> Periodically squid begins to linearly increase the use of the CPU.
> Sometimes this process reaches 100%. At random moment of time the CPU usage is reduced to 5-15%,
> and in the presence of client requests can again start linearly increasing use of CPU.
>
> In the protocols are no error messages.
>
> CPU consumption does not correlate with the number of requests and traffic.
>
> The increase CPU consumption from 0 to 60% occurs in about 4-5 hours, and to 100% for 6-8 hours.
>
> A typical graph of CPU usage can be viewed on http://devel.aanet.ru/tmp/squid-cpu-x.png
>
> With the "perf record -p` pgrep -f squid-1` - sleep 30" I have received the following information:
>
> At 100% CPU load most of the time took 3 calls
>
>   49.15% squid squid [.] MemObject :: dump
>   25.11% squid squid [.] Mem_hdr :: freeDataUpto
>   20.03% squid squid [.] Mem_hdr :: copy
>
> When loading CPU 30-60% most of the time took 3 calls
>
>   37.26% squid squid [.] Mem_node :: dataRange
>   22.61% squid squid [.] Mem_hdr :: NodeCompare
>   17.31% squid squid [.] Mem_hdr :: freeDataUpto
>
> What is it ? Is it possible to somehow fix it?
>
> System: slackware64 14.2
>
> sslbump not used. http only.
>
> Part of config:
>
> memory_pools off
> memory_pools_limit 512 MB
> cache_mem 768 MB
> maximum_object_size_in_memory 64 KB
> cache_dir ufs           /cache/sq_c1 16312 16 256
> cache_dir ufs           /cache/sq_c2 16312 16 256
> cache_dir ufs           /cache/sq_c3 16312 16 256
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High utilization of CPU squid-3.5.23, squid-3.5.24

Eliezer Croitoru
In reply to this post by Vitaly Lavrov
I believe that the squid manager info page should give some clue about the number of concurrent requests.

If it's above some number(300-400 and above) per second then removing the cache_dir from the server for a windows of a day will answer if it's a DISK IO bottle neck or something else.

All The Bests,
Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Vitaly Lavrov
Sent: Wednesday, February 1, 2017 10:56 PM
To: [hidden email]
Subject: [squid-users] High utilization of CPU squid-3.5.23,squid-3.5.24

Periodically squid begins to linearly increase the use of the CPU.
Sometimes this process reaches 100%. At random moment of time the CPU usage is reduced to 5-15%, and in the presence of client requests can again start linearly increasing use of CPU.

In the protocols are no error messages.

CPU consumption does not correlate with the number of requests and traffic.

The increase CPU consumption from 0 to 60% occurs in about 4-5 hours, and to 100% for 6-8 hours.

A typical graph of CPU usage can be viewed on http://devel.aanet.ru/tmp/squid-cpu-x.png

With the "perf record -p` pgrep -f squid-1` - sleep 30" I have received the following information:

At 100% CPU load most of the time took 3 calls

  49.15% squid squid [.] MemObject :: dump
  25.11% squid squid [.] Mem_hdr :: freeDataUpto
  20.03% squid squid [.] Mem_hdr :: copy

When loading CPU 30-60% most of the time took 3 calls

  37.26% squid squid [.] Mem_node :: dataRange
  22.61% squid squid [.] Mem_hdr :: NodeCompare
  17.31% squid squid [.] Mem_hdr :: freeDataUpto

What is it ? Is it possible to somehow fix it?

System: slackware64 14.2

sslbump not used. http only.

Part of config:

memory_pools off
memory_pools_limit 512 MB
cache_mem 768 MB
maximum_object_size_in_memory 64 KB
cache_dir ufs           /cache/sq_c1 16312 16 256
cache_dir ufs           /cache/sq_c2 16312 16 256
cache_dir ufs           /cache/sq_c3 16312 16 256
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High utilization of CPU squid-3.5.23, squid-3.5.24

Yuri Voinov
Yes, it is require to perform extended diagnostics. Including the system
level.

BTW, it can also network IO. And, it is possible that even a slow DNS.
Have to search.


02.02.2017 3:34, Eliezer Croitoru пишет:

> I believe that the squid manager info page should give some clue about the number of concurrent requests.
>
> If it's above some number(300-400 and above) per second then removing the cache_dir from the server for a windows of a day will answer if it's a DISK IO bottle neck or something else.
>
> All The Bests,
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: [hidden email]
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of Vitaly Lavrov
> Sent: Wednesday, February 1, 2017 10:56 PM
> To: [hidden email]
> Subject: [squid-users] High utilization of CPU squid-3.5.23,squid-3.5.24
>
> Periodically squid begins to linearly increase the use of the CPU.
> Sometimes this process reaches 100%. At random moment of time the CPU usage is reduced to 5-15%, and in the presence of client requests can again start linearly increasing use of CPU.
>
> In the protocols are no error messages.
>
> CPU consumption does not correlate with the number of requests and traffic.
>
> The increase CPU consumption from 0 to 60% occurs in about 4-5 hours, and to 100% for 6-8 hours.
>
> A typical graph of CPU usage can be viewed on http://devel.aanet.ru/tmp/squid-cpu-x.png
>
> With the "perf record -p` pgrep -f squid-1` - sleep 30" I have received the following information:
>
> At 100% CPU load most of the time took 3 calls
>
>   49.15% squid squid [.] MemObject :: dump
>   25.11% squid squid [.] Mem_hdr :: freeDataUpto
>   20.03% squid squid [.] Mem_hdr :: copy
>
> When loading CPU 30-60% most of the time took 3 calls
>
>   37.26% squid squid [.] Mem_node :: dataRange
>   22.61% squid squid [.] Mem_hdr :: NodeCompare
>   17.31% squid squid [.] Mem_hdr :: freeDataUpto
>
> What is it ? Is it possible to somehow fix it?
>
> System: slackware64 14.2
>
> sslbump not used. http only.
>
> Part of config:
>
> memory_pools off
> memory_pools_limit 512 MB
> cache_mem 768 MB
> maximum_object_size_in_memory 64 KB
> cache_dir ufs           /cache/sq_c1 16312 16 256
> cache_dir ufs           /cache/sq_c2 16312 16 256
> cache_dir ufs           /cache/sq_c3 16312 16 256
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High utilization of CPU squid-3.5.23, squid-3.5.24

Garri Djavadyan
In reply to this post by Vitaly Lavrov
On Wed, 2017-02-01 at 23:55 +0300, Vitaly Lavrov wrote:

> Periodically squid begins to linearly increase the use of the CPU.
> Sometimes this process reaches 100%. At random moment of time the CPU
> usage is reduced to 5-15%,
> and in the presence of client requests can again start linearly
> increasing use of CPU.
>
> In the protocols are no error messages.
>
> CPU consumption does not correlate with the number of requests and
> traffic.
>
> The increase CPU consumption from 0 to 60% occurs in about 4-5 hours,
> and to 100% for 6-8 hours.
>
> A typical graph of CPU usage can be viewed on http://devel.aanet.ru/t
> mp/squid-cpu-x.png
>
> With the "perf record -p` pgrep -f squid-1` - sleep 30" I have
> received the following information:
>
> At 100% CPU load most of the time took 3 calls
>
>   49.15% squid squid [.] MemObject :: dump
>   25.11% squid squid [.] Mem_hdr :: freeDataUpto
>   20.03% squid squid [.] Mem_hdr :: copy
>
> When loading CPU 30-60% most of the time took 3 calls
>
>   37.26% squid squid [.] Mem_node :: dataRange
>   22.61% squid squid [.] Mem_hdr :: NodeCompare
>   17.31% squid squid [.] Mem_hdr :: freeDataUpto
>
> What is it ? Is it possible to somehow fix it?
>
> System: slackware64 14.2
>
> sslbump not used. http only.
>
> Part of config:
>
> memory_pools off
> memory_pools_limit 512 MB
> cache_mem 768 MB
> maximum_object_size_in_memory 64 KB
> cache_dir ufs           /cache/sq_c1 16312 16 256
> cache_dir ufs           /cache/sq_c2 16312 16 256
> cache_dir ufs           /cache/sq_c3 16312 16 256


Hi Vitaly,

It seems you faced known issue related to linear search through in-
memory nodes. See bug report 4477 [1].

[1] http://bugs.squid-cache.org/show_bug.cgi?id=4477


Garri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High utilization of CPU squid-3.5.23, squid-3.5.24

Vitaly Lavrov
In reply to this post by Yuri Voinov
On 02.02.2017 00:45, Yuri Voinov wrote:
> Yes, it is require to perform extended diagnostics. Including the system
> level.
>
> BTW, it can also network IO. And, it is possible that even a slow DNS.
> Have to search.

squid-3.5.23 or squid-3.5.24 with bug-4606-v3.patch (http://bugs.squid-cache.org/show_bug.cgi?id=4606)

All cache_dir are located on SSD. Read-write up to 2 mbytes/s. IO wait < 3% - no problem.
Network average 20-40Mbit/s peak 110Mbit/s ( gigabit network ). 60-100 request/s
2 local dns-server.

This is very similar to http://bugs.squid-cache.org/show_bug.cgi?id=4477 (no solution)

>
> 02.02.2017 3:34, Eliezer Croitoru пишет:
>> I believe that the squid manager info page should give some clue about the number of concurrent requests.
>>
>> If it's above some number(300-400 and above) per second then removing the cache_dir from the server for a windows of a day will answer if it's a DISK IO bottle neck or something else.
>>
>> All The Bests,
>> Eliezer
>>
>> ----
>> Eliezer Croitoru
>> Linux System Administrator
>> Mobile: +972-5-28704261
>> Email: [hidden email]
>>
>>
>> -----Original Message-----
>> From: squid-users [mailto:[hidden email]] On Behalf Of Vitaly Lavrov
>> Sent: Wednesday, February 1, 2017 10:56 PM
>> To: [hidden email]
>> Subject: [squid-users] High utilization of CPU squid-3.5.23,squid-3.5.24
>>
>> Periodically squid begins to linearly increase the use of the CPU.
>> Sometimes this process reaches 100%. At random moment of time the CPU usage is reduced to 5-15%, and in the presence of client requests can again start linearly increasing use of CPU.
>>
>> In the protocols are no error messages.
>>
>> CPU consumption does not correlate with the number of requests and traffic.
>>
>> The increase CPU consumption from 0 to 60% occurs in about 4-5 hours, and to 100% for 6-8 hours.
>>
>> A typical graph of CPU usage can be viewed on http://devel.aanet.ru/tmp/squid-cpu-x.png
>>
>> With the "perf record -p` pgrep -f squid-1` - sleep 30" I have received the following information:
>>
>> At 100% CPU load most of the time took 3 calls
>>
>>   49.15% squid squid [.] MemObject :: dump
>>   25.11% squid squid [.] Mem_hdr :: freeDataUpto
>>   20.03% squid squid [.] Mem_hdr :: copy
>>
>> When loading CPU 30-60% most of the time took 3 calls
>>
>>   37.26% squid squid [.] Mem_node :: dataRange
>>   22.61% squid squid [.] Mem_hdr :: NodeCompare
>>   17.31% squid squid [.] Mem_hdr :: freeDataUpto
>>
>> What is it ? Is it possible to somehow fix it?
>>
>> System: slackware64 14.2
>>
>> sslbump not used. http only.
>>
>> Part of config:
>>
>> memory_pools off
>> memory_pools_limit 512 MB
>> cache_mem 768 MB
>> maximum_object_size_in_memory 64 KB
>> cache_dir ufs           /cache/sq_c1 16312 16 256
>> cache_dir ufs           /cache/sq_c2 16312 16 256
>> cache_dir ufs           /cache/sq_c3 16312 16 256
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...