Squid Web Proxy Cache › Squid - Users

squid cpu usage

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

11 messages Options

Vieri

squid cpu usage

Hi,

During the last 3 days my Squid process has gone wild with 100% CPU usage once a day. Since this is a production system (never had this issue before), I "quickly restart" the Squid daemon, and that solves the issue until the next day.
Restarting takes quite a while...

Next time I get a 100% squid process that brings my proxy to a crawl HTTP-wise, what can I try in order to get more info, and possibly see the cause of this?
Would squidclient mgr:info shed some light on this?

Squid Object Cache: Version 3.5.27-20170916-re69e56c

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Amos Jeffries

Re: squid cpu usage

Administrator

On 25/10/17 01:34, Vieri wrote:
> Hi,
>
> During the last 3 days my Squid process has gone wild with 100% CPU usage once a day. Since this is a production system (never had this issue before), I "quickly restart" the Squid daemon, and that solves the issue until the next day.
> Restarting takes quite a while...
>
> Next time I get a 100% squid process that brings my proxy to a crawl HTTP-wise, what can I try in order to get more info, and possibly see the cause of this?
> Would squidclient mgr:info shed some light on this?

That one is just the overview, so not likely to show more than that
things are going slowly.

Usually the 100% CPU is an indicator that the machine is swapping. So
check the memory usage when it is happening.

Next is probably the cache garbage collection. mgr:storedir report might
show that. Or an strace showing a lot of cycles in the cache replacement
calls.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Alex Rousskov

Re: squid cpu usage

In reply to this post by Vieri

On 10/24/2017 06:34 AM, Vieri wrote:

> Next time I get a 100% squid process that brings my proxy to a crawl
> HTTP-wise, what can I try in order to get more info, and possibly see
> the cause of this?

Here are two additional tricks to consider:

* If 100% CPU usage lasts for more than a few seconds, then attaching
gdb to the running Squid worker and generating a few backtraces (a few
seconds apart) may identify the code that got stuck. Make sure you give
gdb a non-stripped Squid binary when doing that. Squid wiki should have
more instructions about the whole process.

Please note that the above is less likely to help if your Squid
continues to respond to new requests while you see ~100% CPU usage. You
may increase your chances by blocking new traffic from reaching the
problematic Squid (e.g., by redirecting all traffic to other Squid(s)
that you have in load balancing or hot standby mode OR allowing all
traffic to bypass Squid). If you cannot easily remove a problematic
Squid from user traffic path, then use this problem as an excuse to fix
your deployment architecture so that you can :-).

* Another potentially useful trick is to log mgr:events every minute or
so. Perhaps 100% CPU usage is associated with a particular regularly
scheduled event?

Please note that some modern Squids abuse the events API for frequently
occurring notifications. If you see hundreds or more events in your log,
this trick may be too expensive/intrusive for your environment.

HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Vieri

Re: squid cpu usage

Well, I've just had another issue with my CPU usage gone bonkers.

Here's the strace (compressed with gz):

https://drive.google.com/file/d/0B-tpkY1LkI67WFBaNnVpR2lteEE/view?usp=sharing

There are a lot of "Resource temporarily unavailable" messages.
However, I'm seeing these messages even when my Squid process has been restarted, and everything seems to be OK from a "user experience" point of view.
Do you see anything useful in the trace?
If not, how can I improve the trace next time I have this issue?

Here's what squidclient has to say during the "storm":

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:34:43 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:34:43 GMT
Last-Modified: Thu, 26 Oct 2017 08:34:43 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20170916-re69e56c
Build Info:
Service Name: squid
Start Time: Thu, 26 Oct 2017 02:09:30 GMT
Current Time: Thu, 26 Oct 2017 08:34:43 GMT
Connection information for squid:
Number of clients accessing cache: 471
Number of HTTP requests received: 267968
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 695.6
Average ICP messages per minute since start: 0.0
Select loop called: 10630162 times, 2.174 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 2.3%, 60min: 2.5%
Hits as % of bytes sent: 5min: 10.6%, 60min: -4.4%
Memory hits as % of hit requests: 5min: 22.1%, 60min: 36.5%
Disk hits as % of hit requests: 5min: 4.3%, 60min: 0.3%
Storage Swap size: 29484 KB
Storage Swap capacity: 90.0% used, 10.0% free
Storage Mem size: 28492 KB
Storage Mem capacity: 87.0% used, 13.0% free
Mean Object Size: 22.29 KB
Requests given to unlinkd: 17424
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 2.64709 0.27332
Cache Misses: 1.54242 0.27332
Cache Hits: 0.64968 0.00000
Near Hits: 1.31166 0.22004
Not-Modified Replies: 0.44492 0.00000
DNS Lookups: 0.12472 0.07968
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 23113.055 seconds
CPU Time: 1599.750 seconds
CPU Usage: 6.92%
CPU Usage, 5 minute avg: 99.80%
CPU Usage, 60 minute avg: 32.48%
Maximum Resident Size: 987392 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted: 148456 KB
memPoolAlloc calls: 87698086
memPoolFree calls: 89179152
File descriptor usage for squid:
Maximum number of file descriptors: 1024
Largest file desc currently in use: 913
Number of file desc currently in use: 791
Files queued for open: 0
Available number of file descriptors: 233
Reserved number of file descriptors: 100
Store Disk files open: 0
Internal Data Structures:
1642 StoreEntries
1640 StoreEntries with MemObjects
1375 Hot Object Cache Items
1323 on-disk objects

A little later (still during the "storm"):

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:40:13 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:40:13 GMT
Last-Modified: Thu, 26 Oct 2017 08:40:13 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20170916-re69e56c
Build Info:
Service Name: squid
Start Time: Thu, 26 Oct 2017 02:09:30 GMT
Current Time: Thu, 26 Oct 2017 08:40:13 GMT
Connection information for squid:
Number of clients accessing cache: 472
Number of HTTP requests received: 274183
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 701.8
Average ICP messages per minute since start: 0.0
Select loop called: 10679512 times, 2.195 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 0.2%, 60min: 2.3%
Hits as % of bytes sent: 5min: -3.2%, 60min: -3.6%
Memory hits as % of hit requests: 5min: 9.1%, 60min: 35.2%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.4%
Storage Swap size: 29484 KB
Storage Swap capacity: 90.0% used, 10.0% free
Storage Mem size: 27304 KB
Storage Mem capacity: 83.3% used, 16.7% free
Mean Object Size: 16.25 KB
Requests given to unlinkd: 17424
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 1.05672 0.30459
Cache Misses: 0.94847 0.30459
Cache Hits: 0.00000 0.00000
Near Hits: 0.58309 0.23230
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.11405 0.08334
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 23442.516 seconds
CPU Time: 1888.520 seconds
CPU Usage: 8.06%
CPU Usage, 5 minute avg: 86.51%
CPU Usage, 60 minute avg: 39.31%
Maximum Resident Size: 988768 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted: 151913 KB
memPoolAlloc calls: 89548742
memPoolFree calls: 91062557
File descriptor usage for squid:
Maximum number of file descriptors: 1024
Largest file desc currently in use: 913
Number of file desc currently in use: 814
Files queued for open: 0
Available number of file descriptors: 210
Reserved number of file descriptors: 100
Store Disk files open: 0
Internal Data Structures:
2139 StoreEntries
2135 StoreEntries with MemObjects
1863 Hot Object Cache Items
1814 on-disk objects

Memory info:

TTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:34:51 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:34:51 GMT
Last-Modified: Thu, 26 Oct 2017 08:34:51 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Current memory usage:
Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate
(bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec
64K Buffer 65536 1366 87424 88064 0.03 58.871 1340 85760 88064 0.03 98.097 26 1664 5248 283321 0.317 35.761 21.750
mem_node 4136 7326 29591 30843 1.38 19.926 7096 28662 30843 1.38 96.860 230 929 6802 558119 0.624 4.446 48.467
32K Buffer 32768 129 4128 5984 0.77 2.780 123 3936 5984 0.77 95.349 6 192 2272 49708 0.056 3.137 4.285
ExternalACLEntry 256 16447 4112 4953 4.32 2.769 16447 4112 4953 4.32 100.000 0 0 4953 591 0.001 0.000 5.834
16K Buffer 16384 189 3024 6992 4.35 2.036 167 2672 6992 4.35 88.360 22 352 3152 223171 0.250 7.042 16.461
8K Buffer 8192 368 2944 3464 0.17 1.982 347 2776 3464 0.17 94.293 21 168 1112 288150 0.322 4.546 27.526
Short Strings 40 73543 2873 3066 0.57 1.935 73511 2872 3066 0.57 99.956 32 2 867 50208368 56.164 3.868 3944.297
4K Buffer 4096 497 1988 2696 1.51 1.339 466 1864 2696 1.51 93.763 31 124 1012 397819 0.445 3.138 37.339
HttpHeaderEntry 56 30961 1694 1862 0.57 1.140 30959 1694 1862 0.57 99.994 2 1 590 12908222 14.439 1.392 1001.067
HttpRequest 1784 891 1553 1631 0.02 1.045 885 1542 1631 0.02 99.327 6 11 295 613472 0.686 2.108 52.175
Medium Strings 128 6801 851 954 4.33 0.572 6644 831 954 4.33 97.692 157 20 360 2746616 3.072 0.677 219.037
4KB Strings 4096 208 832 1168 1.42 0.560 143 572 1168 1.42 68.750 65 260 444 262956 0.294 2.074 23.268
HttpReply 280 2884 789 861 0.57 0.531 2865 784 861 0.57 99.341 19 6 291 1030249 1.152 0.556 82.441
2K Buffer 2048 374 748 1416 4.32 0.504 346 692 1416 4.32 92.513 28 56 768 4412257 4.936 17.404 317.415
16KB Strings 16384 44 704 4400 1.71 0.474 40 640 4400 1.71 90.909 4 64 2368 76985 0.086 2.429 7.763
cbdata clientReplyContext (25) 4352 140 595 706 0.08 0.401 122 519 706 0.08 87.143 18 77 489 264790 0.296 2.219 23.215
cbdata ClientSocketContext (24) 4256 140 582 690 0.08 0.392 122 508 690 0.08 87.143 18 75 478 264789 0.296 2.170 23.215
MemObject 328 1682 539 794 4.33 0.363 1663 533 794 4.33 98.870 19 7 339 255955 0.286 0.162 23.520
Long Strings 512 958 479 811 4.35 0.323 950 475 811 4.35 99.165 8 4 369 677454 0.758 0.668 56.843
Comm::Connection 200 1533 300 307 0.28 0.202 1495 292 307 0.28 97.521 38 8 70 606480 0.678 0.234 65.623
cbdata Server (20) 520 549 279 304 0.44 0.188 528 269 304 0.44 96.175 21 11 114 108934 0.122 0.109 12.423
cbdata BodyPipe (38) 152 1659 247 293 0.56 0.166 1640 244 293 0.56 98.855 19 3 109 215397 0.241 0.063 15.348
ClientInfo 448 471 207 207 0.09 0.139 471 207 207 0.09 100.000 0 0 0 0 0.000 0.000 0.007
1KB Strings 1024 189 189 208 0.29 0.127 156 156 208 0.29 82.540 33 33 107 134127 0.150 0.265 13.835
cbdata MemBuf (12) 64 2999 188 201 0.57 0.126 2963 186 201 0.57 98.800 36 3 67 1535006 1.717 0.189 123.213
StoreEntry 104 1683 171 1670 6.42 0.115 1665 170 1670 6.42 98.930 18 2 1478 256816 0.287 0.051 23.520
cbdata ModXact (41) 512 319 160 162 0.12 0.107 308 154 162 0.12 96.552 11 6 16 131784 0.147 0.130 10.001
netdbEntry 168 970 160 165 1.91 0.107 970 160 165 1.91 100.000 0 0 17 261 0.000 0.000 0.180
fqdncache_entry 160 934 146 152 2.16 0.098 926 145 152 2.16 99.143 8 2 8 30025 0.034 0.009 2.861
ipcache_entry 128 927 116 119 2.19 0.078 921 116 119 2.19 99.353 6 1 5 16480 0.018 0.004 1.682
MemBlob 48 2355 111 145 1.30 0.074 2226 105 145 1.30 94.522 129 7 33 3054760 3.417 0.282 267.710
HttpHdrCc 96 1112 105 159 4.35 0.070 1067 101 159 4.35 95.953 45 5 70 485419 0.543 0.090 37.649
cbdata idns_query (9) 8696 12 102 391 2.28 0.069 2 17 391 2.28 16.667 10 85 391 62918 0.070 1.054 5.848
net_db_name 32 2228 70 72 0.16 0.047 2228 70 72 0.16 100.000 0 0 6 493 0.001 0.000 0.275
cbdata ClientHttpRequest (23) 392 174 67 77 0.01 0.045 152 59 77 0.01 87.356 22 9 46 264283 0.296 0.200 23.215
LRU policy node 24 2692 64 384 6.42 0.042 2666 63 384 6.42 99.034 26 1 341 32395 0.036 0.001 4.782
cbdata IdleConnList (18) 4160 15 61 252 1.97 0.041 14 57 252 1.97 93.333 1 5 200 13686 0.015 0.110 0.972
cbdata store_client (31) 160 288 45 56 1.52 0.030 278 44 56 1.52 96.528 10 2 27 275536 0.308 0.085 24.485
NotePairs::Entry 48 927 44 46 2.92 0.029 924 44 46 2.92 99.676 3 1 46 18571 0.021 0.002 1.796
cbdata clientStreamNode (26) 128 280 35 42 0.08 0.024 244 31 42 0.08 87.143 36 5 29 529580 0.592 0.131 46.431
cbdata ServerBump (30) 104 259 27 34 1.52 0.018 240 25 34 1.52 92.664 19 2 17 81041 0.091 0.016 9.000
MD5 digest 16 1683 27 257 6.42 0.018 1665 27 257 6.42 98.930 18 1 228 287894 0.322 0.009 26.086
MimeEntry 128 177 23 23 6.42 0.015 177 23 23 6.42 100.000 0 0 23 1062 0.001 0.000 0.063
cbdata HttpStateData (36) 320 58 19 19 0.07 0.012 37 12 19 0.07 63.793 21 7 15 133511 0.149 0.082 10.258
cbdata FwdState (32) 176 101 18 18 0.28 0.012 59 11 18 0.28 58.416 42 8 13 211946 0.237 0.072 18.843
cbdata helper_server (11) 240 55 13 13 0.09 0.009 55 13 13 0.09 100.000 0 0 9 209 0.000 0.000 0.019
cbdata ConnOpener (34) 136 64 9 9 0.01 0.006 4 1 9 0.01 6.250 60 8 9 98344 0.110 0.026 10.905
cbdata PeerConnector (35) 136 61 9 9 0.01 0.005 18 3 9 0.01 29.508 43 6 9 77220 0.086 0.020 8.531
Acl::AndNode 160 44 7 7 6.42 0.005 44 7 7 6.42 100.000 0 0 7 264 0.000 0.000 0.016
cbdata Iterator (39) 168 31 6 8 0.77 0.003 23 4 8 0.77 74.194 8 2 6 131859 0.148 0.043 9.978
cbdata ACLFilledChecklist (22) 464 11 5 11 2.28 0.003 10 5 11 2.28 90.909 1 1 11 756320 0.846 0.676 66.886
cbdata ModXactLauncher (40) 160 30 5 8 0.77 0.003 20 4 8 0.77 66.667 10 2 5 131871 0.148 0.041 9.978
ACLStrategised 160 19 3 3 6.42 0.002 19 3 3 6.42 100.000 0 0 3 114 0.000 0.000 0.007
cbdata ServiceRep (17) 344 7 3 3 0.78 0.002 7 3 3 0.78 100.000 0 0 0 0 0.000 0.000 0.000
cbdata Tree (4) 216 11 3 3 6.42 0.002 11 3 3 6.42 100.000 0 0 3 66 0.000 0.000 0.004
cbdata AccessCheck (29) 152 15 3 4 0.57 0.001 0 0 4 0.57 0.000 15 3 4 352429 0.394 0.103 29.244
Acl::NotNode 160 14 3 3 6.42 0.001 14 3 3 6.42 100.000 0 0 3 84 0.000 0.000 0.005
acl_ip_data 96 19 2 2 6.42 0.001 19 2 2 6.42 100.000 0 0 2 289 0.000 0.000 0.016
cbdata ClientRequestContext (27) 104 17 2 4 0.08 0.001 11 2 4 0.08 64.706 6 1 4 262994 0.294 0.053 22.940
cbdata WriteRequest (47) 80 20 2 6 4.28 0.001 0 0 6 4.28 0.000 20 2 6 190311 0.213 0.029 15.692
cbdata UFSStoreState (45) 176 8 2 3 0.26 0.001 0 0 3 0.26 0.000 8 2 3 28910 0.032 0.010 2.397
Helper::Request 48 28 2 2 0.08 0.001 16 1 2 0.08 57.143 12 1 2 158681 0.178 0.015 14.377
HttpHdrContRange 24 48 2 2 0.18 0.001 48 2 2 0.18 100.000 0 0 1 410 0.000 0.000 0.036
RegexList 88 10 1 1 6.42 0.001 10 1 1 6.42 100.000 0 0 1 60 0.000 0.000 0.004
cbdata externalAclState (14) 88 10 1 2 0.28 0.001 10 1 2 0.28 100.000 0 0 2 81199 0.091 0.014 6.014
ACLSourceIP 144 6 1 1 6.42 0.001 6 1 1 6.42 100.000 0 0 1 36 0.000 0.000 0.002
cbdata DiskdFile (46) 104 8 1 2 0.26 0.001 0 0 2 0.26 0.000 8 1 2 28910 0.032 0.006 2.397
ev_entry 48 17 1 2 0.24 0.001 14 1 2 0.24 82.353 3 1 1 63038 0.071 0.006 3.019
ACLStrategised 160 5 1 1 6.42 0.001 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
cbdata helper (10) 168 3 1 1 0.78 0.000 3 1 1 0.78 100.000 0 0 1 17 0.000 0.000 0.001
wordlist 16 31 1 2 0.78 0.000 31 1 2 0.78 100.000 0 0 1 837 0.001 0.000 0.064
cbdata CbDataList (2) 40 12 1 1 6.42 0.000 12 1 1 6.42 100.000 0 0 1 72 0.000 0.000 0.004
cbdata ErrorState (37) 240 2 1 3 0.05 0.000 0 0 3 0.05 0.000 2 1 3 53117 0.059 0.025 4.356
ACLExternal 152 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
ACLStringData 56 8 1 1 6.42 0.000 8 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
ACLRegexData 16 25 1 1 6.42 0.000 25 1 1 6.42 100.000 0 0 1 54 0.000 0.000 0.003
cbdata CbDataList (1) 40 10 1 1 6.42 0.000 10 1 1 6.42 100.000 0 0 1 60 0.000 0.000 0.004
cbdata external_acl (5) 192 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
cbdata generic_cbdata (21) 32 12 1 2 2.28 0.000 1 1 2 2.28 8.333 11 1 2 86571 0.097 0.005 7.694
ACLCertificateData 120 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
acl_name_list 72 5 1 1 6.42 0.000 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
cbdata Logfile (15) 352 1 1 1 5.76 0.000 1 1 1 5.76 100.000 0 0 1 6 0.000 0.000 0.001
ACLStrategised 160 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
cbdata TcpAcceptor (19) 104 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
ACLDestinationIP 144 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
ACLUserData 64 4 1 1 6.42 0.000 4 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
cbdata ps_state (33) 256 1 1 1 0.05 0.000 0 0 1 0.05 0.000 1 1 1 215092 0.241 0.106 18.843
cbdata RemovalPolicy (7) 104 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
cbdata statefulhelper (13) 192 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
AclDenyInfoList 32 5 1 1 6.42 0.000 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
ACLStrategised 160 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
ACLProxyAuth 152 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
ACLDomainData 16 9 1 1 6.42 0.000 9 1 1 6.42 100.000 0 0 1 42 0.000 0.000 0.002
cbdata external_acl_data (6) 48 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
external_acl_format 64 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
cbdata StoreSearchHashIndex (28) 104 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 0 0.000 0.000 0.000
ACLHTTPHeaderData 48 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLSslErrorData 16 6 1 1 6.42 0.000 6 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
cbdata nsvc (8) 72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 75 0.000 0.000 0.009
StoreSwapLogData 72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 89577 0.100 0.012 4.858
cbdata RemovalPurgeWalker (44) 72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 1961 0.002 0.000 0.211
cbdata ReadRequest (48) 72 1 1 1 6.30 0.000 0 0 1 6.30 0.000 1 1 1 111 0.000 0.000 0.006
cbdata CbDataList (3) 64 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
dwrite_q 48 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 89578 0.100 0.008 4.858
FwdServer 24 2 1 1 0.05 0.000 0 0 1 0.05 0.000 2 1 1 318128 0.356 0.015 25.710
ACLNoteData 40 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
CacheDigest 40 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLHierCodeData 32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLMethodData 16 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
ACLTimeData 32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLAtStepData 32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLASN 16 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
StoreMetaMD5 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
StoreMetaSTDLFS 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
StoreMetaURL 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
StoreMetaObjSize 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
UFSStoreState::_queued_write 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 193581 0.217 0.012 15.692
StoreMetaVary 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 7514 0.008 0.000 0.565
ACLProtocolData 16 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLServerNameData 16 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
link_list 16 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 193581 0.217 0.006 15.692
dlink_node 24 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.001
cbdata RebuildState (16) 944 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata OptXactLauncher (42) 128 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 1 0.000 0.000 0.000
cbdata OptXact (43) 320 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 1 0.000 0.000 0.000
HttpHdrRange 32 0 0 1 1.82 0.000 0 0 1 1.82 -1.000 0 0 1 314 0.000 0.000 0.020
HttpHdrRangeSpec 16 0 0 1 1.82 0.000 0 0 1 1.82 -1.000 0 0 1 314 0.000 0.000 0.020
cbdata CbDataList (49) 96 0 0 1 6.21 0.000 0 0 1 6.21 -1.000 0 0 1 4662 0.005 0.001 1.421
HttpHdrSc 16 0 0 1 2.84 0.000 0 0 1 2.84 -1.000 0 0 1 12 0.000 0.000 0.005
HttpHdrScTarget 88 0 0 1 2.84 0.000 0 0 1 2.84 -1.000 0 0 1 12 0.000 0.000 0.005
cbdata BodySink (50) 80 0 0 1 2.34 0.000 0 0 1 2.34 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::User 184 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::UserRequest 88 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata StateData (51) 48 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata helper_stateful_server (52) 232 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
Total 1 170288 148501 150273 0.03 100.000 168782 144272 146333 0.03 97.152 1506 4229 18484 87746747 98.156 98.226 31638.278
Cumulative allocated volume: 51.921 GB
Current overhead: 43294 bytes (0.029%)
Idle pool limit: 5.00 MB
Total Pools created: 138
Pools ever used: 135 (shown above)
Currently in use: 102
String Pool Impact
(%strings) (%volume)
Short Strings 90 47
Medium Strings 8 14
Long Strings 1 8
1KB Strings 0 3
4KB Strings 0 9
16KB Strings 0 11
Other Strings 0 9

Large buffers: 0 (0 KB)

Again, a little later:

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:39:42 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:39:42 GMT
Last-Modified: Thu, 26 Oct 2017 08:39:42 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Current memory usage:
Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate
(bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec
64K Buffer 65536 1366 87424 89280 0.01 59.621 1364 87296 89280 0.01 99.854 2 128 5248 288796 0.317 35.771 19.344
mem_node 4136 6798 27458 30843 1.46 18.725 6783 27397 30843 1.46 99.779 15 61 6802 568521 0.624 4.444 38.660
ExternalACLEntry 256 17998 4500 4953 4.40 3.069 17998 4500 4953 4.40 100.000 0 0 4953 591 0.001 0.000 5.325
32K Buffer 32768 136 4352 5984 0.85 2.968 134 4288 5984 0.85 98.529 2 64 2272 51135 0.056 3.167 5.133
Short Strings 40 80835 3158 3208 0.00 2.153 80636 3150 3208 0.00 99.754 199 8 867 51148695 56.142 3.867 3308.818
16K Buffer 16384 169 2704 6992 4.43 1.844 169 2704 6992 4.43 100.000 0 0 3152 227118 0.249 7.033 13.881
8K Buffer 8192 290 2320 3464 0.25 1.582 282 2256 3464 0.25 97.241 8 64 1112 294270 0.323 4.556 21.610
HttpHeaderEntry 56 34149 1868 1899 0.00 1.274 34114 1866 1899 0.00 99.898 35 2 590 13142150 14.425 1.391 835.378
4K Buffer 4096 439 1756 2696 1.59 1.198 409 1636 2696 1.59 93.166 30 120 1012 406215 0.446 3.145 29.469
HttpRequest 1784 873 1521 1631 0.10 1.037 859 1497 1631 0.10 98.396 14 25 295 625953 0.687 2.111 43.460
2K Buffer 2048 539 1078 1416 4.40 0.735 535 1070 1416 4.40 99.258 4 8 768 4472923 4.910 17.313 209.220
Medium Strings 128 7899 988 1002 0.00 0.673 7893 987 1002 0.00 99.924 6 1 360 2798935 3.072 0.677 186.051
16KB Strings 16384 59 944 4400 1.79 0.644 54 864 4400 1.79 91.525 5 80 2368 78587 0.086 2.434 5.827
HttpReply 280 3294 901 911 0.00 0.614 3288 900 911 0.00 99.818 6 2 291 1049980 1.152 0.556 70.385
MemObject 328 2077 666 794 4.41 0.454 2071 664 794 4.41 99.711 6 2 339 261223 0.287 0.162 20.347
4KB Strings 4096 149 596 1168 1.50 0.406 142 568 1168 1.50 95.302 7 28 444 267713 0.294 2.072 16.635
cbdata clientReplyContext (25) 4352 118 502 706 0.16 0.342 109 464 706 0.16 92.373 9 39 489 270088 0.296 2.222 18.736
cbdata ClientSocketContext (24) 4256 118 491 690 0.16 0.334 109 454 690 0.16 92.373 9 38 478 270087 0.296 2.173 18.736
Long Strings 512 751 376 811 4.43 0.256 718 359 811 4.43 95.606 33 17 369 692635 0.760 0.670 52.968
Comm::Connection 200 1499 293 307 0.36 0.200 1464 286 307 0.36 97.665 35 7 70 623501 0.684 0.236 59.278
cbdata Server (20) 520 563 286 304 0.52 0.195 542 276 304 0.52 96.270 21 11 114 111986 0.123 0.110 10.935
cbdata BodyPipe (38) 152 1798 267 293 0.64 0.182 1796 267 293 0.64 99.889 2 1 109 219412 0.241 0.063 15.090
cbdata MemBuf (12) 64 3382 212 215 0.00 0.144 3377 212 215 0.00 99.852 5 1 67 1564605 1.717 0.189 104.318
StoreEntry 104 2081 212 1670 6.50 0.144 2075 211 1670 6.50 99.712 6 1 1478 262085 0.288 0.052 20.347
ClientInfo 448 472 207 207 0.02 0.141 472 207 207 0.02 100.000 0 0 0 0 0.000 0.000 0.003
cbdata ModXact (41) 512 309 155 164 0.07 0.105 308 154 164 0.07 99.676 1 1 16 134215 0.147 0.130 8.549
netdbEntry 168 925 152 165 1.99 0.103 925 152 165 1.99 100.000 0 0 17 286 0.000 0.000 0.196
fqdncache_entry 160 931 146 152 2.24 0.099 920 144 152 2.24 98.818 11 2 8 30868 0.034 0.009 3.083
cbdata idns_query (9) 8696 16 136 391 2.36 0.093 0 0 391 2.36 0.000 16 136 391 64579 0.071 1.061 5.881
ipcache_entry 128 923 116 119 2.28 0.079 923 116 119 2.28 100.000 0 0 5 16903 0.019 0.004 1.528
MemBlob 48 2438 115 145 1.38 0.078 2304 108 145 1.38 94.504 134 7 33 3118212 3.423 0.283 223.259
HttpHdrCc 96 1159 109 159 4.43 0.074 1158 109 159 4.43 99.914 1 1 70 493512 0.542 0.090 28.676
1KB Strings 1024 101 101 208 0.37 0.069 97 97 208 0.37 96.040 4 4 107 137459 0.151 0.266 11.691
LRU policy node 24 3530 83 384 6.50 0.056 3530 83 384 6.50 100.000 0 0 341 33829 0.037 0.002 9.150
net_db_name 32 2187 69 72 0.24 0.047 2187 69 72 0.24 100.000 0 0 6 540 0.001 0.000 0.336
cbdata ClientHttpRequest (23) 392 144 56 77 0.10 0.038 131 51 77 0.10 90.972 13 5 46 269589 0.296 0.200 18.736
cbdata IdleConnList (18) 4160 12 49 252 2.05 0.033 11 45 252 2.05 91.667 1 5 200 13861 0.015 0.109 0.663
NotePairs::Entry 48 995 47 47 0.00 0.032 992 47 47 0.00 99.698 3 1 46 18882 0.021 0.002 1.346
cbdata store_client (31) 160 263 42 56 1.60 0.028 256 40 56 1.60 97.338 7 2 27 281020 0.308 0.085 19.251
MD5 digest 16 2081 33 257 6.50 0.022 2075 33 257 6.50 99.712 6 1 228 294523 0.323 0.009 25.016
cbdata clientStreamNode (26) 128 236 30 42 0.16 0.020 218 28 42 0.16 92.373 18 3 29 540176 0.593 0.131 37.472
cbdata ServerBump (30) 104 234 24 34 1.60 0.016 219 23 34 1.60 93.590 15 2 17 83160 0.091 0.016 7.626
MimeEntry 128 177 23 23 6.50 0.015 177 23 23 6.50 100.000 0 0 23 1062 0.001 0.000 0.000
cbdata helper_server (11) 240 55 13 13 0.17 0.009 55 13 13 0.17 100.000 0 0 9 209 0.000 0.000 0.000
cbdata HttpStateData (36) 320 32 10 19 0.15 0.007 32 10 19 0.15 100.000 0 0 15 135965 0.149 0.082 8.642
cbdata FwdState (32) 176 56 10 19 0.08 0.007 43 8 19 0.08 76.786 13 3 14 216432 0.238 0.072 15.828
Acl::AndNode 160 44 7 7 6.50 0.005 44 7 7 6.50 100.000 0 0 7 264 0.000 0.000 0.000
cbdata ACLFilledChecklist (22) 464 10 5 11 2.36 0.003 1 1 11 2.36 10.000 9 5 11 772759 0.848 0.678 56.648
ACLStrategised 160 19 3 3 6.50 0.002 19 3 3 6.50 100.000 0 0 3 114 0.000 0.000 0.000
cbdata PeerConnector (35) 136 22 3 9 0.08 0.002 10 2 9 0.08 45.455 12 2 9 79214 0.087 0.020 7.172
cbdata Iterator (39) 168 15 3 8 0.85 0.002 15 3 8 0.85 100.000 0 0 6 134291 0.147 0.043 8.549
cbdata ServiceRep (17) 344 7 3 3 0.86 0.002 7 3 3 0.86 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ModXactLauncher (40) 160 15 3 8 0.85 0.002 14 3 8 0.85 93.333 1 1 5 134308 0.147 0.041 8.549
cbdata Tree (4) 216 11 3 3 6.50 0.002 11 3 3 6.50 100.000 0 0 3 66 0.000 0.000 0.000
Acl::NotNode 160 14 3 3 6.50 0.001 14 3 3 6.50 100.000 0 0 3 84 0.000 0.000 0.000
acl_ip_data 96 19 2 2 6.50 0.001 19 2 2 6.50 100.000 0 0 2 289 0.000 0.000 0.000
cbdata ConnOpener (34) 136 11 2 10 0.08 0.001 1 1 10 0.08 9.091 10 2 10 101038 0.111 0.026 9.638
cbdata ClientRequestContext (27) 104 12 2 4 0.16 0.001 1 1 4 0.16 8.333 11 2 4 268308 0.294 0.053 18.499
HttpHdrContRange 24 48 2 2 0.27 0.001 48 2 2 0.27 100.000 0 0 1 410 0.000 0.000 0.000
cbdata AccessCheck (29) 152 7 2 4 0.65 0.001 0 0 4 0.65 0.000 7 2 4 359554 0.395 0.103 24.631
RegexList 88 10 1 1 6.50 0.001 10 1 1 6.50 100.000 0 0 1 60 0.000 0.000 0.000
ACLSourceIP 144 6 1 1 6.50 0.001 6 1 1 6.50 100.000 0 0 1 36 0.000 0.000 0.000
Helper::Request 48 17 1 2 0.17 0.001 4 1 2 0.17 23.529 13 1 2 162246 0.178 0.015 12.498
ACLStrategised 160 5 1 1 6.50 0.001 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
cbdata WriteRequest (47) 80 10 1 6 4.36 0.001 0 0 6 4.36 0.000 10 1 6 196846 0.216 0.030 23.134
cbdata UFSStoreState (45) 176 4 1 3 0.34 0.000 4 1 3 0.34 100.000 0 0 3 30203 0.033 0.010 4.587
ev_entry 48 14 1 2 0.32 0.000 14 1 2 0.32 100.000 0 0 1 63810 0.070 0.006 2.712
cbdata externalAclState (14) 88 7 1 2 0.08 0.000 1 1 2 0.08 14.286 6 1 2 82716 0.091 0.014 5.373
cbdata generic_cbdata (21) 32 16 1 2 2.36 0.000 4 1 2 2.36 25.000 12 1 2 89441 0.098 0.005 10.084
cbdata helper (10) 168 3 1 1 0.86 0.000 3 1 1 0.86 100.000 0 0 1 17 0.000 0.000 0.000
wordlist 16 31 1 2 0.86 0.000 31 1 2 0.86 100.000 0 0 1 837 0.001 0.000 0.000
cbdata CbDataList (2) 40 12 1 1 6.50 0.000 12 1 1 6.50 100.000 0 0 1 72 0.000 0.000 0.000
ACLExternal 152 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
ACLStringData 56 8 1 1 6.50 0.000 8 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
cbdata DiskdFile (46) 104 4 1 2 0.34 0.000 4 1 2 0.34 100.000 0 0 2 30203 0.033 0.006 4.587
ACLRegexData 16 25 1 1 6.50 0.000 25 1 1 6.50 100.000 0 0 1 54 0.000 0.000 0.000
cbdata CbDataList (1) 40 10 1 1 6.50 0.000 10 1 1 6.50 100.000 0 0 1 60 0.000 0.000 0.000
cbdata external_acl (5) 192 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
ACLCertificateData 120 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
acl_name_list 72 5 1 1 6.50 0.000 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
cbdata Logfile (15) 352 1 1 1 5.84 0.000 1 1 1 5.84 100.000 0 0 1 6 0.000 0.000 0.003
ACLStrategised 160 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
cbdata TcpAcceptor (19) 104 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
ACLDestinationIP 144 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
ACLUserData 64 4 1 1 6.50 0.000 4 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
cbdata ps_state (33) 256 1 1 1 0.13 0.000 0 0 1 0.13 0.000 1 1 1 219697 0.241 0.106 15.828
cbdata ErrorState (37) 240 1 1 3 0.13 0.000 0 0 3 0.13 0.000 1 1 3 53986 0.059 0.024 3.008
cbdata RemovalPolicy (7) 104 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
cbdata statefulhelper (13) 192 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
AclDenyInfoList 32 5 1 1 6.50 0.000 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
ACLStrategised 160 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
ACLProxyAuth 152 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
ACLDomainData 16 9 1 1 6.50 0.000 9 1 1 6.50 100.000 0 0 1 42 0.000 0.000 0.000
cbdata external_acl_data (6) 48 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
external_acl_format 64 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
cbdata StoreSearchHashIndex (28) 104 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 0 0.000 0.000 0.000
ACLHTTPHeaderData 48 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLSslErrorData 16 6 1 1 6.50 0.000 6 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
cbdata CbDataList (3) 64 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
FwdServer 24 2 1 1 0.13 0.000 0 0 1 0.13 0.000 2 1 1 324323 0.356 0.015 21.304
ACLNoteData 40 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
CacheDigest 40 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLHierCodeData 32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLMethodData 16 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
ACLTimeData 32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLAtStepData 32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLASN 16 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
StoreMetaMD5 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
StoreMetaSTDLFS 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
StoreMetaURL 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
StoreMetaObjSize 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
UFSStoreState::_queued_write 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 200314 0.220 0.012 23.134
StoreMetaVary 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 7963 0.009 0.000 1.559
ACLProtocolData 16 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLServerNameData 16 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
link_list 16 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 200314 0.220 0.006 23.134
dlink_node 24 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
dwrite_q 48 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 91807 0.101 0.008 7.667
cbdata nsvc (8) 72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 75 0.000 0.000 0.003
cbdata RebuildState (16) 944 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 0 0.000 0.000 0.000
StoreSwapLogData 72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 91806 0.101 0.012 7.667
cbdata OptXactLauncher (42) 128 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 1 0.000 0.000 0.000
cbdata OptXact (43) 320 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 1 0.000 0.000 0.000
cbdata RemovalPurgeWalker (44) 72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 2048 0.002 0.000 0.312
cbdata ReadRequest (48) 72 0 0 1 6.38 0.000 0 0 1 6.38 -1.000 0 0 1 111 0.000 0.000 0.000
HttpHdrRange 32 0 0 1 1.90 0.000 0 0 1 1.90 -1.000 0 0 1 314 0.000 0.000 0.000
HttpHdrRangeSpec 16 0 0 1 1.90 0.000 0 0 1 1.90 -1.000 0 0 1 314 0.000 0.000 0.000
cbdata CbDataList (49) 96 0 0 1 6.29 0.000 0 0 1 6.29 -1.000 0 0 1 4662 0.005 0.001 0.000
HttpHdrSc 16 0 0 1 2.92 0.000 0 0 1 2.92 -1.000 0 0 1 12 0.000 0.000 0.000
HttpHdrScTarget 88 0 0 1 2.92 0.000 0 0 1 2.92 -1.000 0 0 1 12 0.000 0.000 0.000
cbdata BodySink (50) 80 0 0 1 2.42 0.000 0 0 1 2.42 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::User 184 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::UserRequest 88 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata StateData (51) 48 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata helper_stateful_server (52) 232 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
Total 1 185173 146634 151763 0.00 100.000 184369 145755 146812 0.00 99.401 804 880 18484 89409360 98.137 98.206 312807.853
Cumulative allocated volume: 52.910 GB
Current overhead: 43294 bytes (0.029%)
Idle pool limit: 5.00 MB
Total Pools created: 138
Pools ever used: 135 (shown above)
Currently in use: 103
String Pool Impact
(%strings) (%volume)
Short Strings 90 48
Medium Strings 9 15
Long Strings 1 5
1KB Strings 0 1
4KB Strings 0 9
16KB Strings 0 13
Other Strings 0 8

Large buffers: 0 (0 KB)

Finally, storedir:

Store Directory Statistics:
Store Entries : 1673
Maximum Swap Size : 32768 KB
Current Store Swap Size: 29484.00 KB
Current Capacity : 89.98% used, 10.02% free

Store Directory #0 (diskd): /var/cache/squid
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 32768 KB
Current Size: 29484.00 KB
Percent Used: 89.98%
Filemap bits in use: 1338 of 16384 (8%)
Filesystem Space in use: 44360368/192108884 KB (23%)
Filesystem Inodes in use: 454196/12271616 (4%)
Flags: SELECTED
Pending operations: 0
Removal policy: lru
LRU reference age: 0.01 days

# free --bytes
total used free shared buff/cache available
Mem: 33653817344 3982090240 929779712 162263040 28741947392 29162274816
Swap: 37925875712 173297664 37752578048

I read Alex's e-mail too late, so I'll try to attach gdb next time.
The problem is that Squid continues to respond, but slowly.
Having a backup Squid is not that simple in my scenario because I require to do content analysis on ssl-bumped traffic. I'm using TPROXY on the gateway/firewall with Squid. A true backup on this production system would mean having a full conntrackd'ed firewall with Squid as I cannot TPROXY on a redirected machine.
I know Squid 4 is recommended for SSL-Bump, but it's still beta, and I'm not sure I can take that leap now.

I didn't run mgr:events while the CPU was at 100%, but here's what came up afterwards:

# squidclient mgr:events
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 11:02:42 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 11:02:42 GMT
Last-Modified: Thu, 26 Oct 2017 11:02:42 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Last event to run: MaintainSwapSpace

Operation Next Execution Weight Callback Valid?
logfileFlush 0.511 sec 1 yes
MaintainSwapSpace 0.511 sec 1 N/A
fqdncache_purgelru 1.481 sec 1 N/A
idnsCheckQueue 2.507 sec 1 N/A
statAvgTick 3.821 sec 1 N/A
storeDirClean 5.484 sec 1 N/A
memPoolCleanIdlePools 5.484 sec 1 N/A
ipcache_purgelru 6.900 sec 1 N/A
peerRefreshDNS 348.872 sec 1 N/A
netdbSaveState 678.327 sec 1 N/A
Adaptation::Icap::ServiceRep::noteTimeToUpdate 3243.778 sec 0 yes
User Cache Maintenance 3302.980 sec 1 N/A
storeDigestRebuildStart 3303.451 sec 1 N/A
storeDigestRewriteStart 3303.451 sec 1 N/A

# squidclient mgr:events
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 11:08:04 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 11:08:04 GMT
Last-Modified: Thu, 26 Oct 2017 11:08:04 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Last event to run: fqdncache_purgelru

Operation Next Execution Weight Callback Valid?
logfileFlush 0.141 sec 1 yes
MaintainSwapSpace 0.141 sec 1 N/A
idnsCheckQueue 3.212 sec 1 N/A
ipcache_purgelru 5.393 sec 1 N/A
fqdncache_purgelru 9.962 sec 1 N/A
storeDirClean 13.942 sec 1 N/A
memPoolCleanIdlePools 13.942 sec 1 N/A
peerRefreshDNS 27.243 sec 1 N/A
statAvgTick 42.223 sec 1 N/A
netdbSaveState 356.698 sec 1 N/A
Adaptation::Icap::ServiceRep::noteTimeToUpdate 2922.149 sec 0 yes
User Cache Maintenance 2981.351 sec 1 N/A
storeDigestRebuildStart 2981.822 sec 1 N/A
storeDigestRewriteStart 2981.822 sec 1 N/A

I haven't been able to associate the 100% CPU issue with a scheduled event.
I'll keep looking.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Alex Rousskov

Re: squid cpu usage

On 10/26/2017 05:15 AM, Vieri wrote:

> Here's the strace (compressed with gz):
> Do you see anything useful in the trace?

FWIW, nothing jumped at me. As you know, I was asking for a gdb
backtrace, but I no longer think that it is likely to help at this
stage: Your further comments and strace itself confirm that Squid is
_not_ stuck in some tight loop, consuming 100% of CPU. It may be
consuming a lot of CPU cycles, but it is not doing the same compact
operation over and over again, so random gdb backtraces are unlikely to
pinpoint the culprit (if there is one).

> If not, how can I improve the trace next time I have this issue?

Please add (relative) timestamps between system calls (strace -r). It
may help detect periods of time when Squid was not making any system
calls. If possible, I would also add time spend in each system call
(strace -T), just in case, but it is unlikely to be useful in this case.

Please also report how many request did Squid finished handling during
the traced period. You can run mgr:info before/after and/or count
access.log lines.

For mgr:info snapshots, please consider doing the legwork to identify
potentially relevant entries that have changed between the snapshots. I
do not have time to do that (but others on the list might, of course).

I would not bother with mgr:mem snapshots unless mgr:info shows that the
memory has grown a lot during the 100% CPU usage.

Finally, consider enabling Squid debugging for a few seconds of that
100% CPU usage period. Wiki has instructions revolving around sending
Squid a debug signal.

> Having a backup Squid is not that simple in my scenario

It rarely is!

> I didn't run mgr:events while the CPU was at 100%

Please run mgr:events all the time (e.g., every minute) so that you can
catch event(s) (if any) that _lead_ to the 100% CPU usage period. You
can also run mgr:events more often (e.g., every 5 seconds) during the
100% CPU usage period, of course.

HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Vieri

Re: squid cpu usage

Thanks for all the tips.

I should've done this before, but I just took a second look at cache.log and found the following message repeated a few times before the 100% CPU issue:

"Your cache is running out of filedescriptors"

I have no more info because the debug level was low.

If this is the cause then I'm even more worried now because:

# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127521
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 127521
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

# cat /etc/security/limits.d/01_squid.conf
* hard nofile 65535
* soft nofile 16384

It just so happens I ran mgr:filedescriptors while the CPU was at 100%.

Here's the output (stripped of domains and IP addresses):

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:35:16 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:35:16 GMT
Last-Modified: Thu, 26 Oct 2017 08:35:16 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Active file descriptors:
File Type Tout Nread * Nwrite * Remote Address Description
---- ------ ---- -------- -------- --------------------- ------------------------------
5 Log 0 0 0 /var/log/squid/cache.log
6 Pipe 0 0 0 IPC UNIX STREAM Parent
9 Socket 28796 0 0 x.x.x.x:3199 client https start
11 Socket 8 1829* 174558 x.x.x.x:62569 Idle client: Waiting for next request
12 Socket 0 77571 243300 x.x.x.x:1344 x.x.x.x
13 Socket 28798 0 0 x.x.x.x:1428 client https start
14 Socket 113 201* 3316 x.x.x.x:49847 Idle client: Waiting for next request
15 Socket 7614 0* 0 x.x.x.x:56204 client https start
16 Socket 0 70492 72860 x.x.x.x:1344 x.x.x.x
17 Socket 0 184736 412668 x.x.x.x:1344 x.x.x.x
18 Socket 0 269207 467337 x.x.x.x:1344 x.x.x.x
19 Socket 0 70386 70846 x.x.x.x:1344 x.x.x.x
20 Socket 56 319* 452 x.x.x.x:59366 Idle client: Waiting for next request
21 Socket 0 74155 160454 x.x.x.x:1344 x.x.x.x
22 Socket 0 70373 184389 x.x.x.x:1344 x.x.x.x
23 Socket 0 321566 418429 x.x.x.x:1344 x.x.x.x
24 Socket 28783 1633* 0 x.x.x.x:60033 Reading next request
25 Socket 895 1809* 37539 x.x.x.x:1344 x.x.x.x
26 Socket 0 171606 765947 x.x.x.x:1344 x.x.x.x
27 Socket 0 505651 936639 x.x.x.x:1344 x.x.x.x
28 Socket 0 77690 579865 x.x.x.x:1344 x.x.x.x
29 Socket 893 1120129* 1356723 x.x.x.x:1344 x.x.x.x
30 Socket 99 1409* 1067 x.x.x.x:51458 Idle client: Waiting for next request
31 Socket 0 697084 991951 x.x.x.x:1344 x.x.x.x
32 Socket 0 72740 380302 x.x.x.x:1344 x.x.x.x
33 Socket 0 133319 4018875 x.x.x.x:1344 x.x.x.x
34 Socket 885 365084* 418439 x.x.x.x:1344 x.x.x.x
35 Socket 0 230550 976940 x.x.x.x:1344 x.x.x.x
36 Socket 0 72085 159051 x.x.x.x:1344 x.x.x.x
37 Socket 0 142361 344294 x.x.x.x:1344 x.x.x.x
38 Socket 0 1008147 1644576 x.x.x.x:1344 x.x.x.x
39 Socket 0 861819 1334921 x.x.x.x:1344 x.x.x.x
40 Socket 28800 2851* 344 x.x.x.x:60799 Reading next request
41 Socket 0 0 0 [::]:39163 DNS Socket IPv6
42 Socket 0 152497 327514 x.x.x.x:1344 x.x.x.x
43 Socket 0 1139088 1516707 x.x.x.x:1344 x.x.x.x
44 Socket 0 237721 398977 x.x.x.x:1344 x.x.x.x
45 Socket 0 91801 114000 x.x.x.x:1344 x.x.x.x
46 Socket 0 0 0 x.x.x.x:3233 HTTP Request
47 Socket 0 131675 695360 x.x.x.x:1344 x.x.x.x
48 Socket 0 159372 179649 x.x.x.x:1344 x.x.x.x
49 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (187)
50 Socket 0 161* 1378 x.x.x.x:443 https://domain.org
52 Socket 69 286* 1430 x.x.x.x:55771 Idle client: Waiting for next request
53 Socket 8 397* 48788 x.x.x.x:62572 Idle client: Waiting for next request
54 Socket 0 341879 480589 x.x.x.x:1344 x.x.x.x
55 Socket 91 201* 3315 x.x.x.x:56787 Idle client: Waiting for next request
56 Socket 0 726999 1336667 x.x.x.x:1344 x.x.x.x
57 Socket 113 201* 3316 x.x.x.x:51036 Idle client: Waiting for next request
58 Socket 119 556* 3723 x.x.x.x:51851 Idle client: Waiting for next request
59 Socket 99 1322* 766 x.x.x.x:1968 Idle client: Waiting for next request
60 Socket 0 78554 216848 x.x.x.x:1344 x.x.x.x
61 Socket 0 70211 717853 x.x.x.x:1344 x.x.x.x
62 Socket 0 78361 182762 x.x.x.x:1344 x.x.x.x
63 Socket 0 1945692 4239277 x.x.x.x:1344 x.x.x.x
64 Socket 66 1092* 1117 x.x.x.x:50080 Idle client: Waiting for next request
65 Socket 62 202* 286 x.x.x.x:55769 Idle client: Waiting for next request
66 Socket 0 17396* 904 x.x.x.x:443 https://domain.org
67 Socket 0 3536* 1166 x.x.x.x:443 https://domain.org
68 Socket 28798 3038* 797 x.x.x.x:65116 Reading next request
69 Socket 28798 0* 0 x.x.x.x:1426 client https start
70 Socket 28780 0* 0 x.x.x.x:60791 client https start
71 Socket 0 70387 70867 x.x.x.x:1344 x.x.x.x
72 Socket 0 254449 502590 x.x.x.x:1344 x.x.x.x
73 Socket 9 11524* 1152 x.x.x.x:54067 Idle client: Waiting for next request
74 Socket 0 96195 103010 x.x.x.x:1344 x.x.x.x
75 Socket 28 1516* 1291 x.x.x.x:3023 Idle client: Waiting for next request
76 Socket 0 205806 602151 x.x.x.x:1344 x.x.x.x
77 Socket 115 2195* 1248 x.x.x.x:59024 Idle client: Waiting for next request
78 Socket 0 352877 444820 x.x.x.x:1344 x.x.x.x
79 Socket 6 384* 612 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
80 Socket 0 71272 230010 x.x.x.x:1344 x.x.x.x
81 Socket 5 1490* 168706 x.x.x.x:54376 Idle client: Waiting for next request
82 Socket 0 4403371 651548 x.x.x.x:47443 DNS Socket IPv4
83 Socket 0 782173 984402 x.x.x.x:1344 x.x.x.x
84 Pipe 0 0 0 unlinkd -> squid
85 Socket 28798 433* 0 x.x.x.x:1423 Reading next request
86 Socket 5 2211* 1249 x.x.x.x:49400 Idle client: Waiting for next request
87 Pipe 0 0 0 squid -> unlinkd
88 Pipe 0 0 0 squid -> diskd
89 Socket 28792 4914* 2557 x.x.x.x:3381 Reading next request
90 Socket 66 598* 712 x.x.x.x:2262 Idle client: Waiting for next request
91 Socket 0 486500 1052648 x.x.x.x:1344 x.x.x.x
92 Socket 0 84149 459690 x.x.x.x:1344 x.x.x.x
93 Socket 0 446707 703970 x.x.x.x:1344 x.x.x.x
94 Socket 28797 1744* 0 x.x.x.x:49666 Reading next request
95 Socket 0 196505 1356207 x.x.x.x:1344 x.x.x.x
96 Socket 0 71430 747104 x.x.x.x:1344 x.x.x.x
97 Pipe 0 55385864* 83254354 ssl_crtd #1
98 Socket 99 4720* 74394 x.x.x.x:51833 Idle client: Waiting for next request
99 Socket 0 80191 352164 x.x.x.x:1344 x.x.x.x
100 Socket 0 831352 1205057 x.x.x.x:1344 x.x.x.x
101 Socket 0 67263 68047 x.x.x.x:1344 x.x.x.x
102 Socket 28734 0* 0 x.x.x.x:56784 client https start
103 Socket 0 90037 239742 x.x.x.x:1344 x.x.x.x
104 Socket 0 86557 182854 x.x.x.x:1344 x.x.x.x
105 Socket 0 74680 272842 x.x.x.x:1344 x.x.x.x
106 Socket 0 82432 182773 x.x.x.x:1344 x.x.x.x
107 Socket 0 91159 110508 x.x.x.x:1344 x.x.x.x
108 Socket 0 74341 182862 x.x.x.x:1344 x.x.x.x
109 Socket 58 354520* 417712 x.x.x.x:1344 x.x.x.x
110 Socket 896 110923* 238345 x.x.x.x:1344 x.x.x.x
111 Socket 0 367211 830710 x.x.x.x:1344 x.x.x.x
112 Socket 0 799681 2454675 x.x.x.x:1344 x.x.x.x
113 Socket 0 174639 526254 x.x.x.x:1344 x.x.x.x
114 Socket 0 224143 626713 x.x.x.x:1344 x.x.x.x
115 Socket 0 825* 4338 x.x.x.x:443 https://domain.org
116 Socket 50 0 0* x.x.x.x:80 domain.org
117 Socket 28799 0 0 x.x.x.x:3208 client https start
118 Socket 0 29387* 3248 x.x.x.x:443 https://domain.org
119 Socket 69 13796* 1117 x.x.x.x:64434 Idle client: Waiting for next request
120 Socket 0 401186 536770 x.x.x.x:1344 x.x.x.x
121 Socket 0 70420 72431 x.x.x.x:1344 x.x.x.x
122 Socket 0 1248701 2243251 x.x.x.x:1344 x.x.x.x
123 Pipe 0 12387700* 18425650 ssl_crtd #2
124 Socket 0 74579 186204 x.x.x.x:1344 x.x.x.x
125 Socket 0 147688 317832 x.x.x.x:1344 x.x.x.x
126 Socket 0 157570 1000064 x.x.x.x:1344 x.x.x.x
127 Socket 0 484039 932396 x.x.x.x:1344 x.x.x.x
128 Socket 0 72709 204331 x.x.x.x:1344 x.x.x.x
129 Socket 0 121209 121541 x.x.x.x:1344 x.x.x.x
130 Socket 0 89678 203510 x.x.x.x:1344 x.x.x.x
131 Socket 0 1096513 1775872 x.x.x.x:1344 x.x.x.x
132 Socket 28774 1636* 0 x.x.x.x:65117 Reading next request
133 Socket 53 378* 1542 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
134 Socket 108 482* 654 x.x.x.x:59022 Idle client: Waiting for next request
135 Socket 0 67261 67977 x.x.x.x:1344 x.x.x.x
136 Socket 0 353852 1257798 x.x.x.x:1344 x.x.x.x
137 Socket 120 3820* 3424 x.x.x.x:64559 Idle client: Waiting for next request
138 Socket 8 829* 1280 x.x.x.x:3040 Idle client: Waiting for next request
139 Socket 6069 0* 0 x.x.x.x:56177 client https start
140 Socket 0 75215 314184 x.x.x.x:1344 x.x.x.x
141 Socket 0 72428 421624 x.x.x.x:1344 x.x.x.x
142 Socket 0 205459 279606 x.x.x.x:1344 x.x.x.x
143 Socket 0 983140 1528993 x.x.x.x:1344 x.x.x.x
144 Socket 0 72254 128634 x.x.x.x:1344 x.x.x.x
145 Socket 108 470* 654 x.x.x.x:2259 Idle client: Waiting for next request
146 Socket 28798 0 0 x.x.x.x:1839 client https start
147 Socket 28754 6505* 2748 x.x.x.x:2121 Reading next request
148 Socket 0 94387 437023 x.x.x.x:1344 x.x.x.x
149 Socket 112 526* 636 x.x.x.x:54464 Idle client: Waiting for next request
150 Socket 0 291712 418901 x.x.x.x:1344 x.x.x.x
151 Socket 0 260125 466643 x.x.x.x:1344 x.x.x.x
152 Socket 0 75500 192364 x.x.x.x:1344 x.x.x.x
153 Socket 91 6720* 31850 x.x.x.x:51199 Idle client: Waiting for next request
154 Socket 0 340510 3310843 x.x.x.x:1344 x.x.x.x
155 Socket 28799 0 0 x.x.x.x:3206 client https start
156 Socket 0 74500 184502 x.x.x.x:1344 x.x.x.x
157 Socket 888 208562* 377059 x.x.x.x:1344 x.x.x.x
158 Pipe 0 5872190* 8607084 ssl_crtd #3
159 Socket 0 103064 525215 x.x.x.x:1344 x.x.x.x
160 Socket 98 482* 654 x.x.x.x:60031 Idle client: Waiting for next request
161 Socket 62 516* 675 x.x.x.x:51195 Idle client: Waiting for next request
162 Socket 28799 0 0 x.x.x.x:3209 client https start
163 Socket 66 2211* 2150 x.x.x.x:54426 Idle client: Waiting for next request
164 Socket 28798 3348* 798 x.x.x.x:2255 Reading next request
165 Socket 0 222093 321752 x.x.x.x:1344 x.x.x.x
166 Socket 0 126144 442862 x.x.x.x:1344 x.x.x.x
167 Socket 62 16353* 41096 x.x.x.x:3235 Idle client: Waiting for next request
168 Socket 0 67365 68179 x.x.x.x:1344 x.x.x.x
169 Socket 0 9195* 2294 x.x.x.x:443 https://domain.org
170 Socket 113 201* 3316 x.x.x.x:51037 Idle client: Waiting for next request
171 Socket 28800 0 0 x.x.x.x:55497 client https connect
172 Socket 0 126735 522798 x.x.x.x:1344 x.x.x.x
173 Socket 0 27366* 6978 x.x.x.x:443 https://domain.org
174 Socket 28779 3231* 870 x.x.x.x:2142 Reading next request
175 Socket 0 1214190 1308440 x.x.x.x:1344 x.x.x.x
176 Socket 28745 3983* 0 x.x.x.x:1365 Reading next request
177 Socket 27422 0* 0 x.x.x.x:56749 client https start
178 Socket 0 71305 194143 x.x.x.x:1344 x.x.x.x
179 Socket 66 306* 452 x.x.x.x:1353 Idle client: Waiting for next request
180 Socket 28774 1758* 0 x.x.x.x:59375 Reading next request
181 Socket 28798 3618* 609* x.x.x.x:55780 Reading next request
182 Socket 0 463907 4817276 x.x.x.x:1344 x.x.x.x
183 Socket 0 366549 818689 x.x.x.x:1344 x.x.x.x
184 Pipe 0 3597805* 5272955 ssl_crtd #4
185 Socket 0 330910 964399 x.x.x.x:1344 x.x.x.x
186 Socket 113 645* 1205 x.x.x.x:4116 Idle client: Waiting for next request
187 Socket 28800 876* 0* x.x.x.x:62588 Reading next request
188 Socket 0 1484456 1616112 x.x.x.x:1344 x.x.x.x
189 Socket 0 902153 1747543 x.x.x.x:1344 x.x.x.x
190 Socket 99 13466* 1117 x.x.x.x:64687 Idle client: Waiting for next request
191 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (505)
192 Socket 58 277899* 415305 x.x.x.x:1344 x.x.x.x
193 Socket 19 838* 739 x.x.x.x:62578 Idle client: Waiting for next request
194 Socket 58 32594* 53867 x.x.x.x:1344 x.x.x.x
195 Socket 0 74412 748176 x.x.x.x:1344 x.x.x.x
196 Socket 0 294684 832445 x.x.x.x:1344 x.x.x.x
197 Socket 876 431920* 668818 x.x.x.x:1344 x.x.x.x
198 Socket 28744 2487* 0 x.x.x.x:3267 Reading next request
199 Socket 113 201* 3316 x.x.x.x:51238 Idle client: Waiting for next request
200 Socket 0 1043* 886 x.x.x.x:443 https://domain.org
201 Socket 28779 3150* 870 x.x.x.x:2134 Reading next request
202 Socket 104 361* 3527 x.x.x.x:3173 Idle client: Waiting for next request
203 Socket 0 0 0 x.x.x.x:3234 HTTP Request
204 Socket 8 1450* 1308 x.x.x.x:65097 Reading next request
205 Socket 19 566* 784 x.x.x.x:54405 Idle client: Waiting for next request
206 Socket 0 1018301 1251632 x.x.x.x:1344 x.x.x.x
207 Socket 0 243460 725933 x.x.x.x:1344 x.x.x.x
208 Pipe 0 2674774* 3914935 ssl_crtd #5
209 Socket 115 610* 712 x.x.x.x:64689 Idle client: Waiting for next request
210 Socket 0 764* 7298 x.x.x.x:443 https://domain.org
211 Socket 105 1277* 344 x.x.x.x:2283 Idle client: Waiting for next request
212 Socket 0 131679 1220103 x.x.x.x:1344 x.x.x.x
213 Socket 876 30969* 222063 x.x.x.x:1344 x.x.x.x
214 Socket 0 191177 438150 x.x.x.x:1344 x.x.x.x
215 Socket 0 78829 290995 x.x.x.x:1344 x.x.x.x
216 Socket 112 1002* 1328 x.x.x.x:51197 Idle client: Waiting for next request
217 Socket 0 174193* 3658 x.x.x.x:443 https://domain.org
218 Socket 28781 0* 0 x.x.x.x:60795 client https start
219 Pipe 0 1969670* 2885654 ssl_crtd #6
220 Socket 67 5709* 13745 x.x.x.x:1366 Idle client: Waiting for next request
221 Socket 0 570053 628689 x.x.x.x:1344 x.x.x.x
222 Socket 28790 308* 0 x.x.x.x:2899 Reading next request
223 Socket 28798 704* 267112 x.x.x.x:1827 Reading next request
224 Pipe 0 1464643* 2151275 ssl_crtd #7
225 Socket 107 3463* 26520 x.x.x.x:51202 Idle client: Waiting for next request
226 Socket 28799 433* 0 x.x.x.x:1425 Reading next request
227 Socket 118 1424* 344 x.x.x.x:1826 Idle client: Waiting for next request
228 Socket 0 321598 669611 x.x.x.x:1344 x.x.x.x
229 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (686)
230 Socket 87 909* 718 x.x.x.x:1389 Idle client: Waiting for next request
231 Socket 0 70445 89872 x.x.x.x:1344 x.x.x.x
232 Socket 28797 0* 0 x.x.x.x:3389 client https start
233 Socket 0 145988 147306 x.x.x.x:1344 x.x.x.x
234 Socket 0 586261 1258275 x.x.x.x:1344 x.x.x.x
235 Socket 0 285602 466348 x.x.x.x:1344 x.x.x.x
236 Socket 0 131679 3752672 x.x.x.x:1344 x.x.x.x
237 Socket 0 447059 1790353 x.x.x.x:1344 x.x.x.x
238 Socket 0 87910 2010315 x.x.x.x:1344 x.x.x.x
239 Socket 5812 0* 0 x.x.x.x:56171 client https start
240 Socket 28 48115* 1178 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
241 Socket 66 4466* 18377 x.x.x.x:3268 Idle client: Waiting for next request
242 Socket 0 596879 1296258 x.x.x.x:1344 x.x.x.x
243 Socket 0 568370 987503 x.x.x.x:1344 x.x.x.x
244 Pipe 0 1150405* 1697637 ssl_crtd #8
245 Socket 56 485* 682630 x.x.x.x:62580 Idle client: Waiting for next request
246 Socket 0 175607 1683506 x.x.x.x:1344 x.x.x.x
247 Socket 16 1424* 17194 x.x.x.x:54395 Idle client: Waiting for next request
248 Socket 28798 829* 12831* x.x.x.x:2308 Reading next request
249 Socket 56 318* 452 x.x.x.x:62585 Idle client: Waiting for next request
250 Socket 875 1734* 5924 x.x.x.x:443 https://domain.org
251 Socket 5 518* 70059 x.x.x.x:54373 Idle client: Waiting for next request
252 Socket 0 149322 313136 x.x.x.x:1344 x.x.x.x
253 Socket 0 589520 836807 x.x.x.x:1344 x.x.x.x
254 Socket 28797 0 0* x.x.x.x:54932 client https start
255 Socket 28771 1660* 0 x.x.x.x:51201 Reading next request
256 Pipe 0 916891* 1351785 ssl_crtd #9
257 Socket 60 22466* 111364 x.x.x.x:1344 x.x.x.x
258 Socket 28796 3294* 1180 x.x.x.x:59020 Reading next request
259 Socket 0 1055* 872 x.x.x.x:443 https://domain.org
260 Socket 0 14835* 624 x.x.x.x:443 https://domain.org
261 Socket 875 289287* 581694 x.x.x.x:1344 x.x.x.x
262 Socket 7 2437* 10773 x.x.x.x:65303 Idle client: Waiting for next request
263 Socket 28796 0 0 x.x.x.x:3197 client https start
264 Pipe 0 729453* 1059077 ssl_crtd #10
265 Socket 66 4370* 5763 x.x.x.x:3269 Idle client: Waiting for next request
266 Socket 107 3233* 4119 x.x.x.x:64586 Idle client: Waiting for next request
267 Socket 0 212472 431063 x.x.x.x:1344 x.x.x.x
268 Pipe 0 578076* 850440 ssl_crtd #11
269 Socket 113 407* 613 x.x.x.x:51460 Idle client: Waiting for next request
270 Socket 0 1101* 4474 x.x.x.x:443 https://domain.org
271 Socket 0 2563693 3830336 x.x.x.x:1344 x.x.x.x
272 Socket 0 828859 1380776 x.x.x.x:1344 x.x.x.x
273 Socket 0 71319 259076 x.x.x.x:1344 x.x.x.x
274 Socket 105 609* 435 x.x.x.x:3171 Idle client: Waiting for next request
275 Socket 28799 433* 0 x.x.x.x:1427 Reading next request
276 Socket 0 74500 185574 x.x.x.x:1344 x.x.x.x
277 Socket 0 189973 564948 x.x.x.x:1344 x.x.x.x
278 Socket 60 129925* 458165 x.x.x.x:1344 x.x.x.x
279 Socket 28783 0* 0 x.x.x.x:60797 client https start
280 Socket 28797 731* 0 x.x.x.x:54488 Reading next request
281 Socket 0 70962 98876 x.x.x.x:1344 x.x.x.x
282 Socket 0 77789 334559 x.x.x.x:1344 x.x.x.x
283 Socket 115 649* 623 x.x.x.x:54482 Idle client: Waiting for next request
284 Socket 0 131737 182857 x.x.x.x:1344 x.x.x.x
285 Socket 28799 0 0 x.x.x.x:3216 client https start
286 Socket 74 469* 555 x.x.x.x:60744 Idle client: Waiting for next request
287 Pipe 0 85247* 124165 ssl_crtd #1
288 Socket 0 996913 1509983 x.x.x.x:1344 x.x.x.x
289 Pipe 0 466480* 700518 ssl_crtd #12
290 Socket 0 70702 74868 x.x.x.x:1344 x.x.x.x
291 Socket 88 518* 43112 x.x.x.x:60756 Idle client: Waiting for next request
292 Socket 9 10213* 1158 x.x.x.x:54068 Idle client: Waiting for next request
293 Socket 900 269761* 431423* x.x.x.x:1344 x.x.x.x
294 Socket 0 88023 296477 x.x.x.x:1344 x.x.x.x
295 Socket 113 201* 3316 x.x.x.x:51038 Idle client: Waiting for next request
296 Socket 28800 0 0 x.x.x.x:2385 client https start
297 Socket 0 1409562 1861582 x.x.x.x:1344 x.x.x.x
298 Socket 0 74282 713086 x.x.x.x:1344 x.x.x.x
299 Socket 0 74290 182774 x.x.x.x:1344 x.x.x.x
300 Socket 97 2215* 1301 x.x.x.x:51848 Idle client: Waiting for next request
301 Socket 0 85396 343984 x.x.x.x:1344 x.x.x.x
302 Socket 0 78177 143882 x.x.x.x:1344 x.x.x.x
303 Socket 66 5881* 6604 x.x.x.x:1367 Idle client: Waiting for next request
304 Socket 0 2942297 3069551 x.x.x.x:1344 x.x.x.x
305 Pipe 0 364706* 534055 ssl_crtd #13
306 Socket 28758 0 0 x.x.x.x:55095 client https start
307 Socket 118 410* 1231 x.x.x.x:2311 Idle client: Waiting for next request
308 Socket 0 503380 1071795 x.x.x.x:1344 x.x.x.x
309 Socket 0 96839 350661 x.x.x.x:1344 x.x.x.x
310 Socket 0 556795 5837880 x.x.x.x:1344 x.x.x.x
311 Socket 112 254* 787 x.x.x.x:50711 Idle client: Waiting for next request
312 Socket 28800 0 0 x.x.x.x:3220 client https start
313 Socket 0 72449 172781 x.x.x.x:1344 x.x.x.x
314 Socket 0 185849 1747378 x.x.x.x:1344 x.x.x.x
315 Socket 0 0 0 x.x.x.x:3232 HTTP Request
316 Socket 87 201* 3316 x.x.x.x:56871 Idle client: Waiting for next request
317 Socket 9 888* 1066 x.x.x.x:2976 Idle client: Waiting for next request
318 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (269)
319 Socket 0 73737 86419 x.x.x.x:1344 x.x.x.x
320 Socket 0 648346 962465 x.x.x.x:1344 x.x.x.x
321 Socket 0 151927 449211 x.x.x.x:1344 x.x.x.x
322 Socket 0 275* 2398 x.x.x.x:443 https://domain.org
323 Socket 2 354* 466 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
324 Socket 108 470* 654 x.x.x.x:2257 Idle client: Waiting for next request
325 Pipe 0 68850* 103401 ssl_crtd #2
326 Socket 0 0 0 x.x.x.x:3230 client https connect
327 Socket 28773 0* 0 x.x.x.x:56789 client https start
328 Socket 0 242751 910187 x.x.x.x:1344 x.x.x.x
329 Socket 28730 1146* 0 x.x.x.x:2150 Reading next request
330 Pipe 0 309147* 463388 ssl_crtd #14
331 Socket 28783 1366* 0 x.x.x.x:64173 Reading next request
332 Socket 0 531* 1360 x.x.x.x:443 https://domain.org
333 Socket 0 467739 655208 x.x.x.x:1344 x.x.x.x
334 Socket 0 0 0 x.x.x.x:55499 client https connect
335 Socket 10 2143* 880 x.x.x.x:54391 Idle client: Waiting for next request
336 Socket 113 10176* 1157 x.x.x.x:31352 Idle client: Waiting for next request
337 Socket 0 520857 1758549 x.x.x.x:1344 x.x.x.x
338 Socket 0 232302 359424 x.x.x.x:1344 x.x.x.x
339 Socket 59 164364* 422776 x.x.x.x:1344 x.x.x.x
340 Socket 0 1459622 1885753 x.x.x.x:1344 x.x.x.x
341 Socket 28800 0 0 x.x.x.x:60102 client https start
342 Socket 16 201* 3316 x.x.x.x:54400 Idle client: Waiting for next request
343 Socket 28799 433* 0 x.x.x.x:1429 Reading next request
344 Socket 0 170597 572080 x.x.x.x:1344 x.x.x.x
345 Pipe 0 256576* 380163 ssl_crtd #15
346 Socket 16 399* 286 x.x.x.x:61194 Idle client: Waiting for next request
347 Pipe 0 59571* 89389 ssl_crtd #3
348 Socket 19 563* 784 x.x.x.x:54402 Idle client: Waiting for next request
349 Socket 28800 0 0 x.x.x.x:3219 client https start
350 Socket 0 381663 1005072 x.x.x.x:1344 x.x.x.x
351 Socket 28799 2283* 0 x.x.x.x:49667 Reading next request
352 Socket 112 964* 1306 x.x.x.x:51189 Idle client: Waiting for next request
353 Pipe 0 218850* 321361 ssl_crtd #16
354 Socket 106 526* 276 x.x.x.x:51461 Idle client: Waiting for next request
355 Socket 19 573* 784 x.x.x.x:54403 Idle client: Waiting for next request
356 Socket 66 490* 8229 x.x.x.x:3144 Idle client: Waiting for next request
357 Pipe 0 179339* 269018 ssl_crtd #17
358 Socket 28800 0 0 x.x.x.x:3217 client https start
359 Socket 28798 0 0 x.x.x.x:1840 client https start
360 Socket 0 1080220 1947420 x.x.x.x:1344 x.x.x.x
361 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (46)
362 Socket 0 217293 913828 x.x.x.x:1344 x.x.x.x
363 Socket 44 630* 2746 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
364 Socket 0 12335* 15764 x.x.x.x:443 https://domain.org
365 Socket 28800 0 0 x.x.x.x:3221 client https start
366 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (479)
367 Socket 0 343536 972237 x.x.x.x:1344 x.x.x.x
368 Socket 0 1138072 1520823 x.x.x.x:1344 x.x.x.x
369 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (254)
370 Socket 19 563* 784 x.x.x.x:54404 Idle client: Waiting for next request
371 Socket 98 537* 1246 x.x.x.x:54444 Idle client: Waiting for next request
372 Socket 0 860350 1104913 x.x.x.x:1344 x.x.x.x
373 Socket 0 1060* 23048 x.x.x.x:443 https://domain.org
374 Socket 0 1066* 20478 x.x.x.x:443 https://domain.org
375 Socket 28799 0 0 x.x.x.x:3210 client https start
376 Socket 900 0* 4618 x.x.x.x:443 https://domain.org
377 Pipe 0 55913* 81124 ssl_crtd #4
378 Socket 113 201* 3316 x.x.x.x:51039 Idle client: Waiting for next request
379 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (263)
380 Socket 113 201* 3316 x.x.x.x:51040 Idle client: Waiting for next request
381 Socket 46 3906* 24391 x.x.x.x:1344 x.x.x.x
382 Socket 28798 0 0 x.x.x.x:2320 client https start
383 Socket 0 72481 157154 x.x.x.x:1344 x.x.x.x
384 Pipe 0 133145* 199159 ssl_crtd #18
385 Socket 28776 7538* 1630 x.x.x.x:54399 Reading next request
386 Pipe 0 7025* 10746 ssl_crtd #1
387 Socket 86 482* 654 x.x.x.x:49662 Idle client: Waiting for next request
388 Socket 91 1173* 343 x.x.x.x:60760 Idle client: Waiting for next request
389 Socket 0 1560783 3101138 x.x.x.x:1344 x.x.x.x
390 Socket 28796 0 0 x.x.x.x:3198 client https start
391 Socket 0 580399 792546 x.x.x.x:1344 x.x.x.x
392 Socket 0 78307 198904 x.x.x.x:1344 x.x.x.x
393 Socket 28737 0* 0 x.x.x.x:51806 client https start
394 Socket 18 470* 3778 x.x.x.x:3088 Idle client: Waiting for next request
395 Socket 28800 0 0 x.x.x.x:3223 client https start
396 Socket 0 0 0 x.x.x.x:3235 HTTP Request
397 Socket 7 1934* 615 x.x.x.x:54066 Idle client: Waiting for next request
398 Pipe 0 113172* 166273 ssl_crtd #19
399 Socket 893 195206* 214787 x.x.x.x:1344 x.x.x.x
400 Socket 28754 10860* 10436 x.x.x.x:4100 Reading next request
401 Socket 17 201* 1324 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
402 Socket 0 2234270 2503172 x.x.x.x:1344 x.x.x.x
403 Socket 28800 5134* 2537 x.x.x.x:1834 Reading next request
404 Socket 28772 1637* 0 x.x.x.x:51847 Reading next request
405 Socket 54 306* 452 x.x.x.x:31306 Idle client: Waiting for next request
406 Socket 63 819* 1961 x.x.x.x:52887 Idle client: Waiting for next request
407 Socket 28 253* 2550 x.x.x.x:3093 Idle client: Waiting for next request
408 Socket 28793 536* 8935* x.x.x.x:60800 Reading next request
409 Socket 0 67261 67880 x.x.x.x:1344 x.x.x.x
410 Pipe 0 7959* 12590 ssl_crtd #2
411 Pipe 0 37269* 56216 ssl_crtd #5
412 Socket 0 76888 138606 x.x.x.x:1344 x.x.x.x
413 Socket 28800 0 0 x.x.x.x:3225 client https start
414 Socket 98 549* 1141 x.x.x.x:54445 Idle client: Waiting for next request
415 Socket 0 873728 1316366 x.x.x.x:1344 x.x.x.x
416 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (390)
417 Socket 97 482* 654 x.x.x.x:49408 Idle client: Waiting for next request
418 Socket 0 4690935 6430992 x.x.x.x:1344 x.x.x.x
419 Pipe 0 7549* 11042 ssl_crtd #3
420 Socket 0 1058661 1673219 x.x.x.x:1344 x.x.x.x
421 Socket 28796 10874* 125020* x.x.x.x:54421 Reading next request
422 Socket 0 619900 1893341 x.x.x.x:1344 x.x.x.x
423 Socket 0 484497 639048 x.x.x.x:1344 x.x.x.x
424 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (497)
425 Socket 28798 0 0 x.x.x.x:3200 client https start
426 Socket 0 217686 282356 x.x.x.x:1344 x.x.x.x
427 Socket 0 171843 2226884 x.x.x.x:1344 x.x.x.x
428 Socket 0 742382 1291857 x.x.x.x:1344 x.x.x.x
429 Socket 104 249* 776 x.x.x.x:61437 Idle client: Waiting for next request
430 Socket 876 338749* 449700 x.x.x.x:1344 x.x.x.x
431 Socket 0 638966 976750 x.x.x.x:1344 x.x.x.x
432 Socket 0 67626 109037 x.x.x.x:1344 x.x.x.x
433 Socket 28799 0 0 x.x.x.x:3207 client https start
434 Socket 0 539438 762345 x.x.x.x:1344 x.x.x.x
435 Socket 0 529527 740350 x.x.x.x:1344 x.x.x.x
436 Socket 0 133828 244486 x.x.x.x:1344 x.x.x.x
437 Socket 91 273* 3437 x.x.x.x:55776 Idle client: Waiting for next request
438 Socket 28799 0 0 x.x.x.x:3212 client https start
439 Socket 0 175285 360548 x.x.x.x:1344 x.x.x.x
440 Socket 28800 0 0 x.x.x.x:55498 client https connect
441 Socket 900 866564* 1274512* x.x.x.x:1344 x.x.x.x
442 Socket 0 104615 259266 x.x.x.x:1344 x.x.x.x
443 Socket 0 767495 1513964 x.x.x.x:1344 x.x.x.x
444 Socket 113 557* 3604 x.x.x.x:3187 Idle client: Waiting for next request
445 Socket 0 589528 777412 x.x.x.x:1344 x.x.x.x
446 Pipe 0 99772* 143323 ssl_crtd #20
447 Socket 0 51392* 5692 x.x.x.x:443 https://domain.org
448 Socket 0 622* 3326 x.x.x.x:443 https://domain.org
449 Socket 0 178130 479705 x.x.x.x:1344 x.x.x.x
450 Socket 0 75335 249026 x.x.x.x:1344 x.x.x.x
451 Socket 0 584110 1041464 x.x.x.x:1344 x.x.x.x
452 Socket 0 5989855 7840009 x.x.x.x:1344 x.x.x.x
453 Socket 0 70378 183780 x.x.x.x:1344 x.x.x.x
454 Socket 9 397* 817 x.x.x.x:60714 Idle client: Waiting for next request
455 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (615)
456 Socket 0 67473 70236 x.x.x.x:1344 x.x.x.x
457 Socket 113 647* 811 x.x.x.x:54477 Idle client: Waiting for next request
458 Socket 106 277* 3421 x.x.x.x:55778 Idle client: Waiting for next request
459 Socket 0 80521 283037 x.x.x.x:1344 x.x.x.x
460 Socket 98 598* 712 x.x.x.x:50085 Idle client: Waiting for next request
461 Socket 0 167275 298854 x.x.x.x:1344 x.x.x.x
462 Socket 28781 0* 0 x.x.x.x:60792 client https start
463 Socket 115 281* 3428 x.x.x.x:55779 Idle client: Waiting for next request
464 Socket 0 10405* 4874 x.x.x.x:443 https://domain.org
465 Socket 0 1025* 27592 x.x.x.x:443 https://domain.org
466 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (9)
467 Socket 98 555* 2252 x.x.x.x:60770 Idle client: Waiting for next request
468 Socket 0 9354* 2314 x.x.x.x:443 https://domain.org
469 Socket 0 91540 325953 x.x.x.x:1344 x.x.x.x
470 Socket 0 762551 1389788 x.x.x.x:1344 x.x.x.x
471 Socket 74 7830* 12535 x.x.x.x:4097 Idle client: Waiting for next request
472 Socket 0 380016 476604 x.x.x.x:1344 x.x.x.x
473 Socket 885 15670* 85733 x.x.x.x:1344 x.x.x.x
474 Socket 0 406074 456587 x.x.x.x:1344 x.x.x.x
475 Socket 28800 1335* 253 x.x.x.x:3180 Reading next request
476 Socket 46 315* 4617 x.x.x.x:1344 x.x.x.x
477 Socket 58 32530* 49384 x.x.x.x:1344 x.x.x.x
478 Socket 0 172630 242324 x.x.x.x:1344 x.x.x.x
479 Socket 118 387* 3558 x.x.x.x:2316 Idle client: Waiting for next request
480 Socket 0 74290 182781 x.x.x.x:1344 x.x.x.x
481 Socket 0 80770 197294 x.x.x.x:1344 x.x.x.x
482 Socket 0 75048 281261 x.x.x.x:1344 x.x.x.x
483 Socket 0 74554 185351 x.x.x.x:1344 x.x.x.x
484 Socket 97 482* 654 x.x.x.x:49409 Idle client: Waiting for next request
485 Socket 60 1302* 11029 x.x.x.x:1344 x.x.x.x
486 Socket 86 4856* 5810 x.x.x.x:4112 Idle client: Waiting for next request
487 Socket 105 250* 779 x.x.x.x:56421 Idle client: Waiting for next request
488 Socket 0 2063* 4526 x.x.x.x:443 https://domain.org
489 Pipe 0 30386* 47145 ssl_crtd #6
490 Socket 28756 0 0 x.x.x.x:55093 client https start
491 Socket 28800 5190* 1214 x.x.x.x:1835 Reading next request
492 Socket 878 105* 4307 x.x.x.x:1344 x.x.x.x
493 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (232)
494 Socket 900 429 866 x.x.x.x:443 https://domain.org
495 Socket 900 160740* 529879* x.x.x.x:1344 x.x.x.x
496 Socket 0 474171 673500 x.x.x.x:1344 x.x.x.x
497 Socket 113 331* 3485 x.x.x.x:54480 Idle client: Waiting for next request
498 Socket 74 1595* 730 x.x.x.x:49696 Idle client: Waiting for next request
499 Socket 0 559618 865116 x.x.x.x:1344 x.x.x.x
500 Socket 898 595* 3540 x.x.x.x:443 https://domain.org
501 Socket 0 75127 273504 x.x.x.x:1344 x.x.x.x
502 Socket 0 171003 603908 x.x.x.x:1344 x.x.x.x
503 Socket 0 377031 835100 x.x.x.x:1344 x.x.x.x
504 Socket 0 67366 69093 x.x.x.x:1344 x.x.x.x
505 Socket 88 520* 675 x.x.x.x:51841 Idle client: Waiting for next request
506 Socket 0 70420 71693 x.x.x.x:1344 x.x.x.x
507 Socket 0 71047 275124 x.x.x.x:1344 x.x.x.x
508 Socket 0 196158* 23334 x.x.x.x:443 https://domain.org
509 Socket 0 0* 0 ::1:48239 ext_wbinfo_group_acl #1
510 Socket 0 136980 194711 x.x.x.x:1344 x.x.x.x
511 Socket 898 9550* 41835 x.x.x.x:1344 x.x.x.x
512 Socket 0 921556 1321574 x.x.x.x:1344 x.x.x.x
513 Socket 98 537* 1068 x.x.x.x:54446 Idle client: Waiting for next request
514 Socket 0 70273 182857 x.x.x.x:1344 x.x.x.x
515 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (315)
516 Socket 74 507* 1559 x.x.x.x:60741 Idle client: Waiting for next request
517 Socket 87 1531* 29663 x.x.x.x:54429 Idle client: Waiting for next request
518 Socket 87 1085* 9379 x.x.x.x:54430 Idle client: Waiting for next request
519 Socket 0 36043* 842 x.x.x.x:443 https://domain.org
520 Socket 0 156225 280780 x.x.x.x:1344 x.x.x.x
521 Socket 87 1095* 9538 x.x.x.x:54431 Idle client: Waiting for next request
522 Socket 0 1090575 1456044 x.x.x.x:1344 x.x.x.x
523 Socket 0 67263 67988 x.x.x.x:1344 x.x.x.x
524 Socket 0 133693 347078 x.x.x.x:1344 x.x.x.x
525 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (69)
526 Socket 898 311676* 510052 x.x.x.x:1344 x.x.x.x
527 Socket 900 425 866 x.x.x.x:443 https://domain.org
528 Socket 0 0* 0 ::1:59459 ext_wbinfo_group_acl #2
529 Socket 0 78353 989595 x.x.x.x:1344 x.x.x.x
530 Socket 0 1096* 1782 x.x.x.x:443 https://domain.org
531 Socket 28800 0 0 x.x.x.x:3222 client https start
532 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (764)
533 Socket 0 71390 110141 x.x.x.x:1344 x.x.x.x
534 Socket 0 1203421 1809900 x.x.x.x:1344 x.x.x.x
535 Socket 0 70832 76525 x.x.x.x:1344 x.x.x.x
536 Socket 118 417* 1219 x.x.x.x:2310 Idle client: Waiting for next request
537 Socket 77 662* 275 x.x.x.x:1392 Idle client: Waiting for next request
538 Socket 0 76434 486358 x.x.x.x:1344 x.x.x.x
539 Socket 898 445876* 531378 x.x.x.x:1344 x.x.x.x
540 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (203)
541 Socket 0 69878* 1098 x.x.x.x:443 https://domain.org
542 Socket 46 384* 612 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
543 Socket 0 636* 794 x.x.x.x:443 https://domain.org
544 Socket 0 2182761 2218810 x.x.x.x:1344 x.x.x.x
545 Socket 46 150988* 321272 x.x.x.x:1344 x.x.x.x
546 Socket 0 68503 78530 x.x.x.x:1344 x.x.x.x
547 Socket 896 1598* 6692 x.x.x.x:443 https://domain.org
548 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (13)
549 Socket 900 0* 866 x.x.x.x:443 https://domain.org
550 Socket 28799 0 0 x.x.x.x:3203 client https start
551 Socket 59 302774* 595817 x.x.x.x:1344 x.x.x.x
552 Socket 0 515* 1112 x.x.x.x:443 https://domain.org
553 Socket 7 395* 286 x.x.x.x:61191 Idle client: Waiting for next request
554 Socket 0 438847 475815 x.x.x.x:1344 x.x.x.x
555 Socket 0 1666817 2225879 x.x.x.x:1344 x.x.x.x
556 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (625)
557 Socket 0 87652 250185 x.x.x.x:1344 x.x.x.x
558 Socket 0 888616 1629204 x.x.x.x:1344 x.x.x.x
559 Socket 878 2012* 15232 x.x.x.x:443 https://domain.org
560 Socket 85 201* 3316 x.x.x.x:49698 Idle client: Waiting for next request
561 Socket 28774 2910* 1058 x.x.x.x:64166 Reading next request
562 Socket 898 1211* 6180 x.x.x.x:443 https://domain.org
563 Socket 97 940* 1308 x.x.x.x:3376 Idle client: Waiting for next request
564 Socket 0 275* 4748 x.x.x.x:443 https://domain.org
565 Socket 15 1446* 1306 x.x.x.x:51817 Reading next request
566 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (324)
567 Socket 0 594802 827936 x.x.x.x:1344 x.x.x.x
568 Socket 0 303279 409733 x.x.x.x:1344 x.x.x.x
569 Socket 117 1849* 615 x.x.x.x:2304 Idle client: Waiting for next request
570 Socket 0 3110* 7764 x.x.x.x:443 https://domain.org
571 Socket 0 75913* 9544 x.x.x.x:443 https://domain.org
572 Socket 57 198* 286 x.x.x.x:55768 Idle client: Waiting for next request
573 Socket 27 2763* 51668 x.x.x.x:62577 Idle client: Waiting for next request
574 Socket 0 1025* 2184 x.x.x.x:443 https://domain.org
575 Socket 99 11511* 192197 x.x.x.x:50483 Idle client: Waiting for next request
576 Socket 0 0* 0 ::1:37181 ext_wbinfo_group_acl #3
577 Socket 113 1181* 629 x.x.x.x:51203 Idle client: Waiting for next request
578 Socket 0 125227 327390 x.x.x.x:1344 x.x.x.x
579 Socket 0 80821 222391 x.x.x.x:1344 x.x.x.x
580 Socket 28798 0 0 x.x.x.x:1841 client https start
581 Socket 28798 0 0 x.x.x.x:2321 client https start
582 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (396)
583 Socket 0 0* 0 ::1:39025 ext_wbinfo_group_acl #4
584 Socket 28798 0 0 x.x.x.x:2322 client https start
585 Socket 0 67261 67974 x.x.x.x:1344 x.x.x.x
586 Socket 900 192830* 7818 x.x.x.x:443 https://domain.org
587 Socket 0 464123 1899880 x.x.x.x:1344 x.x.x.x
588 Socket 0 74290 182755 x.x.x.x:1344 x.x.x.x
589 Socket 0 647* 1728 x.x.x.x:443 https://domain.org
590 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (145)
591 Socket 28799 0 0 x.x.x.x:2323 client https start
592 Socket 28799 0 0 x.x.x.x:3201 client https start
593 Pipe 0 18335* 25554 ssl_crtd #7
594 Socket 91 292* 452 x.x.x.x:52893 Idle client: Waiting for next request
595 Socket 0 302843 1136496 x.x.x.x:1344 x.x.x.x
596 Socket 28800 0 0 x.x.x.x:2386 client https start
597 Socket 0 86639 207373 x.x.x.x:1344 x.x.x.x
598 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (279)
599 Socket 0 0* 0 ::1:51769 ext_wbinfo_group_acl #5
600 Socket 0 71364 312248 x.x.x.x:1344 x.x.x.x
601 Socket 0 919397 1075737 x.x.x.x:1344 x.x.x.x
602 Socket 0 180943 1358113 x.x.x.x:1344 x.x.x.x
603 Socket 28799 0 0 x.x.x.x:3214 client https start
604 Socket 0 692* 1178 x.x.x.x:443 https://domain.org
605 Socket 62 7966* 12814 x.x.x.x:1230 Idle client: Waiting for next request
606 Socket 0 638* 7316 x.x.x.x:443 https://domain.org
607 Socket 23 306* 452 x.x.x.x:1767 Idle client: Waiting for next request
608 Socket 28799 0 0 x.x.x.x:3205 client https start
609 Socket 0 85591 164291 x.x.x.x:1344 x.x.x.x
610 Socket 0 692* 1198 x.x.x.x:443 https://domain.org
611 Socket 28785 0* 0 x.x.x.x:3183 client https start
612 Socket 0 692* 1178 x.x.x.x:443 https://domain.org
613 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (134)
614 Socket 900 28870* 195741* x.x.x.x:1344 x.x.x.x
615 Socket 28754 0 0 x.x.x.x:55092 client https start
616 Socket 118 390* 36111 x.x.x.x:2319 Idle client: Waiting for next request
617 Socket 98 555* 2252 x.x.x.x:60769 Idle client: Waiting for next request
618 Socket 299 0* 0 x.x.x.x:1828 client http connect
619 Socket 0 473529 793634 x.x.x.x:1344 x.x.x.x
620 Socket 900 450715* 716503 x.x.x.x:1344 x.x.x.x
621 Socket 28799 0 0 x.x.x.x:3202 client https start
622 Socket 0 692* 1184 x.x.x.x:443 https://domain.org
623 Socket 874 16754* 21876 x.x.x.x:443 https://domain.org
624 Socket 28788 4986* 1596 x.x.x.x:49406 Reading next request
625 Socket 98 482* 654 x.x.x.x:64171 Idle client: Waiting for next request
626 Pipe 0 14687* 22194 ssl_crtd #8
627 Socket 59 210* 2635 x.x.x.x:1344 x.x.x.x
628 Socket 0 134507 1462114 x.x.x.x:1344 x.x.x.x
629 Socket 0 108885* 2533583* ::1:34453 squid_url_lookup.pl #1
630 Socket 0 72439 150972 x.x.x.x:1344 x.x.x.x
631 Socket 28799 0 0 x.x.x.x:3204 client https start
632 Socket 879 734* 6486 x.x.x.x:443 https://domain.org
633 Socket 0 1051410 1201280 x.x.x.x:1344 x.x.x.x
634 Socket 28799 0 0 x.x.x.x:3211 client https start
635 Socket 879 734* 6648 x.x.x.x:443 https://domain.org
636 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (611)
637 Socket 0 163962 604624 x.x.x.x:1344 x.x.x.x
638 Socket 0 48696* 856 x.x.x.x:443 https://domain.org
639 Socket 0 729827 1196381 x.x.x.x:1344 x.x.x.x
640 Socket 0 232516 347567 x.x.x.x:1344 x.x.x.x
641 Socket 28798 3754* 128113 x.x.x.x:2303 Reading next request
642 Socket 0 325777 602015 x.x.x.x:1344 x.x.x.x
643 Socket 28799 0 0 x.x.x.x:3213 client https start
644 Socket 18 397* 265 x.x.x.x:1258 Idle client: Waiting for next request
645 Socket 99 3596* 822 x.x.x.x:54417 Idle client: Waiting for next request
646 Pipe 0 14180* 21549 ssl_crtd #9
647 Socket 0 571668 1758812 x.x.x.x:1344 x.x.x.x
648 Socket 0 252* 3372 x.x.x.x:443 https://domain.org
649 Socket 0 68228 116478 x.x.x.x:1344 x.x.x.x
650 Socket 0 582* 1818 x.x.x.x:443 https://domain.org
651 Pipe 0 7520* 11022 ssl_crtd #10
652 Socket 90 435* 285 x.x.x.x:3386 Idle client: Waiting for next request
653 Socket 28799 0 0 x.x.x.x:3215 client https start
654 Socket 99 1110* 2214 x.x.x.x:60771 Idle client: Waiting for next request
655 Socket 28800 0 0 x.x.x.x:3218 client https connect
656 Socket 0 882* 1880 x.x.x.x:443 https://domain.org
657 Socket 59 0 0* x.x.x.x:80 domain.org
658 Socket 0 1283* 4442 x.x.x.x:443 https://domain.org
659 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (807)
660 Socket 90 435* 285 x.x.x.x:3384 Idle client: Waiting for next request
661 Socket 28800 0 0 x.x.x.x:3224 client https start
662 Socket 0 365673 608482 x.x.x.x:1344 x.x.x.x
663 Socket 0 81625 110628 x.x.x.x:1344 x.x.x.x
664 Socket 28800 156* 0* ::1:35046 Reading next request
665 Socket 97 2215* 1279 x.x.x.x:51849 Idle client: Waiting for next request
666 Socket 90 435* 285 x.x.x.x:3385 Idle client: Waiting for next request
667 Socket 0 620* 1220 x.x.x.x:443 https://domain.org
668 Socket 28800 201* 0* x.x.x.x:50010 Reading next request
669 Socket 0 136145 440062 x.x.x.x:1344 x.x.x.x
670 Socket 0 17564* 323089* ::1:33603 squid_url_lookup.pl #2
671 Socket 91 2348* 343 x.x.x.x:60757 Idle client: Waiting for next request
672 Socket 118 421* 17488 x.x.x.x:51046 Idle client: Waiting for next request
673 Socket 28800 0 0 x.x.x.x:3228 client https start
674 Socket 0 168525* 3042 x.x.x.x:443 https://domain.org
675 Socket 0 449451 1179110 x.x.x.x:1344 x.x.x.x
676 Socket 0 81705 176401 x.x.x.x:1344 x.x.x.x
677 Socket 28800 0 0 x.x.x.x:3229 client https connect
678 Socket 0 607338 804760 x.x.x.x:1344 x.x.x.x
679 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (417)
680 Socket 0 7879* 141662 ::1:59195 squid_url_lookup.pl #3
681 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (387)
682 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (393)
683 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (70)
684 Socket 28800 5112* 2235 x.x.x.x:1838 Reading next request
685 Socket 0 586* 990 x.x.x.x:443 https://domain.org
686 Socket 99 482* 654 x.x.x.x:60032 Idle client: Waiting for next request
687 Socket 0 1065* 20404 x.x.x.x:443 https://domain.org
688 Socket 0 4874* 61424 ::1:48333 squid_url_lookup.pl #4
689 Socket 0 225785 342266 x.x.x.x:1344 x.x.x.x
690 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (484)
691 Socket 0 247595 638177 x.x.x.x:1344 x.x.x.x
692 Socket 0 544* 1104 x.x.x.x:443 https://domain.org
693 Socket 0 70387 70862 x.x.x.x:1344 x.x.x.x
694 Socket 27680 0* 0 x.x.x.x:56755 client https start
695 Socket 300 0* 0 x.x.x.x:60802 client http connect
696 Socket 0 0 0 x.x.x.x:3231 HTTP Request
697 Socket 0 67261 67986 x.x.x.x:1344 x.x.x.x
698 Socket 0 1129* 2870 x.x.x.x:443 https://domain.org
700 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (462)
702 Socket 18 402* 266 x.x.x.x:1262 Idle client: Waiting for next request
703 Socket 9 1338* 634 x.x.x.x:80 Idle server: x.x.x.x:80/some_path
704 Socket 58 491* 682630 x.x.x.x:62581 Idle client: Waiting for next request
705 Socket 0 67626 91631 x.x.x.x:1344 x.x.x.x
706 Socket 0 67780 149096 x.x.x.x:1344 x.x.x.x
707 Socket 0 93177 639083 x.x.x.x:1344 x.x.x.x
708 Socket 0 875386 1172221 x.x.x.x:1344 x.x.x.x
709 Socket 876 588* 3324 x.x.x.x:443 https://domain.org
710 Socket 19 444* 384 x.x.x.x:1265 Idle client: Waiting for next request
711 Socket 19 425* 384 x.x.x.x:1266 Idle client: Waiting for next request
712 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (218)
716 Socket 57 420* 158 x.x.x.x:62586 Idle client: Waiting for next request
717 Socket 0 1491* 1066 x.x.x.x:443 https://domain.org
719 Socket 0 169746* 21748 x.x.x.x:443 https://domain.org
720 Socket 0 17102* 2910 x.x.x.x:443 https://domain.org
721 Socket 0 3279* 43331 ::1:42983 squid_url_lookup.pl #5
722 Socket 0 256289 269402 x.x.x.x:1344 x.x.x.x
723 Socket 0 7714* 2384 x.x.x.x:443 https://domain.org
724 Socket 0 1830* 24920 ::1:47149 squid_url_lookup.pl #6
725 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (327)
726 Socket 0 1063154 1790141 x.x.x.x:1344 x.x.x.x
727 Socket 0 616* 13608 ::1:48873 squid_url_lookup.pl #7
729 Socket 18 329* 3506 x.x.x.x:1267 Idle client: Waiting for next request
730 Socket 0 67260 67952 x.x.x.x:1344 x.x.x.x
731 Socket 0 32251* 13544 x.x.x.x:443 https://domain.org
732 Socket 0 12614* 16036 x.x.x.x:443 https://domain.org
733 Socket 0 298* 5090 ::1:59995 squid_url_lookup.pl #8
734 Socket 0 638* 3252 x.x.x.x:443 https://domain.org
735 Socket 0 252* 2900 x.x.x.x:443 https://domain.org
736 Socket 900 2405* 10516 x.x.x.x:443 https://domain.org
737 Socket 0 241* 2542 ::1:44247 squid_url_lookup.pl #9
738 Socket 0 256065 1016479 x.x.x.x:1344 x.x.x.x
740 Socket 0 69188 114543 x.x.x.x:1344 x.x.x.x
741 Socket 97 940* 1308 x.x.x.x:3379 Idle client: Waiting for next request
742 Socket 900 41281* 1514 x.x.x.x:443 https://domain.org
743 Socket 0 681278 1185273 x.x.x.x:1344 x.x.x.x
744 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (306)
745 Socket 900 1082* 10628 x.x.x.x:443 https://domain.org
746 Socket 0 5718* 9764 x.x.x.x:443 https://domain.org
747 Socket 0 620* 1196 x.x.x.x:443 https://domain.org
748 Socket 0 207* 2293 ::1:33591 squid_url_lookup.pl #10
749 Socket 0 0 0 [::]:3128 HTTP Socket
750 Socket 876 592* 3568 x.x.x.x:443 https://domain.org
751 Socket 0 0 0 [::]:3129 HTTP Socket
752 Socket 0 71255 145856 x.x.x.x:1344 x.x.x.x
753 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (569)
754 Socket 0 0 0 [::]:3130 HTTPS Socket
755 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (490)
756 Socket 0 70375 184071 x.x.x.x:1344 x.x.x.x
758 Socket 0 68918 187587 x.x.x.x:1344 x.x.x.x
760 Socket 0 132072 189220 x.x.x.x:1344 x.x.x.x
762 Socket 0 367* 1280 x.x.x.x:443 https://domain.org
763 Socket 0 74474 206574 x.x.x.x:1344 x.x.x.x
764 Socket 87 482* 654 x.x.x.x:49665 Idle client: Waiting for next request
765 Socket 0 0 0 ::1:50813 pinger
766 File 0 0 993744 /var/cache/squid/swap.state
767 Socket 893 2800* 9984 x.x.x.x:443 https://domain.org
769 Socket 0 537* 2414 x.x.x.x:443 https://domain.org
770 Socket 0 628* 3372 x.x.x.x:443 https://domain.org
771 Socket 0 306851 815547 x.x.x.x:1344 x.x.x.x
772 Socket 56 516* 676 x.x.x.x:54415 Idle client: Waiting for next request
773 Socket 0 620* 1196 x.x.x.x:443 https://domain.org
774 Socket 0 43044* 1098 x.x.x.x:443 https://domain.org
775 Socket 0 2458* 568 x.x.x.x:443 https://domain.org
776 Socket 0 173* 794 x.x.x.x:443 https://domain.org
777 Socket 0 131967 194732 x.x.x.x:1344 x.x.x.x
778 Socket 0 30285* 1720 x.x.x.x:443 https://domain.org
780 Socket 898 1213* 6800 x.x.x.x:443 https://domain.org
781 Socket 900 2103* 10472 x.x.x.x:443 https://domain.org
782 Socket 0 3627* 14436 x.x.x.x:443 https://domain.org
783 Socket 0 174* 804 x.x.x.x:443 https://domain.org
784 Socket 0 199486 463497 x.x.x.x:1344 x.x.x.x
785 Socket 0 1099* 4762 x.x.x.x:443 https://domain.org
787 Socket 0 316* 888 x.x.x.x:443 https://domain.org
790 Socket 0 2078* 2324 x.x.x.x:443 https://domain.org
791 Socket 0 316* 850 x.x.x.x:443 https://domain.org
792 Socket 0 2184* 1162 x.x.x.x:443 https://domain.org
793 Socket 0 2184* 1162 x.x.x.x:443 https://domain.org
794 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (202)
796 Socket 0 1290186 1774914 x.x.x.x:1344 x.x.x.x
797 Socket 0 67263 67949 x.x.x.x:1344 x.x.x.x
798 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (160)
799 Socket 99 7140* 3811 x.x.x.x:54406 Idle client: Waiting for next request
800 Socket 0 70622 74503 x.x.x.x:1344 x.x.x.x
801 Socket 27 2355* 1251 x.x.x.x:54407 Idle client: Waiting for next request
802 Socket 0 1154* 1136 x.x.x.x:443 https://domain.org
803 Socket 0 69578 75080 x.x.x.x:1344 x.x.x.x
804 Socket 0 1049* 1160 x.x.x.x:443 https://domain.org
805 Socket 0 976* 1136 x.x.x.x:443 https://domain.org
806 Socket 0 1261* 4482 x.x.x.x:443 https://domain.org
808 Socket 885 589* 3318 x.x.x.x:443 https://domain.org
809 Socket 0 1317* 4482 x.x.x.x:443 https://domain.org
810 Socket 900 82342* 866 x.x.x.x:443 https://domain.org
811 Socket 0 719* 1346 x.x.x.x:443 https://domain.org
812 Socket 0 1113* 1342 x.x.x.x:443 https://domain.org
814 Socket 855 2387* 13320 x.x.x.x:443 https://domain.org
815 Socket 27 500* 286 x.x.x.x:55764 Idle client: Waiting for next request
816 Socket 0 1080317 1867520 x.x.x.x:1344 x.x.x.x
820 Socket 66 1192* 7898 x.x.x.x:4110 Idle client: Waiting for next request
821 Socket 0 1025* 26932 x.x.x.x:443 https://domain.org
822 Socket 0 86033 89065 x.x.x.x:1344 x.x.x.x
823 Socket 59 326800* 558361 x.x.x.x:1344 x.x.x.x
824 Socket 893 5922* 5026 x.x.x.x:443 https://domain.org
826 Socket 0 1317411 1355916 x.x.x.x:1344 x.x.x.x
827 Pipe 0 7431* 9205 ssl_crtd #4
828 Socket 895 6112* 8018 x.x.x.x:443 https://domain.org
829 Socket 0 67263 68003 x.x.x.x:1344 x.x.x.x
830 Socket 0 13653* 11470 x.x.x.x:443 https://domain.org
831 Pipe 0 4461* 5937 ssl_crtd #5
832 Socket 0 98682 230913 x.x.x.x:1344 x.x.x.x
833 Socket 0 18285* 8984 x.x.x.x:443 https://domain.org
834 Socket 0 2769082 3379609 x.x.x.x:1344 x.x.x.x
835 Socket 0 5671* 8792 x.x.x.x:443 https://domain.org
837 Socket 0 6512* 11814 x.x.x.x:443 https://domain.org
838 Pipe 0 4461* 5937 ssl_crtd #6
839 Socket 0 252* 2606 x.x.x.x:443 https://domain.org
840 Socket 0 67366 74962 x.x.x.x:1344 x.x.x.x
842 Socket 0 71312 221653 x.x.x.x:1344 x.x.x.x
846 Pipe 0 4461* 5937 ssl_crtd #7
847 Socket 0 225360 346218 x.x.x.x:1344 x.x.x.x
849 Pipe 0 4461* 5937 ssl_crtd #8
850 Socket 0 8137* 1042 x.x.x.x:443 https://domain.org
851 Socket 885 591* 2784 x.x.x.x:443 https://domain.org
852 Pipe 0 0* 0 ssl_crtd #9
854 Socket 0 3983* 6570 x.x.x.x:443 https://domain.org
857 Pipe 0 0* 0 ssl_crtd #10
869 Socket 0 40620* 32966 x.x.x.x:443 https://domain.org
870 Socket 0 4328494 4704878 x.x.x.x:1344 x.x.x.x
878 Socket 0 70812 1451399 x.x.x.x:1344 x.x.x.x
885 Socket 0 646465 1098283 x.x.x.x:1344 x.x.x.x
891 Socket 0 949722 1844840 x.x.x.x:1344 x.x.x.x
904 Socket 888 1837* 10128 x.x.x.x:443 https://domain.org
909 Socket 0 85268 151920 x.x.x.x:1344 x.x.x.x
911 Socket 831 0* 2354 x.x.x.x:443 https://domain.org
913 Socket 0 70315 70855 x.x.x.x:1344 x.x.x.x

I don't get it. There aren't that many active file descriptors (less than 1000).
Also, on low traffic (late evening) with no CPU or mem issues at all, mgr:filedescriptors shows that there are about 2500 active file descriptors.

The only two things I can think of that could mess with Squid are:

- squid_url_lookup.pl which is an external_acl_type perl script that does a MySQL lookup (ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8)

- c-icap content scanning with clamav and libarchive

If push comes to shove I might consider temporarily disabling one or both just to see what happens, but I'm not sure if that can really be the cause of a Squid process coming down to a crawl.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Alex Rousskov

Re: squid cpu usage

On 10/26/2017 03:58 PM, Vieri wrote:

> I should've done this before, but I just took a second look at
> cache.log and found the following message repeated a few times before
> the 100% CPU issue:

> "Your cache is running out of filedescriptors"

That warning matches the dangerously low number of available descriptor
reported by your mgr:info:

> File descriptor usage for squid:
> Maximum number of file descriptors: 1024
> Largest file desc currently in use: 913
> Number of file desc currently in use: 791
> Files queued for open: 0
> Available number of file descriptors: 233
> Reserved number of file descriptors: 100

> If this is the cause then I'm even more worried now because:
>
> open files (-n) 16384

> # cat /etc/security/limits.d/01_squid.conf
> * hard nofile 65535
> * soft nofile 16384

Your Squid does not know about the above configured limits. It thinks
the limit is 1024.

> I don't get it. There aren't that many active file descriptors (less than 1000).

~900 descriptors is "a lot" when the limit is 1024.

> Also, on low traffic (late evening) with no CPU or mem issues at all,
> mgr:filedescriptors shows that there are about 2500 active file
> descriptors.

Please note that "2500 active descriptors" contradicts the 1024 limit
reported in mgr:info in your earlier email (quoted by me above). Squid
also reports the limit at startup. Here is an example (not from your logs):

> 2017/10/26 15:48:04.744| With 65535 file descriptors available

There are many email threads and probably some wiki documents that
discuss increasing the number of file descriptors available to your
Squid and numerous related caveats.

Cheers,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Vieri

Re: squid cpu usage

> From: Alex Rousskov <[hidden email]>
>
>> # cat /etc/security/limits.d/01_squid.conf
>> * hard nofile 65535
>> * soft nofile 16384
>
> Your Squid does not know about the above configured limits. It thinks the limit is 1024.

That's interesting.

# grep "file descriptors available" cache.log
2017/10/24 11:49:33 kid1| With 1024 file descriptors available
2017/10/24 12:57:10 kid1| With 16384 file descriptors available
2017/10/24 12:57:52 kid1| With 16384 file descriptors available
2017/10/25 05:50:32 kid1| With 1024 file descriptors available
2017/10/25 09:20:15 kid1| With 1024 file descriptors available
2017/10/25 10:00:15 kid1| With 1024 file descriptors available
2017/10/25 12:00:15 kid1| With 1024 file descriptors available
2017/10/25 13:00:16 kid1| With 1024 file descriptors available
2017/10/25 13:20:16 kid1| With 1024 file descriptors available
2017/10/25 14:00:15 kid1| With 1024 file descriptors available
2017/10/26 04:09:30 kid1| With 1024 file descriptors available
2017/10/26 10:57:45 kid1| With 16384 file descriptors available

The issues I reported yesterday came up after 2017/10/26 04:09:30, so here's the relevant part in the log:

2017/10/26 04:09:27 kid1| Reconfiguring Squid Cache (version 3.5.27-20170916-re69e56c)...
2017/10/26 04:09:27 kid1| Closing HTTP port [::]:3128
2017/10/26 04:09:27 kid1| Closing HTTP port [::]:3129
2017/10/26 04:09:27 kid1| Closing HTTPS port [::]:3130
2017/10/26 04:09:27 kid1| Closing Pinger socket on FD 801
2017/10/26 04:09:27 kid1| Logfile: closing log daemon:/var/log/squid/access.log
2017/10/26 04:09:27 kid1| Logfile Daemon: closing log daemon:/var/log/squid/access.log
2017/10/26 04:09:27 kid1| FATAL: dying from an unhandled exception: IsConnOpen(conn)
2017/10/26 04:09:30| Pinger exiting.
2017/10/26 04:09:30 kid1| Set Current Directory to /var/cache/squid
2017/10/26 04:09:30 kid1| Starting Squid Cache version 3.5.27-20170916-re69e56c for x86_64-pc-linux-gnu...
2017/10/26 04:09:30 kid1| Service Name: squid
2017/10/26 04:09:30 kid1| Process ID 22581
2017/10/26 04:09:30 kid1| Process Roles: worker
2017/10/26 04:09:30 kid1| With 1024 file descriptors available

A "reload" was issued at that time.

When my openrc init script reloads, it runs the following command:

# /etc/init.d/squid reload
* Reloading squid with /usr/sbin/squid -k reconfigure -f /etc/squid/squid.conf -n squid ... [ ok ]

I issued another "reload" today, and I didn't come up with any "file descriptors available" messages in cache.log (as expected).
I also checked mgr:info, and all seems OK:

File descriptor usage for squid:
Maximum number of file descriptors: 16384
Largest file desc currently in use: 3067
Number of file desc currently in use: 2901
Files queued for open: 0
Available number of file descriptors: 13483
Reserved number of file descriptors: 100
Store Disk files open: 0

I also grep'ed for the log lines around "2017/10/25 05:50:32 kid1| With 1024 file descriptors available" where I see 1024 file descriptors for the first time.
It was a clean restart (with no "FATAL" error message).
The "restart" command was launched from crontab as a daily job (calling the openrc init script) just like it is launched at boot time from openrc (when the number of file descriptors is "as expected").

In particular, I have: KRB5_KTNAME="/etc/krb5.keytab" /usr/sbin/squid -YC -f /etc/squid/squid.conf -n squid

I'm wondering why Squid only detected 1024 file descriptors available after that restart.

Moving back in time:

- "2017/10/24 12:57:52 kid1| With 16384 file descriptors available" was probably a manual restart on my behalf (passing "restart" to the init script on the console).

- the one right before (2017/10/24 12:57:10) was probably also a manual restart as the init script probably exited non-zero due to an error I found in the log: "FATAL: dying from an unhandled exception: IsConnOpen(conn)"

- moving back a little (11:49:30), I had a "reload" with a fatal error:

2017/10/24 11:49:30.406 kid1| Reconfiguring Squid Cache (version 3.5.27-20170916-re69e56c)...
2017/10/24 11:49:30.406 kid1| Closing HTTP port [::]:3128
2017/10/24 11:49:30.406 kid1| Closing HTTP port [::]:3129
2017/10/24 11:49:30.406 kid1| Closing HTTPS port [::]:3130
2017/10/24 11:49:30.406 kid1| Closing Pinger socket on FD 788
2017/10/24 11:49:30.406 kid1| Logfile: closing log daemon:/var/log/squid/access.log
2017/10/24 11:49:30.406 kid1| Logfile Daemon: closing log daemon:/var/log/squid/access.log
2017/10/24 11:49:30.406 kid1| FATAL: dying from an unhandled exception: IsConnOpen(conn)
2017/10/24 11:49:33 kid1| Set Current Directory to /var/cache/squid
2017/10/24 11:49:33 kid1| Starting Squid Cache version 3.5.27-20170916-re69e56c for x86_64-pc-linux-gnu...
2017/10/24 11:49:33 kid1| Service Name: squid
2017/10/24 11:49:33 kid1| Process ID 30753
2017/10/24 11:49:33 kid1| Process Roles: worker
2017/10/24 11:49:33 kid1| With 1024 file descriptors available

- searching within the older rotated cache.log files I noticed this:

2017/10/20 10:42:12 kid1| With 16384 file descriptors available
2017/10/21 05:50:32 kid1| With 1024 file descriptors available

The restart at 05:50:32 is performed by a cron job.

Conclusion: it seems that the cron daemon I use (https://github.com/cronie-crond/cronie) is not honoring system-wide ulimits defined in /etc/security/limits.d/* as, eg.:
* hard nofile 65535
* soft nofile 16384

Weird.

I'm not going to be using the cron job anymore.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Vieri

Re: squid cpu usage

In reply to this post by Alex Rousskov

Just in case someone else bumps into this issue, I can confirm that ulimits are not honored when running a cron job (at least from cronie).

In my case, if I set ulimits globally with:
* hard nofile 65535
* soft nofile 16384

I get this outside cron:
open files (-n) 16384

However, I get the following within cron:

open files (-n) 1024

as demonstrated by running the following test from crontab:

root ulimit -a >/tmp/ulimit 2>&1 3>&1

So beware.

I guess one can avoid this by setting ulimits within Squid's init script, and calling that within cron.

No idea whether this is a cron bug or feature.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Vieri

Re: squid cpu usage

In reply to this post by Alex Rousskov

Just in case someone else bumps into this, I've had the same issue trying to restart Squid from inittab. System-wide ulimit settings are not honored when running scripts from either crontab or inittab.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Vieri

Re: squid cpu usage

In reply to this post by Alex Rousskov

It's seems logical that the cron daemon needs to be restarted for ulimits to apply.
Init should also be restarted, but "init u" is not enough. A system reboot seems to be inevitable.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users