squid cpu usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

squid cpu usage

Vieri
Hi,

During the last 3 days my Squid process has gone wild with 100% CPU usage once a day. Since this is a production system (never had this issue before), I "quickly restart" the Squid daemon, and that solves the issue until the next day.
Restarting takes quite a while...

Next time I get a 100% squid process that brings my proxy to a crawl HTTP-wise, what can I try in order to get more info, and possibly see the cause of this?
Would squidclient mgr:info shed some light on this?

Squid Object Cache: Version 3.5.27-20170916-re69e56c

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Amos Jeffries
Administrator
On 25/10/17 01:34, Vieri wrote:
> Hi,
>
> During the last 3 days my Squid process has gone wild with 100% CPU usage once a day. Since this is a production system (never had this issue before), I "quickly restart" the Squid daemon, and that solves the issue until the next day.
> Restarting takes quite a while...
>
> Next time I get a 100% squid process that brings my proxy to a crawl HTTP-wise, what can I try in order to get more info, and possibly see the cause of this?
> Would squidclient mgr:info shed some light on this?

That one is just the overview, so not likely to show more than that
things are going slowly.

Usually the 100% CPU is an indicator that the machine is swapping. So
check the memory usage when it is happening.

Next is probably the cache garbage collection. mgr:storedir report might
show that. Or an strace showing a lot of cycles in the cache replacement
calls.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Alex Rousskov
In reply to this post by Vieri
On 10/24/2017 06:34 AM, Vieri wrote:

> Next time I get a 100% squid process that brings my proxy to a crawl
> HTTP-wise, what can I try in order to get more info, and possibly see
> the cause of this?

Here are two additional tricks to consider:

* If 100% CPU usage lasts for more than a few seconds, then attaching
gdb to the running Squid worker and generating a few backtraces (a few
seconds apart) may identify the code that got stuck. Make sure you give
gdb a non-stripped Squid binary when doing that. Squid wiki should have
more instructions about the whole process.

Please note that the above is less likely to help if your Squid
continues to respond to new requests while you see ~100% CPU usage. You
may increase your chances by blocking new traffic from reaching the
problematic Squid (e.g., by redirecting all traffic to other Squid(s)
that you have in load balancing or hot standby mode OR allowing all
traffic to bypass Squid). If you cannot easily remove a problematic
Squid from user traffic path, then use this problem as an excuse to fix
your deployment architecture so that you can :-).

* Another potentially useful trick is to log mgr:events every minute or
so. Perhaps 100% CPU usage is associated with a particular regularly
scheduled event?

Please note that some modern Squids abuse the events API for frequently
occurring notifications. If you see hundreds or more events in your log,
this trick may be too expensive/intrusive for your environment.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Vieri
Well, I've just had another issue with my CPU usage gone bonkers.

Here's the strace (compressed with gz):

https://drive.google.com/file/d/0B-tpkY1LkI67WFBaNnVpR2lteEE/view?usp=sharing

There are a lot of "Resource temporarily unavailable" messages.
However, I'm seeing these messages even when my Squid process has been restarted, and everything seems to be OK from a "user experience" point of view.
Do you see anything useful in the trace?
If not, how can I improve the trace next time I have this issue?

Here's what squidclient has to say during the "storm":

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:34:43 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:34:43 GMT
Last-Modified: Thu, 26 Oct 2017 08:34:43 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20170916-re69e56c
Build Info:
Service Name: squid
Start Time: Thu, 26 Oct 2017 02:09:30 GMT
Current Time: Thu, 26 Oct 2017 08:34:43 GMT
Connection information for squid:
Number of clients accessing cache: 471
Number of HTTP requests received: 267968
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 695.6
Average ICP messages per minute since start: 0.0
Select loop called: 10630162 times, 2.174 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 2.3%, 60min: 2.5%
Hits as % of bytes sent: 5min: 10.6%, 60min: -4.4%
Memory hits as % of hit requests: 5min: 22.1%, 60min: 36.5%
Disk hits as % of hit requests: 5min: 4.3%, 60min: 0.3%
Storage Swap size: 29484 KB
Storage Swap capacity: 90.0% used, 10.0% free
Storage Mem size: 28492 KB
Storage Mem capacity: 87.0% used, 13.0% free
Mean Object Size: 22.29 KB
Requests given to unlinkd: 17424
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   2.64709  0.27332
Cache Misses:          1.54242  0.27332
Cache Hits:            0.64968  0.00000
Near Hits:             1.31166  0.22004
Not-Modified Replies:  0.44492  0.00000
DNS Lookups:           0.12472  0.07968
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time: 23113.055 seconds
CPU Time: 1599.750 seconds
CPU Usage: 6.92%
CPU Usage, 5 minute avg: 99.80%
CPU Usage, 60 minute avg: 32.48%
Maximum Resident Size: 987392 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted:       148456 KB
memPoolAlloc calls:  87698086
memPoolFree calls:   89179152
File descriptor usage for squid:
Maximum number of file descriptors:   1024
Largest file desc currently in use:    913
Number of file desc currently in use:  791
Files queued for open:                   0
Available number of file descriptors:  233
Reserved number of file descriptors:   100
Store Disk files open:                   0
Internal Data Structures:
1642 StoreEntries
1640 StoreEntries with MemObjects
1375 Hot Object Cache Items
1323 on-disk objects

A little later (still during the "storm"):

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:40:13 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:40:13 GMT
Last-Modified: Thu, 26 Oct 2017 08:40:13 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20170916-re69e56c
Build Info:
Service Name: squid
Start Time: Thu, 26 Oct 2017 02:09:30 GMT
Current Time: Thu, 26 Oct 2017 08:40:13 GMT
Connection information for squid:
Number of clients accessing cache: 472
Number of HTTP requests received: 274183
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 701.8
Average ICP messages per minute since start: 0.0
Select loop called: 10679512 times, 2.195 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 0.2%, 60min: 2.3%
Hits as % of bytes sent: 5min: -3.2%, 60min: -3.6%
Memory hits as % of hit requests: 5min: 9.1%, 60min: 35.2%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.4%
Storage Swap size: 29484 KB
Storage Swap capacity: 90.0% used, 10.0% free
Storage Mem size: 27304 KB
Storage Mem capacity: 83.3% used, 16.7% free
Mean Object Size: 16.25 KB
Requests given to unlinkd: 17424
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   1.05672  0.30459
Cache Misses:          0.94847  0.30459
Cache Hits:            0.00000  0.00000
Near Hits:             0.58309  0.23230
Not-Modified Replies:  0.00000  0.00000
DNS Lookups:           0.11405  0.08334
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time: 23442.516 seconds
CPU Time: 1888.520 seconds
CPU Usage: 8.06%
CPU Usage, 5 minute avg: 86.51%
CPU Usage, 60 minute avg: 39.31%
Maximum Resident Size: 988768 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted:       151913 KB
memPoolAlloc calls:  89548742
memPoolFree calls:   91062557
File descriptor usage for squid:
Maximum number of file descriptors:   1024
Largest file desc currently in use:    913
Number of file desc currently in use:  814
Files queued for open:                   0
Available number of file descriptors:  210
Reserved number of file descriptors:   100
Store Disk files open:                   0
Internal Data Structures:
2139 StoreEntries
2135 StoreEntries with MemObjects
1863 Hot Object Cache Items
1814 on-disk objects

Memory info:

TTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:34:51 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:34:51 GMT
Last-Modified: Thu, 26 Oct 2017 08:34:51 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Current memory usage:
Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate
(bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec
64K Buffer           65536 1366 87424 88064 0.03 58.871 1340 85760 88064 0.03 98.097 26 1664 5248 283321 0.317 35.761 21.750
mem_node             4136 7326 29591 30843 1.38 19.926 7096 28662 30843 1.38 96.860 230 929 6802 558119 0.624 4.446 48.467
32K Buffer           32768 129 4128 5984 0.77 2.780 123 3936 5984 0.77 95.349 6 192 2272 49708 0.056 3.137 4.285
ExternalACLEntry      256 16447 4112 4953 4.32 2.769 16447 4112 4953 4.32 100.000 0 0 4953 591 0.001 0.000 5.834
16K Buffer           16384 189 3024 6992 4.35 2.036 167 2672 6992 4.35 88.360 22 352 3152 223171 0.250 7.042 16.461
8K Buffer           8192 368 2944 3464 0.17 1.982 347 2776 3464 0.17 94.293 21 168 1112 288150 0.322 4.546 27.526
Short Strings         40 73543 2873 3066 0.57 1.935 73511 2872 3066 0.57 99.956 32 2 867 50208368 56.164 3.868 3944.297
4K Buffer           4096 497 1988 2696 1.51 1.339 466 1864 2696 1.51 93.763 31 124 1012 397819 0.445 3.138 37.339
HttpHeaderEntry       56 30961 1694 1862 0.57 1.140 30959 1694 1862 0.57 99.994 2 1 590 12908222 14.439 1.392 1001.067
HttpRequest         1784 891 1553 1631 0.02 1.045 885 1542 1631 0.02 99.327 6 11 295 613472 0.686 2.108 52.175
Medium Strings        128 6801 851 954 4.33 0.572 6644 831 954 4.33 97.692 157 20 360 2746616 3.072 0.677 219.037
4KB Strings         4096 208 832 1168 1.42 0.560 143 572 1168 1.42 68.750 65 260 444 262956 0.294 2.074 23.268
HttpReply            280 2884 789 861 0.57 0.531 2865 784 861 0.57 99.341 19 6 291 1030249 1.152 0.556 82.441
2K Buffer           2048 374 748 1416 4.32 0.504 346 692 1416 4.32 92.513 28 56 768 4412257 4.936 17.404 317.415
16KB Strings         16384 44 704 4400 1.71 0.474 40 640 4400 1.71 90.909 4 64 2368 76985 0.086 2.429 7.763
cbdata clientReplyContext (25) 4352 140 595 706 0.08 0.401 122 519 706 0.08 87.143 18 77 489 264790 0.296 2.219 23.215
cbdata ClientSocketContext (24) 4256 140 582 690 0.08 0.392 122 508 690 0.08 87.143 18 75 478 264789 0.296 2.170 23.215
MemObject            328 1682 539 794 4.33 0.363 1663 533 794 4.33 98.870 19 7 339 255955 0.286 0.162 23.520
Long Strings          512 958 479 811 4.35 0.323 950 475 811 4.35 99.165 8 4 369 677454 0.758 0.668 56.843
Comm::Connection      200 1533 300 307 0.28 0.202 1495 292 307 0.28 97.521 38 8 70 606480 0.678 0.234 65.623
cbdata Server (20)    520 549 279 304 0.44 0.188 528 269 304 0.44 96.175 21 11 114 108934 0.122 0.109 12.423
cbdata BodyPipe (38)  152 1659 247 293 0.56 0.166 1640 244 293 0.56 98.855 19 3 109 215397 0.241 0.063 15.348
ClientInfo            448 471 207 207 0.09 0.139 471 207 207 0.09 100.000 0 0 0 0 0.000 0.000 0.007
1KB Strings         1024 189 189 208 0.29 0.127 156 156 208 0.29 82.540 33 33 107 134127 0.150 0.265 13.835
cbdata MemBuf (12)     64 2999 188 201 0.57 0.126 2963 186 201 0.57 98.800 36 3 67 1535006 1.717 0.189 123.213
StoreEntry            104 1683 171 1670 6.42 0.115 1665 170 1670 6.42 98.930 18 2 1478 256816 0.287 0.051 23.520
cbdata ModXact (41)  512 319 160 162 0.12 0.107 308 154 162 0.12 96.552 11 6 16 131784 0.147 0.130 10.001
netdbEntry            168 970 160 165 1.91 0.107 970 160 165 1.91 100.000 0 0 17 261 0.000 0.000 0.180
fqdncache_entry      160 934 146 152 2.16 0.098 926 145 152 2.16 99.143 8 2 8 30025 0.034 0.009 2.861
ipcache_entry        128 927 116 119 2.19 0.078 921 116 119 2.19 99.353 6 1 5 16480 0.018 0.004 1.682
MemBlob               48 2355 111 145 1.30 0.074 2226 105 145 1.30 94.522 129 7 33 3054760 3.417 0.282 267.710
HttpHdrCc             96 1112 105 159 4.35 0.070 1067 101 159 4.35 95.953 45 5 70 485419 0.543 0.090 37.649
cbdata idns_query (9) 8696 12 102 391 2.28 0.069 2 17 391 2.28 16.667 10 85 391 62918 0.070 1.054 5.848
net_db_name           32 2228 70 72 0.16 0.047 2228 70 72 0.16 100.000 0 0 6 493 0.001 0.000 0.275
cbdata ClientHttpRequest (23)  392 174 67 77 0.01 0.045 152 59 77 0.01 87.356 22 9 46 264283 0.296 0.200 23.215
LRU policy node       24 2692 64 384 6.42 0.042 2666 63 384 6.42 99.034 26 1 341 32395 0.036 0.001 4.782
cbdata IdleConnList (18) 4160 15 61 252 1.97 0.041 14 57 252 1.97 93.333 1 5 200 13686 0.015 0.110 0.972
cbdata store_client (31)  160 288 45 56 1.52 0.030 278 44 56 1.52 96.528 10 2 27 275536 0.308 0.085 24.485
NotePairs::Entry       48 927 44 46 2.92 0.029 924 44 46 2.92 99.676 3 1 46 18571 0.021 0.002 1.796
cbdata clientStreamNode (26)  128 280 35 42 0.08 0.024 244 31 42 0.08 87.143 36 5 29 529580 0.592 0.131 46.431
cbdata ServerBump (30)  104 259 27 34 1.52 0.018 240 25 34 1.52 92.664 19 2 17 81041 0.091 0.016 9.000
MD5 digest             16 1683 27 257 6.42 0.018 1665 27 257 6.42 98.930 18 1 228 287894 0.322 0.009 26.086
MimeEntry            128 177 23 23 6.42 0.015 177 23 23 6.42 100.000 0 0 23 1062 0.001 0.000 0.063
cbdata HttpStateData (36)  320 58 19 19 0.07 0.012 37 12 19 0.07 63.793 21 7 15 133511 0.149 0.082 10.258
cbdata FwdState (32)  176 101 18 18 0.28 0.012 59 11 18 0.28 58.416 42 8 13 211946 0.237 0.072 18.843
cbdata helper_server (11)  240 55 13 13 0.09 0.009 55 13 13 0.09 100.000 0 0 9 209 0.000 0.000 0.019
cbdata ConnOpener (34)  136 64 9 9 0.01 0.006 4 1 9 0.01 6.250 60 8 9 98344 0.110 0.026 10.905
cbdata PeerConnector (35)  136 61 9 9 0.01 0.005 18 3 9 0.01 29.508 43 6 9 77220 0.086 0.020 8.531
Acl::AndNode          160 44 7 7 6.42 0.005 44 7 7 6.42 100.000 0 0 7 264 0.000 0.000 0.016
cbdata Iterator (39)  168 31 6 8 0.77 0.003 23 4 8 0.77 74.194 8 2 6 131859 0.148 0.043 9.978
cbdata ACLFilledChecklist (22)  464 11 5 11 2.28 0.003 10 5 11 2.28 90.909 1 1 11 756320 0.846 0.676 66.886
cbdata ModXactLauncher (40)  160 30 5 8 0.77 0.003 20 4 8 0.77 66.667 10 2 5 131871 0.148 0.041 9.978
ACLStrategised        160 19 3 3 6.42 0.002 19 3 3 6.42 100.000 0 0 3 114 0.000 0.000 0.007
cbdata ServiceRep (17)  344 7 3 3 0.78 0.002 7 3 3 0.78 100.000 0 0 0 0 0.000 0.000 0.000
cbdata Tree (4)      216 11 3 3 6.42 0.002 11 3 3 6.42 100.000 0 0 3 66 0.000 0.000 0.004
cbdata AccessCheck (29)  152 15 3 4 0.57 0.001 0 0 4 0.57 0.000 15 3 4 352429 0.394 0.103 29.244
Acl::NotNode          160 14 3 3 6.42 0.001 14 3 3 6.42 100.000 0 0 3 84 0.000 0.000 0.005
acl_ip_data           96 19 2 2 6.42 0.001 19 2 2 6.42 100.000 0 0 2 289 0.000 0.000 0.016
cbdata ClientRequestContext (27)  104 17 2 4 0.08 0.001 11 2 4 0.08 64.706 6 1 4 262994 0.294 0.053 22.940
cbdata WriteRequest (47)   80 20 2 6 4.28 0.001 0 0 6 4.28 0.000 20 2 6 190311 0.213 0.029 15.692
cbdata UFSStoreState (45)  176 8 2 3 0.26 0.001 0 0 3 0.26 0.000 8 2 3 28910 0.032 0.010 2.397
Helper::Request       48 28 2 2 0.08 0.001 16 1 2 0.08 57.143 12 1 2 158681 0.178 0.015 14.377
HttpHdrContRange       24 48 2 2 0.18 0.001 48 2 2 0.18 100.000 0 0 1 410 0.000 0.000 0.036
RegexList             88 10 1 1 6.42 0.001 10 1 1 6.42 100.000 0 0 1 60 0.000 0.000 0.004
cbdata externalAclState (14)   88 10 1 2 0.28 0.001 10 1 2 0.28 100.000 0 0 2 81199 0.091 0.014 6.014
ACLSourceIP          144 6 1 1 6.42 0.001 6 1 1 6.42 100.000 0 0 1 36 0.000 0.000 0.002
cbdata DiskdFile (46)  104 8 1 2 0.26 0.001 0 0 2 0.26 0.000 8 1 2 28910 0.032 0.006 2.397
ev_entry               48 17 1 2 0.24 0.001 14 1 2 0.24 82.353 3 1 1 63038 0.071 0.006 3.019
ACLStrategised        160 5 1 1 6.42 0.001 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
cbdata helper (10)    168 3 1 1 0.78 0.000 3 1 1 0.78 100.000 0 0 1 17 0.000 0.000 0.001
wordlist               16 31 1 2 0.78 0.000 31 1 2 0.78 100.000 0 0 1 837 0.001 0.000 0.064
cbdata CbDataList (2)   40 12 1 1 6.42 0.000 12 1 1 6.42 100.000 0 0 1 72 0.000 0.000 0.004
cbdata ErrorState (37)  240 2 1 3 0.05 0.000 0 0 3 0.05 0.000 2 1 3 53117 0.059 0.025 4.356
ACLExternal          152 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
ACLStringData         56 8 1 1 6.42 0.000 8 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
ACLRegexData           16 25 1 1 6.42 0.000 25 1 1 6.42 100.000 0 0 1 54 0.000 0.000 0.003
cbdata CbDataList (1)   40 10 1 1 6.42 0.000 10 1 1 6.42 100.000 0 0 1 60 0.000 0.000 0.004
cbdata external_acl (5)  192 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
cbdata generic_cbdata (21)   32 12 1 2 2.28 0.000 1 1 2 2.28 8.333 11 1 2 86571 0.097 0.005 7.694
ACLCertificateData    120 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
acl_name_list         72 5 1 1 6.42 0.000 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
cbdata Logfile (15)  352 1 1 1 5.76 0.000 1 1 1 5.76 100.000 0 0 1 6 0.000 0.000 0.001
ACLStrategised        160 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
cbdata TcpAcceptor (19)  104 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
ACLDestinationIP      144 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
ACLUserData           64 4 1 1 6.42 0.000 4 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
cbdata ps_state (33)  256 1 1 1 0.05 0.000 0 0 1 0.05 0.000 1 1 1 215092 0.241 0.106 18.843
cbdata RemovalPolicy (7)  104 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
cbdata statefulhelper (13)  192 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
AclDenyInfoList       32 5 1 1 6.42 0.000 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
ACLStrategised        160 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
ACLProxyAuth          152 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
ACLDomainData         16 9 1 1 6.42 0.000 9 1 1 6.42 100.000 0 0 1 42 0.000 0.000 0.002
cbdata external_acl_data (6)   48 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001
external_acl_format   64 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001
cbdata StoreSearchHashIndex (28)  104 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 0 0.000 0.000 0.000
ACLHTTPHeaderData     48 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLSslErrorData       16 6 1 1 6.42 0.000 6 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002
cbdata nsvc (8)       72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 75 0.000 0.000 0.009
StoreSwapLogData       72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 89577 0.100 0.012 4.858
cbdata RemovalPurgeWalker (44)   72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 1961 0.002 0.000 0.211
cbdata ReadRequest (48)   72 1 1 1 6.30 0.000 0 0 1 6.30 0.000 1 1 1 111 0.000 0.000 0.006
cbdata CbDataList (3)   64 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
dwrite_q               48 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 89578 0.100 0.008 4.858
FwdServer             24 2 1 1 0.05 0.000 0 0 1 0.05 0.000 2 1 1 318128 0.356 0.015 25.710
ACLNoteData           40 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
CacheDigest           40 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLHierCodeData       32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLMethodData         16 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000
ACLTimeData           32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLAtStepData         32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLASN                 16 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
StoreMetaMD5           32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
StoreMetaSTDLFS       32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
StoreMetaURL           32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
StoreMetaObjSize       32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398
UFSStoreState::_queued_write   32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 193581 0.217 0.012 15.692
StoreMetaVary         32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 7514 0.008 0.000 0.565
ACLProtocolData       16 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
ACLServerNameData     16 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000
link_list             16 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 193581 0.217 0.006 15.692
dlink_node             24 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.001
cbdata RebuildState (16)  944 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata OptXactLauncher (42)  128 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 1 0.000 0.000 0.000
cbdata OptXact (43)  320 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 1 0.000 0.000 0.000
HttpHdrRange           32 0 0 1 1.82 0.000 0 0 1 1.82 -1.000 0 0 1 314 0.000 0.000 0.020
HttpHdrRangeSpec       16 0 0 1 1.82 0.000 0 0 1 1.82 -1.000 0 0 1 314 0.000 0.000 0.020
cbdata CbDataList (49)   96 0 0 1 6.21 0.000 0 0 1 6.21 -1.000 0 0 1 4662 0.005 0.001 1.421
HttpHdrSc             16 0 0 1 2.84 0.000 0 0 1 2.84 -1.000 0 0 1 12 0.000 0.000 0.005
HttpHdrScTarget       88 0 0 1 2.84 0.000 0 0 1 2.84 -1.000 0 0 1 12 0.000 0.000 0.005
cbdata BodySink (50)   80 0 0 1 2.34 0.000 0 0 1 2.34 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::User  184 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::UserRequest   88 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata StateData (51)   48 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata helper_stateful_server (52)  232 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000
Total                  1 170288 148501 150273 0.03 100.000 168782 144272 146333 0.03 97.152 1506 4229 18484 87746747 98.156 98.226 31638.278
Cumulative allocated volume: 51.921 GB
Current overhead: 43294 bytes (0.029%)
Idle pool limit: 5.00 MB
Total Pools created: 138
Pools ever used:     135 (shown above)
Currently in use:    102
String Pool Impact
(%strings) (%volume)
Short Strings       90 47
Medium Strings       8 14
Long Strings         1 8
1KB Strings         0 3
4KB Strings         0 9
16KB Strings         0 11
Other Strings       0 9

Large buffers: 0 (0 KB)

Again, a little later:

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:39:42 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:39:42 GMT
Last-Modified: Thu, 26 Oct 2017 08:39:42 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Current memory usage:
Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate
(bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec
64K Buffer           65536 1366 87424 89280 0.01 59.621 1364 87296 89280 0.01 99.854 2 128 5248 288796 0.317 35.771 19.344
mem_node             4136 6798 27458 30843 1.46 18.725 6783 27397 30843 1.46 99.779 15 61 6802 568521 0.624 4.444 38.660
ExternalACLEntry      256 17998 4500 4953 4.40 3.069 17998 4500 4953 4.40 100.000 0 0 4953 591 0.001 0.000 5.325
32K Buffer           32768 136 4352 5984 0.85 2.968 134 4288 5984 0.85 98.529 2 64 2272 51135 0.056 3.167 5.133
Short Strings         40 80835 3158 3208 0.00 2.153 80636 3150 3208 0.00 99.754 199 8 867 51148695 56.142 3.867 3308.818
16K Buffer           16384 169 2704 6992 4.43 1.844 169 2704 6992 4.43 100.000 0 0 3152 227118 0.249 7.033 13.881
8K Buffer           8192 290 2320 3464 0.25 1.582 282 2256 3464 0.25 97.241 8 64 1112 294270 0.323 4.556 21.610
HttpHeaderEntry       56 34149 1868 1899 0.00 1.274 34114 1866 1899 0.00 99.898 35 2 590 13142150 14.425 1.391 835.378
4K Buffer           4096 439 1756 2696 1.59 1.198 409 1636 2696 1.59 93.166 30 120 1012 406215 0.446 3.145 29.469
HttpRequest         1784 873 1521 1631 0.10 1.037 859 1497 1631 0.10 98.396 14 25 295 625953 0.687 2.111 43.460
2K Buffer           2048 539 1078 1416 4.40 0.735 535 1070 1416 4.40 99.258 4 8 768 4472923 4.910 17.313 209.220
Medium Strings        128 7899 988 1002 0.00 0.673 7893 987 1002 0.00 99.924 6 1 360 2798935 3.072 0.677 186.051
16KB Strings         16384 59 944 4400 1.79 0.644 54 864 4400 1.79 91.525 5 80 2368 78587 0.086 2.434 5.827
HttpReply            280 3294 901 911 0.00 0.614 3288 900 911 0.00 99.818 6 2 291 1049980 1.152 0.556 70.385
MemObject            328 2077 666 794 4.41 0.454 2071 664 794 4.41 99.711 6 2 339 261223 0.287 0.162 20.347
4KB Strings         4096 149 596 1168 1.50 0.406 142 568 1168 1.50 95.302 7 28 444 267713 0.294 2.072 16.635
cbdata clientReplyContext (25) 4352 118 502 706 0.16 0.342 109 464 706 0.16 92.373 9 39 489 270088 0.296 2.222 18.736
cbdata ClientSocketContext (24) 4256 118 491 690 0.16 0.334 109 454 690 0.16 92.373 9 38 478 270087 0.296 2.173 18.736
Long Strings          512 751 376 811 4.43 0.256 718 359 811 4.43 95.606 33 17 369 692635 0.760 0.670 52.968
Comm::Connection      200 1499 293 307 0.36 0.200 1464 286 307 0.36 97.665 35 7 70 623501 0.684 0.236 59.278
cbdata Server (20)    520 563 286 304 0.52 0.195 542 276 304 0.52 96.270 21 11 114 111986 0.123 0.110 10.935
cbdata BodyPipe (38)  152 1798 267 293 0.64 0.182 1796 267 293 0.64 99.889 2 1 109 219412 0.241 0.063 15.090
cbdata MemBuf (12)     64 3382 212 215 0.00 0.144 3377 212 215 0.00 99.852 5 1 67 1564605 1.717 0.189 104.318
StoreEntry            104 2081 212 1670 6.50 0.144 2075 211 1670 6.50 99.712 6 1 1478 262085 0.288 0.052 20.347
ClientInfo            448 472 207 207 0.02 0.141 472 207 207 0.02 100.000 0 0 0 0 0.000 0.000 0.003
cbdata ModXact (41)  512 309 155 164 0.07 0.105 308 154 164 0.07 99.676 1 1 16 134215 0.147 0.130 8.549
netdbEntry            168 925 152 165 1.99 0.103 925 152 165 1.99 100.000 0 0 17 286 0.000 0.000 0.196
fqdncache_entry      160 931 146 152 2.24 0.099 920 144 152 2.24 98.818 11 2 8 30868 0.034 0.009 3.083
cbdata idns_query (9) 8696 16 136 391 2.36 0.093 0 0 391 2.36 0.000 16 136 391 64579 0.071 1.061 5.881
ipcache_entry        128 923 116 119 2.28 0.079 923 116 119 2.28 100.000 0 0 5 16903 0.019 0.004 1.528
MemBlob               48 2438 115 145 1.38 0.078 2304 108 145 1.38 94.504 134 7 33 3118212 3.423 0.283 223.259
HttpHdrCc             96 1159 109 159 4.43 0.074 1158 109 159 4.43 99.914 1 1 70 493512 0.542 0.090 28.676
1KB Strings         1024 101 101 208 0.37 0.069 97 97 208 0.37 96.040 4 4 107 137459 0.151 0.266 11.691
LRU policy node       24 3530 83 384 6.50 0.056 3530 83 384 6.50 100.000 0 0 341 33829 0.037 0.002 9.150
net_db_name           32 2187 69 72 0.24 0.047 2187 69 72 0.24 100.000 0 0 6 540 0.001 0.000 0.336
cbdata ClientHttpRequest (23)  392 144 56 77 0.10 0.038 131 51 77 0.10 90.972 13 5 46 269589 0.296 0.200 18.736
cbdata IdleConnList (18) 4160 12 49 252 2.05 0.033 11 45 252 2.05 91.667 1 5 200 13861 0.015 0.109 0.663
NotePairs::Entry       48 995 47 47 0.00 0.032 992 47 47 0.00 99.698 3 1 46 18882 0.021 0.002 1.346
cbdata store_client (31)  160 263 42 56 1.60 0.028 256 40 56 1.60 97.338 7 2 27 281020 0.308 0.085 19.251
MD5 digest             16 2081 33 257 6.50 0.022 2075 33 257 6.50 99.712 6 1 228 294523 0.323 0.009 25.016
cbdata clientStreamNode (26)  128 236 30 42 0.16 0.020 218 28 42 0.16 92.373 18 3 29 540176 0.593 0.131 37.472
cbdata ServerBump (30)  104 234 24 34 1.60 0.016 219 23 34 1.60 93.590 15 2 17 83160 0.091 0.016 7.626
MimeEntry            128 177 23 23 6.50 0.015 177 23 23 6.50 100.000 0 0 23 1062 0.001 0.000 0.000
cbdata helper_server (11)  240 55 13 13 0.17 0.009 55 13 13 0.17 100.000 0 0 9 209 0.000 0.000 0.000
cbdata HttpStateData (36)  320 32 10 19 0.15 0.007 32 10 19 0.15 100.000 0 0 15 135965 0.149 0.082 8.642
cbdata FwdState (32)  176 56 10 19 0.08 0.007 43 8 19 0.08 76.786 13 3 14 216432 0.238 0.072 15.828
Acl::AndNode          160 44 7 7 6.50 0.005 44 7 7 6.50 100.000 0 0 7 264 0.000 0.000 0.000
cbdata ACLFilledChecklist (22)  464 10 5 11 2.36 0.003 1 1 11 2.36 10.000 9 5 11 772759 0.848 0.678 56.648
ACLStrategised        160 19 3 3 6.50 0.002 19 3 3 6.50 100.000 0 0 3 114 0.000 0.000 0.000
cbdata PeerConnector (35)  136 22 3 9 0.08 0.002 10 2 9 0.08 45.455 12 2 9 79214 0.087 0.020 7.172
cbdata Iterator (39)  168 15 3 8 0.85 0.002 15 3 8 0.85 100.000 0 0 6 134291 0.147 0.043 8.549
cbdata ServiceRep (17)  344 7 3 3 0.86 0.002 7 3 3 0.86 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ModXactLauncher (40)  160 15 3 8 0.85 0.002 14 3 8 0.85 93.333 1 1 5 134308 0.147 0.041 8.549
cbdata Tree (4)      216 11 3 3 6.50 0.002 11 3 3 6.50 100.000 0 0 3 66 0.000 0.000 0.000
Acl::NotNode          160 14 3 3 6.50 0.001 14 3 3 6.50 100.000 0 0 3 84 0.000 0.000 0.000
acl_ip_data           96 19 2 2 6.50 0.001 19 2 2 6.50 100.000 0 0 2 289 0.000 0.000 0.000
cbdata ConnOpener (34)  136 11 2 10 0.08 0.001 1 1 10 0.08 9.091 10 2 10 101038 0.111 0.026 9.638
cbdata ClientRequestContext (27)  104 12 2 4 0.16 0.001 1 1 4 0.16 8.333 11 2 4 268308 0.294 0.053 18.499
HttpHdrContRange       24 48 2 2 0.27 0.001 48 2 2 0.27 100.000 0 0 1 410 0.000 0.000 0.000
cbdata AccessCheck (29)  152 7 2 4 0.65 0.001 0 0 4 0.65 0.000 7 2 4 359554 0.395 0.103 24.631
RegexList             88 10 1 1 6.50 0.001 10 1 1 6.50 100.000 0 0 1 60 0.000 0.000 0.000
ACLSourceIP          144 6 1 1 6.50 0.001 6 1 1 6.50 100.000 0 0 1 36 0.000 0.000 0.000
Helper::Request       48 17 1 2 0.17 0.001 4 1 2 0.17 23.529 13 1 2 162246 0.178 0.015 12.498
ACLStrategised        160 5 1 1 6.50 0.001 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
cbdata WriteRequest (47)   80 10 1 6 4.36 0.001 0 0 6 4.36 0.000 10 1 6 196846 0.216 0.030 23.134
cbdata UFSStoreState (45)  176 4 1 3 0.34 0.000 4 1 3 0.34 100.000 0 0 3 30203 0.033 0.010 4.587
ev_entry               48 14 1 2 0.32 0.000 14 1 2 0.32 100.000 0 0 1 63810 0.070 0.006 2.712
cbdata externalAclState (14)   88 7 1 2 0.08 0.000 1 1 2 0.08 14.286 6 1 2 82716 0.091 0.014 5.373
cbdata generic_cbdata (21)   32 16 1 2 2.36 0.000 4 1 2 2.36 25.000 12 1 2 89441 0.098 0.005 10.084
cbdata helper (10)    168 3 1 1 0.86 0.000 3 1 1 0.86 100.000 0 0 1 17 0.000 0.000 0.000
wordlist               16 31 1 2 0.86 0.000 31 1 2 0.86 100.000 0 0 1 837 0.001 0.000 0.000
cbdata CbDataList (2)   40 12 1 1 6.50 0.000 12 1 1 6.50 100.000 0 0 1 72 0.000 0.000 0.000
ACLExternal          152 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
ACLStringData         56 8 1 1 6.50 0.000 8 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
cbdata DiskdFile (46)  104 4 1 2 0.34 0.000 4 1 2 0.34 100.000 0 0 2 30203 0.033 0.006 4.587
ACLRegexData           16 25 1 1 6.50 0.000 25 1 1 6.50 100.000 0 0 1 54 0.000 0.000 0.000
cbdata CbDataList (1)   40 10 1 1 6.50 0.000 10 1 1 6.50 100.000 0 0 1 60 0.000 0.000 0.000
cbdata external_acl (5)  192 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
ACLCertificateData    120 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
acl_name_list         72 5 1 1 6.50 0.000 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
cbdata Logfile (15)  352 1 1 1 5.84 0.000 1 1 1 5.84 100.000 0 0 1 6 0.000 0.000 0.003
ACLStrategised        160 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
cbdata TcpAcceptor (19)  104 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
ACLDestinationIP      144 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
ACLUserData           64 4 1 1 6.50 0.000 4 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
cbdata ps_state (33)  256 1 1 1 0.13 0.000 0 0 1 0.13 0.000 1 1 1 219697 0.241 0.106 15.828
cbdata ErrorState (37)  240 1 1 3 0.13 0.000 0 0 3 0.13 0.000 1 1 3 53986 0.059 0.024 3.008
cbdata RemovalPolicy (7)  104 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
cbdata statefulhelper (13)  192 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
AclDenyInfoList       32 5 1 1 6.50 0.000 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
ACLStrategised        160 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
ACLProxyAuth          152 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
ACLDomainData         16 9 1 1 6.50 0.000 9 1 1 6.50 100.000 0 0 1 42 0.000 0.000 0.000
cbdata external_acl_data (6)   48 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000
external_acl_format   64 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000
cbdata StoreSearchHashIndex (28)  104 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 0 0.000 0.000 0.000
ACLHTTPHeaderData     48 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLSslErrorData       16 6 1 1 6.50 0.000 6 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000
cbdata CbDataList (3)   64 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
FwdServer             24 2 1 1 0.13 0.000 0 0 1 0.13 0.000 2 1 1 324323 0.356 0.015 21.304
ACLNoteData           40 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
CacheDigest           40 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLHierCodeData       32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLMethodData         16 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000
ACLTimeData           32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLAtStepData         32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLASN                 16 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
StoreMetaMD5           32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
StoreMetaSTDLFS       32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
StoreMetaURL           32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
StoreMetaObjSize       32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587
UFSStoreState::_queued_write   32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 200314 0.220 0.012 23.134
StoreMetaVary         32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 7963 0.009 0.000 1.559
ACLProtocolData       16 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
ACLServerNameData     16 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000
link_list             16 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 200314 0.220 0.006 23.134
dlink_node             24 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
dwrite_q               48 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 91807 0.101 0.008 7.667
cbdata nsvc (8)       72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 75 0.000 0.000 0.003
cbdata RebuildState (16)  944 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 0 0.000 0.000 0.000
StoreSwapLogData       72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 91806 0.101 0.012 7.667
cbdata OptXactLauncher (42)  128 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 1 0.000 0.000 0.000
cbdata OptXact (43)  320 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 1 0.000 0.000 0.000
cbdata RemovalPurgeWalker (44)   72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 2048 0.002 0.000 0.312
cbdata ReadRequest (48)   72 0 0 1 6.38 0.000 0 0 1 6.38 -1.000 0 0 1 111 0.000 0.000 0.000
HttpHdrRange           32 0 0 1 1.90 0.000 0 0 1 1.90 -1.000 0 0 1 314 0.000 0.000 0.000
HttpHdrRangeSpec       16 0 0 1 1.90 0.000 0 0 1 1.90 -1.000 0 0 1 314 0.000 0.000 0.000
cbdata CbDataList (49)   96 0 0 1 6.29 0.000 0 0 1 6.29 -1.000 0 0 1 4662 0.005 0.001 0.000
HttpHdrSc             16 0 0 1 2.92 0.000 0 0 1 2.92 -1.000 0 0 1 12 0.000 0.000 0.000
HttpHdrScTarget       88 0 0 1 2.92 0.000 0 0 1 2.92 -1.000 0 0 1 12 0.000 0.000 0.000
cbdata BodySink (50)   80 0 0 1 2.42 0.000 0 0 1 2.42 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::User  184 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
Auth::Negotiate::UserRequest   88 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata StateData (51)   48 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
cbdata helper_stateful_server (52)  232 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000
Total                  1 185173 146634 151763 0.00 100.000 184369 145755 146812 0.00 99.401 804 880 18484 89409360 98.137 98.206 312807.853
Cumulative allocated volume: 52.910 GB
Current overhead: 43294 bytes (0.029%)
Idle pool limit: 5.00 MB
Total Pools created: 138
Pools ever used:     135 (shown above)
Currently in use:    103
String Pool Impact
(%strings) (%volume)
Short Strings       90 48
Medium Strings       9 15
Long Strings         1 5
1KB Strings         0 1
4KB Strings         0 9
16KB Strings         0 13
Other Strings       0 8

Large buffers: 0 (0 KB)

Finally, storedir:

Store Directory Statistics:
Store Entries          : 1673
Maximum Swap Size      : 32768 KB
Current Store Swap Size: 29484.00 KB
Current Capacity       : 89.98% used, 10.02% free

Store Directory #0 (diskd): /var/cache/squid
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 32768 KB
Current Size: 29484.00 KB
Percent Used: 89.98%
Filemap bits in use: 1338 of 16384 (8%)
Filesystem Space in use: 44360368/192108884 KB (23%)
Filesystem Inodes in use: 454196/12271616 (4%)
Flags: SELECTED
Pending operations: 0
Removal policy: lru
LRU reference age: 0.01 days

# free --bytes
total        used        free      shared  buff/cache   available
Mem:    33653817344  3982090240   929779712   162263040 28741947392 29162274816
Swap:   37925875712   173297664 37752578048

I read Alex's e-mail too late, so I'll try to attach gdb next time.
The problem is that Squid continues to respond, but slowly.
Having a backup Squid is not that simple in my scenario because I require to do content analysis on ssl-bumped traffic. I'm using TPROXY on the gateway/firewall with Squid. A true backup on this production system would mean having a full conntrackd'ed firewall with Squid as I cannot TPROXY on a redirected machine.
I know Squid 4 is recommended for SSL-Bump, but it's still beta, and I'm not sure I can take that leap now.

I didn't run mgr:events while the CPU was at 100%, but here's what came up afterwards:

# squidclient mgr:events
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 11:02:42 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 11:02:42 GMT
Last-Modified: Thu, 26 Oct 2017 11:02:42 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Last event to run: MaintainSwapSpace

Operation                       Next Execution  Weight  Callback Valid?
logfileFlush                    0.511 sec           1    yes
MaintainSwapSpace               0.511 sec           1    N/A
fqdncache_purgelru              1.481 sec           1    N/A
idnsCheckQueue                  2.507 sec           1    N/A
statAvgTick                     3.821 sec           1    N/A
storeDirClean                   5.484 sec           1    N/A
memPoolCleanIdlePools           5.484 sec           1    N/A
ipcache_purgelru                6.900 sec           1    N/A
peerRefreshDNS                  348.872 sec         1    N/A
netdbSaveState                  678.327 sec         1    N/A
Adaptation::Icap::ServiceRep::noteTimeToUpdate  3243.778 sec        0    yes
User Cache Maintenance          3302.980 sec        1    N/A
storeDigestRebuildStart         3303.451 sec        1    N/A
storeDigestRewriteStart         3303.451 sec        1    N/A

# squidclient mgr:events
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 11:08:04 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 11:08:04 GMT
Last-Modified: Thu, 26 Oct 2017 11:08:04 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Last event to run: fqdncache_purgelru

Operation                       Next Execution  Weight  Callback Valid?
logfileFlush                    0.141 sec           1    yes
MaintainSwapSpace               0.141 sec           1    N/A
idnsCheckQueue                  3.212 sec           1    N/A
ipcache_purgelru                5.393 sec           1    N/A
fqdncache_purgelru              9.962 sec           1    N/A
storeDirClean                   13.942 sec          1    N/A
memPoolCleanIdlePools           13.942 sec          1    N/A
peerRefreshDNS                  27.243 sec          1    N/A
statAvgTick                     42.223 sec          1    N/A
netdbSaveState                  356.698 sec         1    N/A
Adaptation::Icap::ServiceRep::noteTimeToUpdate  2922.149 sec        0    yes
User Cache Maintenance          2981.351 sec        1    N/A
storeDigestRebuildStart         2981.822 sec        1    N/A
storeDigestRewriteStart         2981.822 sec        1    N/A

I haven't been able to associate the 100% CPU issue with a scheduled event.
I'll keep looking.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Alex Rousskov
On 10/26/2017 05:15 AM, Vieri wrote:

> Here's the strace (compressed with gz):
> Do you see anything useful in the trace?

FWIW, nothing jumped at me. As you know, I was asking for a gdb
backtrace, but I no longer think that it is likely to help at this
stage: Your further comments and strace itself confirm that Squid is
_not_ stuck in some tight loop, consuming 100% of CPU. It may be
consuming a lot of CPU cycles, but it is not doing the same compact
operation over and over again, so random gdb backtraces are unlikely to
pinpoint the culprit (if there is one).


> If not, how can I improve the trace next time I have this issue?

Please add (relative) timestamps between system calls (strace -r). It
may help detect periods of time when Squid was not making any system
calls. If possible, I would also add time spend in each system call
(strace -T), just in case, but it is unlikely to be useful in this case.

Please also report how many request did Squid finished handling during
the traced period. You can run mgr:info before/after and/or count
access.log lines.

For mgr:info snapshots, please consider doing the legwork to identify
potentially relevant entries that have changed between the snapshots. I
do not have time to do that (but others on the list might, of course).

I would not bother with mgr:mem snapshots unless mgr:info shows that the
memory has grown a lot during the 100% CPU usage.

Finally, consider enabling Squid debugging for a few seconds of that
100% CPU usage period. Wiki has instructions revolving around sending
Squid a debug signal.


> Having a backup Squid is not that simple in my scenario

It rarely is!


> I didn't run mgr:events while the CPU was at 100%

Please run mgr:events all the time (e.g., every minute) so that you can
catch event(s) (if any) that _lead_ to the 100% CPU usage period. You
can also run mgr:events more often (e.g., every 5 seconds) during the
100% CPU usage period, of course.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Vieri
Thanks for all the tips.

I should've done this before, but I just took a second look at cache.log and found the following message repeated a few times before the 100% CPU issue:

"Your cache is running out of filedescriptors"

I have no more info because the debug level was low.

If this is the cause then I'm even more worried now because:

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127521
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 127521
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

# cat /etc/security/limits.d/01_squid.conf
* hard nofile 65535
* soft nofile 16384

It just so happens I ran mgr:filedescriptors while the CPU was at 100%.

Here's the output (stripped of domains and IP addresses):

HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 26 Oct 2017 08:35:16 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 26 Oct 2017 08:35:16 GMT
Last-Modified: Thu, 26 Oct 2017 08:35:16 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Active file descriptors:
File Type   Tout Nread  * Nwrite * Remote Address        Description
---- ------ ---- -------- -------- --------------------- ------------------------------
5 Log       0       0        0                        /var/log/squid/cache.log
6 Pipe      0       0        0                        IPC UNIX STREAM Parent
9 Socket 28796       0        0  x.x.x.x:3199    client https start
11 Socket    8    1829*  174558  x.x.x.x:62569  Idle client: Waiting for next request
12 Socket    0   77571   243300  x.x.x.x:1344        x.x.x.x
13 Socket 28798       0        0  x.x.x.x:1428   client https start
14 Socket  113     201*    3316  x.x.x.x:49847  Idle client: Waiting for next request
15 Socket 7614       0*       0  x.x.x.x:56204   client https start
16 Socket    0   70492    72860  x.x.x.x:1344        x.x.x.x
17 Socket    0  184736   412668  x.x.x.x:1344        x.x.x.x
18 Socket    0  269207   467337  x.x.x.x:1344        x.x.x.x
19 Socket    0   70386    70846  x.x.x.x:1344        x.x.x.x
20 Socket   56     319*     452  x.x.x.x:59366   Idle client: Waiting for next request
21 Socket    0   74155   160454  x.x.x.x:1344        x.x.x.x
22 Socket    0   70373   184389  x.x.x.x:1344        x.x.x.x
23 Socket    0  321566   418429  x.x.x.x:1344        x.x.x.x
24 Socket 28783    1633*       0  x.x.x.x:60033  Reading next request
25 Socket  895    1809*   37539  x.x.x.x:1344        x.x.x.x
26 Socket    0  171606   765947  x.x.x.x:1344        x.x.x.x
27 Socket    0  505651   936639  x.x.x.x:1344        x.x.x.x
28 Socket    0   77690   579865  x.x.x.x:1344        x.x.x.x
29 Socket  893 1120129* 1356723  x.x.x.x:1344        x.x.x.x
30 Socket   99    1409*    1067  x.x.x.x:51458  Idle client: Waiting for next request
31 Socket    0  697084   991951  x.x.x.x:1344        x.x.x.x
32 Socket    0   72740   380302  x.x.x.x:1344        x.x.x.x
33 Socket    0  133319  4018875  x.x.x.x:1344        x.x.x.x
34 Socket  885  365084*  418439  x.x.x.x:1344        x.x.x.x
35 Socket    0  230550   976940  x.x.x.x:1344        x.x.x.x
36 Socket    0   72085   159051  x.x.x.x:1344        x.x.x.x
37 Socket    0  142361   344294  x.x.x.x:1344        x.x.x.x
38 Socket    0 1008147  1644576  x.x.x.x:1344        x.x.x.x
39 Socket    0  861819  1334921  x.x.x.x:1344        x.x.x.x
40 Socket 28800    2851*     344  x.x.x.x:60799  Reading next request
41 Socket    0       0        0  [::]:39163            DNS Socket IPv6
42 Socket    0  152497   327514  x.x.x.x:1344        x.x.x.x
43 Socket    0 1139088  1516707  x.x.x.x:1344        x.x.x.x
44 Socket    0  237721   398977  x.x.x.x:1344        x.x.x.x
45 Socket    0   91801   114000  x.x.x.x:1344        x.x.x.x
46 Socket    0       0        0  x.x.x.x:3233    HTTP Request
47 Socket    0  131675   695360  x.x.x.x:1344        x.x.x.x
48 Socket    0  159372   179649  x.x.x.x:1344        x.x.x.x
49 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (187)
50 Socket    0     161*    1378  x.x.x.x:443      https://domain.org
52 Socket   69     286*    1430  x.x.x.x:55771   Idle client: Waiting for next request
53 Socket    8     397*   48788  x.x.x.x:62572  Idle client: Waiting for next request
54 Socket    0  341879   480589  x.x.x.x:1344        x.x.x.x
55 Socket   91     201*    3315  x.x.x.x:56787   Idle client: Waiting for next request
56 Socket    0  726999  1336667  x.x.x.x:1344        x.x.x.x
57 Socket  113     201*    3316  x.x.x.x:51036  Idle client: Waiting for next request
58 Socket  119     556*    3723  x.x.x.x:51851   Idle client: Waiting for next request
59 Socket   99    1322*     766  x.x.x.x:1968    Idle client: Waiting for next request
60 Socket    0   78554   216848  x.x.x.x:1344        x.x.x.x
61 Socket    0   70211   717853  x.x.x.x:1344        x.x.x.x
62 Socket    0   78361   182762  x.x.x.x:1344        x.x.x.x
63 Socket    0 1945692  4239277  x.x.x.x:1344        x.x.x.x
64 Socket   66    1092*    1117  x.x.x.x:50080   Idle client: Waiting for next request
65 Socket   62     202*     286  x.x.x.x:55769   Idle client: Waiting for next request
66 Socket    0   17396*     904  x.x.x.x:443    https://domain.org
67 Socket    0    3536*    1166  x.x.x.x:443     https://domain.org
68 Socket 28798    3038*     797  x.x.x.x:65116  Reading next request
69 Socket 28798       0*       0  x.x.x.x:1426   client https start
70 Socket 28780       0*       0  x.x.x.x:60791  client https start
71 Socket    0   70387    70867  x.x.x.x:1344        x.x.x.x
72 Socket    0  254449   502590  x.x.x.x:1344        x.x.x.x
73 Socket    9   11524*    1152  x.x.x.x:54067  Idle client: Waiting for next request
74 Socket    0   96195   103010  x.x.x.x:1344        x.x.x.x
75 Socket   28    1516*    1291  x.x.x.x:3023    Idle client: Waiting for next request
76 Socket    0  205806   602151  x.x.x.x:1344        x.x.x.x
77 Socket  115    2195*    1248  x.x.x.x:59024  Idle client: Waiting for next request
78 Socket    0  352877   444820  x.x.x.x:1344        x.x.x.x
79 Socket    6     384*     612  x.x.x.x:80      Idle server: x.x.x.x:80/some_path
80 Socket    0   71272   230010  x.x.x.x:1344        x.x.x.x
81 Socket    5    1490*  168706  x.x.x.x:54376  Idle client: Waiting for next request
82 Socket    0 4403371   651548  x.x.x.x:47443         DNS Socket IPv4
83 Socket    0  782173   984402  x.x.x.x:1344        x.x.x.x
84 Pipe      0       0        0                        unlinkd -> squid
85 Socket 28798     433*       0  x.x.x.x:1423   Reading next request
86 Socket    5    2211*    1249  x.x.x.x:49400  Idle client: Waiting for next request
87 Pipe      0       0        0                        squid -> unlinkd
88 Pipe      0       0        0                        squid -> diskd
89 Socket 28792    4914*    2557  x.x.x.x:3381   Reading next request
90 Socket   66     598*     712  x.x.x.x:2262   Idle client: Waiting for next request
91 Socket    0  486500  1052648  x.x.x.x:1344        x.x.x.x
92 Socket    0   84149   459690  x.x.x.x:1344        x.x.x.x
93 Socket    0  446707   703970  x.x.x.x:1344        x.x.x.x
94 Socket 28797    1744*       0  x.x.x.x:49666  Reading next request
95 Socket    0  196505  1356207  x.x.x.x:1344        x.x.x.x
96 Socket    0   71430   747104  x.x.x.x:1344        x.x.x.x
97 Pipe      0 55385864* 83254354                        ssl_crtd #1
98 Socket   99    4720*   74394  x.x.x.x:51833   Idle client: Waiting for next request
99 Socket    0   80191   352164  x.x.x.x:1344        x.x.x.x
100 Socket    0  831352  1205057  x.x.x.x:1344        x.x.x.x
101 Socket    0   67263    68047  x.x.x.x:1344        x.x.x.x
102 Socket 28734       0*       0  x.x.x.x:56784   client https start
103 Socket    0   90037   239742  x.x.x.x:1344        x.x.x.x
104 Socket    0   86557   182854  x.x.x.x:1344        x.x.x.x
105 Socket    0   74680   272842  x.x.x.x:1344        x.x.x.x
106 Socket    0   82432   182773  x.x.x.x:1344        x.x.x.x
107 Socket    0   91159   110508  x.x.x.x:1344        x.x.x.x
108 Socket    0   74341   182862  x.x.x.x:1344        x.x.x.x
109 Socket   58  354520*  417712  x.x.x.x:1344        x.x.x.x
110 Socket  896  110923*  238345  x.x.x.x:1344        x.x.x.x
111 Socket    0  367211   830710  x.x.x.x:1344        x.x.x.x
112 Socket    0  799681  2454675  x.x.x.x:1344        x.x.x.x
113 Socket    0  174639   526254  x.x.x.x:1344        x.x.x.x
114 Socket    0  224143   626713  x.x.x.x:1344        x.x.x.x
115 Socket    0     825*    4338  x.x.x.x:443    https://domain.org
116 Socket   50       0        0* x.x.x.x:80       domain.org
117 Socket 28799       0        0  x.x.x.x:3208    client https start
118 Socket    0   29387*    3248  x.x.x.x:443   https://domain.org
119 Socket   69   13796*    1117  x.x.x.x:64434  Idle client: Waiting for next request
120 Socket    0  401186   536770  x.x.x.x:1344        x.x.x.x
121 Socket    0   70420    72431  x.x.x.x:1344        x.x.x.x
122 Socket    0 1248701  2243251  x.x.x.x:1344        x.x.x.x
123 Pipe      0 12387700* 18425650                        ssl_crtd #2
124 Socket    0   74579   186204  x.x.x.x:1344        x.x.x.x
125 Socket    0  147688   317832  x.x.x.x:1344        x.x.x.x
126 Socket    0  157570  1000064  x.x.x.x:1344        x.x.x.x
127 Socket    0  484039   932396  x.x.x.x:1344        x.x.x.x
128 Socket    0   72709   204331  x.x.x.x:1344        x.x.x.x
129 Socket    0  121209   121541  x.x.x.x:1344        x.x.x.x
130 Socket    0   89678   203510  x.x.x.x:1344        x.x.x.x
131 Socket    0 1096513  1775872  x.x.x.x:1344        x.x.x.x
132 Socket 28774    1636*       0  x.x.x.x:65117  Reading next request
133 Socket   53     378*    1542  x.x.x.x:80        Idle server: x.x.x.x:80/some_path
134 Socket  108     482*     654  x.x.x.x:59022  Idle client: Waiting for next request
135 Socket    0   67261    67977  x.x.x.x:1344        x.x.x.x
136 Socket    0  353852  1257798  x.x.x.x:1344        x.x.x.x
137 Socket  120    3820*    3424  x.x.x.x:64559  Idle client: Waiting for next request
138 Socket    8     829*    1280  x.x.x.x:3040    Idle client: Waiting for next request
139 Socket 6069       0*       0  x.x.x.x:56177   client https start
140 Socket    0   75215   314184  x.x.x.x:1344        x.x.x.x
141 Socket    0   72428   421624  x.x.x.x:1344        x.x.x.x
142 Socket    0  205459   279606  x.x.x.x:1344        x.x.x.x
143 Socket    0  983140  1528993  x.x.x.x:1344        x.x.x.x
144 Socket    0   72254   128634  x.x.x.x:1344        x.x.x.x
145 Socket  108     470*     654  x.x.x.x:2259   Idle client: Waiting for next request
146 Socket 28798       0        0  x.x.x.x:1839   client https start
147 Socket 28754    6505*    2748  x.x.x.x:2121   Reading next request
148 Socket    0   94387   437023  x.x.x.x:1344        x.x.x.x
149 Socket  112     526*     636  x.x.x.x:54464  Idle client: Waiting for next request
150 Socket    0  291712   418901  x.x.x.x:1344        x.x.x.x
151 Socket    0  260125   466643  x.x.x.x:1344        x.x.x.x
152 Socket    0   75500   192364  x.x.x.x:1344        x.x.x.x
153 Socket   91    6720*   31850  x.x.x.x:51199   Idle client: Waiting for next request
154 Socket    0  340510  3310843  x.x.x.x:1344        x.x.x.x
155 Socket 28799       0        0  x.x.x.x:3206    client https start
156 Socket    0   74500   184502  x.x.x.x:1344        x.x.x.x
157 Socket  888  208562*  377059  x.x.x.x:1344        x.x.x.x
158 Pipe      0 5872190* 8607084                        ssl_crtd #3
159 Socket    0  103064   525215  x.x.x.x:1344        x.x.x.x
160 Socket   98     482*     654  x.x.x.x:60031  Idle client: Waiting for next request
161 Socket   62     516*     675  x.x.x.x:51195   Idle client: Waiting for next request
162 Socket 28799       0        0  x.x.x.x:3209    client https start
163 Socket   66    2211*    2150  x.x.x.x:54426  Idle client: Waiting for next request
164 Socket 28798    3348*     798  x.x.x.x:2255   Reading next request
165 Socket    0  222093   321752  x.x.x.x:1344        x.x.x.x
166 Socket    0  126144   442862  x.x.x.x:1344        x.x.x.x
167 Socket   62   16353*   41096  x.x.x.x:3235    Idle client: Waiting for next request
168 Socket    0   67365    68179  x.x.x.x:1344        x.x.x.x
169 Socket    0    9195*    2294  x.x.x.x:443   https://domain.org
170 Socket  113     201*    3316  x.x.x.x:51037  Idle client: Waiting for next request
171 Socket 28800       0        0  x.x.x.x:55497  client https connect
172 Socket    0  126735   522798  x.x.x.x:1344        x.x.x.x
173 Socket    0   27366*    6978  x.x.x.x:443    https://domain.org
174 Socket 28779    3231*     870  x.x.x.x:2142   Reading next request
175 Socket    0 1214190  1308440  x.x.x.x:1344        x.x.x.x
176 Socket 28745    3983*       0  x.x.x.x:1365   Reading next request
177 Socket 27422       0*       0  x.x.x.x:56749   client https start
178 Socket    0   71305   194143  x.x.x.x:1344        x.x.x.x
179 Socket   66     306*     452  x.x.x.x:1353   Idle client: Waiting for next request
180 Socket 28774    1758*       0  x.x.x.x:59375   Reading next request
181 Socket 28798    3618*     609* x.x.x.x:55780   Reading next request
182 Socket    0  463907  4817276  x.x.x.x:1344        x.x.x.x
183 Socket    0  366549   818689  x.x.x.x:1344        x.x.x.x
184 Pipe      0 3597805* 5272955                        ssl_crtd #4
185 Socket    0  330910   964399  x.x.x.x:1344        x.x.x.x
186 Socket  113     645*    1205  x.x.x.x:4116    Idle client: Waiting for next request
187 Socket 28800     876*       0* x.x.x.x:62588  Reading next request
188 Socket    0 1484456  1616112  x.x.x.x:1344        x.x.x.x
189 Socket    0  902153  1747543  x.x.x.x:1344        x.x.x.x
190 Socket   99   13466*    1117  x.x.x.x:64687   Idle client: Waiting for next request
191 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (505)
192 Socket   58  277899*  415305  x.x.x.x:1344        x.x.x.x
193 Socket   19     838*     739  x.x.x.x:62578  Idle client: Waiting for next request
194 Socket   58   32594*   53867  x.x.x.x:1344        x.x.x.x
195 Socket    0   74412   748176  x.x.x.x:1344        x.x.x.x
196 Socket    0  294684   832445  x.x.x.x:1344        x.x.x.x
197 Socket  876  431920*  668818  x.x.x.x:1344        x.x.x.x
198 Socket 28744    2487*       0  x.x.x.x:3267    Reading next request
199 Socket  113     201*    3316  x.x.x.x:51238  Idle client: Waiting for next request
200 Socket    0    1043*     886  x.x.x.x:443    https://domain.org
201 Socket 28779    3150*     870  x.x.x.x:2134   Reading next request
202 Socket  104     361*    3527  x.x.x.x:3173    Idle client: Waiting for next request
203 Socket    0       0        0  x.x.x.x:3234    HTTP Request
204 Socket    8    1450*    1308  x.x.x.x:65097  Reading next request
205 Socket   19     566*     784  x.x.x.x:54405  Idle client: Waiting for next request
206 Socket    0 1018301  1251632  x.x.x.x:1344        x.x.x.x
207 Socket    0  243460   725933  x.x.x.x:1344        x.x.x.x
208 Pipe      0 2674774* 3914935                        ssl_crtd #5
209 Socket  115     610*     712  x.x.x.x:64689   Idle client: Waiting for next request
210 Socket    0     764*    7298  x.x.x.x:443    https://domain.org
211 Socket  105    1277*     344  x.x.x.x:2283   Idle client: Waiting for next request
212 Socket    0  131679  1220103  x.x.x.x:1344        x.x.x.x
213 Socket  876   30969*  222063  x.x.x.x:1344        x.x.x.x
214 Socket    0  191177   438150  x.x.x.x:1344        x.x.x.x
215 Socket    0   78829   290995  x.x.x.x:1344        x.x.x.x
216 Socket  112    1002*    1328  x.x.x.x:51197   Idle client: Waiting for next request
217 Socket    0  174193*    3658  x.x.x.x:443    https://domain.org
218 Socket 28781       0*       0  x.x.x.x:60795  client https start
219 Pipe      0 1969670* 2885654                        ssl_crtd #6
220 Socket   67    5709*   13745  x.x.x.x:1366   Idle client: Waiting for next request
221 Socket    0  570053   628689  x.x.x.x:1344        x.x.x.x
222 Socket 28790     308*       0  x.x.x.x:2899   Reading next request
223 Socket 28798     704*  267112  x.x.x.x:1827    Reading next request
224 Pipe      0 1464643* 2151275                        ssl_crtd #7
225 Socket  107    3463*   26520  x.x.x.x:51202   Idle client: Waiting for next request
226 Socket 28799     433*       0  x.x.x.x:1425   Reading next request
227 Socket  118    1424*     344  x.x.x.x:1826    Idle client: Waiting for next request
228 Socket    0  321598   669611  x.x.x.x:1344        x.x.x.x
229 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (686)
230 Socket   87     909*     718  x.x.x.x:1389   Idle client: Waiting for next request
231 Socket    0   70445    89872  x.x.x.x:1344        x.x.x.x
232 Socket 28797       0*       0  x.x.x.x:3389   client https start
233 Socket    0  145988   147306  x.x.x.x:1344        x.x.x.x
234 Socket    0  586261  1258275  x.x.x.x:1344        x.x.x.x
235 Socket    0  285602   466348  x.x.x.x:1344        x.x.x.x
236 Socket    0  131679  3752672  x.x.x.x:1344        x.x.x.x
237 Socket    0  447059  1790353  x.x.x.x:1344        x.x.x.x
238 Socket    0   87910  2010315  x.x.x.x:1344        x.x.x.x
239 Socket 5812       0*       0  x.x.x.x:56171   client https start
240 Socket   28   48115*    1178  x.x.x.x:80        Idle server: x.x.x.x:80/some_path
241 Socket   66    4466*   18377  x.x.x.x:3268    Idle client: Waiting for next request
242 Socket    0  596879  1296258  x.x.x.x:1344        x.x.x.x
243 Socket    0  568370   987503  x.x.x.x:1344        x.x.x.x
244 Pipe      0 1150405* 1697637                        ssl_crtd #8
245 Socket   56     485*  682630  x.x.x.x:62580  Idle client: Waiting for next request
246 Socket    0  175607  1683506  x.x.x.x:1344        x.x.x.x
247 Socket   16    1424*   17194  x.x.x.x:54395  Idle client: Waiting for next request
248 Socket 28798     829*   12831* x.x.x.x:2308   Reading next request
249 Socket   56     318*     452  x.x.x.x:62585  Idle client: Waiting for next request
250 Socket  875    1734*    5924  x.x.x.x:443    https://domain.org
251 Socket    5     518*   70059  x.x.x.x:54373  Idle client: Waiting for next request
252 Socket    0  149322   313136  x.x.x.x:1344        x.x.x.x
253 Socket    0  589520   836807  x.x.x.x:1344        x.x.x.x
254 Socket 28797       0        0* x.x.x.x:54932   client https start
255 Socket 28771    1660*       0  x.x.x.x:51201   Reading next request
256 Pipe      0  916891* 1351785                        ssl_crtd #9
257 Socket   60   22466*  111364  x.x.x.x:1344        x.x.x.x
258 Socket 28796    3294*    1180  x.x.x.x:59020  Reading next request
259 Socket    0    1055*     872  x.x.x.x:443    https://domain.org
260 Socket    0   14835*     624  x.x.x.x:443    https://domain.org
261 Socket  875  289287*  581694  x.x.x.x:1344        x.x.x.x
262 Socket    7    2437*   10773  x.x.x.x:65303  Idle client: Waiting for next request
263 Socket 28796       0        0  x.x.x.x:3197    client https start
264 Pipe      0  729453* 1059077                        ssl_crtd #10
265 Socket   66    4370*    5763  x.x.x.x:3269    Idle client: Waiting for next request
266 Socket  107    3233*    4119  x.x.x.x:64586  Idle client: Waiting for next request
267 Socket    0  212472   431063  x.x.x.x:1344        x.x.x.x
268 Pipe      0  578076*  850440                        ssl_crtd #11
269 Socket  113     407*     613  x.x.x.x:51460  Idle client: Waiting for next request
270 Socket    0    1101*    4474  x.x.x.x:443    https://domain.org
271 Socket    0 2563693  3830336  x.x.x.x:1344        x.x.x.x
272 Socket    0  828859  1380776  x.x.x.x:1344        x.x.x.x
273 Socket    0   71319   259076  x.x.x.x:1344        x.x.x.x
274 Socket  105     609*     435  x.x.x.x:3171    Idle client: Waiting for next request
275 Socket 28799     433*       0  x.x.x.x:1427   Reading next request
276 Socket    0   74500   185574  x.x.x.x:1344        x.x.x.x
277 Socket    0  189973   564948  x.x.x.x:1344        x.x.x.x
278 Socket   60  129925*  458165  x.x.x.x:1344        x.x.x.x
279 Socket 28783       0*       0  x.x.x.x:60797  client https start
280 Socket 28797     731*       0  x.x.x.x:54488  Reading next request
281 Socket    0   70962    98876  x.x.x.x:1344        x.x.x.x
282 Socket    0   77789   334559  x.x.x.x:1344        x.x.x.x
283 Socket  115     649*     623  x.x.x.x:54482  Idle client: Waiting for next request
284 Socket    0  131737   182857  x.x.x.x:1344        x.x.x.x
285 Socket 28799       0        0  x.x.x.x:3216    client https start
286 Socket   74     469*     555  x.x.x.x:60744  Idle client: Waiting for next request
287 Pipe      0   85247*  124165                        ssl_crtd #1
288 Socket    0  996913  1509983  x.x.x.x:1344        x.x.x.x
289 Pipe      0  466480*  700518                        ssl_crtd #12
290 Socket    0   70702    74868  x.x.x.x:1344        x.x.x.x
291 Socket   88     518*   43112  x.x.x.x:60756  Idle client: Waiting for next request
292 Socket    9   10213*    1158  x.x.x.x:54068  Idle client: Waiting for next request
293 Socket  900  269761*  431423* x.x.x.x:1344        x.x.x.x
294 Socket    0   88023   296477  x.x.x.x:1344        x.x.x.x
295 Socket  113     201*    3316  x.x.x.x:51038  Idle client: Waiting for next request
296 Socket 28800       0        0  x.x.x.x:2385   client https start
297 Socket    0 1409562  1861582  x.x.x.x:1344        x.x.x.x
298 Socket    0   74282   713086  x.x.x.x:1344        x.x.x.x
299 Socket    0   74290   182774  x.x.x.x:1344        x.x.x.x
300 Socket   97    2215*    1301  x.x.x.x:51848   Idle client: Waiting for next request
301 Socket    0   85396   343984  x.x.x.x:1344        x.x.x.x
302 Socket    0   78177   143882  x.x.x.x:1344        x.x.x.x
303 Socket   66    5881*    6604  x.x.x.x:1367   Idle client: Waiting for next request
304 Socket    0 2942297  3069551  x.x.x.x:1344        x.x.x.x
305 Pipe      0  364706*  534055                        ssl_crtd #13
306 Socket 28758       0        0  x.x.x.x:55095  client https start
307 Socket  118     410*    1231  x.x.x.x:2311   Idle client: Waiting for next request
308 Socket    0  503380  1071795  x.x.x.x:1344        x.x.x.x
309 Socket    0   96839   350661  x.x.x.x:1344        x.x.x.x
310 Socket    0  556795  5837880  x.x.x.x:1344        x.x.x.x
311 Socket  112     254*     787  x.x.x.x:50711  Idle client: Waiting for next request
312 Socket 28800       0        0  x.x.x.x:3220    client https start
313 Socket    0   72449   172781  x.x.x.x:1344        x.x.x.x
314 Socket    0  185849  1747378  x.x.x.x:1344        x.x.x.x
315 Socket    0       0        0  x.x.x.x:3232    HTTP Request
316 Socket   87     201*    3316  x.x.x.x:56871  Idle client: Waiting for next request
317 Socket    9     888*    1066  x.x.x.x:2976    Idle client: Waiting for next request
318 Socket    0       0*       0  x.x.x.x:443       x.x.x.x pinned connection for x.x.x.x (269)
319 Socket    0   73737    86419  x.x.x.x:1344        x.x.x.x
320 Socket    0  648346   962465  x.x.x.x:1344        x.x.x.x
321 Socket    0  151927   449211  x.x.x.x:1344        x.x.x.x
322 Socket    0     275*    2398  x.x.x.x:443      https://domain.org
323 Socket    2     354*     466  x.x.x.x:80     Idle server: x.x.x.x:80/some_path
324 Socket  108     470*     654  x.x.x.x:2257   Idle client: Waiting for next request
325 Pipe      0   68850*  103401                        ssl_crtd #2
326 Socket    0       0        0  x.x.x.x:3230    client https connect
327 Socket 28773       0*       0  x.x.x.x:56789   client https start
328 Socket    0  242751   910187  x.x.x.x:1344        x.x.x.x
329 Socket 28730    1146*       0  x.x.x.x:2150   Reading next request
330 Pipe      0  309147*  463388                        ssl_crtd #14
331 Socket 28783    1366*       0  x.x.x.x:64173  Reading next request
332 Socket    0     531*    1360  x.x.x.x:443    https://domain.org
333 Socket    0  467739   655208  x.x.x.x:1344        x.x.x.x
334 Socket    0       0        0  x.x.x.x:55499  client https connect
335 Socket   10    2143*     880  x.x.x.x:54391  Idle client: Waiting for next request
336 Socket  113   10176*    1157  x.x.x.x:31352   Idle client: Waiting for next request
337 Socket    0  520857  1758549  x.x.x.x:1344        x.x.x.x
338 Socket    0  232302   359424  x.x.x.x:1344        x.x.x.x
339 Socket   59  164364*  422776  x.x.x.x:1344        x.x.x.x
340 Socket    0 1459622  1885753  x.x.x.x:1344        x.x.x.x
341 Socket 28800       0        0  x.x.x.x:60102  client https start
342 Socket   16     201*    3316  x.x.x.x:54400  Idle client: Waiting for next request
343 Socket 28799     433*       0  x.x.x.x:1429   Reading next request
344 Socket    0  170597   572080  x.x.x.x:1344        x.x.x.x
345 Pipe      0  256576*  380163                        ssl_crtd #15
346 Socket   16     399*     286  x.x.x.x:61194   Idle client: Waiting for next request
347 Pipe      0   59571*   89389                        ssl_crtd #3
348 Socket   19     563*     784  x.x.x.x:54402  Idle client: Waiting for next request
349 Socket 28800       0        0  x.x.x.x:3219    client https start
350 Socket    0  381663  1005072  x.x.x.x:1344        x.x.x.x
351 Socket 28799    2283*       0  x.x.x.x:49667  Reading next request
352 Socket  112     964*    1306  x.x.x.x:51189   Idle client: Waiting for next request
353 Pipe      0  218850*  321361                        ssl_crtd #16
354 Socket  106     526*     276  x.x.x.x:51461  Idle client: Waiting for next request
355 Socket   19     573*     784  x.x.x.x:54403  Idle client: Waiting for next request
356 Socket   66     490*    8229  x.x.x.x:3144    Idle client: Waiting for next request
357 Pipe      0  179339*  269018                        ssl_crtd #17
358 Socket 28800       0        0  x.x.x.x:3217    client https start
359 Socket 28798       0        0  x.x.x.x:1840   client https start
360 Socket    0 1080220  1947420  x.x.x.x:1344        x.x.x.x
361 Socket    0       0        0  x.x.x.x:443     x.x.x.x pinned connection for x.x.x.x (46)
362 Socket    0  217293   913828  x.x.x.x:1344        x.x.x.x
363 Socket   44     630*    2746  x.x.x.x:80     Idle server: x.x.x.x:80/some_path
364 Socket    0   12335*   15764  x.x.x.x:443      https://domain.org
365 Socket 28800       0        0  x.x.x.x:3221    client https start
366 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (479)
367 Socket    0  343536   972237  x.x.x.x:1344        x.x.x.x
368 Socket    0 1138072  1520823  x.x.x.x:1344        x.x.x.x
369 Socket    0       0*       0  x.x.x.x:443       x.x.x.x pinned connection for x.x.x.x (254)
370 Socket   19     563*     784  x.x.x.x:54404  Idle client: Waiting for next request
371 Socket   98     537*    1246  x.x.x.x:54444  Idle client: Waiting for next request
372 Socket    0  860350  1104913  x.x.x.x:1344        x.x.x.x
373 Socket    0    1060*   23048  x.x.x.x:443    https://domain.org
374 Socket    0    1066*   20478  x.x.x.x:443    https://domain.org
375 Socket 28799       0        0  x.x.x.x:3210    client https start
376 Socket  900       0*    4618  x.x.x.x:443    https://domain.org
377 Pipe      0   55913*   81124                        ssl_crtd #4
378 Socket  113     201*    3316  x.x.x.x:51039  Idle client: Waiting for next request
379 Socket    0       0*       0  x.x.x.x:443      x.x.x.x pinned connection for x.x.x.x (263)
380 Socket  113     201*    3316  x.x.x.x:51040  Idle client: Waiting for next request
381 Socket   46    3906*   24391  x.x.x.x:1344        x.x.x.x
382 Socket 28798       0        0  x.x.x.x:2320   client https start
383 Socket    0   72481   157154  x.x.x.x:1344        x.x.x.x
384 Pipe      0  133145*  199159                        ssl_crtd #18
385 Socket 28776    7538*    1630  x.x.x.x:54399  Reading next request
386 Pipe      0    7025*   10746                        ssl_crtd #1
387 Socket   86     482*     654  x.x.x.x:49662  Idle client: Waiting for next request
388 Socket   91    1173*     343  x.x.x.x:60760  Idle client: Waiting for next request
389 Socket    0 1560783  3101138  x.x.x.x:1344        x.x.x.x
390 Socket 28796       0        0  x.x.x.x:3198    client https start
391 Socket    0  580399   792546  x.x.x.x:1344        x.x.x.x
392 Socket    0   78307   198904  x.x.x.x:1344        x.x.x.x
393 Socket 28737       0*       0  x.x.x.x:51806  client https start
394 Socket   18     470*    3778  x.x.x.x:3088    Idle client: Waiting for next request
395 Socket 28800       0        0  x.x.x.x:3223    client https start
396 Socket    0       0        0  x.x.x.x:3235    HTTP Request
397 Socket    7    1934*     615  x.x.x.x:54066  Idle client: Waiting for next request
398 Pipe      0  113172*  166273                        ssl_crtd #19
399 Socket  893  195206*  214787  x.x.x.x:1344        x.x.x.x
400 Socket 28754   10860*   10436  x.x.x.x:4100    Reading next request
401 Socket   17     201*    1324  x.x.x.x:80       Idle server: x.x.x.x:80/some_path
402 Socket    0 2234270  2503172  x.x.x.x:1344        x.x.x.x
403 Socket 28800    5134*    2537  x.x.x.x:1834   Reading next request
404 Socket 28772    1637*       0  x.x.x.x:51847   Reading next request
405 Socket   54     306*     452  x.x.x.x:31306   Idle client: Waiting for next request
406 Socket   63     819*    1961  x.x.x.x:52887  Idle client: Waiting for next request
407 Socket   28     253*    2550  x.x.x.x:3093    Idle client: Waiting for next request
408 Socket 28793     536*    8935* x.x.x.x:60800  Reading next request
409 Socket    0   67261    67880  x.x.x.x:1344        x.x.x.x
410 Pipe      0    7959*   12590                        ssl_crtd #2
411 Pipe      0   37269*   56216                        ssl_crtd #5
412 Socket    0   76888   138606  x.x.x.x:1344        x.x.x.x
413 Socket 28800       0        0  x.x.x.x:3225    client https start
414 Socket   98     549*    1141  x.x.x.x:54445  Idle client: Waiting for next request
415 Socket    0  873728  1316366  x.x.x.x:1344        x.x.x.x
416 Socket    0       0*       0  x.x.x.x:443      x.x.x.x pinned connection for x.x.x.x (390)
417 Socket   97     482*     654  x.x.x.x:49408  Idle client: Waiting for next request
418 Socket    0 4690935  6430992  x.x.x.x:1344        x.x.x.x
419 Pipe      0    7549*   11042                        ssl_crtd #3
420 Socket    0 1058661  1673219  x.x.x.x:1344        x.x.x.x
421 Socket 28796   10874*  125020* x.x.x.x:54421  Reading next request
422 Socket    0  619900  1893341  x.x.x.x:1344        x.x.x.x
423 Socket    0  484497   639048  x.x.x.x:1344        x.x.x.x
424 Socket    0       0*       0  x.x.x.x:443      x.x.x.x pinned connection for x.x.x.x (497)
425 Socket 28798       0        0  x.x.x.x:3200    client https start
426 Socket    0  217686   282356  x.x.x.x:1344        x.x.x.x
427 Socket    0  171843  2226884  x.x.x.x:1344        x.x.x.x
428 Socket    0  742382  1291857  x.x.x.x:1344        x.x.x.x
429 Socket  104     249*     776  x.x.x.x:61437   Idle client: Waiting for next request
430 Socket  876  338749*  449700  x.x.x.x:1344        x.x.x.x
431 Socket    0  638966   976750  x.x.x.x:1344        x.x.x.x
432 Socket    0   67626   109037  x.x.x.x:1344        x.x.x.x
433 Socket 28799       0        0  x.x.x.x:3207    client https start
434 Socket    0  539438   762345  x.x.x.x:1344        x.x.x.x
435 Socket    0  529527   740350  x.x.x.x:1344        x.x.x.x
436 Socket    0  133828   244486  x.x.x.x:1344        x.x.x.x
437 Socket   91     273*    3437  x.x.x.x:55776   Idle client: Waiting for next request
438 Socket 28799       0        0  x.x.x.x:3212    client https start
439 Socket    0  175285   360548  x.x.x.x:1344        x.x.x.x
440 Socket 28800       0        0  x.x.x.x:55498  client https connect
441 Socket  900  866564* 1274512* x.x.x.x:1344        x.x.x.x
442 Socket    0  104615   259266  x.x.x.x:1344        x.x.x.x
443 Socket    0  767495  1513964  x.x.x.x:1344        x.x.x.x
444 Socket  113     557*    3604  x.x.x.x:3187    Idle client: Waiting for next request
445 Socket    0  589528   777412  x.x.x.x:1344        x.x.x.x
446 Pipe      0   99772*  143323                        ssl_crtd #20
447 Socket    0   51392*    5692  x.x.x.x:443    https://domain.org
448 Socket    0     622*    3326  x.x.x.x:443    https://domain.org
449 Socket    0  178130   479705  x.x.x.x:1344        x.x.x.x
450 Socket    0   75335   249026  x.x.x.x:1344        x.x.x.x
451 Socket    0  584110  1041464  x.x.x.x:1344        x.x.x.x
452 Socket    0 5989855  7840009  x.x.x.x:1344        x.x.x.x
453 Socket    0   70378   183780  x.x.x.x:1344        x.x.x.x
454 Socket    9     397*     817  x.x.x.x:60714  Idle client: Waiting for next request
455 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (615)
456 Socket    0   67473    70236  x.x.x.x:1344        x.x.x.x
457 Socket  113     647*     811  x.x.x.x:54477  Idle client: Waiting for next request
458 Socket  106     277*    3421  x.x.x.x:55778   Idle client: Waiting for next request
459 Socket    0   80521   283037  x.x.x.x:1344        x.x.x.x
460 Socket   98     598*     712  x.x.x.x:50085   Idle client: Waiting for next request
461 Socket    0  167275   298854  x.x.x.x:1344        x.x.x.x
462 Socket 28781       0*       0  x.x.x.x:60792  client https start
463 Socket  115     281*    3428  x.x.x.x:55779   Idle client: Waiting for next request
464 Socket    0   10405*    4874  x.x.x.x:443    https://domain.org
465 Socket    0    1025*   27592  x.x.x.x:443    https://domain.org
466 Socket    0       0*       0  x.x.x.x:443     x.x.x.x pinned connection for x.x.x.x (9)
467 Socket   98     555*    2252  x.x.x.x:60770  Idle client: Waiting for next request
468 Socket    0    9354*    2314  x.x.x.x:443   https://domain.org
469 Socket    0   91540   325953  x.x.x.x:1344        x.x.x.x
470 Socket    0  762551  1389788  x.x.x.x:1344        x.x.x.x
471 Socket   74    7830*   12535  x.x.x.x:4097    Idle client: Waiting for next request
472 Socket    0  380016   476604  x.x.x.x:1344        x.x.x.x
473 Socket  885   15670*   85733  x.x.x.x:1344        x.x.x.x
474 Socket    0  406074   456587  x.x.x.x:1344        x.x.x.x
475 Socket 28800    1335*     253  x.x.x.x:3180    Reading next request
476 Socket   46     315*    4617  x.x.x.x:1344        x.x.x.x
477 Socket   58   32530*   49384  x.x.x.x:1344        x.x.x.x
478 Socket    0  172630   242324  x.x.x.x:1344        x.x.x.x
479 Socket  118     387*    3558  x.x.x.x:2316   Idle client: Waiting for next request
480 Socket    0   74290   182781  x.x.x.x:1344        x.x.x.x
481 Socket    0   80770   197294  x.x.x.x:1344        x.x.x.x
482 Socket    0   75048   281261  x.x.x.x:1344        x.x.x.x
483 Socket    0   74554   185351  x.x.x.x:1344        x.x.x.x
484 Socket   97     482*     654  x.x.x.x:49409  Idle client: Waiting for next request
485 Socket   60    1302*   11029  x.x.x.x:1344        x.x.x.x
486 Socket   86    4856*    5810  x.x.x.x:4112    Idle client: Waiting for next request
487 Socket  105     250*     779  x.x.x.x:56421  Idle client: Waiting for next request
488 Socket    0    2063*    4526  x.x.x.x:443    https://domain.org
489 Pipe      0   30386*   47145                        ssl_crtd #6
490 Socket 28756       0        0  x.x.x.x:55093  client https start
491 Socket 28800    5190*    1214  x.x.x.x:1835   Reading next request
492 Socket  878     105*    4307  x.x.x.x:1344        x.x.x.x
493 Socket    0       0*       0  x.x.x.x:443      x.x.x.x pinned connection for x.x.x.x (232)
494 Socket  900     429      866  x.x.x.x:443   https://domain.org
495 Socket  900  160740*  529879* x.x.x.x:1344        x.x.x.x
496 Socket    0  474171   673500  x.x.x.x:1344        x.x.x.x
497 Socket  113     331*    3485  x.x.x.x:54480  Idle client: Waiting for next request
498 Socket   74    1595*     730  x.x.x.x:49696  Idle client: Waiting for next request
499 Socket    0  559618   865116  x.x.x.x:1344        x.x.x.x
500 Socket  898     595*    3540  x.x.x.x:443    https://domain.org
501 Socket    0   75127   273504  x.x.x.x:1344        x.x.x.x
502 Socket    0  171003   603908  x.x.x.x:1344        x.x.x.x
503 Socket    0  377031   835100  x.x.x.x:1344        x.x.x.x
504 Socket    0   67366    69093  x.x.x.x:1344        x.x.x.x
505 Socket   88     520*     675  x.x.x.x:51841   Idle client: Waiting for next request
506 Socket    0   70420    71693  x.x.x.x:1344        x.x.x.x
507 Socket    0   71047   275124  x.x.x.x:1344        x.x.x.x
508 Socket    0  196158*   23334  x.x.x.x:443    https://domain.org
509 Socket    0       0*       0  ::1:48239             ext_wbinfo_group_acl #1
510 Socket    0  136980   194711  x.x.x.x:1344        x.x.x.x
511 Socket  898    9550*   41835  x.x.x.x:1344        x.x.x.x
512 Socket    0  921556  1321574  x.x.x.x:1344        x.x.x.x
513 Socket   98     537*    1068  x.x.x.x:54446  Idle client: Waiting for next request
514 Socket    0   70273   182857  x.x.x.x:1344        x.x.x.x
515 Socket    0       0        0  x.x.x.x:443     x.x.x.x pinned connection for x.x.x.x (315)
516 Socket   74     507*    1559  x.x.x.x:60741  Idle client: Waiting for next request
517 Socket   87    1531*   29663  x.x.x.x:54429  Idle client: Waiting for next request
518 Socket   87    1085*    9379  x.x.x.x:54430  Idle client: Waiting for next request
519 Socket    0   36043*     842  x.x.x.x:443      https://domain.org
520 Socket    0  156225   280780  x.x.x.x:1344        x.x.x.x
521 Socket   87    1095*    9538  x.x.x.x:54431  Idle client: Waiting for next request
522 Socket    0 1090575  1456044  x.x.x.x:1344        x.x.x.x
523 Socket    0   67263    67988  x.x.x.x:1344        x.x.x.x
524 Socket    0  133693   347078  x.x.x.x:1344        x.x.x.x
525 Socket    0       0*       0  x.x.x.x:443   x.x.x.x pinned connection for x.x.x.x (69)
526 Socket  898  311676*  510052  x.x.x.x:1344        x.x.x.x
527 Socket  900     425      866  x.x.x.x:443   https://domain.org
528 Socket    0       0*       0  ::1:59459             ext_wbinfo_group_acl #2
529 Socket    0   78353   989595  x.x.x.x:1344        x.x.x.x
530 Socket    0    1096*    1782  x.x.x.x:443    https://domain.org
531 Socket 28800       0        0  x.x.x.x:3222    client https start
532 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (764)
533 Socket    0   71390   110141  x.x.x.x:1344        x.x.x.x
534 Socket    0 1203421  1809900  x.x.x.x:1344        x.x.x.x
535 Socket    0   70832    76525  x.x.x.x:1344        x.x.x.x
536 Socket  118     417*    1219  x.x.x.x:2310   Idle client: Waiting for next request
537 Socket   77     662*     275  x.x.x.x:1392   Idle client: Waiting for next request
538 Socket    0   76434   486358  x.x.x.x:1344        x.x.x.x
539 Socket  898  445876*  531378  x.x.x.x:1344        x.x.x.x
540 Socket    0       0        0  x.x.x.x:443     x.x.x.x pinned connection for x.x.x.x (203)
541 Socket    0   69878*    1098  x.x.x.x:443    https://domain.org
542 Socket   46     384*     612  x.x.x.x:80      Idle server: x.x.x.x:80/some_path
543 Socket    0     636*     794  x.x.x.x:443    https://domain.org
544 Socket    0 2182761  2218810  x.x.x.x:1344        x.x.x.x
545 Socket   46  150988*  321272  x.x.x.x:1344        x.x.x.x
546 Socket    0   68503    78530  x.x.x.x:1344        x.x.x.x
547 Socket  896    1598*    6692  x.x.x.x:443    https://domain.org
548 Socket    0       0*       0  x.x.x.x:443   x.x.x.x pinned connection for x.x.x.x (13)
549 Socket  900       0*     866  x.x.x.x:443   https://domain.org
550 Socket 28799       0        0  x.x.x.x:3203    client https start
551 Socket   59  302774*  595817  x.x.x.x:1344        x.x.x.x
552 Socket    0     515*    1112  x.x.x.x:443     https://domain.org
553 Socket    7     395*     286  x.x.x.x:61191   Idle client: Waiting for next request
554 Socket    0  438847   475815  x.x.x.x:1344        x.x.x.x
555 Socket    0 1666817  2225879  x.x.x.x:1344        x.x.x.x
556 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (625)
557 Socket    0   87652   250185  x.x.x.x:1344        x.x.x.x
558 Socket    0  888616  1629204  x.x.x.x:1344        x.x.x.x
559 Socket  878    2012*   15232  x.x.x.x:443    https://domain.org
560 Socket   85     201*    3316  x.x.x.x:49698  Idle client: Waiting for next request
561 Socket 28774    2910*    1058  x.x.x.x:64166  Reading next request
562 Socket  898    1211*    6180  x.x.x.x:443    https://domain.org
563 Socket   97     940*    1308  x.x.x.x:3376   Idle client: Waiting for next request
564 Socket    0     275*    4748  x.x.x.x:443      https://domain.org
565 Socket   15    1446*    1306  x.x.x.x:51817   Reading next request
566 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (324)
567 Socket    0  594802   827936  x.x.x.x:1344        x.x.x.x
568 Socket    0  303279   409733  x.x.x.x:1344        x.x.x.x
569 Socket  117    1849*     615  x.x.x.x:2304   Idle client: Waiting for next request
570 Socket    0    3110*    7764  x.x.x.x:443      https://domain.org
571 Socket    0   75913*    9544  x.x.x.x:443    https://domain.org
572 Socket   57     198*     286  x.x.x.x:55768   Idle client: Waiting for next request
573 Socket   27    2763*   51668  x.x.x.x:62577  Idle client: Waiting for next request
574 Socket    0    1025*    2184  x.x.x.x:443    https://domain.org
575 Socket   99   11511*  192197  x.x.x.x:50483   Idle client: Waiting for next request
576 Socket    0       0*       0  ::1:37181             ext_wbinfo_group_acl #3
577 Socket  113    1181*     629  x.x.x.x:51203   Idle client: Waiting for next request
578 Socket    0  125227   327390  x.x.x.x:1344        x.x.x.x
579 Socket    0   80821   222391  x.x.x.x:1344        x.x.x.x
580 Socket 28798       0        0  x.x.x.x:1841   client https start
581 Socket 28798       0        0  x.x.x.x:2321   client https start
582 Socket    0       0        0  x.x.x.x:443     x.x.x.x pinned connection for x.x.x.x (396)
583 Socket    0       0*       0  ::1:39025             ext_wbinfo_group_acl #4
584 Socket 28798       0        0  x.x.x.x:2322   client https start
585 Socket    0   67261    67974  x.x.x.x:1344        x.x.x.x
586 Socket  900  192830*    7818  x.x.x.x:443     https://domain.org
587 Socket    0  464123  1899880  x.x.x.x:1344        x.x.x.x
588 Socket    0   74290   182755  x.x.x.x:1344        x.x.x.x
589 Socket    0     647*    1728  x.x.x.x:443    https://domain.org
590 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (145)
591 Socket 28799       0        0  x.x.x.x:2323   client https start
592 Socket 28799       0        0  x.x.x.x:3201    client https start
593 Pipe      0   18335*   25554                        ssl_crtd #7
594 Socket   91     292*     452  x.x.x.x:52893  Idle client: Waiting for next request
595 Socket    0  302843  1136496  x.x.x.x:1344        x.x.x.x
596 Socket 28800       0        0  x.x.x.x:2386   client https start
597 Socket    0   86639   207373  x.x.x.x:1344        x.x.x.x
598 Socket    0       0*       0  x.x.x.x:443      x.x.x.x pinned connection for x.x.x.x (279)
599 Socket    0       0*       0  ::1:51769             ext_wbinfo_group_acl #5
600 Socket    0   71364   312248  x.x.x.x:1344        x.x.x.x
601 Socket    0  919397  1075737  x.x.x.x:1344        x.x.x.x
602 Socket    0  180943  1358113  x.x.x.x:1344        x.x.x.x
603 Socket 28799       0        0  x.x.x.x:3214    client https start
604 Socket    0     692*    1178  x.x.x.x:443    https://domain.org
605 Socket   62    7966*   12814  x.x.x.x:1230   Idle client: Waiting for next request
606 Socket    0     638*    7316  x.x.x.x:443    https://domain.org
607 Socket   23     306*     452  x.x.x.x:1767    Idle client: Waiting for next request
608 Socket 28799       0        0  x.x.x.x:3205    client https start
609 Socket    0   85591   164291  x.x.x.x:1344        x.x.x.x
610 Socket    0     692*    1198  x.x.x.x:443    https://domain.org
611 Socket 28785       0*       0  x.x.x.x:3183    client https start
612 Socket    0     692*    1178  x.x.x.x:443    https://domain.org
613 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (134)
614 Socket  900   28870*  195741* x.x.x.x:1344        x.x.x.x
615 Socket 28754       0        0  x.x.x.x:55092  client https start
616 Socket  118     390*   36111  x.x.x.x:2319   Idle client: Waiting for next request
617 Socket   98     555*    2252  x.x.x.x:60769  Idle client: Waiting for next request
618 Socket  299       0*       0  x.x.x.x:1828    client http connect
619 Socket    0  473529   793634  x.x.x.x:1344        x.x.x.x
620 Socket  900  450715*  716503  x.x.x.x:1344        x.x.x.x
621 Socket 28799       0        0  x.x.x.x:3202    client https start
622 Socket    0     692*    1184  x.x.x.x:443    https://domain.org
623 Socket  874   16754*   21876  x.x.x.x:443      https://domain.org
624 Socket 28788    4986*    1596  x.x.x.x:49406  Reading next request
625 Socket   98     482*     654  x.x.x.x:64171  Idle client: Waiting for next request
626 Pipe      0   14687*   22194                        ssl_crtd #8
627 Socket   59     210*    2635  x.x.x.x:1344        x.x.x.x
628 Socket    0  134507  1462114  x.x.x.x:1344        x.x.x.x
629 Socket    0  108885* 2533583* ::1:34453             squid_url_lookup.pl #1
630 Socket    0   72439   150972  x.x.x.x:1344        x.x.x.x
631 Socket 28799       0        0  x.x.x.x:3204    client https start
632 Socket  879     734*    6486  x.x.x.x:443    https://domain.org
633 Socket    0 1051410  1201280  x.x.x.x:1344        x.x.x.x
634 Socket 28799       0        0  x.x.x.x:3211    client https start
635 Socket  879     734*    6648  x.x.x.x:443    https://domain.org
636 Socket    0       0*       0  x.x.x.x:443     x.x.x.x pinned connection for x.x.x.x (611)
637 Socket    0  163962   604624  x.x.x.x:1344        x.x.x.x
638 Socket    0   48696*     856  x.x.x.x:443     https://domain.org
639 Socket    0  729827  1196381  x.x.x.x:1344        x.x.x.x
640 Socket    0  232516   347567  x.x.x.x:1344        x.x.x.x
641 Socket 28798    3754*  128113  x.x.x.x:2303   Reading next request
642 Socket    0  325777   602015  x.x.x.x:1344        x.x.x.x
643 Socket 28799       0        0  x.x.x.x:3213    client https start
644 Socket   18     397*     265  x.x.x.x:1258   Idle client: Waiting for next request
645 Socket   99    3596*     822  x.x.x.x:54417  Idle client: Waiting for next request
646 Pipe      0   14180*   21549                        ssl_crtd #9
647 Socket    0  571668  1758812  x.x.x.x:1344        x.x.x.x
648 Socket    0     252*    3372  x.x.x.x:443    https://domain.org
649 Socket    0   68228   116478  x.x.x.x:1344        x.x.x.x
650 Socket    0     582*    1818  x.x.x.x:443   https://domain.org
651 Pipe      0    7520*   11022                        ssl_crtd #10
652 Socket   90     435*     285  x.x.x.x:3386   Idle client: Waiting for next request
653 Socket 28799       0        0  x.x.x.x:3215    client https start
654 Socket   99    1110*    2214  x.x.x.x:60771  Idle client: Waiting for next request
655 Socket 28800       0        0  x.x.x.x:3218    client https connect
656 Socket    0     882*    1880  x.x.x.x:443    https://domain.org
657 Socket   59       0        0* x.x.x.x:80        domain.org
658 Socket    0    1283*    4442  x.x.x.x:443    https://domain.org
659 Socket    0       0        0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (807)
660 Socket   90     435*     285  x.x.x.x:3384   Idle client: Waiting for next request
661 Socket 28800       0        0  x.x.x.x:3224    client https start
662 Socket    0  365673   608482  x.x.x.x:1344        x.x.x.x
663 Socket    0   81625   110628  x.x.x.x:1344        x.x.x.x
664 Socket 28800     156*       0* ::1:35046             Reading next request
665 Socket   97    2215*    1279  x.x.x.x:51849   Idle client: Waiting for next request
666 Socket   90     435*     285  x.x.x.x:3385   Idle client: Waiting for next request
667 Socket    0     620*    1220  x.x.x.x:443    https://domain.org
668 Socket 28800     201*       0* x.x.x.x:50010  Reading next request
669 Socket    0  136145   440062  x.x.x.x:1344        x.x.x.x
670 Socket    0   17564*  323089* ::1:33603             squid_url_lookup.pl #2
671 Socket   91    2348*     343  x.x.x.x:60757  Idle client: Waiting for next request
672 Socket  118     421*   17488  x.x.x.x:51046  Idle client: Waiting for next request
673 Socket 28800       0        0  x.x.x.x:3228    client https start
674 Socket    0  168525*    3042  x.x.x.x:443    https://domain.org
675 Socket    0  449451  1179110  x.x.x.x:1344        x.x.x.x
676 Socket    0   81705   176401  x.x.x.x:1344        x.x.x.x
677 Socket 28800       0        0  x.x.x.x:3229    client https connect
678 Socket    0  607338   804760  x.x.x.x:1344        x.x.x.x
679 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (417)
680 Socket    0    7879*  141662  ::1:59195             squid_url_lookup.pl #3
681 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (387)
682 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (393)
683 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (70)
684 Socket 28800    5112*    2235  x.x.x.x:1838   Reading next request
685 Socket    0     586*     990  x.x.x.x:443    https://domain.org
686 Socket   99     482*     654  x.x.x.x:60032  Idle client: Waiting for next request
687 Socket    0    1065*   20404  x.x.x.x:443    https://domain.org
688 Socket    0    4874*   61424  ::1:48333             squid_url_lookup.pl #4
689 Socket    0  225785   342266  x.x.x.x:1344        x.x.x.x
690 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (484)
691 Socket    0  247595   638177  x.x.x.x:1344        x.x.x.x
692 Socket    0     544*    1104  x.x.x.x:443    https://domain.org
693 Socket    0   70387    70862  x.x.x.x:1344        x.x.x.x
694 Socket 27680       0*       0  x.x.x.x:56755   client https start
695 Socket  300       0*       0  x.x.x.x:60802  client http connect
696 Socket    0       0        0  x.x.x.x:3231    HTTP Request
697 Socket    0   67261    67986  x.x.x.x:1344        x.x.x.x
698 Socket    0    1129*    2870  x.x.x.x:443    https://domain.org
700 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (462)
702 Socket   18     402*     266  x.x.x.x:1262   Idle client: Waiting for next request
703 Socket    9    1338*     634  x.x.x.x:80     Idle server: x.x.x.x:80/some_path
704 Socket   58     491*  682630  x.x.x.x:62581  Idle client: Waiting for next request
705 Socket    0   67626    91631  x.x.x.x:1344        x.x.x.x
706 Socket    0   67780   149096  x.x.x.x:1344        x.x.x.x
707 Socket    0   93177   639083  x.x.x.x:1344        x.x.x.x
708 Socket    0  875386  1172221  x.x.x.x:1344        x.x.x.x
709 Socket  876     588*    3324  x.x.x.x:443    https://domain.org
710 Socket   19     444*     384  x.x.x.x:1265   Idle client: Waiting for next request
711 Socket   19     425*     384  x.x.x.x:1266   Idle client: Waiting for next request
712 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (218)
716 Socket   57     420*     158  x.x.x.x:62586  Idle client: Waiting for next request
717 Socket    0    1491*    1066  x.x.x.x:443      https://domain.org
719 Socket    0  169746*   21748  x.x.x.x:443    https://domain.org
720 Socket    0   17102*    2910  x.x.x.x:443    https://domain.org
721 Socket    0    3279*   43331  ::1:42983             squid_url_lookup.pl #5
722 Socket    0  256289   269402  x.x.x.x:1344        x.x.x.x
723 Socket    0    7714*    2384  x.x.x.x:443    https://domain.org
724 Socket    0    1830*   24920  ::1:47149             squid_url_lookup.pl #6
725 Socket    0       0*       0  x.x.x.x:443      x.x.x.x pinned connection for x.x.x.x (327)
726 Socket    0 1063154  1790141  x.x.x.x:1344        x.x.x.x
727 Socket    0     616*   13608  ::1:48873             squid_url_lookup.pl #7
729 Socket   18     329*    3506  x.x.x.x:1267   Idle client: Waiting for next request
730 Socket    0   67260    67952  x.x.x.x:1344        x.x.x.x
731 Socket    0   32251*   13544  x.x.x.x:443    https://domain.org
732 Socket    0   12614*   16036  x.x.x.x:443      https://domain.org
733 Socket    0     298*    5090  ::1:59995             squid_url_lookup.pl #8
734 Socket    0     638*    3252  x.x.x.x:443     https://domain.org
735 Socket    0     252*    2900  x.x.x.x:443    https://domain.org
736 Socket  900    2405*   10516  x.x.x.x:443    https://domain.org
737 Socket    0     241*    2542  ::1:44247             squid_url_lookup.pl #9
738 Socket    0  256065  1016479  x.x.x.x:1344        x.x.x.x
740 Socket    0   69188   114543  x.x.x.x:1344        x.x.x.x
741 Socket   97     940*    1308  x.x.x.x:3379   Idle client: Waiting for next request
742 Socket  900   41281*    1514  x.x.x.x:443    https://domain.org
743 Socket    0  681278  1185273  x.x.x.x:1344        x.x.x.x
744 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (306)
745 Socket  900    1082*   10628  x.x.x.x:443    https://domain.org
746 Socket    0    5718*    9764  x.x.x.x:443      https://domain.org
747 Socket    0     620*    1196  x.x.x.x:443    https://domain.org
748 Socket    0     207*    2293  ::1:33591             squid_url_lookup.pl #10
749 Socket    0       0        0  [::]:3128             HTTP Socket
750 Socket  876     592*    3568  x.x.x.x:443    https://domain.org
751 Socket    0       0        0  [::]:3129             HTTP Socket
752 Socket    0   71255   145856  x.x.x.x:1344        x.x.x.x
753 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (569)
754 Socket    0       0        0  [::]:3130             HTTPS Socket
755 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (490)
756 Socket    0   70375   184071  x.x.x.x:1344        x.x.x.x
758 Socket    0   68918   187587  x.x.x.x:1344        x.x.x.x
760 Socket    0  132072   189220  x.x.x.x:1344        x.x.x.x
762 Socket    0     367*    1280  x.x.x.x:443       https://domain.org
763 Socket    0   74474   206574  x.x.x.x:1344        x.x.x.x
764 Socket   87     482*     654  x.x.x.x:49665  Idle client: Waiting for next request
765 Socket    0       0        0  ::1:50813             pinger
766 File      0       0   993744                        /var/cache/squid/swap.state
767 Socket  893    2800*    9984  x.x.x.x:443    https://domain.org
769 Socket    0     537*    2414  x.x.x.x:443    https://domain.org
770 Socket    0     628*    3372  x.x.x.x:443    https://domain.org
771 Socket    0  306851   815547  x.x.x.x:1344        x.x.x.x
772 Socket   56     516*     676  x.x.x.x:54415  Idle client: Waiting for next request
773 Socket    0     620*    1196  x.x.x.x:443    https://domain.org
774 Socket    0   43044*    1098  x.x.x.x:443       https://domain.org
775 Socket    0    2458*     568  x.x.x.x:443     https://domain.org
776 Socket    0     173*     794  x.x.x.x:443    https://domain.org
777 Socket    0  131967   194732  x.x.x.x:1344        x.x.x.x
778 Socket    0   30285*    1720  x.x.x.x:443     https://domain.org
780 Socket  898    1213*    6800  x.x.x.x:443    https://domain.org
781 Socket  900    2103*   10472  x.x.x.x:443    https://domain.org
782 Socket    0    3627*   14436  x.x.x.x:443    https://domain.org
783 Socket    0     174*     804  x.x.x.x:443    https://domain.org
784 Socket    0  199486   463497  x.x.x.x:1344        x.x.x.x
785 Socket    0    1099*    4762  x.x.x.x:443    https://domain.org
787 Socket    0     316*     888  x.x.x.x:443   https://domain.org
790 Socket    0    2078*    2324  x.x.x.x:443      https://domain.org
791 Socket    0     316*     850  x.x.x.x:443   https://domain.org
792 Socket    0    2184*    1162  x.x.x.x:443      https://domain.org
793 Socket    0    2184*    1162  x.x.x.x:443      https://domain.org
794 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (202)
796 Socket    0 1290186  1774914  x.x.x.x:1344        x.x.x.x
797 Socket    0   67263    67949  x.x.x.x:1344        x.x.x.x
798 Socket    0       0*       0  x.x.x.x:443    x.x.x.x pinned connection for x.x.x.x (160)
799 Socket   99    7140*    3811  x.x.x.x:54406  Idle client: Waiting for next request
800 Socket    0   70622    74503  x.x.x.x:1344        x.x.x.x
801 Socket   27    2355*    1251  x.x.x.x:54407  Idle client: Waiting for next request
802 Socket    0    1154*    1136  x.x.x.x:443   https://domain.org
803 Socket    0   69578    75080  x.x.x.x:1344        x.x.x.x
804 Socket    0    1049*    1160  x.x.x.x:443   https://domain.org
805 Socket    0     976*    1136  x.x.x.x:443   https://domain.org
806 Socket    0    1261*    4482  x.x.x.x:443    https://domain.org
808 Socket  885     589*    3318  x.x.x.x:443    https://domain.org
809 Socket    0    1317*    4482  x.x.x.x:443    https://domain.org
810 Socket  900   82342*     866  x.x.x.x:443   https://domain.org
811 Socket    0     719*    1346  x.x.x.x:443    https://domain.org
812 Socket    0    1113*    1342  x.x.x.x:443     https://domain.org
814 Socket  855    2387*   13320  x.x.x.x:443    https://domain.org
815 Socket   27     500*     286  x.x.x.x:55764   Idle client: Waiting for next request
816 Socket    0 1080317  1867520  x.x.x.x:1344        x.x.x.x
820 Socket   66    1192*    7898  x.x.x.x:4110    Idle client: Waiting for next request
821 Socket    0    1025*   26932  x.x.x.x:443    https://domain.org
822 Socket    0   86033    89065  x.x.x.x:1344        x.x.x.x
823 Socket   59  326800*  558361  x.x.x.x:1344        x.x.x.x
824 Socket  893    5922*    5026  x.x.x.x:443      https://domain.org
826 Socket    0 1317411  1355916  x.x.x.x:1344        x.x.x.x
827 Pipe      0    7431*    9205                        ssl_crtd #4
828 Socket  895    6112*    8018  x.x.x.x:443      https://domain.org
829 Socket    0   67263    68003  x.x.x.x:1344        x.x.x.x
830 Socket    0   13653*   11470  x.x.x.x:443      https://domain.org
831 Pipe      0    4461*    5937                        ssl_crtd #5
832 Socket    0   98682   230913  x.x.x.x:1344        x.x.x.x
833 Socket    0   18285*    8984  x.x.x.x:443      https://domain.org
834 Socket    0 2769082  3379609  x.x.x.x:1344        x.x.x.x
835 Socket    0    5671*    8792  x.x.x.x:443      https://domain.org
837 Socket    0    6512*   11814  x.x.x.x:443      https://domain.org
838 Pipe      0    4461*    5937                        ssl_crtd #6
839 Socket    0     252*    2606  x.x.x.x:443    https://domain.org
840 Socket    0   67366    74962  x.x.x.x:1344        x.x.x.x
842 Socket    0   71312   221653  x.x.x.x:1344        x.x.x.x
846 Pipe      0    4461*    5937                        ssl_crtd #7
847 Socket    0  225360   346218  x.x.x.x:1344        x.x.x.x
849 Pipe      0    4461*    5937                        ssl_crtd #8
850 Socket    0    8137*    1042  x.x.x.x:443     https://domain.org
851 Socket  885     591*    2784  x.x.x.x:443    https://domain.org
852 Pipe      0       0*       0                        ssl_crtd #9
854 Socket    0    3983*    6570  x.x.x.x:443    https://domain.org
857 Pipe      0       0*       0                        ssl_crtd #10
869 Socket    0   40620*   32966  x.x.x.x:443      https://domain.org
870 Socket    0 4328494  4704878  x.x.x.x:1344        x.x.x.x
878 Socket    0   70812  1451399  x.x.x.x:1344        x.x.x.x
885 Socket    0  646465  1098283  x.x.x.x:1344        x.x.x.x
891 Socket    0  949722  1844840  x.x.x.x:1344        x.x.x.x
904 Socket  888    1837*   10128  x.x.x.x:443    https://domain.org
909 Socket    0   85268   151920  x.x.x.x:1344        x.x.x.x
911 Socket  831       0*    2354  x.x.x.x:443    https://domain.org
913 Socket    0   70315    70855  x.x.x.x:1344        x.x.x.x


I don't get it. There aren't that many active file descriptors (less than 1000).
Also, on low traffic (late evening) with no CPU or mem issues at all, mgr:filedescriptors shows that there are about 2500 active file descriptors.

The only two things I can think of that could mess with Squid are:

- squid_url_lookup.pl which is an external_acl_type perl script that does a MySQL lookup (ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8)

- c-icap content scanning with clamav and libarchive

If push comes to shove I might consider temporarily disabling one or both just to see what happens, but I'm not sure if that can really be the cause of a Squid process coming down to a crawl.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Alex Rousskov
On 10/26/2017 03:58 PM, Vieri wrote:

> I should've done this before, but I just took a second look at
> cache.log and found the following message repeated a few times before
> the 100% CPU issue:

> "Your cache is running out of filedescriptors"

That warning matches the dangerously low number of available descriptor
reported by your mgr:info:

> File descriptor usage for squid:
> Maximum number of file descriptors:   1024
> Largest file desc currently in use:    913
> Number of file desc currently in use:  791
> Files queued for open:                   0
> Available number of file descriptors:  233
> Reserved number of file descriptors:   100


> If this is the cause then I'm even more worried now because:
>
> open files                      (-n) 16384

> # cat /etc/security/limits.d/01_squid.conf
> * hard nofile 65535
> * soft nofile 16384

Your Squid does not know about the above configured limits. It thinks
the limit is 1024.


> I don't get it. There aren't that many active file descriptors (less than 1000).

~900 descriptors is "a lot" when the limit is 1024.


> Also, on low traffic (late evening) with no CPU or mem issues at all,
> mgr:filedescriptors shows that there are about 2500 active file
> descriptors.

Please note that "2500 active descriptors" contradicts the 1024 limit
reported in mgr:info in your earlier email (quoted by me above). Squid
also reports the limit at startup. Here is an example (not from your logs):

> 2017/10/26 15:48:04.744| With 65535 file descriptors available


There are many email threads and probably some wiki documents that
discuss increasing the number of file descriptors available to your
Squid and numerous related caveats.


Cheers,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Vieri

> From: Alex Rousskov <[hidden email]>
>
>> # cat /etc/security/limits.d/01_squid.conf
>> * hard nofile 65535
>> * soft nofile 16384
>
> Your Squid does not know about the above configured limits. It thinks the limit is 1024.


That's interesting.

# grep "file descriptors available" cache.log
2017/10/24 11:49:33 kid1| With 1024 file descriptors available
2017/10/24 12:57:10 kid1| With 16384 file descriptors available
2017/10/24 12:57:52 kid1| With 16384 file descriptors available
2017/10/25 05:50:32 kid1| With 1024 file descriptors available
2017/10/25 09:20:15 kid1| With 1024 file descriptors available
2017/10/25 10:00:15 kid1| With 1024 file descriptors available
2017/10/25 12:00:15 kid1| With 1024 file descriptors available
2017/10/25 13:00:16 kid1| With 1024 file descriptors available
2017/10/25 13:20:16 kid1| With 1024 file descriptors available
2017/10/25 14:00:15 kid1| With 1024 file descriptors available
2017/10/26 04:09:30 kid1| With 1024 file descriptors available
2017/10/26 10:57:45 kid1| With 16384 file descriptors available

The issues I reported yesterday came up after 2017/10/26 04:09:30, so here's the relevant part in the log:

2017/10/26 04:09:27 kid1| Reconfiguring Squid Cache (version 3.5.27-20170916-re69e56c)...
2017/10/26 04:09:27 kid1| Closing HTTP port [::]:3128
2017/10/26 04:09:27 kid1| Closing HTTP port [::]:3129
2017/10/26 04:09:27 kid1| Closing HTTPS port [::]:3130
2017/10/26 04:09:27 kid1| Closing Pinger socket on FD 801
2017/10/26 04:09:27 kid1| Logfile: closing log daemon:/var/log/squid/access.log
2017/10/26 04:09:27 kid1| Logfile Daemon: closing log daemon:/var/log/squid/access.log
2017/10/26 04:09:27 kid1| FATAL: dying from an unhandled exception: IsConnOpen(conn)
2017/10/26 04:09:30| Pinger exiting.
2017/10/26 04:09:30 kid1| Set Current Directory to /var/cache/squid
2017/10/26 04:09:30 kid1| Starting Squid Cache version 3.5.27-20170916-re69e56c for x86_64-pc-linux-gnu...
2017/10/26 04:09:30 kid1| Service Name: squid
2017/10/26 04:09:30 kid1| Process ID 22581
2017/10/26 04:09:30 kid1| Process Roles: worker
2017/10/26 04:09:30 kid1| With 1024 file descriptors available

A "reload" was issued at that time.

When my openrc init script reloads, it runs the following command:

# /etc/init.d/squid reload
* Reloading squid with /usr/sbin/squid -k reconfigure -f /etc/squid/squid.conf -n squid ...                                                            [ ok ]

I issued another "reload" today, and I didn't come up with any "file descriptors available" messages in cache.log (as expected).
I also checked mgr:info, and all seems OK:

File descriptor usage for squid:
Maximum number of file descriptors:   16384
Largest file desc currently in use:   3067
Number of file desc currently in use: 2901
Files queued for open:                   0
Available number of file descriptors: 13483
Reserved number of file descriptors:   100
Store Disk files open:                   0

I also grep'ed for the log lines around "2017/10/25 05:50:32 kid1| With 1024 file descriptors available" where I see 1024 file descriptors for the first time.
It was a clean restart (with no "FATAL" error message).
The "restart" command was launched from crontab as a daily job (calling the openrc init script) just like it is launched at boot time from openrc (when the number of file descriptors is "as expected").

In particular, I have: KRB5_KTNAME="/etc/krb5.keytab" /usr/sbin/squid -YC -f /etc/squid/squid.conf -n squid

I'm wondering why Squid only detected 1024 file descriptors available after that restart.

Moving back in time:

- "2017/10/24 12:57:52 kid1| With 16384 file descriptors available" was probably a manual restart on my behalf (passing "restart" to the init script on the console).

- the one right before (2017/10/24 12:57:10) was probably also a manual restart as the init script probably exited non-zero due to an error I found in the log: "FATAL: dying from an unhandled exception: IsConnOpen(conn)"

- moving back a little (11:49:30), I had a "reload" with a fatal error:

2017/10/24 11:49:30.406 kid1| Reconfiguring Squid Cache (version 3.5.27-20170916-re69e56c)...
2017/10/24 11:49:30.406 kid1| Closing HTTP port [::]:3128
2017/10/24 11:49:30.406 kid1| Closing HTTP port [::]:3129
2017/10/24 11:49:30.406 kid1| Closing HTTPS port [::]:3130
2017/10/24 11:49:30.406 kid1| Closing Pinger socket on FD 788
2017/10/24 11:49:30.406 kid1| Logfile: closing log daemon:/var/log/squid/access.log
2017/10/24 11:49:30.406 kid1| Logfile Daemon: closing log daemon:/var/log/squid/access.log
2017/10/24 11:49:30.406 kid1| FATAL: dying from an unhandled exception: IsConnOpen(conn)
2017/10/24 11:49:33 kid1| Set Current Directory to /var/cache/squid
2017/10/24 11:49:33 kid1| Starting Squid Cache version 3.5.27-20170916-re69e56c for x86_64-pc-linux-gnu...
2017/10/24 11:49:33 kid1| Service Name: squid
2017/10/24 11:49:33 kid1| Process ID 30753
2017/10/24 11:49:33 kid1| Process Roles: worker
2017/10/24 11:49:33 kid1| With 1024 file descriptors available

- searching within the older rotated cache.log files I noticed this:

2017/10/20 10:42:12 kid1| With 16384 file descriptors available
2017/10/21 05:50:32 kid1| With 1024 file descriptors available

The restart at 05:50:32 is performed by a cron job.

Conclusion: it seems that the cron daemon I use (https://github.com/cronie-crond/cronie) is not honoring system-wide ulimits defined in /etc/security/limits.d/* as, eg.:
* hard nofile 65535
* soft nofile 16384

Weird.

I'm not going to be using the cron job anymore.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Vieri
In reply to this post by Alex Rousskov
Just in case someone else bumps into this issue, I can confirm that ulimits are not honored when running a cron job (at least from cronie).

In my case, if I set ulimits globally with:
* hard nofile 65535
* soft nofile 16384

I get this outside cron:
open files                      (-n) 16384

However, I get the following within cron:

open files                      (-n) 1024

as demonstrated by running the following test from crontab:

root ulimit -a >/tmp/ulimit 2>&1 3>&1

So beware.

I guess one can avoid this by setting ulimits within Squid's init script, and calling that within cron.

No idea whether this is a cron bug or feature.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Vieri
In reply to this post by Alex Rousskov
Just in case someone else bumps into this, I've had the same issue trying to restart Squid from inittab. System-wide ulimit settings are not honored when running scripts from either crontab or inittab.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cpu usage

Vieri
In reply to this post by Alex Rousskov
It's seems logical that the cron daemon needs to be restarted for ulimits to apply.
Init should also be restarted, but "init u" is not enough. A system reboot seems to be inevitable.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users