squid cpu usage
Locked
 
|
Hi,
During the last 3 days my Squid process has gone wild with 100% CPU usage once a day. Since this is a production system (never had this issue before), I "quickly restart" the Squid daemon, and that solves the issue until the next day. Restarting takes quite a while... Next time I get a 100% squid process that brings my proxy to a crawl HTTP-wise, what can I try in order to get more info, and possibly see the cause of this? Would squidclient mgr:info shed some light on this? Squid Object Cache: Version 3.5.27-20170916-re69e56c Thanks, Vieri _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
Administrator
|
On 25/10/17 01:34, Vieri wrote:
> Hi, > > During the last 3 days my Squid process has gone wild with 100% CPU usage once a day. Since this is a production system (never had this issue before), I "quickly restart" the Squid daemon, and that solves the issue until the next day. > Restarting takes quite a while... > > Next time I get a 100% squid process that brings my proxy to a crawl HTTP-wise, what can I try in order to get more info, and possibly see the cause of this? > Would squidclient mgr:info shed some light on this? That one is just the overview, so not likely to show more than that things are going slowly. Usually the 100% CPU is an indicator that the machine is swapping. So check the memory usage when it is happening. Next is probably the cache garbage collection. mgr:storedir report might show that. Or an strace showing a lot of cycles in the cache replacement calls. Amos _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
In reply to this post by Vieri
On 10/24/2017 06:34 AM, Vieri wrote:
> Next time I get a 100% squid process that brings my proxy to a crawl > HTTP-wise, what can I try in order to get more info, and possibly see > the cause of this? Here are two additional tricks to consider: * If 100% CPU usage lasts for more than a few seconds, then attaching gdb to the running Squid worker and generating a few backtraces (a few seconds apart) may identify the code that got stuck. Make sure you give gdb a non-stripped Squid binary when doing that. Squid wiki should have more instructions about the whole process. Please note that the above is less likely to help if your Squid continues to respond to new requests while you see ~100% CPU usage. You may increase your chances by blocking new traffic from reaching the problematic Squid (e.g., by redirecting all traffic to other Squid(s) that you have in load balancing or hot standby mode OR allowing all traffic to bypass Squid). If you cannot easily remove a problematic Squid from user traffic path, then use this problem as an excuse to fix your deployment architecture so that you can :-). * Another potentially useful trick is to log mgr:events every minute or so. Perhaps 100% CPU usage is associated with a particular regularly scheduled event? Please note that some modern Squids abuse the events API for frequently occurring notifications. If you see hundreds or more events in your log, this trick may be too expensive/intrusive for your environment. HTH, Alex. _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
Well, I've just had another issue with my CPU usage gone bonkers.
Here's the strace (compressed with gz): https://drive.google.com/file/d/0B-tpkY1LkI67WFBaNnVpR2lteEE/view?usp=sharing There are a lot of "Resource temporarily unavailable" messages. However, I'm seeing these messages even when my Squid process has been restarted, and everything seems to be OK from a "user experience" point of view. Do you see anything useful in the trace? If not, how can I improve the trace next time I have this issue? Here's what squidclient has to say during the "storm": HTTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Thu, 26 Oct 2017 08:34:43 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 26 Oct 2017 08:34:43 GMT Last-Modified: Thu, 26 Oct 2017 08:34:43 GMT X-Cache: MISS from inf-fw2 X-Cache-Lookup: MISS from inf-fw2:3128 Connection: close Squid Object Cache: Version 3.5.27-20170916-re69e56c Build Info: Service Name: squid Start Time: Thu, 26 Oct 2017 02:09:30 GMT Current Time: Thu, 26 Oct 2017 08:34:43 GMT Connection information for squid: Number of clients accessing cache: 471 Number of HTTP requests received: 267968 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 695.6 Average ICP messages per minute since start: 0.0 Select loop called: 10630162 times, 2.174 ms avg Cache information for squid: Hits as % of all requests: 5min: 2.3%, 60min: 2.5% Hits as % of bytes sent: 5min: 10.6%, 60min: -4.4% Memory hits as % of hit requests: 5min: 22.1%, 60min: 36.5% Disk hits as % of hit requests: 5min: 4.3%, 60min: 0.3% Storage Swap size: 29484 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 28492 KB Storage Mem capacity: 87.0% used, 13.0% free Mean Object Size: 22.29 KB Requests given to unlinkd: 17424 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 2.64709 0.27332 Cache Misses: 1.54242 0.27332 Cache Hits: 0.64968 0.00000 Near Hits: 1.31166 0.22004 Not-Modified Replies: 0.44492 0.00000 DNS Lookups: 0.12472 0.07968 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 23113.055 seconds CPU Time: 1599.750 seconds CPU Usage: 6.92% CPU Usage, 5 minute avg: 99.80% CPU Usage, 60 minute avg: 32.48% Maximum Resident Size: 987392 KB Page faults with physical i/o: 0 Memory accounted for: Total accounted: 148456 KB memPoolAlloc calls: 87698086 memPoolFree calls: 89179152 File descriptor usage for squid: Maximum number of file descriptors: 1024 Largest file desc currently in use: 913 Number of file desc currently in use: 791 Files queued for open: 0 Available number of file descriptors: 233 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 1642 StoreEntries 1640 StoreEntries with MemObjects 1375 Hot Object Cache Items 1323 on-disk objects A little later (still during the "storm"): HTTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Thu, 26 Oct 2017 08:40:13 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 26 Oct 2017 08:40:13 GMT Last-Modified: Thu, 26 Oct 2017 08:40:13 GMT X-Cache: MISS from inf-fw2 X-Cache-Lookup: MISS from inf-fw2:3128 Connection: close Squid Object Cache: Version 3.5.27-20170916-re69e56c Build Info: Service Name: squid Start Time: Thu, 26 Oct 2017 02:09:30 GMT Current Time: Thu, 26 Oct 2017 08:40:13 GMT Connection information for squid: Number of clients accessing cache: 472 Number of HTTP requests received: 274183 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 701.8 Average ICP messages per minute since start: 0.0 Select loop called: 10679512 times, 2.195 ms avg Cache information for squid: Hits as % of all requests: 5min: 0.2%, 60min: 2.3% Hits as % of bytes sent: 5min: -3.2%, 60min: -3.6% Memory hits as % of hit requests: 5min: 9.1%, 60min: 35.2% Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.4% Storage Swap size: 29484 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 27304 KB Storage Mem capacity: 83.3% used, 16.7% free Mean Object Size: 16.25 KB Requests given to unlinkd: 17424 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 1.05672 0.30459 Cache Misses: 0.94847 0.30459 Cache Hits: 0.00000 0.00000 Near Hits: 0.58309 0.23230 Not-Modified Replies: 0.00000 0.00000 DNS Lookups: 0.11405 0.08334 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 23442.516 seconds CPU Time: 1888.520 seconds CPU Usage: 8.06% CPU Usage, 5 minute avg: 86.51% CPU Usage, 60 minute avg: 39.31% Maximum Resident Size: 988768 KB Page faults with physical i/o: 0 Memory accounted for: Total accounted: 151913 KB memPoolAlloc calls: 89548742 memPoolFree calls: 91062557 File descriptor usage for squid: Maximum number of file descriptors: 1024 Largest file desc currently in use: 913 Number of file desc currently in use: 814 Files queued for open: 0 Available number of file descriptors: 210 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 2139 StoreEntries 2135 StoreEntries with MemObjects 1863 Hot Object Cache Items 1814 on-disk objects Memory info: TTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Thu, 26 Oct 2017 08:34:51 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 26 Oct 2017 08:34:51 GMT Last-Modified: Thu, 26 Oct 2017 08:34:51 GMT X-Cache: MISS from inf-fw2 X-Cache-Lookup: MISS from inf-fw2:3128 Connection: close Current memory usage: Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate (bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec 64K Buffer 65536 1366 87424 88064 0.03 58.871 1340 85760 88064 0.03 98.097 26 1664 5248 283321 0.317 35.761 21.750 mem_node 4136 7326 29591 30843 1.38 19.926 7096 28662 30843 1.38 96.860 230 929 6802 558119 0.624 4.446 48.467 32K Buffer 32768 129 4128 5984 0.77 2.780 123 3936 5984 0.77 95.349 6 192 2272 49708 0.056 3.137 4.285 ExternalACLEntry 256 16447 4112 4953 4.32 2.769 16447 4112 4953 4.32 100.000 0 0 4953 591 0.001 0.000 5.834 16K Buffer 16384 189 3024 6992 4.35 2.036 167 2672 6992 4.35 88.360 22 352 3152 223171 0.250 7.042 16.461 8K Buffer 8192 368 2944 3464 0.17 1.982 347 2776 3464 0.17 94.293 21 168 1112 288150 0.322 4.546 27.526 Short Strings 40 73543 2873 3066 0.57 1.935 73511 2872 3066 0.57 99.956 32 2 867 50208368 56.164 3.868 3944.297 4K Buffer 4096 497 1988 2696 1.51 1.339 466 1864 2696 1.51 93.763 31 124 1012 397819 0.445 3.138 37.339 HttpHeaderEntry 56 30961 1694 1862 0.57 1.140 30959 1694 1862 0.57 99.994 2 1 590 12908222 14.439 1.392 1001.067 HttpRequest 1784 891 1553 1631 0.02 1.045 885 1542 1631 0.02 99.327 6 11 295 613472 0.686 2.108 52.175 Medium Strings 128 6801 851 954 4.33 0.572 6644 831 954 4.33 97.692 157 20 360 2746616 3.072 0.677 219.037 4KB Strings 4096 208 832 1168 1.42 0.560 143 572 1168 1.42 68.750 65 260 444 262956 0.294 2.074 23.268 HttpReply 280 2884 789 861 0.57 0.531 2865 784 861 0.57 99.341 19 6 291 1030249 1.152 0.556 82.441 2K Buffer 2048 374 748 1416 4.32 0.504 346 692 1416 4.32 92.513 28 56 768 4412257 4.936 17.404 317.415 16KB Strings 16384 44 704 4400 1.71 0.474 40 640 4400 1.71 90.909 4 64 2368 76985 0.086 2.429 7.763 cbdata clientReplyContext (25) 4352 140 595 706 0.08 0.401 122 519 706 0.08 87.143 18 77 489 264790 0.296 2.219 23.215 cbdata ClientSocketContext (24) 4256 140 582 690 0.08 0.392 122 508 690 0.08 87.143 18 75 478 264789 0.296 2.170 23.215 MemObject 328 1682 539 794 4.33 0.363 1663 533 794 4.33 98.870 19 7 339 255955 0.286 0.162 23.520 Long Strings 512 958 479 811 4.35 0.323 950 475 811 4.35 99.165 8 4 369 677454 0.758 0.668 56.843 Comm::Connection 200 1533 300 307 0.28 0.202 1495 292 307 0.28 97.521 38 8 70 606480 0.678 0.234 65.623 cbdata Server (20) 520 549 279 304 0.44 0.188 528 269 304 0.44 96.175 21 11 114 108934 0.122 0.109 12.423 cbdata BodyPipe (38) 152 1659 247 293 0.56 0.166 1640 244 293 0.56 98.855 19 3 109 215397 0.241 0.063 15.348 ClientInfo 448 471 207 207 0.09 0.139 471 207 207 0.09 100.000 0 0 0 0 0.000 0.000 0.007 1KB Strings 1024 189 189 208 0.29 0.127 156 156 208 0.29 82.540 33 33 107 134127 0.150 0.265 13.835 cbdata MemBuf (12) 64 2999 188 201 0.57 0.126 2963 186 201 0.57 98.800 36 3 67 1535006 1.717 0.189 123.213 StoreEntry 104 1683 171 1670 6.42 0.115 1665 170 1670 6.42 98.930 18 2 1478 256816 0.287 0.051 23.520 cbdata ModXact (41) 512 319 160 162 0.12 0.107 308 154 162 0.12 96.552 11 6 16 131784 0.147 0.130 10.001 netdbEntry 168 970 160 165 1.91 0.107 970 160 165 1.91 100.000 0 0 17 261 0.000 0.000 0.180 fqdncache_entry 160 934 146 152 2.16 0.098 926 145 152 2.16 99.143 8 2 8 30025 0.034 0.009 2.861 ipcache_entry 128 927 116 119 2.19 0.078 921 116 119 2.19 99.353 6 1 5 16480 0.018 0.004 1.682 MemBlob 48 2355 111 145 1.30 0.074 2226 105 145 1.30 94.522 129 7 33 3054760 3.417 0.282 267.710 HttpHdrCc 96 1112 105 159 4.35 0.070 1067 101 159 4.35 95.953 45 5 70 485419 0.543 0.090 37.649 cbdata idns_query (9) 8696 12 102 391 2.28 0.069 2 17 391 2.28 16.667 10 85 391 62918 0.070 1.054 5.848 net_db_name 32 2228 70 72 0.16 0.047 2228 70 72 0.16 100.000 0 0 6 493 0.001 0.000 0.275 cbdata ClientHttpRequest (23) 392 174 67 77 0.01 0.045 152 59 77 0.01 87.356 22 9 46 264283 0.296 0.200 23.215 LRU policy node 24 2692 64 384 6.42 0.042 2666 63 384 6.42 99.034 26 1 341 32395 0.036 0.001 4.782 cbdata IdleConnList (18) 4160 15 61 252 1.97 0.041 14 57 252 1.97 93.333 1 5 200 13686 0.015 0.110 0.972 cbdata store_client (31) 160 288 45 56 1.52 0.030 278 44 56 1.52 96.528 10 2 27 275536 0.308 0.085 24.485 NotePairs::Entry 48 927 44 46 2.92 0.029 924 44 46 2.92 99.676 3 1 46 18571 0.021 0.002 1.796 cbdata clientStreamNode (26) 128 280 35 42 0.08 0.024 244 31 42 0.08 87.143 36 5 29 529580 0.592 0.131 46.431 cbdata ServerBump (30) 104 259 27 34 1.52 0.018 240 25 34 1.52 92.664 19 2 17 81041 0.091 0.016 9.000 MD5 digest 16 1683 27 257 6.42 0.018 1665 27 257 6.42 98.930 18 1 228 287894 0.322 0.009 26.086 MimeEntry 128 177 23 23 6.42 0.015 177 23 23 6.42 100.000 0 0 23 1062 0.001 0.000 0.063 cbdata HttpStateData (36) 320 58 19 19 0.07 0.012 37 12 19 0.07 63.793 21 7 15 133511 0.149 0.082 10.258 cbdata FwdState (32) 176 101 18 18 0.28 0.012 59 11 18 0.28 58.416 42 8 13 211946 0.237 0.072 18.843 cbdata helper_server (11) 240 55 13 13 0.09 0.009 55 13 13 0.09 100.000 0 0 9 209 0.000 0.000 0.019 cbdata ConnOpener (34) 136 64 9 9 0.01 0.006 4 1 9 0.01 6.250 60 8 9 98344 0.110 0.026 10.905 cbdata PeerConnector (35) 136 61 9 9 0.01 0.005 18 3 9 0.01 29.508 43 6 9 77220 0.086 0.020 8.531 Acl::AndNode 160 44 7 7 6.42 0.005 44 7 7 6.42 100.000 0 0 7 264 0.000 0.000 0.016 cbdata Iterator (39) 168 31 6 8 0.77 0.003 23 4 8 0.77 74.194 8 2 6 131859 0.148 0.043 9.978 cbdata ACLFilledChecklist (22) 464 11 5 11 2.28 0.003 10 5 11 2.28 90.909 1 1 11 756320 0.846 0.676 66.886 cbdata ModXactLauncher (40) 160 30 5 8 0.77 0.003 20 4 8 0.77 66.667 10 2 5 131871 0.148 0.041 9.978 ACLStrategised 160 19 3 3 6.42 0.002 19 3 3 6.42 100.000 0 0 3 114 0.000 0.000 0.007 cbdata ServiceRep (17) 344 7 3 3 0.78 0.002 7 3 3 0.78 100.000 0 0 0 0 0.000 0.000 0.000 cbdata Tree (4) 216 11 3 3 6.42 0.002 11 3 3 6.42 100.000 0 0 3 66 0.000 0.000 0.004 cbdata AccessCheck (29) 152 15 3 4 0.57 0.001 0 0 4 0.57 0.000 15 3 4 352429 0.394 0.103 29.244 Acl::NotNode 160 14 3 3 6.42 0.001 14 3 3 6.42 100.000 0 0 3 84 0.000 0.000 0.005 acl_ip_data 96 19 2 2 6.42 0.001 19 2 2 6.42 100.000 0 0 2 289 0.000 0.000 0.016 cbdata ClientRequestContext (27) 104 17 2 4 0.08 0.001 11 2 4 0.08 64.706 6 1 4 262994 0.294 0.053 22.940 cbdata WriteRequest (47) 80 20 2 6 4.28 0.001 0 0 6 4.28 0.000 20 2 6 190311 0.213 0.029 15.692 cbdata UFSStoreState (45) 176 8 2 3 0.26 0.001 0 0 3 0.26 0.000 8 2 3 28910 0.032 0.010 2.397 Helper::Request 48 28 2 2 0.08 0.001 16 1 2 0.08 57.143 12 1 2 158681 0.178 0.015 14.377 HttpHdrContRange 24 48 2 2 0.18 0.001 48 2 2 0.18 100.000 0 0 1 410 0.000 0.000 0.036 RegexList 88 10 1 1 6.42 0.001 10 1 1 6.42 100.000 0 0 1 60 0.000 0.000 0.004 cbdata externalAclState (14) 88 10 1 2 0.28 0.001 10 1 2 0.28 100.000 0 0 2 81199 0.091 0.014 6.014 ACLSourceIP 144 6 1 1 6.42 0.001 6 1 1 6.42 100.000 0 0 1 36 0.000 0.000 0.002 cbdata DiskdFile (46) 104 8 1 2 0.26 0.001 0 0 2 0.26 0.000 8 1 2 28910 0.032 0.006 2.397 ev_entry 48 17 1 2 0.24 0.001 14 1 2 0.24 82.353 3 1 1 63038 0.071 0.006 3.019 ACLStrategised 160 5 1 1 6.42 0.001 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002 cbdata helper (10) 168 3 1 1 0.78 0.000 3 1 1 0.78 100.000 0 0 1 17 0.000 0.000 0.001 wordlist 16 31 1 2 0.78 0.000 31 1 2 0.78 100.000 0 0 1 837 0.001 0.000 0.064 cbdata CbDataList (2) 40 12 1 1 6.42 0.000 12 1 1 6.42 100.000 0 0 1 72 0.000 0.000 0.004 cbdata ErrorState (37) 240 2 1 3 0.05 0.000 0 0 3 0.05 0.000 2 1 3 53117 0.059 0.025 4.356 ACLExternal 152 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001 ACLStringData 56 8 1 1 6.42 0.000 8 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001 ACLRegexData 16 25 1 1 6.42 0.000 25 1 1 6.42 100.000 0 0 1 54 0.000 0.000 0.003 cbdata CbDataList (1) 40 10 1 1 6.42 0.000 10 1 1 6.42 100.000 0 0 1 60 0.000 0.000 0.004 cbdata external_acl (5) 192 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001 cbdata generic_cbdata (21) 32 12 1 2 2.28 0.000 1 1 2 2.28 8.333 11 1 2 86571 0.097 0.005 7.694 ACLCertificateData 120 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 acl_name_list 72 5 1 1 6.42 0.000 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002 cbdata Logfile (15) 352 1 1 1 5.76 0.000 1 1 1 5.76 100.000 0 0 1 6 0.000 0.000 0.001 ACLStrategised 160 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001 cbdata TcpAcceptor (19) 104 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001 ACLDestinationIP 144 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001 ACLUserData 64 4 1 1 6.42 0.000 4 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000 cbdata ps_state (33) 256 1 1 1 0.05 0.000 0 0 1 0.05 0.000 1 1 1 215092 0.241 0.106 18.843 cbdata RemovalPolicy (7) 104 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata statefulhelper (13) 192 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 AclDenyInfoList 32 5 1 1 6.42 0.000 5 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002 ACLStrategised 160 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000 ACLProxyAuth 152 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000 ACLDomainData 16 9 1 1 6.42 0.000 9 1 1 6.42 100.000 0 0 1 42 0.000 0.000 0.002 cbdata external_acl_data (6) 48 3 1 1 6.42 0.000 3 1 1 6.42 100.000 0 0 1 18 0.000 0.000 0.001 external_acl_format 64 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 12 0.000 0.000 0.001 cbdata StoreSearchHashIndex (28) 104 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 0 0.000 0.000 0.000 ACLHTTPHeaderData 48 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLSslErrorData 16 6 1 1 6.42 0.000 6 1 1 6.42 100.000 0 0 1 30 0.000 0.000 0.002 cbdata nsvc (8) 72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 75 0.000 0.000 0.009 StoreSwapLogData 72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 89577 0.100 0.012 4.858 cbdata RemovalPurgeWalker (44) 72 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 1961 0.002 0.000 0.211 cbdata ReadRequest (48) 72 1 1 1 6.30 0.000 0 0 1 6.30 0.000 1 1 1 111 0.000 0.000 0.006 cbdata CbDataList (3) 64 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000 dwrite_q 48 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 89578 0.100 0.008 4.858 FwdServer 24 2 1 1 0.05 0.000 0 0 1 0.05 0.000 2 1 1 318128 0.356 0.015 25.710 ACLNoteData 40 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 CacheDigest 40 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLHierCodeData 32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLMethodData 16 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 1 6 0.000 0.000 0.000 ACLTimeData 32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLAtStepData 32 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLASN 16 2 1 1 6.42 0.000 2 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 StoreMetaMD5 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398 StoreMetaSTDLFS 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398 StoreMetaURL 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398 StoreMetaObjSize 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 29043 0.032 0.002 2.398 UFSStoreState::_queued_write 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 193581 0.217 0.012 15.692 StoreMetaVary 32 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 7514 0.008 0.000 0.565 ACLProtocolData 16 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLServerNameData 16 1 1 1 6.42 0.000 1 1 1 6.42 100.000 0 0 0 0 0.000 0.000 0.000 link_list 16 1 1 1 6.42 0.000 0 0 1 6.42 0.000 1 1 1 193581 0.217 0.006 15.692 dlink_node 24 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.001 cbdata RebuildState (16) 944 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 0 0.000 0.000 0.000 cbdata OptXactLauncher (42) 128 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 1 0.000 0.000 0.000 cbdata OptXact (43) 320 0 0 1 6.42 0.000 0 0 1 6.42 -1.000 0 0 1 1 0.000 0.000 0.000 HttpHdrRange 32 0 0 1 1.82 0.000 0 0 1 1.82 -1.000 0 0 1 314 0.000 0.000 0.020 HttpHdrRangeSpec 16 0 0 1 1.82 0.000 0 0 1 1.82 -1.000 0 0 1 314 0.000 0.000 0.020 cbdata CbDataList (49) 96 0 0 1 6.21 0.000 0 0 1 6.21 -1.000 0 0 1 4662 0.005 0.001 1.421 HttpHdrSc 16 0 0 1 2.84 0.000 0 0 1 2.84 -1.000 0 0 1 12 0.000 0.000 0.005 HttpHdrScTarget 88 0 0 1 2.84 0.000 0 0 1 2.84 -1.000 0 0 1 12 0.000 0.000 0.005 cbdata BodySink (50) 80 0 0 1 2.34 0.000 0 0 1 2.34 -1.000 0 0 1 0 0.000 0.000 0.000 Auth::Negotiate::User 184 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000 Auth::Negotiate::UserRequest 88 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000 cbdata StateData (51) 48 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000 cbdata helper_stateful_server (52) 232 0 0 1 1.63 0.000 0 0 1 1.63 -1.000 0 0 1 0 0.000 0.000 0.000 Total 1 170288 148501 150273 0.03 100.000 168782 144272 146333 0.03 97.152 1506 4229 18484 87746747 98.156 98.226 31638.278 Cumulative allocated volume: 51.921 GB Current overhead: 43294 bytes (0.029%) Idle pool limit: 5.00 MB Total Pools created: 138 Pools ever used: 135 (shown above) Currently in use: 102 String Pool Impact (%strings) (%volume) Short Strings 90 47 Medium Strings 8 14 Long Strings 1 8 1KB Strings 0 3 4KB Strings 0 9 16KB Strings 0 11 Other Strings 0 9 Large buffers: 0 (0 KB) Again, a little later: HTTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Thu, 26 Oct 2017 08:39:42 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 26 Oct 2017 08:39:42 GMT Last-Modified: Thu, 26 Oct 2017 08:39:42 GMT X-Cache: MISS from inf-fw2 X-Cache-Lookup: MISS from inf-fw2:3128 Connection: close Current memory usage: Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate (bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec 64K Buffer 65536 1366 87424 89280 0.01 59.621 1364 87296 89280 0.01 99.854 2 128 5248 288796 0.317 35.771 19.344 mem_node 4136 6798 27458 30843 1.46 18.725 6783 27397 30843 1.46 99.779 15 61 6802 568521 0.624 4.444 38.660 ExternalACLEntry 256 17998 4500 4953 4.40 3.069 17998 4500 4953 4.40 100.000 0 0 4953 591 0.001 0.000 5.325 32K Buffer 32768 136 4352 5984 0.85 2.968 134 4288 5984 0.85 98.529 2 64 2272 51135 0.056 3.167 5.133 Short Strings 40 80835 3158 3208 0.00 2.153 80636 3150 3208 0.00 99.754 199 8 867 51148695 56.142 3.867 3308.818 16K Buffer 16384 169 2704 6992 4.43 1.844 169 2704 6992 4.43 100.000 0 0 3152 227118 0.249 7.033 13.881 8K Buffer 8192 290 2320 3464 0.25 1.582 282 2256 3464 0.25 97.241 8 64 1112 294270 0.323 4.556 21.610 HttpHeaderEntry 56 34149 1868 1899 0.00 1.274 34114 1866 1899 0.00 99.898 35 2 590 13142150 14.425 1.391 835.378 4K Buffer 4096 439 1756 2696 1.59 1.198 409 1636 2696 1.59 93.166 30 120 1012 406215 0.446 3.145 29.469 HttpRequest 1784 873 1521 1631 0.10 1.037 859 1497 1631 0.10 98.396 14 25 295 625953 0.687 2.111 43.460 2K Buffer 2048 539 1078 1416 4.40 0.735 535 1070 1416 4.40 99.258 4 8 768 4472923 4.910 17.313 209.220 Medium Strings 128 7899 988 1002 0.00 0.673 7893 987 1002 0.00 99.924 6 1 360 2798935 3.072 0.677 186.051 16KB Strings 16384 59 944 4400 1.79 0.644 54 864 4400 1.79 91.525 5 80 2368 78587 0.086 2.434 5.827 HttpReply 280 3294 901 911 0.00 0.614 3288 900 911 0.00 99.818 6 2 291 1049980 1.152 0.556 70.385 MemObject 328 2077 666 794 4.41 0.454 2071 664 794 4.41 99.711 6 2 339 261223 0.287 0.162 20.347 4KB Strings 4096 149 596 1168 1.50 0.406 142 568 1168 1.50 95.302 7 28 444 267713 0.294 2.072 16.635 cbdata clientReplyContext (25) 4352 118 502 706 0.16 0.342 109 464 706 0.16 92.373 9 39 489 270088 0.296 2.222 18.736 cbdata ClientSocketContext (24) 4256 118 491 690 0.16 0.334 109 454 690 0.16 92.373 9 38 478 270087 0.296 2.173 18.736 Long Strings 512 751 376 811 4.43 0.256 718 359 811 4.43 95.606 33 17 369 692635 0.760 0.670 52.968 Comm::Connection 200 1499 293 307 0.36 0.200 1464 286 307 0.36 97.665 35 7 70 623501 0.684 0.236 59.278 cbdata Server (20) 520 563 286 304 0.52 0.195 542 276 304 0.52 96.270 21 11 114 111986 0.123 0.110 10.935 cbdata BodyPipe (38) 152 1798 267 293 0.64 0.182 1796 267 293 0.64 99.889 2 1 109 219412 0.241 0.063 15.090 cbdata MemBuf (12) 64 3382 212 215 0.00 0.144 3377 212 215 0.00 99.852 5 1 67 1564605 1.717 0.189 104.318 StoreEntry 104 2081 212 1670 6.50 0.144 2075 211 1670 6.50 99.712 6 1 1478 262085 0.288 0.052 20.347 ClientInfo 448 472 207 207 0.02 0.141 472 207 207 0.02 100.000 0 0 0 0 0.000 0.000 0.003 cbdata ModXact (41) 512 309 155 164 0.07 0.105 308 154 164 0.07 99.676 1 1 16 134215 0.147 0.130 8.549 netdbEntry 168 925 152 165 1.99 0.103 925 152 165 1.99 100.000 0 0 17 286 0.000 0.000 0.196 fqdncache_entry 160 931 146 152 2.24 0.099 920 144 152 2.24 98.818 11 2 8 30868 0.034 0.009 3.083 cbdata idns_query (9) 8696 16 136 391 2.36 0.093 0 0 391 2.36 0.000 16 136 391 64579 0.071 1.061 5.881 ipcache_entry 128 923 116 119 2.28 0.079 923 116 119 2.28 100.000 0 0 5 16903 0.019 0.004 1.528 MemBlob 48 2438 115 145 1.38 0.078 2304 108 145 1.38 94.504 134 7 33 3118212 3.423 0.283 223.259 HttpHdrCc 96 1159 109 159 4.43 0.074 1158 109 159 4.43 99.914 1 1 70 493512 0.542 0.090 28.676 1KB Strings 1024 101 101 208 0.37 0.069 97 97 208 0.37 96.040 4 4 107 137459 0.151 0.266 11.691 LRU policy node 24 3530 83 384 6.50 0.056 3530 83 384 6.50 100.000 0 0 341 33829 0.037 0.002 9.150 net_db_name 32 2187 69 72 0.24 0.047 2187 69 72 0.24 100.000 0 0 6 540 0.001 0.000 0.336 cbdata ClientHttpRequest (23) 392 144 56 77 0.10 0.038 131 51 77 0.10 90.972 13 5 46 269589 0.296 0.200 18.736 cbdata IdleConnList (18) 4160 12 49 252 2.05 0.033 11 45 252 2.05 91.667 1 5 200 13861 0.015 0.109 0.663 NotePairs::Entry 48 995 47 47 0.00 0.032 992 47 47 0.00 99.698 3 1 46 18882 0.021 0.002 1.346 cbdata store_client (31) 160 263 42 56 1.60 0.028 256 40 56 1.60 97.338 7 2 27 281020 0.308 0.085 19.251 MD5 digest 16 2081 33 257 6.50 0.022 2075 33 257 6.50 99.712 6 1 228 294523 0.323 0.009 25.016 cbdata clientStreamNode (26) 128 236 30 42 0.16 0.020 218 28 42 0.16 92.373 18 3 29 540176 0.593 0.131 37.472 cbdata ServerBump (30) 104 234 24 34 1.60 0.016 219 23 34 1.60 93.590 15 2 17 83160 0.091 0.016 7.626 MimeEntry 128 177 23 23 6.50 0.015 177 23 23 6.50 100.000 0 0 23 1062 0.001 0.000 0.000 cbdata helper_server (11) 240 55 13 13 0.17 0.009 55 13 13 0.17 100.000 0 0 9 209 0.000 0.000 0.000 cbdata HttpStateData (36) 320 32 10 19 0.15 0.007 32 10 19 0.15 100.000 0 0 15 135965 0.149 0.082 8.642 cbdata FwdState (32) 176 56 10 19 0.08 0.007 43 8 19 0.08 76.786 13 3 14 216432 0.238 0.072 15.828 Acl::AndNode 160 44 7 7 6.50 0.005 44 7 7 6.50 100.000 0 0 7 264 0.000 0.000 0.000 cbdata ACLFilledChecklist (22) 464 10 5 11 2.36 0.003 1 1 11 2.36 10.000 9 5 11 772759 0.848 0.678 56.648 ACLStrategised 160 19 3 3 6.50 0.002 19 3 3 6.50 100.000 0 0 3 114 0.000 0.000 0.000 cbdata PeerConnector (35) 136 22 3 9 0.08 0.002 10 2 9 0.08 45.455 12 2 9 79214 0.087 0.020 7.172 cbdata Iterator (39) 168 15 3 8 0.85 0.002 15 3 8 0.85 100.000 0 0 6 134291 0.147 0.043 8.549 cbdata ServiceRep (17) 344 7 3 3 0.86 0.002 7 3 3 0.86 100.000 0 0 0 0 0.000 0.000 0.000 cbdata ModXactLauncher (40) 160 15 3 8 0.85 0.002 14 3 8 0.85 93.333 1 1 5 134308 0.147 0.041 8.549 cbdata Tree (4) 216 11 3 3 6.50 0.002 11 3 3 6.50 100.000 0 0 3 66 0.000 0.000 0.000 Acl::NotNode 160 14 3 3 6.50 0.001 14 3 3 6.50 100.000 0 0 3 84 0.000 0.000 0.000 acl_ip_data 96 19 2 2 6.50 0.001 19 2 2 6.50 100.000 0 0 2 289 0.000 0.000 0.000 cbdata ConnOpener (34) 136 11 2 10 0.08 0.001 1 1 10 0.08 9.091 10 2 10 101038 0.111 0.026 9.638 cbdata ClientRequestContext (27) 104 12 2 4 0.16 0.001 1 1 4 0.16 8.333 11 2 4 268308 0.294 0.053 18.499 HttpHdrContRange 24 48 2 2 0.27 0.001 48 2 2 0.27 100.000 0 0 1 410 0.000 0.000 0.000 cbdata AccessCheck (29) 152 7 2 4 0.65 0.001 0 0 4 0.65 0.000 7 2 4 359554 0.395 0.103 24.631 RegexList 88 10 1 1 6.50 0.001 10 1 1 6.50 100.000 0 0 1 60 0.000 0.000 0.000 ACLSourceIP 144 6 1 1 6.50 0.001 6 1 1 6.50 100.000 0 0 1 36 0.000 0.000 0.000 Helper::Request 48 17 1 2 0.17 0.001 4 1 2 0.17 23.529 13 1 2 162246 0.178 0.015 12.498 ACLStrategised 160 5 1 1 6.50 0.001 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000 cbdata WriteRequest (47) 80 10 1 6 4.36 0.001 0 0 6 4.36 0.000 10 1 6 196846 0.216 0.030 23.134 cbdata UFSStoreState (45) 176 4 1 3 0.34 0.000 4 1 3 0.34 100.000 0 0 3 30203 0.033 0.010 4.587 ev_entry 48 14 1 2 0.32 0.000 14 1 2 0.32 100.000 0 0 1 63810 0.070 0.006 2.712 cbdata externalAclState (14) 88 7 1 2 0.08 0.000 1 1 2 0.08 14.286 6 1 2 82716 0.091 0.014 5.373 cbdata generic_cbdata (21) 32 16 1 2 2.36 0.000 4 1 2 2.36 25.000 12 1 2 89441 0.098 0.005 10.084 cbdata helper (10) 168 3 1 1 0.86 0.000 3 1 1 0.86 100.000 0 0 1 17 0.000 0.000 0.000 wordlist 16 31 1 2 0.86 0.000 31 1 2 0.86 100.000 0 0 1 837 0.001 0.000 0.000 cbdata CbDataList (2) 40 12 1 1 6.50 0.000 12 1 1 6.50 100.000 0 0 1 72 0.000 0.000 0.000 ACLExternal 152 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000 ACLStringData 56 8 1 1 6.50 0.000 8 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000 cbdata DiskdFile (46) 104 4 1 2 0.34 0.000 4 1 2 0.34 100.000 0 0 2 30203 0.033 0.006 4.587 ACLRegexData 16 25 1 1 6.50 0.000 25 1 1 6.50 100.000 0 0 1 54 0.000 0.000 0.000 cbdata CbDataList (1) 40 10 1 1 6.50 0.000 10 1 1 6.50 100.000 0 0 1 60 0.000 0.000 0.000 cbdata external_acl (5) 192 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000 ACLCertificateData 120 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 acl_name_list 72 5 1 1 6.50 0.000 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000 cbdata Logfile (15) 352 1 1 1 5.84 0.000 1 1 1 5.84 100.000 0 0 1 6 0.000 0.000 0.003 ACLStrategised 160 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000 cbdata TcpAcceptor (19) 104 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000 ACLDestinationIP 144 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000 ACLUserData 64 4 1 1 6.50 0.000 4 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000 cbdata ps_state (33) 256 1 1 1 0.13 0.000 0 0 1 0.13 0.000 1 1 1 219697 0.241 0.106 15.828 cbdata ErrorState (37) 240 1 1 3 0.13 0.000 0 0 3 0.13 0.000 1 1 3 53986 0.059 0.024 3.008 cbdata RemovalPolicy (7) 104 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 cbdata statefulhelper (13) 192 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 AclDenyInfoList 32 5 1 1 6.50 0.000 5 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000 ACLStrategised 160 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000 ACLProxyAuth 152 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000 ACLDomainData 16 9 1 1 6.50 0.000 9 1 1 6.50 100.000 0 0 1 42 0.000 0.000 0.000 cbdata external_acl_data (6) 48 3 1 1 6.50 0.000 3 1 1 6.50 100.000 0 0 1 18 0.000 0.000 0.000 external_acl_format 64 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 12 0.000 0.000 0.000 cbdata StoreSearchHashIndex (28) 104 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 0 0.000 0.000 0.000 ACLHTTPHeaderData 48 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 ACLSslErrorData 16 6 1 1 6.50 0.000 6 1 1 6.50 100.000 0 0 1 30 0.000 0.000 0.000 cbdata CbDataList (3) 64 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000 FwdServer 24 2 1 1 0.13 0.000 0 0 1 0.13 0.000 2 1 1 324323 0.356 0.015 21.304 ACLNoteData 40 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 CacheDigest 40 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 ACLHierCodeData 32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 ACLMethodData 16 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 1 6 0.000 0.000 0.000 ACLTimeData 32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 ACLAtStepData 32 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 ACLASN 16 2 1 1 6.50 0.000 2 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 StoreMetaMD5 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587 StoreMetaSTDLFS 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587 StoreMetaURL 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587 StoreMetaObjSize 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 30374 0.033 0.002 4.587 UFSStoreState::_queued_write 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 200314 0.220 0.012 23.134 StoreMetaVary 32 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 7963 0.009 0.000 1.559 ACLProtocolData 16 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 ACLServerNameData 16 1 1 1 6.50 0.000 1 1 1 6.50 100.000 0 0 0 0 0.000 0.000 0.000 link_list 16 1 1 1 6.50 0.000 0 0 1 6.50 0.000 1 1 1 200314 0.220 0.006 23.134 dlink_node 24 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000 dwrite_q 48 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 91807 0.101 0.008 7.667 cbdata nsvc (8) 72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 75 0.000 0.000 0.003 cbdata RebuildState (16) 944 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 0 0.000 0.000 0.000 StoreSwapLogData 72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 91806 0.101 0.012 7.667 cbdata OptXactLauncher (42) 128 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 1 0.000 0.000 0.000 cbdata OptXact (43) 320 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 1 0.000 0.000 0.000 cbdata RemovalPurgeWalker (44) 72 0 0 1 6.50 0.000 0 0 1 6.50 -1.000 0 0 1 2048 0.002 0.000 0.312 cbdata ReadRequest (48) 72 0 0 1 6.38 0.000 0 0 1 6.38 -1.000 0 0 1 111 0.000 0.000 0.000 HttpHdrRange 32 0 0 1 1.90 0.000 0 0 1 1.90 -1.000 0 0 1 314 0.000 0.000 0.000 HttpHdrRangeSpec 16 0 0 1 1.90 0.000 0 0 1 1.90 -1.000 0 0 1 314 0.000 0.000 0.000 cbdata CbDataList (49) 96 0 0 1 6.29 0.000 0 0 1 6.29 -1.000 0 0 1 4662 0.005 0.001 0.000 HttpHdrSc 16 0 0 1 2.92 0.000 0 0 1 2.92 -1.000 0 0 1 12 0.000 0.000 0.000 HttpHdrScTarget 88 0 0 1 2.92 0.000 0 0 1 2.92 -1.000 0 0 1 12 0.000 0.000 0.000 cbdata BodySink (50) 80 0 0 1 2.42 0.000 0 0 1 2.42 -1.000 0 0 1 0 0.000 0.000 0.000 Auth::Negotiate::User 184 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000 Auth::Negotiate::UserRequest 88 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000 cbdata StateData (51) 48 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000 cbdata helper_stateful_server (52) 232 0 0 1 1.71 0.000 0 0 1 1.71 -1.000 0 0 1 0 0.000 0.000 0.000 Total 1 185173 146634 151763 0.00 100.000 184369 145755 146812 0.00 99.401 804 880 18484 89409360 98.137 98.206 312807.853 Cumulative allocated volume: 52.910 GB Current overhead: 43294 bytes (0.029%) Idle pool limit: 5.00 MB Total Pools created: 138 Pools ever used: 135 (shown above) Currently in use: 103 String Pool Impact (%strings) (%volume) Short Strings 90 48 Medium Strings 9 15 Long Strings 1 5 1KB Strings 0 1 4KB Strings 0 9 16KB Strings 0 13 Other Strings 0 8 Large buffers: 0 (0 KB) Finally, storedir: Store Directory Statistics: Store Entries : 1673 Maximum Swap Size : 32768 KB Current Store Swap Size: 29484.00 KB Current Capacity : 89.98% used, 10.02% free Store Directory #0 (diskd): /var/cache/squid FS Block Size 4096 Bytes First level subdirectories: 16 Second level subdirectories: 256 Maximum Size: 32768 KB Current Size: 29484.00 KB Percent Used: 89.98% Filemap bits in use: 1338 of 16384 (8%) Filesystem Space in use: 44360368/192108884 KB (23%) Filesystem Inodes in use: 454196/12271616 (4%) Flags: SELECTED Pending operations: 0 Removal policy: lru LRU reference age: 0.01 days # free --bytes total used free shared buff/cache available Mem: 33653817344 3982090240 929779712 162263040 28741947392 29162274816 Swap: 37925875712 173297664 37752578048 I read Alex's e-mail too late, so I'll try to attach gdb next time. The problem is that Squid continues to respond, but slowly. Having a backup Squid is not that simple in my scenario because I require to do content analysis on ssl-bumped traffic. I'm using TPROXY on the gateway/firewall with Squid. A true backup on this production system would mean having a full conntrackd'ed firewall with Squid as I cannot TPROXY on a redirected machine. I know Squid 4 is recommended for SSL-Bump, but it's still beta, and I'm not sure I can take that leap now. I didn't run mgr:events while the CPU was at 100%, but here's what came up afterwards: # squidclient mgr:events HTTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Thu, 26 Oct 2017 11:02:42 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 26 Oct 2017 11:02:42 GMT Last-Modified: Thu, 26 Oct 2017 11:02:42 GMT X-Cache: MISS from inf-fw2 X-Cache-Lookup: MISS from inf-fw2:3128 Connection: close Last event to run: MaintainSwapSpace Operation Next Execution Weight Callback Valid? logfileFlush 0.511 sec 1 yes MaintainSwapSpace 0.511 sec 1 N/A fqdncache_purgelru 1.481 sec 1 N/A idnsCheckQueue 2.507 sec 1 N/A statAvgTick 3.821 sec 1 N/A storeDirClean 5.484 sec 1 N/A memPoolCleanIdlePools 5.484 sec 1 N/A ipcache_purgelru 6.900 sec 1 N/A peerRefreshDNS 348.872 sec 1 N/A netdbSaveState 678.327 sec 1 N/A Adaptation::Icap::ServiceRep::noteTimeToUpdate 3243.778 sec 0 yes User Cache Maintenance 3302.980 sec 1 N/A storeDigestRebuildStart 3303.451 sec 1 N/A storeDigestRewriteStart 3303.451 sec 1 N/A # squidclient mgr:events HTTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Thu, 26 Oct 2017 11:08:04 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 26 Oct 2017 11:08:04 GMT Last-Modified: Thu, 26 Oct 2017 11:08:04 GMT X-Cache: MISS from inf-fw2 X-Cache-Lookup: MISS from inf-fw2:3128 Connection: close Last event to run: fqdncache_purgelru Operation Next Execution Weight Callback Valid? logfileFlush 0.141 sec 1 yes MaintainSwapSpace 0.141 sec 1 N/A idnsCheckQueue 3.212 sec 1 N/A ipcache_purgelru 5.393 sec 1 N/A fqdncache_purgelru 9.962 sec 1 N/A storeDirClean 13.942 sec 1 N/A memPoolCleanIdlePools 13.942 sec 1 N/A peerRefreshDNS 27.243 sec 1 N/A statAvgTick 42.223 sec 1 N/A netdbSaveState 356.698 sec 1 N/A Adaptation::Icap::ServiceRep::noteTimeToUpdate 2922.149 sec 0 yes User Cache Maintenance 2981.351 sec 1 N/A storeDigestRebuildStart 2981.822 sec 1 N/A storeDigestRewriteStart 2981.822 sec 1 N/A I haven't been able to associate the 100% CPU issue with a scheduled event. I'll keep looking. Thanks, Vieri _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
On 10/26/2017 05:15 AM, Vieri wrote:
> Here's the strace (compressed with gz): > Do you see anything useful in the trace? FWIW, nothing jumped at me. As you know, I was asking for a gdb backtrace, but I no longer think that it is likely to help at this stage: Your further comments and strace itself confirm that Squid is _not_ stuck in some tight loop, consuming 100% of CPU. It may be consuming a lot of CPU cycles, but it is not doing the same compact operation over and over again, so random gdb backtraces are unlikely to pinpoint the culprit (if there is one). > If not, how can I improve the trace next time I have this issue? Please add (relative) timestamps between system calls (strace -r). It may help detect periods of time when Squid was not making any system calls. If possible, I would also add time spend in each system call (strace -T), just in case, but it is unlikely to be useful in this case. Please also report how many request did Squid finished handling during the traced period. You can run mgr:info before/after and/or count access.log lines. For mgr:info snapshots, please consider doing the legwork to identify potentially relevant entries that have changed between the snapshots. I do not have time to do that (but others on the list might, of course). I would not bother with mgr:mem snapshots unless mgr:info shows that the memory has grown a lot during the 100% CPU usage. Finally, consider enabling Squid debugging for a few seconds of that 100% CPU usage period. Wiki has instructions revolving around sending Squid a debug signal. > Having a backup Squid is not that simple in my scenario It rarely is! > I didn't run mgr:events while the CPU was at 100% Please run mgr:events all the time (e.g., every minute) so that you can catch event(s) (if any) that _lead_ to the 100% CPU usage period. You can also run mgr:events more often (e.g., every 5 seconds) during the 100% CPU usage period, of course. HTH, Alex. _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
Thanks for all the tips.
I should've done this before, but I just took a second look at cache.log and found the following message repeated a few times before the 100% CPU issue: "Your cache is running out of filedescriptors" I have no more info because the debug level was low. If this is the cause then I'm even more worried now because: # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127521 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 16384 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 127521 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited # cat /etc/security/limits.d/01_squid.conf * hard nofile 65535 * soft nofile 16384 It just so happens I ran mgr:filedescriptors while the CPU was at 100%. Here's the output (stripped of domains and IP addresses): HTTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Thu, 26 Oct 2017 08:35:16 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 26 Oct 2017 08:35:16 GMT Last-Modified: Thu, 26 Oct 2017 08:35:16 GMT X-Cache: MISS from inf-fw2 X-Cache-Lookup: MISS from inf-fw2:3128 Connection: close Active file descriptors: File Type Tout Nread * Nwrite * Remote Address Description ---- ------ ---- -------- -------- --------------------- ------------------------------ 5 Log 0 0 0 /var/log/squid/cache.log 6 Pipe 0 0 0 IPC UNIX STREAM Parent 9 Socket 28796 0 0 x.x.x.x:3199 client https start 11 Socket 8 1829* 174558 x.x.x.x:62569 Idle client: Waiting for next request 12 Socket 0 77571 243300 x.x.x.x:1344 x.x.x.x 13 Socket 28798 0 0 x.x.x.x:1428 client https start 14 Socket 113 201* 3316 x.x.x.x:49847 Idle client: Waiting for next request 15 Socket 7614 0* 0 x.x.x.x:56204 client https start 16 Socket 0 70492 72860 x.x.x.x:1344 x.x.x.x 17 Socket 0 184736 412668 x.x.x.x:1344 x.x.x.x 18 Socket 0 269207 467337 x.x.x.x:1344 x.x.x.x 19 Socket 0 70386 70846 x.x.x.x:1344 x.x.x.x 20 Socket 56 319* 452 x.x.x.x:59366 Idle client: Waiting for next request 21 Socket 0 74155 160454 x.x.x.x:1344 x.x.x.x 22 Socket 0 70373 184389 x.x.x.x:1344 x.x.x.x 23 Socket 0 321566 418429 x.x.x.x:1344 x.x.x.x 24 Socket 28783 1633* 0 x.x.x.x:60033 Reading next request 25 Socket 895 1809* 37539 x.x.x.x:1344 x.x.x.x 26 Socket 0 171606 765947 x.x.x.x:1344 x.x.x.x 27 Socket 0 505651 936639 x.x.x.x:1344 x.x.x.x 28 Socket 0 77690 579865 x.x.x.x:1344 x.x.x.x 29 Socket 893 1120129* 1356723 x.x.x.x:1344 x.x.x.x 30 Socket 99 1409* 1067 x.x.x.x:51458 Idle client: Waiting for next request 31 Socket 0 697084 991951 x.x.x.x:1344 x.x.x.x 32 Socket 0 72740 380302 x.x.x.x:1344 x.x.x.x 33 Socket 0 133319 4018875 x.x.x.x:1344 x.x.x.x 34 Socket 885 365084* 418439 x.x.x.x:1344 x.x.x.x 35 Socket 0 230550 976940 x.x.x.x:1344 x.x.x.x 36 Socket 0 72085 159051 x.x.x.x:1344 x.x.x.x 37 Socket 0 142361 344294 x.x.x.x:1344 x.x.x.x 38 Socket 0 1008147 1644576 x.x.x.x:1344 x.x.x.x 39 Socket 0 861819 1334921 x.x.x.x:1344 x.x.x.x 40 Socket 28800 2851* 344 x.x.x.x:60799 Reading next request 41 Socket 0 0 0 [::]:39163 DNS Socket IPv6 42 Socket 0 152497 327514 x.x.x.x:1344 x.x.x.x 43 Socket 0 1139088 1516707 x.x.x.x:1344 x.x.x.x 44 Socket 0 237721 398977 x.x.x.x:1344 x.x.x.x 45 Socket 0 91801 114000 x.x.x.x:1344 x.x.x.x 46 Socket 0 0 0 x.x.x.x:3233 HTTP Request 47 Socket 0 131675 695360 x.x.x.x:1344 x.x.x.x 48 Socket 0 159372 179649 x.x.x.x:1344 x.x.x.x 49 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (187) 50 Socket 0 161* 1378 x.x.x.x:443 https://domain.org 52 Socket 69 286* 1430 x.x.x.x:55771 Idle client: Waiting for next request 53 Socket 8 397* 48788 x.x.x.x:62572 Idle client: Waiting for next request 54 Socket 0 341879 480589 x.x.x.x:1344 x.x.x.x 55 Socket 91 201* 3315 x.x.x.x:56787 Idle client: Waiting for next request 56 Socket 0 726999 1336667 x.x.x.x:1344 x.x.x.x 57 Socket 113 201* 3316 x.x.x.x:51036 Idle client: Waiting for next request 58 Socket 119 556* 3723 x.x.x.x:51851 Idle client: Waiting for next request 59 Socket 99 1322* 766 x.x.x.x:1968 Idle client: Waiting for next request 60 Socket 0 78554 216848 x.x.x.x:1344 x.x.x.x 61 Socket 0 70211 717853 x.x.x.x:1344 x.x.x.x 62 Socket 0 78361 182762 x.x.x.x:1344 x.x.x.x 63 Socket 0 1945692 4239277 x.x.x.x:1344 x.x.x.x 64 Socket 66 1092* 1117 x.x.x.x:50080 Idle client: Waiting for next request 65 Socket 62 202* 286 x.x.x.x:55769 Idle client: Waiting for next request 66 Socket 0 17396* 904 x.x.x.x:443 https://domain.org 67 Socket 0 3536* 1166 x.x.x.x:443 https://domain.org 68 Socket 28798 3038* 797 x.x.x.x:65116 Reading next request 69 Socket 28798 0* 0 x.x.x.x:1426 client https start 70 Socket 28780 0* 0 x.x.x.x:60791 client https start 71 Socket 0 70387 70867 x.x.x.x:1344 x.x.x.x 72 Socket 0 254449 502590 x.x.x.x:1344 x.x.x.x 73 Socket 9 11524* 1152 x.x.x.x:54067 Idle client: Waiting for next request 74 Socket 0 96195 103010 x.x.x.x:1344 x.x.x.x 75 Socket 28 1516* 1291 x.x.x.x:3023 Idle client: Waiting for next request 76 Socket 0 205806 602151 x.x.x.x:1344 x.x.x.x 77 Socket 115 2195* 1248 x.x.x.x:59024 Idle client: Waiting for next request 78 Socket 0 352877 444820 x.x.x.x:1344 x.x.x.x 79 Socket 6 384* 612 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 80 Socket 0 71272 230010 x.x.x.x:1344 x.x.x.x 81 Socket 5 1490* 168706 x.x.x.x:54376 Idle client: Waiting for next request 82 Socket 0 4403371 651548 x.x.x.x:47443 DNS Socket IPv4 83 Socket 0 782173 984402 x.x.x.x:1344 x.x.x.x 84 Pipe 0 0 0 unlinkd -> squid 85 Socket 28798 433* 0 x.x.x.x:1423 Reading next request 86 Socket 5 2211* 1249 x.x.x.x:49400 Idle client: Waiting for next request 87 Pipe 0 0 0 squid -> unlinkd 88 Pipe 0 0 0 squid -> diskd 89 Socket 28792 4914* 2557 x.x.x.x:3381 Reading next request 90 Socket 66 598* 712 x.x.x.x:2262 Idle client: Waiting for next request 91 Socket 0 486500 1052648 x.x.x.x:1344 x.x.x.x 92 Socket 0 84149 459690 x.x.x.x:1344 x.x.x.x 93 Socket 0 446707 703970 x.x.x.x:1344 x.x.x.x 94 Socket 28797 1744* 0 x.x.x.x:49666 Reading next request 95 Socket 0 196505 1356207 x.x.x.x:1344 x.x.x.x 96 Socket 0 71430 747104 x.x.x.x:1344 x.x.x.x 97 Pipe 0 55385864* 83254354 ssl_crtd #1 98 Socket 99 4720* 74394 x.x.x.x:51833 Idle client: Waiting for next request 99 Socket 0 80191 352164 x.x.x.x:1344 x.x.x.x 100 Socket 0 831352 1205057 x.x.x.x:1344 x.x.x.x 101 Socket 0 67263 68047 x.x.x.x:1344 x.x.x.x 102 Socket 28734 0* 0 x.x.x.x:56784 client https start 103 Socket 0 90037 239742 x.x.x.x:1344 x.x.x.x 104 Socket 0 86557 182854 x.x.x.x:1344 x.x.x.x 105 Socket 0 74680 272842 x.x.x.x:1344 x.x.x.x 106 Socket 0 82432 182773 x.x.x.x:1344 x.x.x.x 107 Socket 0 91159 110508 x.x.x.x:1344 x.x.x.x 108 Socket 0 74341 182862 x.x.x.x:1344 x.x.x.x 109 Socket 58 354520* 417712 x.x.x.x:1344 x.x.x.x 110 Socket 896 110923* 238345 x.x.x.x:1344 x.x.x.x 111 Socket 0 367211 830710 x.x.x.x:1344 x.x.x.x 112 Socket 0 799681 2454675 x.x.x.x:1344 x.x.x.x 113 Socket 0 174639 526254 x.x.x.x:1344 x.x.x.x 114 Socket 0 224143 626713 x.x.x.x:1344 x.x.x.x 115 Socket 0 825* 4338 x.x.x.x:443 https://domain.org 116 Socket 50 0 0* x.x.x.x:80 domain.org 117 Socket 28799 0 0 x.x.x.x:3208 client https start 118 Socket 0 29387* 3248 x.x.x.x:443 https://domain.org 119 Socket 69 13796* 1117 x.x.x.x:64434 Idle client: Waiting for next request 120 Socket 0 401186 536770 x.x.x.x:1344 x.x.x.x 121 Socket 0 70420 72431 x.x.x.x:1344 x.x.x.x 122 Socket 0 1248701 2243251 x.x.x.x:1344 x.x.x.x 123 Pipe 0 12387700* 18425650 ssl_crtd #2 124 Socket 0 74579 186204 x.x.x.x:1344 x.x.x.x 125 Socket 0 147688 317832 x.x.x.x:1344 x.x.x.x 126 Socket 0 157570 1000064 x.x.x.x:1344 x.x.x.x 127 Socket 0 484039 932396 x.x.x.x:1344 x.x.x.x 128 Socket 0 72709 204331 x.x.x.x:1344 x.x.x.x 129 Socket 0 121209 121541 x.x.x.x:1344 x.x.x.x 130 Socket 0 89678 203510 x.x.x.x:1344 x.x.x.x 131 Socket 0 1096513 1775872 x.x.x.x:1344 x.x.x.x 132 Socket 28774 1636* 0 x.x.x.x:65117 Reading next request 133 Socket 53 378* 1542 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 134 Socket 108 482* 654 x.x.x.x:59022 Idle client: Waiting for next request 135 Socket 0 67261 67977 x.x.x.x:1344 x.x.x.x 136 Socket 0 353852 1257798 x.x.x.x:1344 x.x.x.x 137 Socket 120 3820* 3424 x.x.x.x:64559 Idle client: Waiting for next request 138 Socket 8 829* 1280 x.x.x.x:3040 Idle client: Waiting for next request 139 Socket 6069 0* 0 x.x.x.x:56177 client https start 140 Socket 0 75215 314184 x.x.x.x:1344 x.x.x.x 141 Socket 0 72428 421624 x.x.x.x:1344 x.x.x.x 142 Socket 0 205459 279606 x.x.x.x:1344 x.x.x.x 143 Socket 0 983140 1528993 x.x.x.x:1344 x.x.x.x 144 Socket 0 72254 128634 x.x.x.x:1344 x.x.x.x 145 Socket 108 470* 654 x.x.x.x:2259 Idle client: Waiting for next request 146 Socket 28798 0 0 x.x.x.x:1839 client https start 147 Socket 28754 6505* 2748 x.x.x.x:2121 Reading next request 148 Socket 0 94387 437023 x.x.x.x:1344 x.x.x.x 149 Socket 112 526* 636 x.x.x.x:54464 Idle client: Waiting for next request 150 Socket 0 291712 418901 x.x.x.x:1344 x.x.x.x 151 Socket 0 260125 466643 x.x.x.x:1344 x.x.x.x 152 Socket 0 75500 192364 x.x.x.x:1344 x.x.x.x 153 Socket 91 6720* 31850 x.x.x.x:51199 Idle client: Waiting for next request 154 Socket 0 340510 3310843 x.x.x.x:1344 x.x.x.x 155 Socket 28799 0 0 x.x.x.x:3206 client https start 156 Socket 0 74500 184502 x.x.x.x:1344 x.x.x.x 157 Socket 888 208562* 377059 x.x.x.x:1344 x.x.x.x 158 Pipe 0 5872190* 8607084 ssl_crtd #3 159 Socket 0 103064 525215 x.x.x.x:1344 x.x.x.x 160 Socket 98 482* 654 x.x.x.x:60031 Idle client: Waiting for next request 161 Socket 62 516* 675 x.x.x.x:51195 Idle client: Waiting for next request 162 Socket 28799 0 0 x.x.x.x:3209 client https start 163 Socket 66 2211* 2150 x.x.x.x:54426 Idle client: Waiting for next request 164 Socket 28798 3348* 798 x.x.x.x:2255 Reading next request 165 Socket 0 222093 321752 x.x.x.x:1344 x.x.x.x 166 Socket 0 126144 442862 x.x.x.x:1344 x.x.x.x 167 Socket 62 16353* 41096 x.x.x.x:3235 Idle client: Waiting for next request 168 Socket 0 67365 68179 x.x.x.x:1344 x.x.x.x 169 Socket 0 9195* 2294 x.x.x.x:443 https://domain.org 170 Socket 113 201* 3316 x.x.x.x:51037 Idle client: Waiting for next request 171 Socket 28800 0 0 x.x.x.x:55497 client https connect 172 Socket 0 126735 522798 x.x.x.x:1344 x.x.x.x 173 Socket 0 27366* 6978 x.x.x.x:443 https://domain.org 174 Socket 28779 3231* 870 x.x.x.x:2142 Reading next request 175 Socket 0 1214190 1308440 x.x.x.x:1344 x.x.x.x 176 Socket 28745 3983* 0 x.x.x.x:1365 Reading next request 177 Socket 27422 0* 0 x.x.x.x:56749 client https start 178 Socket 0 71305 194143 x.x.x.x:1344 x.x.x.x 179 Socket 66 306* 452 x.x.x.x:1353 Idle client: Waiting for next request 180 Socket 28774 1758* 0 x.x.x.x:59375 Reading next request 181 Socket 28798 3618* 609* x.x.x.x:55780 Reading next request 182 Socket 0 463907 4817276 x.x.x.x:1344 x.x.x.x 183 Socket 0 366549 818689 x.x.x.x:1344 x.x.x.x 184 Pipe 0 3597805* 5272955 ssl_crtd #4 185 Socket 0 330910 964399 x.x.x.x:1344 x.x.x.x 186 Socket 113 645* 1205 x.x.x.x:4116 Idle client: Waiting for next request 187 Socket 28800 876* 0* x.x.x.x:62588 Reading next request 188 Socket 0 1484456 1616112 x.x.x.x:1344 x.x.x.x 189 Socket 0 902153 1747543 x.x.x.x:1344 x.x.x.x 190 Socket 99 13466* 1117 x.x.x.x:64687 Idle client: Waiting for next request 191 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (505) 192 Socket 58 277899* 415305 x.x.x.x:1344 x.x.x.x 193 Socket 19 838* 739 x.x.x.x:62578 Idle client: Waiting for next request 194 Socket 58 32594* 53867 x.x.x.x:1344 x.x.x.x 195 Socket 0 74412 748176 x.x.x.x:1344 x.x.x.x 196 Socket 0 294684 832445 x.x.x.x:1344 x.x.x.x 197 Socket 876 431920* 668818 x.x.x.x:1344 x.x.x.x 198 Socket 28744 2487* 0 x.x.x.x:3267 Reading next request 199 Socket 113 201* 3316 x.x.x.x:51238 Idle client: Waiting for next request 200 Socket 0 1043* 886 x.x.x.x:443 https://domain.org 201 Socket 28779 3150* 870 x.x.x.x:2134 Reading next request 202 Socket 104 361* 3527 x.x.x.x:3173 Idle client: Waiting for next request 203 Socket 0 0 0 x.x.x.x:3234 HTTP Request 204 Socket 8 1450* 1308 x.x.x.x:65097 Reading next request 205 Socket 19 566* 784 x.x.x.x:54405 Idle client: Waiting for next request 206 Socket 0 1018301 1251632 x.x.x.x:1344 x.x.x.x 207 Socket 0 243460 725933 x.x.x.x:1344 x.x.x.x 208 Pipe 0 2674774* 3914935 ssl_crtd #5 209 Socket 115 610* 712 x.x.x.x:64689 Idle client: Waiting for next request 210 Socket 0 764* 7298 x.x.x.x:443 https://domain.org 211 Socket 105 1277* 344 x.x.x.x:2283 Idle client: Waiting for next request 212 Socket 0 131679 1220103 x.x.x.x:1344 x.x.x.x 213 Socket 876 30969* 222063 x.x.x.x:1344 x.x.x.x 214 Socket 0 191177 438150 x.x.x.x:1344 x.x.x.x 215 Socket 0 78829 290995 x.x.x.x:1344 x.x.x.x 216 Socket 112 1002* 1328 x.x.x.x:51197 Idle client: Waiting for next request 217 Socket 0 174193* 3658 x.x.x.x:443 https://domain.org 218 Socket 28781 0* 0 x.x.x.x:60795 client https start 219 Pipe 0 1969670* 2885654 ssl_crtd #6 220 Socket 67 5709* 13745 x.x.x.x:1366 Idle client: Waiting for next request 221 Socket 0 570053 628689 x.x.x.x:1344 x.x.x.x 222 Socket 28790 308* 0 x.x.x.x:2899 Reading next request 223 Socket 28798 704* 267112 x.x.x.x:1827 Reading next request 224 Pipe 0 1464643* 2151275 ssl_crtd #7 225 Socket 107 3463* 26520 x.x.x.x:51202 Idle client: Waiting for next request 226 Socket 28799 433* 0 x.x.x.x:1425 Reading next request 227 Socket 118 1424* 344 x.x.x.x:1826 Idle client: Waiting for next request 228 Socket 0 321598 669611 x.x.x.x:1344 x.x.x.x 229 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (686) 230 Socket 87 909* 718 x.x.x.x:1389 Idle client: Waiting for next request 231 Socket 0 70445 89872 x.x.x.x:1344 x.x.x.x 232 Socket 28797 0* 0 x.x.x.x:3389 client https start 233 Socket 0 145988 147306 x.x.x.x:1344 x.x.x.x 234 Socket 0 586261 1258275 x.x.x.x:1344 x.x.x.x 235 Socket 0 285602 466348 x.x.x.x:1344 x.x.x.x 236 Socket 0 131679 3752672 x.x.x.x:1344 x.x.x.x 237 Socket 0 447059 1790353 x.x.x.x:1344 x.x.x.x 238 Socket 0 87910 2010315 x.x.x.x:1344 x.x.x.x 239 Socket 5812 0* 0 x.x.x.x:56171 client https start 240 Socket 28 48115* 1178 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 241 Socket 66 4466* 18377 x.x.x.x:3268 Idle client: Waiting for next request 242 Socket 0 596879 1296258 x.x.x.x:1344 x.x.x.x 243 Socket 0 568370 987503 x.x.x.x:1344 x.x.x.x 244 Pipe 0 1150405* 1697637 ssl_crtd #8 245 Socket 56 485* 682630 x.x.x.x:62580 Idle client: Waiting for next request 246 Socket 0 175607 1683506 x.x.x.x:1344 x.x.x.x 247 Socket 16 1424* 17194 x.x.x.x:54395 Idle client: Waiting for next request 248 Socket 28798 829* 12831* x.x.x.x:2308 Reading next request 249 Socket 56 318* 452 x.x.x.x:62585 Idle client: Waiting for next request 250 Socket 875 1734* 5924 x.x.x.x:443 https://domain.org 251 Socket 5 518* 70059 x.x.x.x:54373 Idle client: Waiting for next request 252 Socket 0 149322 313136 x.x.x.x:1344 x.x.x.x 253 Socket 0 589520 836807 x.x.x.x:1344 x.x.x.x 254 Socket 28797 0 0* x.x.x.x:54932 client https start 255 Socket 28771 1660* 0 x.x.x.x:51201 Reading next request 256 Pipe 0 916891* 1351785 ssl_crtd #9 257 Socket 60 22466* 111364 x.x.x.x:1344 x.x.x.x 258 Socket 28796 3294* 1180 x.x.x.x:59020 Reading next request 259 Socket 0 1055* 872 x.x.x.x:443 https://domain.org 260 Socket 0 14835* 624 x.x.x.x:443 https://domain.org 261 Socket 875 289287* 581694 x.x.x.x:1344 x.x.x.x 262 Socket 7 2437* 10773 x.x.x.x:65303 Idle client: Waiting for next request 263 Socket 28796 0 0 x.x.x.x:3197 client https start 264 Pipe 0 729453* 1059077 ssl_crtd #10 265 Socket 66 4370* 5763 x.x.x.x:3269 Idle client: Waiting for next request 266 Socket 107 3233* 4119 x.x.x.x:64586 Idle client: Waiting for next request 267 Socket 0 212472 431063 x.x.x.x:1344 x.x.x.x 268 Pipe 0 578076* 850440 ssl_crtd #11 269 Socket 113 407* 613 x.x.x.x:51460 Idle client: Waiting for next request 270 Socket 0 1101* 4474 x.x.x.x:443 https://domain.org 271 Socket 0 2563693 3830336 x.x.x.x:1344 x.x.x.x 272 Socket 0 828859 1380776 x.x.x.x:1344 x.x.x.x 273 Socket 0 71319 259076 x.x.x.x:1344 x.x.x.x 274 Socket 105 609* 435 x.x.x.x:3171 Idle client: Waiting for next request 275 Socket 28799 433* 0 x.x.x.x:1427 Reading next request 276 Socket 0 74500 185574 x.x.x.x:1344 x.x.x.x 277 Socket 0 189973 564948 x.x.x.x:1344 x.x.x.x 278 Socket 60 129925* 458165 x.x.x.x:1344 x.x.x.x 279 Socket 28783 0* 0 x.x.x.x:60797 client https start 280 Socket 28797 731* 0 x.x.x.x:54488 Reading next request 281 Socket 0 70962 98876 x.x.x.x:1344 x.x.x.x 282 Socket 0 77789 334559 x.x.x.x:1344 x.x.x.x 283 Socket 115 649* 623 x.x.x.x:54482 Idle client: Waiting for next request 284 Socket 0 131737 182857 x.x.x.x:1344 x.x.x.x 285 Socket 28799 0 0 x.x.x.x:3216 client https start 286 Socket 74 469* 555 x.x.x.x:60744 Idle client: Waiting for next request 287 Pipe 0 85247* 124165 ssl_crtd #1 288 Socket 0 996913 1509983 x.x.x.x:1344 x.x.x.x 289 Pipe 0 466480* 700518 ssl_crtd #12 290 Socket 0 70702 74868 x.x.x.x:1344 x.x.x.x 291 Socket 88 518* 43112 x.x.x.x:60756 Idle client: Waiting for next request 292 Socket 9 10213* 1158 x.x.x.x:54068 Idle client: Waiting for next request 293 Socket 900 269761* 431423* x.x.x.x:1344 x.x.x.x 294 Socket 0 88023 296477 x.x.x.x:1344 x.x.x.x 295 Socket 113 201* 3316 x.x.x.x:51038 Idle client: Waiting for next request 296 Socket 28800 0 0 x.x.x.x:2385 client https start 297 Socket 0 1409562 1861582 x.x.x.x:1344 x.x.x.x 298 Socket 0 74282 713086 x.x.x.x:1344 x.x.x.x 299 Socket 0 74290 182774 x.x.x.x:1344 x.x.x.x 300 Socket 97 2215* 1301 x.x.x.x:51848 Idle client: Waiting for next request 301 Socket 0 85396 343984 x.x.x.x:1344 x.x.x.x 302 Socket 0 78177 143882 x.x.x.x:1344 x.x.x.x 303 Socket 66 5881* 6604 x.x.x.x:1367 Idle client: Waiting for next request 304 Socket 0 2942297 3069551 x.x.x.x:1344 x.x.x.x 305 Pipe 0 364706* 534055 ssl_crtd #13 306 Socket 28758 0 0 x.x.x.x:55095 client https start 307 Socket 118 410* 1231 x.x.x.x:2311 Idle client: Waiting for next request 308 Socket 0 503380 1071795 x.x.x.x:1344 x.x.x.x 309 Socket 0 96839 350661 x.x.x.x:1344 x.x.x.x 310 Socket 0 556795 5837880 x.x.x.x:1344 x.x.x.x 311 Socket 112 254* 787 x.x.x.x:50711 Idle client: Waiting for next request 312 Socket 28800 0 0 x.x.x.x:3220 client https start 313 Socket 0 72449 172781 x.x.x.x:1344 x.x.x.x 314 Socket 0 185849 1747378 x.x.x.x:1344 x.x.x.x 315 Socket 0 0 0 x.x.x.x:3232 HTTP Request 316 Socket 87 201* 3316 x.x.x.x:56871 Idle client: Waiting for next request 317 Socket 9 888* 1066 x.x.x.x:2976 Idle client: Waiting for next request 318 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (269) 319 Socket 0 73737 86419 x.x.x.x:1344 x.x.x.x 320 Socket 0 648346 962465 x.x.x.x:1344 x.x.x.x 321 Socket 0 151927 449211 x.x.x.x:1344 x.x.x.x 322 Socket 0 275* 2398 x.x.x.x:443 https://domain.org 323 Socket 2 354* 466 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 324 Socket 108 470* 654 x.x.x.x:2257 Idle client: Waiting for next request 325 Pipe 0 68850* 103401 ssl_crtd #2 326 Socket 0 0 0 x.x.x.x:3230 client https connect 327 Socket 28773 0* 0 x.x.x.x:56789 client https start 328 Socket 0 242751 910187 x.x.x.x:1344 x.x.x.x 329 Socket 28730 1146* 0 x.x.x.x:2150 Reading next request 330 Pipe 0 309147* 463388 ssl_crtd #14 331 Socket 28783 1366* 0 x.x.x.x:64173 Reading next request 332 Socket 0 531* 1360 x.x.x.x:443 https://domain.org 333 Socket 0 467739 655208 x.x.x.x:1344 x.x.x.x 334 Socket 0 0 0 x.x.x.x:55499 client https connect 335 Socket 10 2143* 880 x.x.x.x:54391 Idle client: Waiting for next request 336 Socket 113 10176* 1157 x.x.x.x:31352 Idle client: Waiting for next request 337 Socket 0 520857 1758549 x.x.x.x:1344 x.x.x.x 338 Socket 0 232302 359424 x.x.x.x:1344 x.x.x.x 339 Socket 59 164364* 422776 x.x.x.x:1344 x.x.x.x 340 Socket 0 1459622 1885753 x.x.x.x:1344 x.x.x.x 341 Socket 28800 0 0 x.x.x.x:60102 client https start 342 Socket 16 201* 3316 x.x.x.x:54400 Idle client: Waiting for next request 343 Socket 28799 433* 0 x.x.x.x:1429 Reading next request 344 Socket 0 170597 572080 x.x.x.x:1344 x.x.x.x 345 Pipe 0 256576* 380163 ssl_crtd #15 346 Socket 16 399* 286 x.x.x.x:61194 Idle client: Waiting for next request 347 Pipe 0 59571* 89389 ssl_crtd #3 348 Socket 19 563* 784 x.x.x.x:54402 Idle client: Waiting for next request 349 Socket 28800 0 0 x.x.x.x:3219 client https start 350 Socket 0 381663 1005072 x.x.x.x:1344 x.x.x.x 351 Socket 28799 2283* 0 x.x.x.x:49667 Reading next request 352 Socket 112 964* 1306 x.x.x.x:51189 Idle client: Waiting for next request 353 Pipe 0 218850* 321361 ssl_crtd #16 354 Socket 106 526* 276 x.x.x.x:51461 Idle client: Waiting for next request 355 Socket 19 573* 784 x.x.x.x:54403 Idle client: Waiting for next request 356 Socket 66 490* 8229 x.x.x.x:3144 Idle client: Waiting for next request 357 Pipe 0 179339* 269018 ssl_crtd #17 358 Socket 28800 0 0 x.x.x.x:3217 client https start 359 Socket 28798 0 0 x.x.x.x:1840 client https start 360 Socket 0 1080220 1947420 x.x.x.x:1344 x.x.x.x 361 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (46) 362 Socket 0 217293 913828 x.x.x.x:1344 x.x.x.x 363 Socket 44 630* 2746 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 364 Socket 0 12335* 15764 x.x.x.x:443 https://domain.org 365 Socket 28800 0 0 x.x.x.x:3221 client https start 366 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (479) 367 Socket 0 343536 972237 x.x.x.x:1344 x.x.x.x 368 Socket 0 1138072 1520823 x.x.x.x:1344 x.x.x.x 369 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (254) 370 Socket 19 563* 784 x.x.x.x:54404 Idle client: Waiting for next request 371 Socket 98 537* 1246 x.x.x.x:54444 Idle client: Waiting for next request 372 Socket 0 860350 1104913 x.x.x.x:1344 x.x.x.x 373 Socket 0 1060* 23048 x.x.x.x:443 https://domain.org 374 Socket 0 1066* 20478 x.x.x.x:443 https://domain.org 375 Socket 28799 0 0 x.x.x.x:3210 client https start 376 Socket 900 0* 4618 x.x.x.x:443 https://domain.org 377 Pipe 0 55913* 81124 ssl_crtd #4 378 Socket 113 201* 3316 x.x.x.x:51039 Idle client: Waiting for next request 379 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (263) 380 Socket 113 201* 3316 x.x.x.x:51040 Idle client: Waiting for next request 381 Socket 46 3906* 24391 x.x.x.x:1344 x.x.x.x 382 Socket 28798 0 0 x.x.x.x:2320 client https start 383 Socket 0 72481 157154 x.x.x.x:1344 x.x.x.x 384 Pipe 0 133145* 199159 ssl_crtd #18 385 Socket 28776 7538* 1630 x.x.x.x:54399 Reading next request 386 Pipe 0 7025* 10746 ssl_crtd #1 387 Socket 86 482* 654 x.x.x.x:49662 Idle client: Waiting for next request 388 Socket 91 1173* 343 x.x.x.x:60760 Idle client: Waiting for next request 389 Socket 0 1560783 3101138 x.x.x.x:1344 x.x.x.x 390 Socket 28796 0 0 x.x.x.x:3198 client https start 391 Socket 0 580399 792546 x.x.x.x:1344 x.x.x.x 392 Socket 0 78307 198904 x.x.x.x:1344 x.x.x.x 393 Socket 28737 0* 0 x.x.x.x:51806 client https start 394 Socket 18 470* 3778 x.x.x.x:3088 Idle client: Waiting for next request 395 Socket 28800 0 0 x.x.x.x:3223 client https start 396 Socket 0 0 0 x.x.x.x:3235 HTTP Request 397 Socket 7 1934* 615 x.x.x.x:54066 Idle client: Waiting for next request 398 Pipe 0 113172* 166273 ssl_crtd #19 399 Socket 893 195206* 214787 x.x.x.x:1344 x.x.x.x 400 Socket 28754 10860* 10436 x.x.x.x:4100 Reading next request 401 Socket 17 201* 1324 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 402 Socket 0 2234270 2503172 x.x.x.x:1344 x.x.x.x 403 Socket 28800 5134* 2537 x.x.x.x:1834 Reading next request 404 Socket 28772 1637* 0 x.x.x.x:51847 Reading next request 405 Socket 54 306* 452 x.x.x.x:31306 Idle client: Waiting for next request 406 Socket 63 819* 1961 x.x.x.x:52887 Idle client: Waiting for next request 407 Socket 28 253* 2550 x.x.x.x:3093 Idle client: Waiting for next request 408 Socket 28793 536* 8935* x.x.x.x:60800 Reading next request 409 Socket 0 67261 67880 x.x.x.x:1344 x.x.x.x 410 Pipe 0 7959* 12590 ssl_crtd #2 411 Pipe 0 37269* 56216 ssl_crtd #5 412 Socket 0 76888 138606 x.x.x.x:1344 x.x.x.x 413 Socket 28800 0 0 x.x.x.x:3225 client https start 414 Socket 98 549* 1141 x.x.x.x:54445 Idle client: Waiting for next request 415 Socket 0 873728 1316366 x.x.x.x:1344 x.x.x.x 416 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (390) 417 Socket 97 482* 654 x.x.x.x:49408 Idle client: Waiting for next request 418 Socket 0 4690935 6430992 x.x.x.x:1344 x.x.x.x 419 Pipe 0 7549* 11042 ssl_crtd #3 420 Socket 0 1058661 1673219 x.x.x.x:1344 x.x.x.x 421 Socket 28796 10874* 125020* x.x.x.x:54421 Reading next request 422 Socket 0 619900 1893341 x.x.x.x:1344 x.x.x.x 423 Socket 0 484497 639048 x.x.x.x:1344 x.x.x.x 424 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (497) 425 Socket 28798 0 0 x.x.x.x:3200 client https start 426 Socket 0 217686 282356 x.x.x.x:1344 x.x.x.x 427 Socket 0 171843 2226884 x.x.x.x:1344 x.x.x.x 428 Socket 0 742382 1291857 x.x.x.x:1344 x.x.x.x 429 Socket 104 249* 776 x.x.x.x:61437 Idle client: Waiting for next request 430 Socket 876 338749* 449700 x.x.x.x:1344 x.x.x.x 431 Socket 0 638966 976750 x.x.x.x:1344 x.x.x.x 432 Socket 0 67626 109037 x.x.x.x:1344 x.x.x.x 433 Socket 28799 0 0 x.x.x.x:3207 client https start 434 Socket 0 539438 762345 x.x.x.x:1344 x.x.x.x 435 Socket 0 529527 740350 x.x.x.x:1344 x.x.x.x 436 Socket 0 133828 244486 x.x.x.x:1344 x.x.x.x 437 Socket 91 273* 3437 x.x.x.x:55776 Idle client: Waiting for next request 438 Socket 28799 0 0 x.x.x.x:3212 client https start 439 Socket 0 175285 360548 x.x.x.x:1344 x.x.x.x 440 Socket 28800 0 0 x.x.x.x:55498 client https connect 441 Socket 900 866564* 1274512* x.x.x.x:1344 x.x.x.x 442 Socket 0 104615 259266 x.x.x.x:1344 x.x.x.x 443 Socket 0 767495 1513964 x.x.x.x:1344 x.x.x.x 444 Socket 113 557* 3604 x.x.x.x:3187 Idle client: Waiting for next request 445 Socket 0 589528 777412 x.x.x.x:1344 x.x.x.x 446 Pipe 0 99772* 143323 ssl_crtd #20 447 Socket 0 51392* 5692 x.x.x.x:443 https://domain.org 448 Socket 0 622* 3326 x.x.x.x:443 https://domain.org 449 Socket 0 178130 479705 x.x.x.x:1344 x.x.x.x 450 Socket 0 75335 249026 x.x.x.x:1344 x.x.x.x 451 Socket 0 584110 1041464 x.x.x.x:1344 x.x.x.x 452 Socket 0 5989855 7840009 x.x.x.x:1344 x.x.x.x 453 Socket 0 70378 183780 x.x.x.x:1344 x.x.x.x 454 Socket 9 397* 817 x.x.x.x:60714 Idle client: Waiting for next request 455 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (615) 456 Socket 0 67473 70236 x.x.x.x:1344 x.x.x.x 457 Socket 113 647* 811 x.x.x.x:54477 Idle client: Waiting for next request 458 Socket 106 277* 3421 x.x.x.x:55778 Idle client: Waiting for next request 459 Socket 0 80521 283037 x.x.x.x:1344 x.x.x.x 460 Socket 98 598* 712 x.x.x.x:50085 Idle client: Waiting for next request 461 Socket 0 167275 298854 x.x.x.x:1344 x.x.x.x 462 Socket 28781 0* 0 x.x.x.x:60792 client https start 463 Socket 115 281* 3428 x.x.x.x:55779 Idle client: Waiting for next request 464 Socket 0 10405* 4874 x.x.x.x:443 https://domain.org 465 Socket 0 1025* 27592 x.x.x.x:443 https://domain.org 466 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (9) 467 Socket 98 555* 2252 x.x.x.x:60770 Idle client: Waiting for next request 468 Socket 0 9354* 2314 x.x.x.x:443 https://domain.org 469 Socket 0 91540 325953 x.x.x.x:1344 x.x.x.x 470 Socket 0 762551 1389788 x.x.x.x:1344 x.x.x.x 471 Socket 74 7830* 12535 x.x.x.x:4097 Idle client: Waiting for next request 472 Socket 0 380016 476604 x.x.x.x:1344 x.x.x.x 473 Socket 885 15670* 85733 x.x.x.x:1344 x.x.x.x 474 Socket 0 406074 456587 x.x.x.x:1344 x.x.x.x 475 Socket 28800 1335* 253 x.x.x.x:3180 Reading next request 476 Socket 46 315* 4617 x.x.x.x:1344 x.x.x.x 477 Socket 58 32530* 49384 x.x.x.x:1344 x.x.x.x 478 Socket 0 172630 242324 x.x.x.x:1344 x.x.x.x 479 Socket 118 387* 3558 x.x.x.x:2316 Idle client: Waiting for next request 480 Socket 0 74290 182781 x.x.x.x:1344 x.x.x.x 481 Socket 0 80770 197294 x.x.x.x:1344 x.x.x.x 482 Socket 0 75048 281261 x.x.x.x:1344 x.x.x.x 483 Socket 0 74554 185351 x.x.x.x:1344 x.x.x.x 484 Socket 97 482* 654 x.x.x.x:49409 Idle client: Waiting for next request 485 Socket 60 1302* 11029 x.x.x.x:1344 x.x.x.x 486 Socket 86 4856* 5810 x.x.x.x:4112 Idle client: Waiting for next request 487 Socket 105 250* 779 x.x.x.x:56421 Idle client: Waiting for next request 488 Socket 0 2063* 4526 x.x.x.x:443 https://domain.org 489 Pipe 0 30386* 47145 ssl_crtd #6 490 Socket 28756 0 0 x.x.x.x:55093 client https start 491 Socket 28800 5190* 1214 x.x.x.x:1835 Reading next request 492 Socket 878 105* 4307 x.x.x.x:1344 x.x.x.x 493 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (232) 494 Socket 900 429 866 x.x.x.x:443 https://domain.org 495 Socket 900 160740* 529879* x.x.x.x:1344 x.x.x.x 496 Socket 0 474171 673500 x.x.x.x:1344 x.x.x.x 497 Socket 113 331* 3485 x.x.x.x:54480 Idle client: Waiting for next request 498 Socket 74 1595* 730 x.x.x.x:49696 Idle client: Waiting for next request 499 Socket 0 559618 865116 x.x.x.x:1344 x.x.x.x 500 Socket 898 595* 3540 x.x.x.x:443 https://domain.org 501 Socket 0 75127 273504 x.x.x.x:1344 x.x.x.x 502 Socket 0 171003 603908 x.x.x.x:1344 x.x.x.x 503 Socket 0 377031 835100 x.x.x.x:1344 x.x.x.x 504 Socket 0 67366 69093 x.x.x.x:1344 x.x.x.x 505 Socket 88 520* 675 x.x.x.x:51841 Idle client: Waiting for next request 506 Socket 0 70420 71693 x.x.x.x:1344 x.x.x.x 507 Socket 0 71047 275124 x.x.x.x:1344 x.x.x.x 508 Socket 0 196158* 23334 x.x.x.x:443 https://domain.org 509 Socket 0 0* 0 ::1:48239 ext_wbinfo_group_acl #1 510 Socket 0 136980 194711 x.x.x.x:1344 x.x.x.x 511 Socket 898 9550* 41835 x.x.x.x:1344 x.x.x.x 512 Socket 0 921556 1321574 x.x.x.x:1344 x.x.x.x 513 Socket 98 537* 1068 x.x.x.x:54446 Idle client: Waiting for next request 514 Socket 0 70273 182857 x.x.x.x:1344 x.x.x.x 515 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (315) 516 Socket 74 507* 1559 x.x.x.x:60741 Idle client: Waiting for next request 517 Socket 87 1531* 29663 x.x.x.x:54429 Idle client: Waiting for next request 518 Socket 87 1085* 9379 x.x.x.x:54430 Idle client: Waiting for next request 519 Socket 0 36043* 842 x.x.x.x:443 https://domain.org 520 Socket 0 156225 280780 x.x.x.x:1344 x.x.x.x 521 Socket 87 1095* 9538 x.x.x.x:54431 Idle client: Waiting for next request 522 Socket 0 1090575 1456044 x.x.x.x:1344 x.x.x.x 523 Socket 0 67263 67988 x.x.x.x:1344 x.x.x.x 524 Socket 0 133693 347078 x.x.x.x:1344 x.x.x.x 525 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (69) 526 Socket 898 311676* 510052 x.x.x.x:1344 x.x.x.x 527 Socket 900 425 866 x.x.x.x:443 https://domain.org 528 Socket 0 0* 0 ::1:59459 ext_wbinfo_group_acl #2 529 Socket 0 78353 989595 x.x.x.x:1344 x.x.x.x 530 Socket 0 1096* 1782 x.x.x.x:443 https://domain.org 531 Socket 28800 0 0 x.x.x.x:3222 client https start 532 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (764) 533 Socket 0 71390 110141 x.x.x.x:1344 x.x.x.x 534 Socket 0 1203421 1809900 x.x.x.x:1344 x.x.x.x 535 Socket 0 70832 76525 x.x.x.x:1344 x.x.x.x 536 Socket 118 417* 1219 x.x.x.x:2310 Idle client: Waiting for next request 537 Socket 77 662* 275 x.x.x.x:1392 Idle client: Waiting for next request 538 Socket 0 76434 486358 x.x.x.x:1344 x.x.x.x 539 Socket 898 445876* 531378 x.x.x.x:1344 x.x.x.x 540 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (203) 541 Socket 0 69878* 1098 x.x.x.x:443 https://domain.org 542 Socket 46 384* 612 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 543 Socket 0 636* 794 x.x.x.x:443 https://domain.org 544 Socket 0 2182761 2218810 x.x.x.x:1344 x.x.x.x 545 Socket 46 150988* 321272 x.x.x.x:1344 x.x.x.x 546 Socket 0 68503 78530 x.x.x.x:1344 x.x.x.x 547 Socket 896 1598* 6692 x.x.x.x:443 https://domain.org 548 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (13) 549 Socket 900 0* 866 x.x.x.x:443 https://domain.org 550 Socket 28799 0 0 x.x.x.x:3203 client https start 551 Socket 59 302774* 595817 x.x.x.x:1344 x.x.x.x 552 Socket 0 515* 1112 x.x.x.x:443 https://domain.org 553 Socket 7 395* 286 x.x.x.x:61191 Idle client: Waiting for next request 554 Socket 0 438847 475815 x.x.x.x:1344 x.x.x.x 555 Socket 0 1666817 2225879 x.x.x.x:1344 x.x.x.x 556 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (625) 557 Socket 0 87652 250185 x.x.x.x:1344 x.x.x.x 558 Socket 0 888616 1629204 x.x.x.x:1344 x.x.x.x 559 Socket 878 2012* 15232 x.x.x.x:443 https://domain.org 560 Socket 85 201* 3316 x.x.x.x:49698 Idle client: Waiting for next request 561 Socket 28774 2910* 1058 x.x.x.x:64166 Reading next request 562 Socket 898 1211* 6180 x.x.x.x:443 https://domain.org 563 Socket 97 940* 1308 x.x.x.x:3376 Idle client: Waiting for next request 564 Socket 0 275* 4748 x.x.x.x:443 https://domain.org 565 Socket 15 1446* 1306 x.x.x.x:51817 Reading next request 566 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (324) 567 Socket 0 594802 827936 x.x.x.x:1344 x.x.x.x 568 Socket 0 303279 409733 x.x.x.x:1344 x.x.x.x 569 Socket 117 1849* 615 x.x.x.x:2304 Idle client: Waiting for next request 570 Socket 0 3110* 7764 x.x.x.x:443 https://domain.org 571 Socket 0 75913* 9544 x.x.x.x:443 https://domain.org 572 Socket 57 198* 286 x.x.x.x:55768 Idle client: Waiting for next request 573 Socket 27 2763* 51668 x.x.x.x:62577 Idle client: Waiting for next request 574 Socket 0 1025* 2184 x.x.x.x:443 https://domain.org 575 Socket 99 11511* 192197 x.x.x.x:50483 Idle client: Waiting for next request 576 Socket 0 0* 0 ::1:37181 ext_wbinfo_group_acl #3 577 Socket 113 1181* 629 x.x.x.x:51203 Idle client: Waiting for next request 578 Socket 0 125227 327390 x.x.x.x:1344 x.x.x.x 579 Socket 0 80821 222391 x.x.x.x:1344 x.x.x.x 580 Socket 28798 0 0 x.x.x.x:1841 client https start 581 Socket 28798 0 0 x.x.x.x:2321 client https start 582 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (396) 583 Socket 0 0* 0 ::1:39025 ext_wbinfo_group_acl #4 584 Socket 28798 0 0 x.x.x.x:2322 client https start 585 Socket 0 67261 67974 x.x.x.x:1344 x.x.x.x 586 Socket 900 192830* 7818 x.x.x.x:443 https://domain.org 587 Socket 0 464123 1899880 x.x.x.x:1344 x.x.x.x 588 Socket 0 74290 182755 x.x.x.x:1344 x.x.x.x 589 Socket 0 647* 1728 x.x.x.x:443 https://domain.org 590 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (145) 591 Socket 28799 0 0 x.x.x.x:2323 client https start 592 Socket 28799 0 0 x.x.x.x:3201 client https start 593 Pipe 0 18335* 25554 ssl_crtd #7 594 Socket 91 292* 452 x.x.x.x:52893 Idle client: Waiting for next request 595 Socket 0 302843 1136496 x.x.x.x:1344 x.x.x.x 596 Socket 28800 0 0 x.x.x.x:2386 client https start 597 Socket 0 86639 207373 x.x.x.x:1344 x.x.x.x 598 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (279) 599 Socket 0 0* 0 ::1:51769 ext_wbinfo_group_acl #5 600 Socket 0 71364 312248 x.x.x.x:1344 x.x.x.x 601 Socket 0 919397 1075737 x.x.x.x:1344 x.x.x.x 602 Socket 0 180943 1358113 x.x.x.x:1344 x.x.x.x 603 Socket 28799 0 0 x.x.x.x:3214 client https start 604 Socket 0 692* 1178 x.x.x.x:443 https://domain.org 605 Socket 62 7966* 12814 x.x.x.x:1230 Idle client: Waiting for next request 606 Socket 0 638* 7316 x.x.x.x:443 https://domain.org 607 Socket 23 306* 452 x.x.x.x:1767 Idle client: Waiting for next request 608 Socket 28799 0 0 x.x.x.x:3205 client https start 609 Socket 0 85591 164291 x.x.x.x:1344 x.x.x.x 610 Socket 0 692* 1198 x.x.x.x:443 https://domain.org 611 Socket 28785 0* 0 x.x.x.x:3183 client https start 612 Socket 0 692* 1178 x.x.x.x:443 https://domain.org 613 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (134) 614 Socket 900 28870* 195741* x.x.x.x:1344 x.x.x.x 615 Socket 28754 0 0 x.x.x.x:55092 client https start 616 Socket 118 390* 36111 x.x.x.x:2319 Idle client: Waiting for next request 617 Socket 98 555* 2252 x.x.x.x:60769 Idle client: Waiting for next request 618 Socket 299 0* 0 x.x.x.x:1828 client http connect 619 Socket 0 473529 793634 x.x.x.x:1344 x.x.x.x 620 Socket 900 450715* 716503 x.x.x.x:1344 x.x.x.x 621 Socket 28799 0 0 x.x.x.x:3202 client https start 622 Socket 0 692* 1184 x.x.x.x:443 https://domain.org 623 Socket 874 16754* 21876 x.x.x.x:443 https://domain.org 624 Socket 28788 4986* 1596 x.x.x.x:49406 Reading next request 625 Socket 98 482* 654 x.x.x.x:64171 Idle client: Waiting for next request 626 Pipe 0 14687* 22194 ssl_crtd #8 627 Socket 59 210* 2635 x.x.x.x:1344 x.x.x.x 628 Socket 0 134507 1462114 x.x.x.x:1344 x.x.x.x 629 Socket 0 108885* 2533583* ::1:34453 squid_url_lookup.pl #1 630 Socket 0 72439 150972 x.x.x.x:1344 x.x.x.x 631 Socket 28799 0 0 x.x.x.x:3204 client https start 632 Socket 879 734* 6486 x.x.x.x:443 https://domain.org 633 Socket 0 1051410 1201280 x.x.x.x:1344 x.x.x.x 634 Socket 28799 0 0 x.x.x.x:3211 client https start 635 Socket 879 734* 6648 x.x.x.x:443 https://domain.org 636 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (611) 637 Socket 0 163962 604624 x.x.x.x:1344 x.x.x.x 638 Socket 0 48696* 856 x.x.x.x:443 https://domain.org 639 Socket 0 729827 1196381 x.x.x.x:1344 x.x.x.x 640 Socket 0 232516 347567 x.x.x.x:1344 x.x.x.x 641 Socket 28798 3754* 128113 x.x.x.x:2303 Reading next request 642 Socket 0 325777 602015 x.x.x.x:1344 x.x.x.x 643 Socket 28799 0 0 x.x.x.x:3213 client https start 644 Socket 18 397* 265 x.x.x.x:1258 Idle client: Waiting for next request 645 Socket 99 3596* 822 x.x.x.x:54417 Idle client: Waiting for next request 646 Pipe 0 14180* 21549 ssl_crtd #9 647 Socket 0 571668 1758812 x.x.x.x:1344 x.x.x.x 648 Socket 0 252* 3372 x.x.x.x:443 https://domain.org 649 Socket 0 68228 116478 x.x.x.x:1344 x.x.x.x 650 Socket 0 582* 1818 x.x.x.x:443 https://domain.org 651 Pipe 0 7520* 11022 ssl_crtd #10 652 Socket 90 435* 285 x.x.x.x:3386 Idle client: Waiting for next request 653 Socket 28799 0 0 x.x.x.x:3215 client https start 654 Socket 99 1110* 2214 x.x.x.x:60771 Idle client: Waiting for next request 655 Socket 28800 0 0 x.x.x.x:3218 client https connect 656 Socket 0 882* 1880 x.x.x.x:443 https://domain.org 657 Socket 59 0 0* x.x.x.x:80 domain.org 658 Socket 0 1283* 4442 x.x.x.x:443 https://domain.org 659 Socket 0 0 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (807) 660 Socket 90 435* 285 x.x.x.x:3384 Idle client: Waiting for next request 661 Socket 28800 0 0 x.x.x.x:3224 client https start 662 Socket 0 365673 608482 x.x.x.x:1344 x.x.x.x 663 Socket 0 81625 110628 x.x.x.x:1344 x.x.x.x 664 Socket 28800 156* 0* ::1:35046 Reading next request 665 Socket 97 2215* 1279 x.x.x.x:51849 Idle client: Waiting for next request 666 Socket 90 435* 285 x.x.x.x:3385 Idle client: Waiting for next request 667 Socket 0 620* 1220 x.x.x.x:443 https://domain.org 668 Socket 28800 201* 0* x.x.x.x:50010 Reading next request 669 Socket 0 136145 440062 x.x.x.x:1344 x.x.x.x 670 Socket 0 17564* 323089* ::1:33603 squid_url_lookup.pl #2 671 Socket 91 2348* 343 x.x.x.x:60757 Idle client: Waiting for next request 672 Socket 118 421* 17488 x.x.x.x:51046 Idle client: Waiting for next request 673 Socket 28800 0 0 x.x.x.x:3228 client https start 674 Socket 0 168525* 3042 x.x.x.x:443 https://domain.org 675 Socket 0 449451 1179110 x.x.x.x:1344 x.x.x.x 676 Socket 0 81705 176401 x.x.x.x:1344 x.x.x.x 677 Socket 28800 0 0 x.x.x.x:3229 client https connect 678 Socket 0 607338 804760 x.x.x.x:1344 x.x.x.x 679 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (417) 680 Socket 0 7879* 141662 ::1:59195 squid_url_lookup.pl #3 681 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (387) 682 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (393) 683 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (70) 684 Socket 28800 5112* 2235 x.x.x.x:1838 Reading next request 685 Socket 0 586* 990 x.x.x.x:443 https://domain.org 686 Socket 99 482* 654 x.x.x.x:60032 Idle client: Waiting for next request 687 Socket 0 1065* 20404 x.x.x.x:443 https://domain.org 688 Socket 0 4874* 61424 ::1:48333 squid_url_lookup.pl #4 689 Socket 0 225785 342266 x.x.x.x:1344 x.x.x.x 690 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (484) 691 Socket 0 247595 638177 x.x.x.x:1344 x.x.x.x 692 Socket 0 544* 1104 x.x.x.x:443 https://domain.org 693 Socket 0 70387 70862 x.x.x.x:1344 x.x.x.x 694 Socket 27680 0* 0 x.x.x.x:56755 client https start 695 Socket 300 0* 0 x.x.x.x:60802 client http connect 696 Socket 0 0 0 x.x.x.x:3231 HTTP Request 697 Socket 0 67261 67986 x.x.x.x:1344 x.x.x.x 698 Socket 0 1129* 2870 x.x.x.x:443 https://domain.org 700 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (462) 702 Socket 18 402* 266 x.x.x.x:1262 Idle client: Waiting for next request 703 Socket 9 1338* 634 x.x.x.x:80 Idle server: x.x.x.x:80/some_path 704 Socket 58 491* 682630 x.x.x.x:62581 Idle client: Waiting for next request 705 Socket 0 67626 91631 x.x.x.x:1344 x.x.x.x 706 Socket 0 67780 149096 x.x.x.x:1344 x.x.x.x 707 Socket 0 93177 639083 x.x.x.x:1344 x.x.x.x 708 Socket 0 875386 1172221 x.x.x.x:1344 x.x.x.x 709 Socket 876 588* 3324 x.x.x.x:443 https://domain.org 710 Socket 19 444* 384 x.x.x.x:1265 Idle client: Waiting for next request 711 Socket 19 425* 384 x.x.x.x:1266 Idle client: Waiting for next request 712 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (218) 716 Socket 57 420* 158 x.x.x.x:62586 Idle client: Waiting for next request 717 Socket 0 1491* 1066 x.x.x.x:443 https://domain.org 719 Socket 0 169746* 21748 x.x.x.x:443 https://domain.org 720 Socket 0 17102* 2910 x.x.x.x:443 https://domain.org 721 Socket 0 3279* 43331 ::1:42983 squid_url_lookup.pl #5 722 Socket 0 256289 269402 x.x.x.x:1344 x.x.x.x 723 Socket 0 7714* 2384 x.x.x.x:443 https://domain.org 724 Socket 0 1830* 24920 ::1:47149 squid_url_lookup.pl #6 725 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (327) 726 Socket 0 1063154 1790141 x.x.x.x:1344 x.x.x.x 727 Socket 0 616* 13608 ::1:48873 squid_url_lookup.pl #7 729 Socket 18 329* 3506 x.x.x.x:1267 Idle client: Waiting for next request 730 Socket 0 67260 67952 x.x.x.x:1344 x.x.x.x 731 Socket 0 32251* 13544 x.x.x.x:443 https://domain.org 732 Socket 0 12614* 16036 x.x.x.x:443 https://domain.org 733 Socket 0 298* 5090 ::1:59995 squid_url_lookup.pl #8 734 Socket 0 638* 3252 x.x.x.x:443 https://domain.org 735 Socket 0 252* 2900 x.x.x.x:443 https://domain.org 736 Socket 900 2405* 10516 x.x.x.x:443 https://domain.org 737 Socket 0 241* 2542 ::1:44247 squid_url_lookup.pl #9 738 Socket 0 256065 1016479 x.x.x.x:1344 x.x.x.x 740 Socket 0 69188 114543 x.x.x.x:1344 x.x.x.x 741 Socket 97 940* 1308 x.x.x.x:3379 Idle client: Waiting for next request 742 Socket 900 41281* 1514 x.x.x.x:443 https://domain.org 743 Socket 0 681278 1185273 x.x.x.x:1344 x.x.x.x 744 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (306) 745 Socket 900 1082* 10628 x.x.x.x:443 https://domain.org 746 Socket 0 5718* 9764 x.x.x.x:443 https://domain.org 747 Socket 0 620* 1196 x.x.x.x:443 https://domain.org 748 Socket 0 207* 2293 ::1:33591 squid_url_lookup.pl #10 749 Socket 0 0 0 [::]:3128 HTTP Socket 750 Socket 876 592* 3568 x.x.x.x:443 https://domain.org 751 Socket 0 0 0 [::]:3129 HTTP Socket 752 Socket 0 71255 145856 x.x.x.x:1344 x.x.x.x 753 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (569) 754 Socket 0 0 0 [::]:3130 HTTPS Socket 755 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (490) 756 Socket 0 70375 184071 x.x.x.x:1344 x.x.x.x 758 Socket 0 68918 187587 x.x.x.x:1344 x.x.x.x 760 Socket 0 132072 189220 x.x.x.x:1344 x.x.x.x 762 Socket 0 367* 1280 x.x.x.x:443 https://domain.org 763 Socket 0 74474 206574 x.x.x.x:1344 x.x.x.x 764 Socket 87 482* 654 x.x.x.x:49665 Idle client: Waiting for next request 765 Socket 0 0 0 ::1:50813 pinger 766 File 0 0 993744 /var/cache/squid/swap.state 767 Socket 893 2800* 9984 x.x.x.x:443 https://domain.org 769 Socket 0 537* 2414 x.x.x.x:443 https://domain.org 770 Socket 0 628* 3372 x.x.x.x:443 https://domain.org 771 Socket 0 306851 815547 x.x.x.x:1344 x.x.x.x 772 Socket 56 516* 676 x.x.x.x:54415 Idle client: Waiting for next request 773 Socket 0 620* 1196 x.x.x.x:443 https://domain.org 774 Socket 0 43044* 1098 x.x.x.x:443 https://domain.org 775 Socket 0 2458* 568 x.x.x.x:443 https://domain.org 776 Socket 0 173* 794 x.x.x.x:443 https://domain.org 777 Socket 0 131967 194732 x.x.x.x:1344 x.x.x.x 778 Socket 0 30285* 1720 x.x.x.x:443 https://domain.org 780 Socket 898 1213* 6800 x.x.x.x:443 https://domain.org 781 Socket 900 2103* 10472 x.x.x.x:443 https://domain.org 782 Socket 0 3627* 14436 x.x.x.x:443 https://domain.org 783 Socket 0 174* 804 x.x.x.x:443 https://domain.org 784 Socket 0 199486 463497 x.x.x.x:1344 x.x.x.x 785 Socket 0 1099* 4762 x.x.x.x:443 https://domain.org 787 Socket 0 316* 888 x.x.x.x:443 https://domain.org 790 Socket 0 2078* 2324 x.x.x.x:443 https://domain.org 791 Socket 0 316* 850 x.x.x.x:443 https://domain.org 792 Socket 0 2184* 1162 x.x.x.x:443 https://domain.org 793 Socket 0 2184* 1162 x.x.x.x:443 https://domain.org 794 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (202) 796 Socket 0 1290186 1774914 x.x.x.x:1344 x.x.x.x 797 Socket 0 67263 67949 x.x.x.x:1344 x.x.x.x 798 Socket 0 0* 0 x.x.x.x:443 x.x.x.x pinned connection for x.x.x.x (160) 799 Socket 99 7140* 3811 x.x.x.x:54406 Idle client: Waiting for next request 800 Socket 0 70622 74503 x.x.x.x:1344 x.x.x.x 801 Socket 27 2355* 1251 x.x.x.x:54407 Idle client: Waiting for next request 802 Socket 0 1154* 1136 x.x.x.x:443 https://domain.org 803 Socket 0 69578 75080 x.x.x.x:1344 x.x.x.x 804 Socket 0 1049* 1160 x.x.x.x:443 https://domain.org 805 Socket 0 976* 1136 x.x.x.x:443 https://domain.org 806 Socket 0 1261* 4482 x.x.x.x:443 https://domain.org 808 Socket 885 589* 3318 x.x.x.x:443 https://domain.org 809 Socket 0 1317* 4482 x.x.x.x:443 https://domain.org 810 Socket 900 82342* 866 x.x.x.x:443 https://domain.org 811 Socket 0 719* 1346 x.x.x.x:443 https://domain.org 812 Socket 0 1113* 1342 x.x.x.x:443 https://domain.org 814 Socket 855 2387* 13320 x.x.x.x:443 https://domain.org 815 Socket 27 500* 286 x.x.x.x:55764 Idle client: Waiting for next request 816 Socket 0 1080317 1867520 x.x.x.x:1344 x.x.x.x 820 Socket 66 1192* 7898 x.x.x.x:4110 Idle client: Waiting for next request 821 Socket 0 1025* 26932 x.x.x.x:443 https://domain.org 822 Socket 0 86033 89065 x.x.x.x:1344 x.x.x.x 823 Socket 59 326800* 558361 x.x.x.x:1344 x.x.x.x 824 Socket 893 5922* 5026 x.x.x.x:443 https://domain.org 826 Socket 0 1317411 1355916 x.x.x.x:1344 x.x.x.x 827 Pipe 0 7431* 9205 ssl_crtd #4 828 Socket 895 6112* 8018 x.x.x.x:443 https://domain.org 829 Socket 0 67263 68003 x.x.x.x:1344 x.x.x.x 830 Socket 0 13653* 11470 x.x.x.x:443 https://domain.org 831 Pipe 0 4461* 5937 ssl_crtd #5 832 Socket 0 98682 230913 x.x.x.x:1344 x.x.x.x 833 Socket 0 18285* 8984 x.x.x.x:443 https://domain.org 834 Socket 0 2769082 3379609 x.x.x.x:1344 x.x.x.x 835 Socket 0 5671* 8792 x.x.x.x:443 https://domain.org 837 Socket 0 6512* 11814 x.x.x.x:443 https://domain.org 838 Pipe 0 4461* 5937 ssl_crtd #6 839 Socket 0 252* 2606 x.x.x.x:443 https://domain.org 840 Socket 0 67366 74962 x.x.x.x:1344 x.x.x.x 842 Socket 0 71312 221653 x.x.x.x:1344 x.x.x.x 846 Pipe 0 4461* 5937 ssl_crtd #7 847 Socket 0 225360 346218 x.x.x.x:1344 x.x.x.x 849 Pipe 0 4461* 5937 ssl_crtd #8 850 Socket 0 8137* 1042 x.x.x.x:443 https://domain.org 851 Socket 885 591* 2784 x.x.x.x:443 https://domain.org 852 Pipe 0 0* 0 ssl_crtd #9 854 Socket 0 3983* 6570 x.x.x.x:443 https://domain.org 857 Pipe 0 0* 0 ssl_crtd #10 869 Socket 0 40620* 32966 x.x.x.x:443 https://domain.org 870 Socket 0 4328494 4704878 x.x.x.x:1344 x.x.x.x 878 Socket 0 70812 1451399 x.x.x.x:1344 x.x.x.x 885 Socket 0 646465 1098283 x.x.x.x:1344 x.x.x.x 891 Socket 0 949722 1844840 x.x.x.x:1344 x.x.x.x 904 Socket 888 1837* 10128 x.x.x.x:443 https://domain.org 909 Socket 0 85268 151920 x.x.x.x:1344 x.x.x.x 911 Socket 831 0* 2354 x.x.x.x:443 https://domain.org 913 Socket 0 70315 70855 x.x.x.x:1344 x.x.x.x I don't get it. There aren't that many active file descriptors (less than 1000). Also, on low traffic (late evening) with no CPU or mem issues at all, mgr:filedescriptors shows that there are about 2500 active file descriptors. The only two things I can think of that could mess with Squid are: - squid_url_lookup.pl which is an external_acl_type perl script that does a MySQL lookup (ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8) - c-icap content scanning with clamav and libarchive If push comes to shove I might consider temporarily disabling one or both just to see what happens, but I'm not sure if that can really be the cause of a Squid process coming down to a crawl. Thanks, Vieri _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
On 10/26/2017 03:58 PM, Vieri wrote:
> I should've done this before, but I just took a second look at > cache.log and found the following message repeated a few times before > the 100% CPU issue: > "Your cache is running out of filedescriptors" That warning matches the dangerously low number of available descriptor reported by your mgr:info: > File descriptor usage for squid: > Maximum number of file descriptors: 1024 > Largest file desc currently in use: 913 > Number of file desc currently in use: 791 > Files queued for open: 0 > Available number of file descriptors: 233 > Reserved number of file descriptors: 100 > If this is the cause then I'm even more worried now because: > > open files (-n) 16384 > # cat /etc/security/limits.d/01_squid.conf > * hard nofile 65535 > * soft nofile 16384 Your Squid does not know about the above configured limits. It thinks the limit is 1024. > I don't get it. There aren't that many active file descriptors (less than 1000). ~900 descriptors is "a lot" when the limit is 1024. > Also, on low traffic (late evening) with no CPU or mem issues at all, > mgr:filedescriptors shows that there are about 2500 active file > descriptors. Please note that "2500 active descriptors" contradicts the 1024 limit reported in mgr:info in your earlier email (quoted by me above). Squid also reports the limit at startup. Here is an example (not from your logs): > 2017/10/26 15:48:04.744| With 65535 file descriptors available There are many email threads and probably some wiki documents that discuss increasing the number of file descriptors available to your Squid and numerous related caveats. Cheers, Alex. _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
> From: Alex Rousskov <[hidden email]> > >> # cat /etc/security/limits.d/01_squid.conf >> * hard nofile 65535 >> * soft nofile 16384 > > Your Squid does not know about the above configured limits. It thinks the limit is 1024. That's interesting. # grep "file descriptors available" cache.log 2017/10/24 11:49:33 kid1| With 1024 file descriptors available 2017/10/24 12:57:10 kid1| With 16384 file descriptors available 2017/10/24 12:57:52 kid1| With 16384 file descriptors available 2017/10/25 05:50:32 kid1| With 1024 file descriptors available 2017/10/25 09:20:15 kid1| With 1024 file descriptors available 2017/10/25 10:00:15 kid1| With 1024 file descriptors available 2017/10/25 12:00:15 kid1| With 1024 file descriptors available 2017/10/25 13:00:16 kid1| With 1024 file descriptors available 2017/10/25 13:20:16 kid1| With 1024 file descriptors available 2017/10/25 14:00:15 kid1| With 1024 file descriptors available 2017/10/26 04:09:30 kid1| With 1024 file descriptors available 2017/10/26 10:57:45 kid1| With 16384 file descriptors available The issues I reported yesterday came up after 2017/10/26 04:09:30, so here's the relevant part in the log: 2017/10/26 04:09:27 kid1| Reconfiguring Squid Cache (version 3.5.27-20170916-re69e56c)... 2017/10/26 04:09:27 kid1| Closing HTTP port [::]:3128 2017/10/26 04:09:27 kid1| Closing HTTP port [::]:3129 2017/10/26 04:09:27 kid1| Closing HTTPS port [::]:3130 2017/10/26 04:09:27 kid1| Closing Pinger socket on FD 801 2017/10/26 04:09:27 kid1| Logfile: closing log daemon:/var/log/squid/access.log 2017/10/26 04:09:27 kid1| Logfile Daemon: closing log daemon:/var/log/squid/access.log 2017/10/26 04:09:27 kid1| FATAL: dying from an unhandled exception: IsConnOpen(conn) 2017/10/26 04:09:30| Pinger exiting. 2017/10/26 04:09:30 kid1| Set Current Directory to /var/cache/squid 2017/10/26 04:09:30 kid1| Starting Squid Cache version 3.5.27-20170916-re69e56c for x86_64-pc-linux-gnu... 2017/10/26 04:09:30 kid1| Service Name: squid 2017/10/26 04:09:30 kid1| Process ID 22581 2017/10/26 04:09:30 kid1| Process Roles: worker 2017/10/26 04:09:30 kid1| With 1024 file descriptors available A "reload" was issued at that time. When my openrc init script reloads, it runs the following command: # /etc/init.d/squid reload * Reloading squid with /usr/sbin/squid -k reconfigure -f /etc/squid/squid.conf -n squid ... [ ok ] I issued another "reload" today, and I didn't come up with any "file descriptors available" messages in cache.log (as expected). I also checked mgr:info, and all seems OK: File descriptor usage for squid: Maximum number of file descriptors: 16384 Largest file desc currently in use: 3067 Number of file desc currently in use: 2901 Files queued for open: 0 Available number of file descriptors: 13483 Reserved number of file descriptors: 100 Store Disk files open: 0 I also grep'ed for the log lines around "2017/10/25 05:50:32 kid1| With 1024 file descriptors available" where I see 1024 file descriptors for the first time. It was a clean restart (with no "FATAL" error message). The "restart" command was launched from crontab as a daily job (calling the openrc init script) just like it is launched at boot time from openrc (when the number of file descriptors is "as expected"). In particular, I have: KRB5_KTNAME="/etc/krb5.keytab" /usr/sbin/squid -YC -f /etc/squid/squid.conf -n squid I'm wondering why Squid only detected 1024 file descriptors available after that restart. Moving back in time: - "2017/10/24 12:57:52 kid1| With 16384 file descriptors available" was probably a manual restart on my behalf (passing "restart" to the init script on the console). - the one right before (2017/10/24 12:57:10) was probably also a manual restart as the init script probably exited non-zero due to an error I found in the log: "FATAL: dying from an unhandled exception: IsConnOpen(conn)" - moving back a little (11:49:30), I had a "reload" with a fatal error: 2017/10/24 11:49:30.406 kid1| Reconfiguring Squid Cache (version 3.5.27-20170916-re69e56c)... 2017/10/24 11:49:30.406 kid1| Closing HTTP port [::]:3128 2017/10/24 11:49:30.406 kid1| Closing HTTP port [::]:3129 2017/10/24 11:49:30.406 kid1| Closing HTTPS port [::]:3130 2017/10/24 11:49:30.406 kid1| Closing Pinger socket on FD 788 2017/10/24 11:49:30.406 kid1| Logfile: closing log daemon:/var/log/squid/access.log 2017/10/24 11:49:30.406 kid1| Logfile Daemon: closing log daemon:/var/log/squid/access.log 2017/10/24 11:49:30.406 kid1| FATAL: dying from an unhandled exception: IsConnOpen(conn) 2017/10/24 11:49:33 kid1| Set Current Directory to /var/cache/squid 2017/10/24 11:49:33 kid1| Starting Squid Cache version 3.5.27-20170916-re69e56c for x86_64-pc-linux-gnu... 2017/10/24 11:49:33 kid1| Service Name: squid 2017/10/24 11:49:33 kid1| Process ID 30753 2017/10/24 11:49:33 kid1| Process Roles: worker 2017/10/24 11:49:33 kid1| With 1024 file descriptors available - searching within the older rotated cache.log files I noticed this: 2017/10/20 10:42:12 kid1| With 16384 file descriptors available 2017/10/21 05:50:32 kid1| With 1024 file descriptors available The restart at 05:50:32 is performed by a cron job. Conclusion: it seems that the cron daemon I use (https://github.com/cronie-crond/cronie) is not honoring system-wide ulimits defined in /etc/security/limits.d/* as, eg.: * hard nofile 65535 * soft nofile 16384 Weird. I'm not going to be using the cron job anymore. Thanks, Vieri _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
In reply to this post by Alex Rousskov
Just in case someone else bumps into this issue, I can confirm that ulimits are not honored when running a cron job (at least from cronie).
In my case, if I set ulimits globally with: * hard nofile 65535 * soft nofile 16384 I get this outside cron: open files (-n) 16384 However, I get the following within cron: open files (-n) 1024 as demonstrated by running the following test from crontab: root ulimit -a >/tmp/ulimit 2>&1 3>&1 So beware. I guess one can avoid this by setting ulimits within Squid's init script, and calling that within cron. No idea whether this is a cron bug or feature. Vieri _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
In reply to this post by Alex Rousskov
Just in case someone else bumps into this, I've had the same issue trying to restart Squid from inittab. System-wide ulimit settings are not honored when running scripts from either crontab or inittab.
_______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
In reply to this post by Alex Rousskov
It's seems logical that the cron daemon needs to be restarted for ulimits to apply.
Init should also be restarted, but "init u" is not enough. A system reboot seems to be inevitable. _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
«
Return to Squid - Users
|
1 view|%1 views
| Free forum by Nabble | Edit this page |