TCP out of memory

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

TCP out of memory

Vieri
Hi,

I need to restart Squid once a week because I see "TCP out of memory" messages in syslog.

I see lots of open file descriptors of type "127.0.0.1:1344".

There could be an issue with the c-icap service.

As suggested previously, I dumped a packet trace here:

https://drive.google.com/file/d/1qCkH6YYa7fgeYzm-AoJEpXTDVpzILCQ9/view?usp=sharing

Can anyone please take a look at it? I'm trying to determine whether c-icap is closing connections properly.
Maybe the dump's time range is too short to see anything useful?
I also tried looking at the c-icap logs, but unfortunately I don't see anything (or I don't know how to interpret them correctly).


Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Amos Jeffries
Administrator
On 18/12/17 21:02, Vieri wrote:

> Hi,
>
> I need to restart Squid once a week because I see "TCP out of memory" messages in syslog.
>
> I see lots of open file descriptors of type "127.0.0.1:1344".
>
> There could be an issue with the c-icap service.
>
> As suggested previously, I dumped a packet trace here:
>
> https://drive.google.com/file/d/1qCkH6YYa7fgeYzm-AoJEpXTDVpzILCQ9/view?usp=sharing
>
> Can anyone please take a look at it? I'm trying to determine whether c-icap is closing connections properly.
> Maybe the dump's time range is too short to see anything useful?
> I also tried looking at the c-icap logs, but unfortunately I don't see anything (or I don't know how to interpret them correctly).
>

What is your ICAP configuration in squid.conf?

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri

________________________________
From: Amos Jeffries <[hidden email]>
>
> What is your ICAP configuration in squid.conf?


icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024
icap_service squidclamav respmod_precache bypass=0 icap://127.0.0.1:1344/clamav
adaptation_access squidclamav allow all

icap_service_failure_limit -1
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri
In reply to this post by Amos Jeffries
BTW, I set icap_service_failure_limit -1 because If I don't the HTTP clients get the ERR_ICAP_FAILURE page for *a very long time* if I restart the c-icap service.
I have no idea how much they would have to wait because I can't afford users seeing this page for more than a minute.
The following test was done in 2 minutes with the default icap_service_failure_limit:

# /etc/init.d/squid reload

# /etc/init.d/c-icap restart

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 21 Dec 2017 08:07:07 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 21 Dec 2017 08:07:07 GMT
Last-Modified: Thu, 21 Dec 2017 08:07:07 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20171101-re69e56c
Build Info:
Service Name: squid
Start Time:     Mon, 18 Dec 2017 08:29:33 GMT
Current Time:   Thu, 21 Dec 2017 08:07:07 GMT
Connection information for squid:
Number of clients accessing cache:      569
Number of HTTP requests received:       3381083
Number of ICP messages received:        0
Number of ICP messages sent:    0
Number of queued ICP replies:   0
Number of HTCP messages received:       0
Number of HTCP messages sent:   0
Request failure ratio:   0.00
Average HTTP requests per minute since start:   786.7
Average ICP messages per minute since start:    0.0
Select loop called: 93431818 times, 2.760 ms avg
Cache information for squid:
Hits as % of all requests:      5min: 0.9%, 60min: 2.0%
Hits as % of bytes sent:        5min: 5.2%, 60min: 9.5%
Memory hits as % of hit requests:       5min: 88.4%, 60min: 60.2%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
Storage Swap size:      29156 KB
Storage Swap capacity:  89.0% used, 11.0% free
Storage Mem size:       30648 KB
Storage Mem capacity:   93.5% used,  6.5% free
Mean Object Size:       18.03 KB
Requests given to unlinkd:      49975
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   0.05633  0.04047
Cache Misses:          0.10281  0.11465
Cache Hits:            0.00000  0.00000
Near Hits:             0.01745  0.01955
Not-Modified Replies:  0.00000  0.00000
DNS Lookups:           0.04048  0.03868
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time:        257853.454 seconds
CPU Time:       6005.270 seconds
CPU Usage:      2.33%
CPU Usage, 5 minute avg:        6.76%
CPU Usage, 60 minute avg:       2.99%
Maximum Resident Size: 5549424 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted:       1004217 KB
memPoolAlloc calls: 980681181
memPoolFree calls:  999432183
File descriptor usage for squid:
Maximum number of file descriptors:   65536
Largest file desc currently in use:   4399
Number of file desc currently in use: 4052
Files queued for open:                   0
Available number of file descriptors: 61484
Reserved number of file descriptors:   100
Store Disk files open:                   0
Internal Data Structures:
2074 StoreEntries
1911 StoreEntries with MemObjects
1687 Hot Object Cache Items
1617 on-disk objects

Two minutes later and clients are still seeing ERR_ICAP_FAILURE, so I'm setting back to icap_service_failure_limit -1.

# /etc/init.d/squid reload

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 21 Dec 2017 08:09:26 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 21 Dec 2017 08:09:26 GMT
Last-Modified: Thu, 21 Dec 2017 08:09:26 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20171101-re69e56c
Build Info:
Service Name: squid
Start Time:     Mon, 18 Dec 2017 08:29:33 GMT
Current Time:   Thu, 21 Dec 2017 08:09:26 GMT
Connection information for squid:
Number of clients accessing cache:      569
Number of HTTP requests received:       3382478
Number of ICP messages received:        0
Number of ICP messages sent:    0
Number of queued ICP replies:   0
Number of HTCP messages received:       0
Number of HTCP messages sent:   0
Request failure ratio:   0.00
Average HTTP requests per minute since start:   786.6
Average ICP messages per minute since start:    0.0
Select loop called: 93457469 times, 2.761 ms avg
Cache information for squid:
Hits as % of all requests:      5min: 1.2%, 60min: 2.0%
Hits as % of bytes sent:        5min: 10.9%, 60min: 9.7%
Memory hits as % of hit requests:       5min: 88.3%, 60min: 60.1%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
Storage Swap size:      29156 KB
Storage Swap capacity:  89.0% used, 11.0% free
Storage Mem size:       30648 KB
Storage Mem capacity:   93.5% used,  6.5% free
Mean Object Size:       18.03 KB
Requests given to unlinkd:      49975
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   0.05046  0.04047
Cache Misses:          0.11465  0.11465
Cache Hits:            0.00000  0.00000
Near Hits:             0.00000  0.01955
Not-Modified Replies:  0.00000  0.00000
DNS Lookups:           0.04237  0.03868
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time:        257992.456 seconds
CPU Time:       6009.530 seconds
CPU Usage:      2.33%
CPU Usage, 5 minute avg:        4.90%
CPU Usage, 60 minute avg:       2.98%
Maximum Resident Size: 5549728 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted:       997578 KB
memPoolAlloc calls: 980907766
memPoolFree calls:  999669215
File descriptor usage for squid:
Maximum number of file descriptors:   65536
Largest file desc currently in use:   4399
Number of file desc currently in use: 3676
Files queued for open:                   0
Available number of file descriptors: 61860
Reserved number of file descriptors:   100
Store Disk files open:                   0
Internal Data Structures:
1895 StoreEntries
1732 StoreEntries with MemObjects
1687 Hot Object Cache Items
1617 on-disk objects


Clients are now browsing, and squid/c-icap are apparently communicating.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users