TCP out of memory

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

TCP out of memory

Vieri
Hi again,

I haven't taken a look at Squid's source code, but I guess that when Squid communicates with a c-icap service it acts as a typical socket client, right?
eg. connect(), write(), read(), close()

Does Squid consider forcing disconnection (close()) if the read() is "too long"?
Is there such a timeout? Is it configurable in squid.conf (only for the c-icap connection)?


Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Amos Jeffries
Administrator
On 04/01/18 21:51, Vieri wrote:
> Hi again,
>
> I haven't taken a look at Squid's source code, but I guess that when Squid communicates with a c-icap service it acts as a typical socket client, right?
> eg. connect(), write(), read(), close()

Uh, Those are system calls for receiving TCP connections and data I/O.

Squid uses ICAP protocol to communicate with ICAP services. ICAP
operates as a layer over TCP. So in a way yes, and in a way no.


>
> Does Squid consider forcing disconnection (close()) if the read() is "too long"?

If you mean "too long" in terms of memory size - there is no such thing
in TCP. Squid provides a buffer and tells the kernel how much space is
available there. The OS writes up to that much, no more, maybe less.

If your title "TCP out of memory" is an error you are seeing somewhere.
That is entirely your system kernel and networking stacks issue. Squid
has nothing to do with the internal memory management of TCP traffic.


> Is there such a timeout? Is it configurable in squid.conf (only for the c-icap connection)?
>

What timeout you speak of?

If you mean "too long" earlier in terms of duration to perform a read()
- there is also no such thing. The OS tells Squid when data is ready and
the data copy from OS memory to Squid buffer takes trivial amount of
time to actually happen.


Timeouts can happen between different parts of the ICAP protocol waiting
for I/O to happen for related message parts. But those have nothing to
do with TCP memory unless you are suffering a bad case of buffer bloat
in the network itself.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri
The open sockets to 127.0.0.1:1344 keep increasing steadily even on high network usage, but they do not decrease when there's little or no traffic.
So, day after day the overall number keeps growing until I have to restart squid once or twice a week.

In other words, this value keeps growing:
Largest file desc currently in use:   xxxx
This other value can decrease at times, but in the long run it keeps growing too:
Number of file desc currently in use: xxxx

I tried changing squid parameters such as:

icap_io_timeout time-units
icap_service_failure_limit
icap_persistent_connections on

I also tried changing c-icap parameters such as:

Timeout
MaxKeepAliveRequests
KeepAliveTimeout
StartServers
MaxServers
MinSpareThreads
MaxSpareThreads
ThreadsPerChild
MaxRequestsPerChild

However, I'm still seeing the same behavior so I reverted back to defaults.

This is a small sample of an icap_log taken from Squid:

1515056868.505      9 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff ICAP_OPT/200 353 OPTIONS icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.506     10 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.506     10 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.511   1624 10.215.246.143 ICAP_MOD/200 306783 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.547   1450 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.560     64 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.560     64 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.561      0 10.215.248.99 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.561     65 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.615      5 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.660      3 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.669      3 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.681      6 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.701      5 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.712     18 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.720      0 10.215.246.143 ICAP_MOD/200 489 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.739     29 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.740      0 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.750      0 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.774      0 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.848      0 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.865      0 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.866      6 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.879      0 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.891      0 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.893      2 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056868.906      2 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.132      0 10.215.246.218 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.149      0 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.316      6 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.322      3 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.329      1 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.338      1 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.346      1 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.354      1 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.408      1 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.421      2 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.431      1 10.215.246.143 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.444      4 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.452      3 10.215.246.143 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.487      0 10.215.145.222 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.489      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.737      0 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056869.966    108 10.215.248.31 ICAP_MOD/200 65761 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056870.419      3 10.215.248.31 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056870.452      3 10.215.248.31 ICAP_MOD/200 23154 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056870.831      2 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056871.197      3 10.215.248.31 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056871.317      0 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056871.371  26103 10.215.247.234 ICAP_MOD/200 914 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056871.910      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056871.952  27493 10.215.246.245 ICAP_MOD/200 905 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.020      1 10.215.144.48 ICAP_MOD/200 10205 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.048      3 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.112      1 10.215.144.171 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.172    111 10.215.248.31 ICAP_MOD/200 46966 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.257  60004 10.215.246.224 ICAP_MOD/200 6588 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.260      0 10.215.246.224 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.394      1 10.215.246.224 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.473      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.546      1 10.215.246.224 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.672     54 10.215.248.31 ICAP_MOD/200 23153 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.788      2 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.856      3 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.914      3 10.215.248.31 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056872.989      1 10.215.248.99 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056873.155      0 10.215.247.182 ICAP_MOD/200 993 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056873.239      0 10.215.247.182 ICAP_MOD/200 993 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056873.272      0 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056873.970      0 10.215.248.31 ICAP_MOD/200 2764 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.021      0 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.192      0 10.215.248.31 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.206      0 10.215.248.99 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.226  59983 10.215.144.222 ICAP_MOD/200 6550 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.228      0 10.215.144.222 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.328      0 10.215.248.99 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.351      1 10.215.144.222 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.363      1 10.215.247.132 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.502      2 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.513      0 10.215.144.222 ICAP_ECHO/204 130 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.617      1 10.215.145.40 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.789      0 10.215.247.132 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056874.995      0 10.215.145.40 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056875.024      1 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056875.039      0 10.215.248.99 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056875.412      0 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056875.582      0 10.215.248.152 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056875.700      2 10.215.145.136 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056876.291      2 10.215.248.31 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.020     24 10.215.247.182 ICAP_MOD/200 65493 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.134      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.187      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.358      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.452      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.502      0 10.215.247.182 ICAP_ECHO/204 105 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.536  26334 10.215.246.136 ICAP_MOD/200 914 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -
1515056877.809  25511 10.215.247.120 ICAP_MOD/200 916 RESPMOD icap://127.0.0.1:1344/clamav - -/127.0.0.1 -

Yes, the title refers to:
kernel: TCP: out of memory -- consider tuning tcp_mem

I'm trying to find out what's wrong on this system even though restarting Squid twice a week at night isn't too bad in my case.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Amos Jeffries
Administrator
On 05/01/18 21:59, Vieri wrote:
> The open sockets to 127.0.0.1:1344 keep increasing steadily even on high network usage, but they do not decrease when there's little or no traffic.
> So, day after day the overall number keeps growing until I have to restart squid once or twice a week.
>
> In other words, this value keeps growing:
> Largest file desc currently in use:   xxxx
> This other value can decrease at times, but in the long run it keeps growing too:
> Number of file desc currently in use: xxxx
>

Ah. What does the cachemgr "filedescriptors" report show when there are
a lot starting to accumulate?

And, are you able to get a cache.log trace with "debug_options 93,6" ?

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri

________________________________
From: Amos Jeffries <[hidden email]>

>> The open sockets to 127.0.0.1:1344 keep increasing steadily even on high network usage, but they do not decrease when there's
>> little or no traffic.>> So, day after day the overall number keeps growing until I have to restart squid once or twice a week.
>>
>> In other words, this value keeps growing:
>> Largest file desc currently in use:   xxxx
>> This other value can decrease at times, but in the long run it keeps growing too:
>> Number of file desc currently in use: xxxx
>>
> Ah. What does the cachemgr "filedescriptors" report show when there are
> a lot starting to accumulate?
>
> And, are you able to get a cache.log trace with "debug_options 93,6" ?


Here's my cache.log:

https://drive.google.com/file/d/1I8R5sCsIGhYa69QmGrOoHVITuom4uW0k/view?usp=sharing

squidclient's filedescriptors:

https://drive.google.com/file/d/1o6zn-o0atqeqFGSMRhPA9r1AAFJpnpBZ/view?usp=sharing

The info page:

https://drive.google.com/file/d/11iWqjgdt2KK1yWPMsr5o-IyWGyKS7joc/view?usp=sharing

The open fds are at around 7k, but they can easily reach 12k or 13k. That's when I start running into trouble.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Amos Jeffries
Administrator
On 08/01/18 11:13, Vieri wrote:

>
> ________________________________
> From: Amos Jeffries <[hidden email]>
>
>>> The open sockets to 127.0.0.1:1344 keep increasing steadily even on high network usage, but they do not decrease when there's
>>> little or no traffic.>> So, day after day the overall number keeps growing until I have to restart squid once or twice a week.
>>>
>>> In other words, this value keeps growing:
>>> Largest file desc currently in use:   xxxx
>>> This other value can decrease at times, but in the long run it keeps growing too:
>>> Number of file desc currently in use: xxxx
>>>
>> Ah. What does the cachemgr "filedescriptors" report show when there are
>> a lot starting to accumulate?
>>
>> And, are you able to get a cache.log trace with "debug_options 93,6" ?
>
>
> Here's my cache.log:
>
> https://drive.google.com/file/d/1I8R5sCsIGhYa69QmGrOoHVITuom4uW0k/view?usp=sharing
>
> squidclient's filedescriptors:
>
> https://drive.google.com/file/d/1o6zn-o0atqeqFGSMRhPA9r1AAFJpnpBZ/view?usp=sharing
>
> The info page:
>
> https://drive.google.com/file/d/11iWqjgdt2KK1yWPMsr5o-IyWGyKS7joc/view?usp=sharing
>
> The open fds are at around 7k, but they can easily reach 12k or 13k. That's when I start running into trouble.
>

Thank you.

I have only taken a brief look, but so far it looks like the problematic
sockets are not participating in any ICAP activity. That implies they
are possibly TCP connections which never complete their opening
sequence, or at least the result of connection attempts does not make it
back to the ICAP code somehow.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri

________________________________
From: Amos Jeffries <[hidden email]>
>
> I have only taken a brief look, but so far it looks like the problematic

> sockets are not participating in any ICAP activity.

Do you see that from the cache.log, or from ":filedescriptors"?
If I list my current filedescriptors right now, I get this:

# squidclient mgr:filedescriptors | grep "127.0.0.1:1344"
20 Socket  899      25*   10001  127.0.0.1:1344        127.0.0.1
30 Socket    0   71648    72210  127.0.0.1:1344        127.0.0.1
38 Socket  900       0*    2564  127.0.0.1:1344        127.0.0.1
102 Socket    0   67222    67689  127.0.0.1:1344        127.0.0.1
107 Socket    0  102679   203677  127.0.0.1:1344        127.0.0.1
113 Socket    0   67222    67709  127.0.0.1:1344        127.0.0.1
115 Socket  886       0*    2588  127.0.0.1:1344        127.0.0.1
116 Socket  873      25*   10395  127.0.0.1:1344        127.0.0.1
129 Socket    0  114892   144095  127.0.0.1:1344        127.0.0.1
134 Socket  900      25*    8863  127.0.0.1:1344        127.0.0.1
160 Socket    0   67222    67687  127.0.0.1:1344        127.0.0.1
165 Socket    0   77833    78401  127.0.0.1:1344        127.0.0.1
166 Socket    0   67222    67702  127.0.0.1:1344        127.0.0.1
175 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
176 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
212 Socket    0   67222    67742  127.0.0.1:1344        127.0.0.1
213 Socket  878       0*    2533  127.0.0.1:1344        127.0.0.1
226 Socket  873       0*    2531  127.0.0.1:1344        127.0.0.1
236 Socket    0   78332   180786  127.0.0.1:1344        127.0.0.1
244 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
281 Socket    0   67222    67685  127.0.0.1:1344        127.0.0.1
285 Socket    0   78253   149568  127.0.0.1:1344        127.0.0.1
298 Socket    0   77833    78451  127.0.0.1:1344        127.0.0.1
305 Socket    0   74366   168309  127.0.0.1:1344        127.0.0.1
307 Socket    0  114519   115068  127.0.0.1:1344        127.0.0.1
326 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
327 Socket    0   67222    67687  127.0.0.1:1344        127.0.0.1
365 Socket    0   70248   114918  127.0.0.1:1344        127.0.0.1
372 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
390 Socket    0   77833    78483  127.0.0.1:1344        127.0.0.1
404 Socket    0   90022    90703  127.0.0.1:1344        127.0.0.1
464 Socket    0   78253   144095  127.0.0.1:1344        127.0.0.1
472 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
480 Socket  891       0*    2514  127.0.0.1:1344        127.0.0.1
491 Socket    0   67222    67685  127.0.0.1:1344        127.0.0.1
509 Socket    0   67222    67687  127.0.0.1:1344        127.0.0.1
512 Socket    0   67222    67703  127.0.0.1:1344        127.0.0.1
528 Socket    0  131176   155548  127.0.0.1:1344        127.0.0.1
536 Socket    0   70111   134058  127.0.0.1:1344        127.0.0.1
547 Socket    0   67222    67689  127.0.0.1:1344        127.0.0.1
554 Socket    0  131860   152673  127.0.0.1:1344        127.0.0.1
570 Socket    0   67222    67707  127.0.0.1:1344        127.0.0.1
572 Socket  893       0*    2706  127.0.0.1:1344        127.0.0.1
596 Socket    0   78390   114864  127.0.0.1:1344        127.0.0.1
602 Socket    0   67222    67691  127.0.0.1:1344        127.0.0.1
624 Socket    0   72678    73442  127.0.0.1:1344        127.0.0.1
631 Socket    0   71646    72250  127.0.0.1:1344        127.0.0.1
635 Socket    0  104333   104896  127.0.0.1:1344        127.0.0.1
641 Socket    0   67222    67687  127.0.0.1:1344        127.0.0.1
646 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
662 Socket    0   67222    67698  127.0.0.1:1344        127.0.0.1
674 Socket    0   67222    67691  127.0.0.1:1344        127.0.0.1
678 Socket    0   67222    67687  127.0.0.1:1344        127.0.0.1
687 Socket    0   67222    67702  127.0.0.1:1344        127.0.0.1
730 Socket    0   67222    67691  127.0.0.1:1344        127.0.0.1
767 Socket    0   74465   152811  127.0.0.1:1344        127.0.0.1
772 Socket    0   67217    67747  127.0.0.1:1344        127.0.0.1
815 Socket    0   77864    78246  127.0.0.1:1344        127.0.0.1
848 Socket    0   67222    67743  127.0.0.1:1344        127.0.0.1
865 Socket    0   67222    67747  127.0.0.1:1344        127.0.0.1
890 Socket    0   67222    67699  127.0.0.1:1344        127.0.0.1
943 Socket    0   77833    78501  127.0.0.1:1344        127.0.0.1
1008 Socket    0   74212    78383  127.0.0.1:1344        127.0.0.1
1018 Socket    0   74466    90630  127.0.0.1:1344        127.0.0.1
1099 Socket    0   67222    67687  127.0.0.1:1344        127.0.0.1
1124 Socket    0   67222    67683  127.0.0.1:1344        127.0.0.1
1167 Socket    0   67222    67687  127.0.0.1:1344        127.0.0.1
1273 Socket    0   67258    67879  127.0.0.1:1344        127.0.0.1
1337 Socket    0   74243    78265  127.0.0.1:1344        127.0.0.1


Both Nread and Nwrite seem to be well over 0.

> That implies they > are possibly TCP connections which never complete their opening
> sequence, or at least the result of connection attempts does not make it
> back to the ICAP code somehow.


ICAP and Squid are both on localhost. I'd like to find out why this is happening.


I believe I already posted a tcpdump trace of the ICAP traffic, but I don't know if you had a chance to take a look at it. I had a quick look, but I'm not familiar with the ICAP protocol. In any case, I probably would see a lot of OPTIONS, REQMOD, RESPMOD methods, but I don't know if I would clearly detect initial TCP issues.


Anyway, here's a dumb question. Can't Squid "tell" when a TCP connection to an ICAP server has never completed correctly after x timeout, and close it down/reset it?
I'm using default values in squid.conf for the following:
connect_timeout
icap_connect_timeout
peer_connect_timeout

The docs say:
#  TAG: icap_connect_timeout
#       This parameter specifies how long to wait for the TCP connect to
#       the requested ICAP server to complete before giving up and either
#       terminating the HTTP transaction or bypassing the failure.


BTW I guess it's just a typo error because instead of an "HTTP transaction" I should read "ICAP transaction", right?

Anyway, I have "bypass=0" for the ICAP service so I guess it should honor connect_timeout.
The default is connect_timeout 1 minute.


With a 1-minute timeout it may be hard to see the sockets close when there's plenty of traffic, but I think I should see a substantial drop of open sockets when traffic is low (eg. at night). However, I don't see it.


What could I try?

Thank you very much for your time.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Amos Jeffries
Administrator
On 09/01/18 22:40, Vieri wrote:
>
> ________________________________
> From: Amos Jeffries <[hidden email]>
>>
>> I have only taken a brief look, but so far it looks like the problematic
>
>> sockets are not participating in any ICAP activity.
>
> Do you see that from the cache.log, or from ":filedescriptors"?

I went through the cache.log looking for the FD numbers mentioned to
correlate with the filedescriptors report lines. The first dozen I found
all seemed to be either in some clearly described state (not that
"127.0.0.1" description).

My deeper look was going to make a script to do the reverse check. For
each entry in the filedescriptors, seeing if there was any ICAP mentions
and print where in the cache.log to look.


>
>
> Both Nread and Nwrite seem to be well over 0.
>

That I think is really odd for sockets to port 1344 which are not having
any ICAP protocol activity happening.


>> That implies they > are possibly TCP connections which never complete their opening
>> sequence, or at least the result of connection attempts does not make it
>> back to the ICAP code somehow.
>
>
> ICAP and Squid are both on localhost. I'd like to find out why this is happening.
>


That is why I specifically asked for 93,* trace. The 93,* traces should
not contains any of the HTTP traffic to Squid just the ICAP.

If you can get an 11,* trace as well separately to see if any of the
stuck port 1344 FDs are using HTTP. There should be none. But if there
are that could well be the problem on those ones.


>
> I believe I already posted a tcpdump trace of the ICAP traffic, but I don't know if you had a chance to take a look at it. I had a quick look, but I'm not familiar with the ICAP protocol. In any case, I probably would see a lot of OPTIONS, REQMOD, RESPMOD methods, but I don't know if I would clearly detect initial TCP issues.
>

You mean the trace in your "TCP out of memory" thread last month?
What I'm seeing in there is connections being used for ICAP, then the
ICAP service requesting keep-alive which Squid honors. Nothing obviously
broken.


>
> Anyway, here's a dumb question. Can't Squid "tell" when a TCP connection to an ICAP server has never completed correctly after x timeout, and close it down/reset it?
> I'm using default values in squid.conf for the following:
> connect_timeout
> icap_connect_timeout
> peer_connect_timeout
>
> The docs say:
> #  TAG: icap_connect_timeout
> #       This parameter specifies how long to wait for the TCP connect to
> #       the requested ICAP server to complete before giving up and either
> #       terminating the HTTP transaction or bypassing the failure.
>
>
> BTW I guess it's just a typo error because instead of an "HTTP transaction" I should read "ICAP transaction", right?

Nope. If ICAP fails for any reason and is mandatory (bypass=0) the HTTP
transaction which is trying to use it gets broken. So the HTTP request
(transaction) needs to be aborted with an error.
That does not necessarily mean the TCP connection used by the HTTP
though, just the request/reply transaction.


>
> Anyway, I have "bypass=0" for the ICAP service so I guess it should honor connect_timeout.
> The default is connect_timeout 1 minute.
>

No, icap_connect_timeout is the only one that should be relevant on
connections to ICAP services.

connect_timeout is for HTP origin servers, and peer_connect_timeout is
for cache_peers.


>
> With a 1-minute timeout it may be hard to see the sockets close when there's plenty of traffic, but I think I should see a substantial drop of open sockets when traffic is low (eg. at night). However, I don't see it.
>

If you mean the *connect_timeout, that only affects the time Squid waits
for the outgoing TCP SYN/SYN+ACK process to complete. Persistent and
other connections already opened have other timeouts applied depending
on what they are being used for.

The *idle_pconn_timeout directives are more likely to result in closed
connections for overnight etc. cases if clients do not actively end them
anyway.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri
Hi,

I don't know how to cleanly seperate the 93,* from the 11,* log lines. I posted the following:


https://drive.google.com/file/d/1PRJOc6czrA0QEDHkqn3MrmNh08K8JajR/view?usp=sharing

It contains a cache.log generated by:
debug_options rotate=1 ALL,0 93,6 11,6

I also ran :info and :filedescriptors when I applied the new debug_options (*1), and again when I reverted back the debug_options (*2).

I'm using c-icap with squidclamav. I'll try to use c-icap-modules instead asap so I can hopefully remove a few variables in the issue (if it keeps giving me this issue then it must be a c-icap service flaw).

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri
In reply to this post by Amos Jeffries
Hi,

Just a quick follow-up on this.

I dropped squidclamav so I could test c-icap-modules's clamd service instead.
The only difference between the two is that squidclamav was using unix sockets while c-icap-modules is using clamd.

At first, the results were good. The open fd numbers were fluctuating, but within the 1k-2k limit during the first days. However, today I'm getting 4k, and it's only day 5. I suspect I'll be getting 10k+ numbers within another week or two. That's when I'll have to restart squid if I don't want the system to come to a network crawl.

I'm posting info and filedescriptors here:

https://drive.google.com/file/d/1V7Horvvak62U-HjSh5pVEBvVnZhu-iQY/view?usp=sharing

https://drive.google.com/file/d/1P1DAX-dOfW0fzt1sAeyT35brQyoPVodX/view?usp=sharing

By the way, what does "Largest file desc currently in use" mean exactly? Should this value also drop (eventually) under sane conditions?

So I guess moving from squidclamav to c-icap-modules did improve things, but I'm still facing something wrong. I could try moving back to squidclamav in "clamd mode" instead of unix sockets just to see if I get the same partial improvement as the one I've witnessed this week.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Amos Jeffries
Administrator
On 17/01/18 02:37, Vieri wrote:

> Hi,
>
> Just a quick follow-up on this.
>
> I dropped squidclamav so I could test c-icap-modules's clamd service instead.
> The only difference between the two is that squidclamav was using unix sockets while c-icap-modules is using clamd.
>
> At first, the results were good. The open fd numbers were fluctuating, but within the 1k-2k limit during the first days. However, today I'm getting 4k, and it's only day 5. I suspect I'll be getting 10k+ numbers within another week or two. That's when I'll have to restart squid if I don't want the system to come to a network crawl.
>
> I'm posting info and filedescriptors here:
>
> https://drive.google.com/file/d/1V7Horvvak62U-HjSh5pVEBvVnZhu-iQY/view?usp=sharing
>
> https://drive.google.com/file/d/1P1DAX-dOfW0fzt1sAeyT35brQyoPVodX/view?usp=sharing
>

Sorry I have a bit of a distraction going on ATM so have not got to that
detailed check yet. Good to hear you found a slightly better situation
though.


> By the way, what does "Largest file desc currently in use" mean exactly? Should this value also drop (eventually) under sane conditions?

The OS assigns FD numbers and prefers to assign with a strong bias
towards the lowest values. So that can be seen as a fluctuating "water
level" of approximately how many FD are currently in use. If there are a
few very long-lived connections and many short ones it may be variably
incorrect - but is good enough for a rough guide of FD usage.

In normal network conditions it should rise and fall with your peak vs
off-peak traffic times. I expect with your particular trouble it will
mostly just go upwards.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri
________________________________
From: Amos Jeffries <[hidden email]>
>
> Sorry I have a bit of a distraction going on ATM so have not got to that

> detailed check yet. Good to hear you found a slightly better situation > though.
[...]
> In normal network conditions it should rise and fall with your peak vs
> off-peak traffic times. I expect with your particular trouble it will
> mostly just go upwards.


No worries. I'd like to confirm that I'm still seeing the same issue with c-icap-modules, even though it's slightly better in that the FD numbers grow slower, at least at first.
I must say that it seems to be growing faster now. I had 4k two days ago, now I have:
Largest file desc currently in use:   6664
Number of file desc currently in use: 6270
So it seesm that the more days go by, the faster the FD numbers rise.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri
In reply to this post by Amos Jeffries
Hi,

I just wanted to add some information to this topic, although I'm not sure if it's related.


I noticed that if I set bypass=1 in squid.conf (regarding ICAP), and if I stop the local clamd service (not the c-icap service), then the clients see Squid's ERR_ICAP_FAILURE page.
Is this expected?

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Alex Rousskov
On 01/27/2018 10:33 AM, Vieri wrote:

> I noticed that if I set bypass=1 in squid.conf (regarding ICAP), and
> if I stop the local clamd service (not the c-icap service), then the
> clients see Squid's ERR_ICAP_FAILURE page. Is this expected?

Difficult to say for sure without knowing what went wrong between Squid
and c-icap: Not all ICAP errors can be bypassed. Consider sharing a
(link to) compressed ALL,9 cache.log collected while reproducing the
problem using a single HTTP transaction through an otherwise idle Squid.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Yuri Voinov

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
 
He's just disabled icap-based service without disabling icap itself. So
- yes - this is as expected.

Vieri, bupass=1 is different thing. This permit squid to bypass
adaptation in case of overloading icap service. And irrelevant thing you
done.

27.01.2018 23:41, Alex Rousskov пишет:
> On 01/27/2018 10:33 AM, Vieri wrote: > >> I noticed that if I set bypass=1 in squid.conf (regarding ICAP),
and >> if I stop the local clamd service (not the c-icap service), then
the >> clients see Squid's ERR_ICAP_FAILURE page. Is this expected? > >
Difficult to say for sure without knowing what went wrong between Squid
> and c-icap: Not all ICAP errors can be bypassed. Consider sharing a >
(link to) compressed ALL,9 cache.log collected while reproducing the >
problem using a single HTTP transaction through an otherwise idle Squid.
> > Alex. > _______________________________________________ >
squid-users mailing list > [hidden email] >
http://lists.squid-cache.org/listinfo/squid-users
- --
*****************************
* C++20 : Bug to the future *
*****************************
-----BEGIN PGP SIGNATURE-----
 
iQGzBAEBCAAdFiEEUAYDxOalHloZaGP7S+6Uoz43Q6cFAlpsuzEACgkQS+6Uoz43
Q6cogwv8D1lqLBtJeIMIbwgv/u6OBEtYAZbx7hAQpj4Hic8SBYsnHJdxUtHYchkC
pDFcrxPyp0b/59U26ngg5pObOHT5tynWYgH2tGp0/raJgPddD2G+xKyztvpawuz8
c1zHvUmyYhwHM96T99eTsI0r4qUq+e91krhK+6JxvSHh0CM8NnwUGycOhKBqNRHE
HGWOjXCLUf9QTw/C4QG2slpO5TJbVvKJjO+lnLyTxBZr+hFmORjbsAXMOsz83Txb
k0u4XuipQOqu3CcfLE/rSZPUX/r5YRQiV9roZdfy/IjYOdsU0+SvnO37RZLr9Z26
hKuu0OaZKGYcKtzy20lQJanIDzUOv7sDvNfqM3tWGTRNnxRD/1S3s2eFyuSZSx7/
0G4rbs/oj/Y7Ksa9mlGW02QNmjOmrUMB1iVwQu3IBWeef7WUQtgvFV+JuSGY8+q/
V605z627s6S8+fHAPzZyLGt/J0kMLWGbImOFUIIBBA+0g1yx41bUswo2feEivzzb
CwRJXfhc
=ZP4V
-----END PGP SIGNATURE-----

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Alex Rousskov
On 01/27/2018 10:47 AM, Yuri wrote:

> He's just disabled icap-based service without disabling icap itself. So
> - yes - this is as expected.

The above logic is flawed: Vieri told Squid to bypass (bypassable) ICAP
errors, but Squid did not bypass an ICAP error. Whether that outcome is
expected depends on whether that specific ICAP error was bypassable.

Yes, I understand that c-icap did not successfully process the message
after clamd went down, but that fact is not important here. What is
important here is how c-icap relayed that problem to Squid. That part I
do not know, so I cannot say whether Squid just could not bypass this
particular ICAP problem (i.e., Squid behavior is expected) or there is a
bug in Squid's bypass code.


> bupass=1 permit squid to bypass
> adaptation in case of overloading icap service.

Yes, but service overload is _not_ the only problem that bypass=1 can
bypass. The documentation for that option describes the option scope:

> If set to 'on' or '1', the ICAP service is treated as
> optional. If the service cannot be reached or malfunctions,
> Squid will try to ignore any errors and process the message as
> if the service was not enabled. No all ICAP errors can be
> bypassed.  If set to 0, the ICAP service is treated as
> essential and all ICAP errors will result in an error page
> returned to the HTTP client.
>
> Bypass is off by default: services are treated as essential.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP out of memory

Vieri
Hi,

I reproduced the problem, and saw that the c-icap server (or its squidclamav module) reports a 500 internal server error when clamd is down. I guess that's not bypassable?


The c-icap server log reports:

Mon Jan 29 08:30:35 2018, 5134/1290311424, squidclamav.c(1934) dconnect: Mon Jan 29 08:30:35 2018, 5134/1290311424, entering.
Mon Jan 29 08:30:35 2018, 5134/1290311424, squidclamav.c(2015) connectINET: Mon Jan 29 08:30:35 2018, 5134/1290311424, ERROR Can't connect on 127.0.0.1:3310.
Mon Jan 29 08:30:35 2018, 5134/1290311424, squidclamav.c(2015) connectINET: Mon Jan 29 08:30:35 2018, 5134/1290311424, ERROR Can't connect on 127.0.0.1:3310.
Mon Jan 29 08:30:35 2018, 5134/1290311424, squidclamav.c(744) squidclamav_end_of_data_handler: Mon Jan 29 08:30:35 2018, 5134/1290311424, ERROR Can't connect to Clamd daemon.
Mon Jan 29 08:30:35 2018, 5134/1290311424, An error occured in end-of-data handler !return code : -1, req->allow204=1, req->allow206=0


Here's Squid's log:

https://drive.google.com/file/d/18HmM8pOuDQmE4W_vwmSncXEeJSvgDjDo/view?usp=sharing

I was hoping I could relate this to the original topic, but I'm afraid they are two different issues.


Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: ICAP 500 is not bypassed

Alex Rousskov
On 01/29/2018 01:01 AM, Vieri wrote:

> I reproduced the problem, and saw that the c-icap server (or its
> squidclamav module) reports a 500 internal server error when clamd is
> down. I guess that's not bypassable?

The ICAP 500 status code does not preclude bypass. In fact, your Squid
attempts to bypass this ICAP server error:


> 2018/01/29 08:30:01.479 ... bypassing ... exception: Unsupported ICAP status code


Unfortunately, Squid bypass code then triggers an internal exception
which kills the bypass attempt and ends the HTTP transaction with
ICAP_ERR_OTHER:

> 2018/01/29 08:30:01.480 ... Throw: ModXact.cc:1848: exception: !disabled()
> 2018/01/29 08:30:01.480 ... Warning: reseting outcome: from ICAP_ECHO to ICAP_ERR_OTHER

That second exception looks like a Squid bug to me, but I have not
investigated it enough to be sure about that designation (not to mention
propose a fix). Your next steps revolve around these standard options:

https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: ICAP 500 is not bypassed

Vieri
Alex, thanks for your time.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users