I've been running squid & c-icap for years, and only recently have I had a severe system slowdown.
My kswapd0 process was at a constant 100% CPU usage level until I forced restarting of both squid and c-icap.
I've been using several Squid versions over the years, but the only differences I know of between my previous recent setup that worked and the current setup that has "failed" once (for now) are:
- upgraded from 5.0.4-20201125-r5fadc09ee to Version 5.0.5-20210223-r4af19cc24
- set cgroups for both squid and c-icap services with just one setting: cpu.shares 512
- upgraded to c-icap 0.5.8
Given the stressful situation I only had time to notice that kswapd0 was at 100%, that top reported that all swap space was being used, and that the whole server was very sluggish. The additional problem is that the system is a router and uses TPROXY with squid sslbump so I don't think I can virtualize the web proxying services. Hence the use of cgroups to try to contain squid, c-icap and clamav. I have yet to define a cgroup for memory usage.
Restarting Squid and c-icap alone (not clamd) immediately solved the kswapd0 "gone wild" issue.
Mem usage went back to something like:
# free -h
total used free shared buff/cache available
Mem: 31Gi 9.2Gi 21Gi 48Mi 1.0Gi 21Gi
Swap: 35Gi 1.7Gi 33Gi
I only have debug_options rotate=1 ALL,1 in my squid config file, and sifting through cache.log doesn't give me any clues.
If this were to happen again (not sure when or if) what should I try to search for?
On Tuesday, March 30, 2021, 8:01:30 AM GMT+2, Amos Jeffries <[hidden email]> wrote:
>> If this were to happen again (not sure when or if) what should I try to search for?
> Output of the "squidclient mgr:mem", "top" and "ps waux" commands would
> be good.
> Those will show how Squid is using the memory it has, what processes are
> using the most memory, and what processes are running. Most memory
> issues can be triaged with that info.
Will do, thanks. I have a script that tries to "predict" when these problems are about to happen. It runs something like
timeout 30 squidclient mgr:info
and if it actually times out then it restarts both squid and c-icap.
So I'm afraid I might not get anything out of "squidclient mgr:mem", but I will run top -b -n 1 and ps waux.