kswapd0 and memory usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

kswapd0 and memory usage

Vieri
Hi,

I've been running squid & c-icap for years, and only recently have I had a severe system slowdown.

My kswapd0 process was at a constant 100% CPU usage level until I forced restarting of both squid and c-icap.

I've been using several Squid versions over the years, but the only differences I know of between my previous recent setup that worked and the current setup that has "failed" once (for now)  are:

- upgraded from 5.0.4-20201125-r5fadc09ee to Version 5.0.5-20210223-r4af19cc24

- set cgroups for both squid and c-icap services with just one setting: cpu.shares 512

- upgraded to c-icap 0.5.8

Given the stressful situation I only had time to notice that kswapd0 was at 100%, that top reported that all swap space was being used, and that the whole server was very sluggish. The additional problem is that the system is a router and uses TPROXY with squid sslbump so I don't think I can virtualize the web proxying services. Hence the use of cgroups to try to contain squid, c-icap and clamav. I have yet to define a cgroup for memory usage.

Restarting Squid and c-icap alone (not clamd) immediately solved the kswapd0 "gone wild" issue.
Mem usage went back to something like:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       9.2Gi        21Gi        48Mi       1.0Gi        21Gi
Swap:          35Gi       1.7Gi        33Gi

I only have debug_options rotate=1 ALL,1 in my squid config file, and sifting through cache.log doesn't give me any clues.

If this were to happen again (not sure when or if) what should I try to search for?

Regards,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: kswapd0 and memory usage

Amos Jeffries
Administrator
On 30/03/21 4:00 am, Vieri wrote:
>
> If this were to happen again (not sure when or if) what should I try to search for?
>

Output of the "squidclient mgr:mem", "top" and "ps waux" commands would
be good.

Those will show how Squid is using the memory it has, what processes are
using the most memory, and what processes are running. Most memory
issues can be triaged with that info.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: kswapd0 and memory usage

Vieri
 On Tuesday, March 30, 2021, 8:01:30 AM GMT+2, Amos Jeffries <[hidden email]> wrote:

>> If this were to happen again (not sure when or if) what should I try to search for?
>
> Output of the "squidclient mgr:mem", "top" and "ps waux" commands would
> be good.
>
> Those will show how Squid is using the memory it has, what processes are
> using the most memory, and what processes are running. Most memory
> issues can be triaged with that info.

Will do, thanks. I have a script that tries to "predict" when these problems are about to happen. It runs something like
timeout 30 squidclient mgr:info
and if it actually times out then it restarts both squid and c-icap.
So I'm afraid I might not get anything out of "squidclient mgr:mem", but I will run top -b -n 1 and ps waux.

Thanks,

Vieri



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users