SQUID memory error after vm.swappines changed from 60 to 10

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
Hi, I hope that someone can explain what happened, why squid stopped working.
The problem is related to  memory/swap handling.

After we changed vm.swappiness parameter from 60 to 10 (tuning
attempt, to lower a disk usage, because we have only 4 disks in a
RAID10, so disk subsystem  is a weak link), we got a lot of errors in
cache.log.
The problems started after scheduled logrotate after  2AM.
Squid ran out of memory, auth helpers stopped working.
It's weird because we didn't disable swap, but behavior is like we did.
After an error, we increased parameter from 10 to 40.

The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT cores).
We have 2800 users, using  kerberos authentication, squidguard for
filtering, ldap authorization.
When problem appeared memory was still 3GB free (free column), ram
(caching) was filled to 15GB, so 21 GB ram filled, 3GB free.

Thanks for help,


errors from cache.log.

2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
2017/11/08 02:55:27 kid1| logfileRotate: daemon:/var/log/squid/access.log
2017/11/08 02:55:27 kid1| logfileRotate: daemon:/var/log/squid/access.log
2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
'squidGuard' processes
2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard' process.
2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
'negotiate_kerberos_auth' processes
2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
2017/11/08 02:55:28 kid1| WARNING: Cannot run
'/usr/lib/squid/negotiate_kerberos_auth' process.
2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
2017/11/08 02:55:28 kid1| WARNING: Cannot run
'/usr/lib/squid/negotiate_kerberos_auth' process.
2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
2017/11/08 02:55:28 kid1| WARNING: Cannot run
'/usr/lib/squid/negotiate_kerberos_auth' process.

external ACL 'memberof' queue overload. Using stale result.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Marcus Kool
There is definitely a problem with available memory because Squid cannot fork.
So start with looking at how much memory Squid and its helpers use.
Do do have other processes on this system that consume a lot of memory ?

Also note that ufdbGuard uses less memory that squidGuard.
If there are 30 helpers squidguard uses 300% more memory than ufdbGuard.

Look at the wiki for more information about memory usage:
https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an expired certificate but it is safe to go ahead)

Marcus


On 08/11/17 07:26, Bike dernikov1 wrote:

> Hi, I hope that someone can explain what happened, why squid stopped working.
> The problem is related to  memory/swap handling.
>
> After we changed vm.swappiness parameter from 60 to 10 (tuning
> attempt, to lower a disk usage, because we have only 4 disks in a
> RAID10, so disk subsystem  is a weak link), we got a lot of errors in
> cache.log.
> The problems started after scheduled logrotate after  2AM.
> Squid ran out of memory, auth helpers stopped working.
> It's weird because we didn't disable swap, but behavior is like we did.
> After an error, we increased parameter from 10 to 40.
>
> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT cores).
> We have 2800 users, using  kerberos authentication, squidguard for
> filtering, ldap authorization.
> When problem appeared memory was still 3GB free (free column), ram
> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>
> Thanks for help,
>
>
> errors from cache.log.
>
> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
> 2017/11/08 02:55:27 kid1| logfileRotate: daemon:/var/log/squid/access.log
> 2017/11/08 02:55:27 kid1| logfileRotate: daemon:/var/log/squid/access.log
> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
> 'squidGuard' processes
> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard' process.
> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
> 'negotiate_kerberos_auth' processes
> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
> '/usr/lib/squid/negotiate_kerberos_auth' process.
> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
> '/usr/lib/squid/negotiate_kerberos_auth' process.
> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
> '/usr/lib/squid/negotiate_kerberos_auth' process.
>
> external ACL 'memberof' queue overload. Using stale result.
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Marcus Kool

On 08/11/17 11:36, Bike dernikov1 wrote:
> Hi,
>
> We stumbled on ufdbGuard, but licence/price was problem, we didn't
> read documentation carefully.
yes, ufdbguard is free.

> We will definitely try ufdbGuard, but we are now in process of moving
> squid/squidguard to production, so we can't test on production (angry
> users, Internet must work :)).
>
> Memory compsumption:squid use largest part of memory  (12GB now,
> second proces use 300MB memory), 14GB used by all process. So squid
> use over 80% of total used memory.
> So no there are not any problematic process. But we changed swappiness
> settings.
Did you monitor Squid for growth (it can start with 12 GB and grow slowly) ?

Squid cannot fork and higher swappiness increases the amount of memory that the OS can use to copy processes.
It makes me think that you have the memory overcommit set to 2 (no overcommit).
What is the output of the following command ?
    sysctl  -a | grep overcommit

> Advice for some settings:
> We have absolute max peak of  2500 users which user squid (of 2800),
> what are recomended settings for:
> negotiate_kerberos_children start/idle
> squidguard helpers.

I have little experience with kerberos, but most likely this is not the issue.
When Squid cannot fork the helpers, helper settings do not matter much.

For 2500 users you probably need 32-64 squidguard helpers.

Marcus

> Thanks for help,
>
> On Wed, Nov 8, 2017 at 10:53 AM, Marcus Kool
> <[hidden email]> wrote:
>> There is definitely a problem with available memory because Squid cannot
>> fork.
>> So start with looking at how much memory Squid and its helpers use.
>> Do do have other processes on this system that consume a lot of memory ?
>>
>> Also note that ufdbGuard uses less memory that squidGuard.
>> If there are 30 helpers squidguard uses 300% more memory than ufdbGuard.
>>
>> Look at the wiki for more information about memory usage:
>> https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an
>> expired certificate but it is safe to go ahead)
>>
>> Marcus
>>
>>
>>
>> On 08/11/17 07:26, Bike dernikov1 wrote:
>>>
>>> Hi, I hope that someone can explain what happened, why squid stopped
>>> working.
>>> The problem is related to  memory/swap handling.
>>>
>>> After we changed vm.swappiness parameter from 60 to 10 (tuning
>>> attempt, to lower a disk usage, because we have only 4 disks in a
>>> RAID10, so disk subsystem  is a weak link), we got a lot of errors in
>>> cache.log.
>>> The problems started after scheduled logrotate after  2AM.
>>> Squid ran out of memory, auth helpers stopped working.
>>> It's weird because we didn't disable swap, but behavior is like we did.
>>> After an error, we increased parameter from 10 to 40.
>>>
>>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT
>>> cores).
>>> We have 2800 users, using  kerberos authentication, squidguard for
>>> filtering, ldap authorization.
>>> When problem appeared memory was still 3GB free (free column), ram
>>> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>>>
>>> Thanks for help,
>>>
>>>
>>> errors from cache.log.
>>>
>>> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
>>> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
>>> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
>>> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
>>> 2017/11/08 02:55:27 kid1| logfileRotate: daemon:/var/log/squid/access.log
>>> 2017/11/08 02:55:27 kid1| logfileRotate: daemon:/var/log/squid/access.log
>>> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
>>> 'squidGuard' processes
>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard'
>>> process.
>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
>>> 'negotiate_kerberos_auth' processes
>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>
>>> external ACL 'memberof' queue overload. Using stale result.
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Alex Rousskov
In reply to this post by Bike dernikov1
On 11/08/2017 02:26 AM, Bike dernikov1 wrote:

> I hope that someone can explain what happened, why squid stopped working.

I can suggest a working theory: You did not have enough RAM before
vm.swappiness changes and the same insufficient RAM problem led to
failed system calls after you told the OS to free RAM (into swap) less
aggressively. The changes effectively decreased the amount of
immediately available RAM.

If Squid is the primary service on your server, then consider:

1. Removing RAID (for Squid disk cache).

2. Removing swap (or at least not counting on it and treating
   any non-trivial swap usage as a system misconfiguration).

3. Identify the primary RAM consumers and
   the factors that influence their memory usage.

The above actions may not solve your "insufficient RAM" problem (unless
RAID was the primary culprit) but they will give you a better starting
point for finding the solution. Others on the list can help you get
there, especially if you have specific questions.


Good luck,

Alex.

> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT cores).
> We have 2800 users, using  kerberos authentication, squidguard for
> filtering, ldap authorization.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Marcus Kool
On Wed, Nov 8, 2017 at 3:48 PM, Marcus Kool <[hidden email]> wrote:

>
> On 08/11/17 11:36, Bike dernikov1 wrote:
>>
>> Hi,
>>
>> We stumbled on ufdbGuard, but licence/price was problem, we didn't
>> read documentation carefully.
>
> yes, ufdbguard is free.
>
>> We will definitely try ufdbGuard, but we are now in process of moving
>> squid/squidguard to production, so we can't test on production (angry
>> users, Internet must work :)).
>>
>> Memory compsumption:squid use largest part of memory  (12GB now,
>> second proces use 300MB memory), 14GB used by all process. So squid
>> use over 80% of total used memory.
>> So no there are not any problematic process. But we changed swappiness
>> settings.
>
> Did you monitor Squid for growth (it can start with 12 GB and grow slowly) ?

Yes we are monitoring continuosly.
Now:
Output from free -m.

           total       used    free   shared  buff/cache  available
Mem:  24101     20507  256    146      3337         3034
Swap: 24561      5040   19521

vm.swappiness=40

Memory by process:
squid  Virt       RES   SHR  MEM%
           22,9G  18.7   8164   79,6
squidguard two process  300MB boths,.

CPU 0.33 0.37 0.43

> Squid cannot fork and higher swappiness increases the amount of memory that
> the OS can use to copy processes.
> It makes me think that you have the memory overcommit set to 2 (no
> overcommit).
> What is the output of the following command ?
>    sysctl  -a | grep overcommit

Command output:

vm.nr_overcommit_hugepages = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50

cat /proc/sys/vm/overcommit_memory
0


>> Advice for some settings:
>> We have absolute max peak of  2500 users which user squid (of 2800),
>> what are recomended settings for:
>> negotiate_kerberos_children start/idle
>> squidguard helpers.
>
>
> I have little experience with kerberos, but most likely this is not the
> issue.
> When Squid cannot fork the helpers, helper settings do not matter much.

> For 2500 users you probably need 32-64 squidguard helpers.

Can you confirm: For 2500 users:

url_rewrite children X (squidguard)  32-64 will be ok ? We have set
much larger number.

For  helper:
negotitate_kerberos_auth

auth_param negotiate children X startup Y idle Z. What X, Y, Z are
best for our user number ?

We disabled kerberos replay cache because of disk performance (4 SAS
DISK  15K, RAID 10) (iowait jumped high, and CPU load jumped to min
40 max 200).
We don't use disk caching.

Thanks for help,

> Marcus
>
>
>> Thanks for help,
>>
>> On Wed, Nov 8, 2017 at 10:53 AM, Marcus Kool
>> <[hidden email]> wrote:
>>>
>>> There is definitely a problem with available memory because Squid cannot
>>> fork.
>>> So start with looking at how much memory Squid and its helpers use.
>>> Do do have other processes on this system that consume a lot of memory ?
>>>
>>> Also note that ufdbGuard uses less memory that squidGuard.
>>> If there are 30 helpers squidguard uses 300% more memory than ufdbGuard.
>>>
>>> Look at the wiki for more information about memory usage:
>>> https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an
>>> expired certificate but it is safe to go ahead)
>>>
>>> Marcus
>>>
>>>
>>>
>>> On 08/11/17 07:26, Bike dernikov1 wrote:
>>>>
>>>>
>>>> Hi, I hope that someone can explain what happened, why squid stopped
>>>> working.
>>>> The problem is related to  memory/swap handling.
>>>>
>>>> After we changed vm.swappiness parameter from 60 to 10 (tuning
>>>> attempt, to lower a disk usage, because we have only 4 disks in a
>>>> RAID10, so disk subsystem  is a weak link), we got a lot of errors in
>>>> cache.log.
>>>> The problems started after scheduled logrotate after  2AM.
>>>> Squid ran out of memory, auth helpers stopped working.
>>>> It's weird because we didn't disable swap, but behavior is like we did.
>>>> After an error, we increased parameter from 10 to 40.
>>>>
>>>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT
>>>> cores).
>>>> We have 2800 users, using  kerberos authentication, squidguard for
>>>> filtering, ldap authorization.
>>>> When problem appeared memory was still 3GB free (free column), ram
>>>> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>>>>
>>>> Thanks for help,
>>>>
>>>>
>>>> errors from cache.log.
>>>>
>>>> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
>>>> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
>>>> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
>>>> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>> daemon:/var/log/squid/access.log
>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>> daemon:/var/log/squid/access.log
>>>> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
>>>> 'squidGuard' processes
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard'
>>>> process.
>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
>>>> 'negotiate_kerberos_auth' processes
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>
>>>> external ACL 'memberof' queue overload. Using stale result.
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> [hidden email]
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Alex Rousskov
>>On 11/08/2017 02:26 AM, Bike dernikov1 wrote:

>> I hope that someone can explain what happened, why squid stopped working.

>I can suggest a working theory: You did not have enough RAM before
>vm.swappiness changes and the same insufficient RAM problem led to
>failed system calls after you told the OS to free RAM (into swap) less
>aggressively. The changes effectively decreased the amount of
>immediately available RAM.

From "free -m output" (available column)  i thought that we have
enough memory, because  as I can remember, available never dropped
under 2GB.
As I understood  that system with  vm.swappines set to 10, will start
draining ram to swap after 90% memory used.
So why didn't swap hop in ? That we set vm.swappines to 0 i would
understand error.
Because errors started with  logrotate, i suspect that  problem
exploded because of logrotate and swap started to write to a disk at
same time,
and because of slow disks (small IOPS) it could not drain memory fast
enough, so available memory vanished and hell lose from there.


>If Squid is the primary service on your server, then consider:

It isn't only service but use over 90-95 percent memory resources.

>1. Removing RAID (for Squid disk cache).

We don't use disk cache because of slow disks, compared to  "high"
bandwith (we have 500Mbit/s)

>2. Removing swap (or at least not counting on it and treating
>  any non-trivial swap usage as a system misconfiguration).

What to search ? Key words, vm.swappines got us in problem, as we
tried to tune  ?

>3. Identify the primary RAM consumers and
>  the factors that influence their memory usage.

Squid, we suspect on kerberos negotiator, squid helpers, we will tune
up down setting over two weeks.
We have problems with squid.conf reconfiguration, because users lose
connection (they got squid error screen for moment) so it is nightmare
if we changing configuration in work time.

>The above actions may not solve your "insufficient RAM" problem (unless
>RAID was the primary culprit) but they will give you a better starting
>point for finding the solution. Others on the list can help you get
>there, especially if you have specific questions.
>
>
>Good luck,
>
>Alex.

Thanks for theory, and help.


On Wed, Nov 8, 2017 at 4:08 PM, Alex Rousskov
<[hidden email]> wrote:

> On 11/08/2017 02:26 AM, Bike dernikov1 wrote:
>
>> I hope that someone can explain what happened, why squid stopped working.
>
> I can suggest a working theory: You did not have enough RAM before
> vm.swappiness changes and the same insufficient RAM problem led to
> failed system calls after you told the OS to free RAM (into swap) less
> aggressively. The changes effectively decreased the amount of
> immediately available RAM.
>
> If Squid is the primary service on your server, then consider:
>
> 1. Removing RAID (for Squid disk cache).
>
> 2. Removing swap (or at least not counting on it and treating
>    any non-trivial swap usage as a system misconfiguration).
>
> 3. Identify the primary RAM consumers and
>    the factors that influence their memory usage.
>
> The above actions may not solve your "insufficient RAM" problem (unless
> RAID was the primary culprit) but they will give you a better starting
> point for finding the solution. Others on the list can help you get
> there, especially if you have specific questions.
>
>
> Good luck,
>
> Alex.
>
>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT cores).
>> We have 2800 users, using  kerberos authentication, squidguard for
>> filtering, ldap authorization.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Alex Rousskov
On 11/09/2017 06:49 AM, Bike dernikov1 wrote:
>>> On 11/08/2017 02:26 AM, Bike dernikov1 wrote:

> So why didn't swap hop in?

As I implied earlier, I do not recommend researching swap-related
details because I do not recommend using (or relying on) swap.


>> 2. Removing swap (or at least not counting on it and treating
>>  any non-trivial swap usage as a system misconfiguration).

> What to search ? Key words, vm.swappines got us in problem, as we
> tried to tune  ?

I do not understand the question. I am sure you know how to turn swap
off (i.e., completely disable it).


> We have problems with squid.conf reconfiguration, because users lose
> connection (they got squid error screen for moment) so it is nightmare
> if we changing configuration in work time.

Yes, this is a common complaint and a known Squid problem.

http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Marcus Kool
In reply to this post by Bike dernikov1


On 09/11/17 11:04, Bike dernikov1 wrote:
[snip]

>>> Memory compsumption:squid use largest part of memory  (12GB now,
>>> second proces use 300MB memory), 14GB used by all process. So squid
>>> use over 80% of total used memory.
>>> So no there are not any problematic process. But we changed swappiness
>>> settings.
>>
>> Did you monitor Squid for growth (it can start with 12 GB and grow slowly) ?
>
> Yes we are monitoring continuosly.
> Now:
> Output from free -m.
>
>             total       used    free   shared  buff/cache  available
> Mem:  24101     20507  256    146      3337         3034
> Swap: 24561      5040   19521
>
> vm.swappiness=40
>
> Memory by process:
> squid  Virt       RES   SHR  MEM%
>             22,9G  18.7   8164   79,6

Hmm. Squid grew from 12 GB to 18.7 GB (23 GB virtual).

With vm.swappiness=40 Linux starts to page out parts of processes when they occupy more than 60% of the memory.
This is a potential bottleneck and I would have also decreased vm.swappiness to 10 as you did.

My guess is that Squid starts too many helpers in a short time frame and that because of paging there are too many forks in progress simultaneously which causes the memory exhaustion.

I suggest to reduce the memory cache of Squid by 50% and set vm.swappiness to 20.
And then observe:
- total memory use
- total swap usage (should be lower than the 5 GB that you have now)
- number of helper processes that are started in short time frames
And then in small steps increase the memory cache and maybe further reduce vm.swappiness to 10.

> squidguard two process  300MB boths,.
>
> CPU 0.33 0.37 0.43
>
>> Squid cannot fork and higher swappiness increases the amount of memory that
>> the OS can use to copy processes.
>> It makes me think that you have the memory overcommit set to 2 (no
>> overcommit).
>> What is the output of the following command ?
>>     sysctl  -a | grep overcommit
>
> Command output:
>
> vm.nr_overcommit_hugepages = 0
> vm.overcommit_kbytes = 0
> vm.overcommit_memory = 0
> vm.overcommit_ratio = 50
>
> cat /proc/sys/vm/overcommit_memory
> 0

The overcommit settings look fine.

>
>>> Advice for some settings:
>>> We have absolute max peak of  2500 users which user squid (of 2800),
>>> what are recomended settings for:
>>> negotiate_kerberos_children start/idle
>>> squidguard helpers.
>>
>>
>> I have little experience with kerberos, but most likely this is not the
>> issue.
>> When Squid cannot fork the helpers, helper settings do not matter much.
>
>> For 2500 users you probably need 32-64 squidguard helpers.
>
> Can you confirm: For 2500 users:
>
> url_rewrite children X (squidguard)  32-64 will be ok ? We have set
> much larger number.

Did I understand it correctly that earlier in this reply you said that there are two squidguard processes (300 MB each).
ufdbGuard is faster than squidGuard and has multithreaded helpers.  ufdbGuard needs less helpers than squidGuard.

If you have a much larger number than 64 url rewrite helpers than I suggest to switch to ufdbGuard as soon as possible since the memory usage is then at least 600% less.

> For  helper:
> negotitate_kerberos_auth
>
> auth_param negotiate children X startup Y idle Z. What X, Y, Z are
> best for our user number ?
>
> We disabled kerberos replay cache because of disk performance (4 SAS
> DISK  15K, RAID 10) (iowait jumped high, and CPU load jumped to min
> 40 max 200).
> We don't use disk caching.
>
> Thanks for help,
>
>> Marcus
>>
>>
>>> Thanks for help,
>>>
>>> On Wed, Nov 8, 2017 at 10:53 AM, Marcus Kool
>>> <[hidden email]> wrote:
>>>>
>>>> There is definitely a problem with available memory because Squid cannot
>>>> fork.
>>>> So start with looking at how much memory Squid and its helpers use.
>>>> Do do have other processes on this system that consume a lot of memory ?
>>>>
>>>> Also note that ufdbGuard uses less memory that squidGuard.
>>>> If there are 30 helpers squidguard uses 300% more memory than ufdbGuard.
>>>>
>>>> Look at the wiki for more information about memory usage:
>>>> https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an
>>>> expired certificate but it is safe to go ahead)
>>>>
>>>> Marcus
>>>>
>>>>
>>>>
>>>> On 08/11/17 07:26, Bike dernikov1 wrote:
>>>>>
>>>>>
>>>>> Hi, I hope that someone can explain what happened, why squid stopped
>>>>> working.
>>>>> The problem is related to  memory/swap handling.
>>>>>
>>>>> After we changed vm.swappiness parameter from 60 to 10 (tuning
>>>>> attempt, to lower a disk usage, because we have only 4 disks in a
>>>>> RAID10, so disk subsystem  is a weak link), we got a lot of errors in
>>>>> cache.log.
>>>>> The problems started after scheduled logrotate after  2AM.
>>>>> Squid ran out of memory, auth helpers stopped working.
>>>>> It's weird because we didn't disable swap, but behavior is like we did.
>>>>> After an error, we increased parameter from 10 to 40.
>>>>>
>>>>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT
>>>>> cores).
>>>>> We have 2800 users, using  kerberos authentication, squidguard for
>>>>> filtering, ldap authorization.
>>>>> When problem appeared memory was still 3GB free (free column), ram
>>>>> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>>>>>
>>>>> Thanks for help,
>>>>>
>>>>>
>>>>> errors from cache.log.
>>>>>
>>>>> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
>>>>> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
>>>>> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
>>>>> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>> daemon:/var/log/squid/access.log
>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>> daemon:/var/log/squid/access.log
>>>>> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
>>>>> 'squidGuard' processes
>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard'
>>>>> process.
>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
>>>>> 'negotiate_kerberos_auth' processes
>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>
>>>>> external ACL 'memberof' queue overload. Using stale result.
>>>>> _______________________________________________
>>>>> squid-users mailing list
>>>>> [hidden email]
>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>>
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> [hidden email]
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
>>>
>>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Alex Rousskov
On Thu, Nov 9, 2017 at 4:03 PM, Alex Rousskov
<[hidden email]> wrote:

> On 11/09/2017 06:49 AM, Bike dernikov1 wrote:
>>>> On 11/08/2017 02:26 AM, Bike dernikov1 wrote:
>
>> So why didn't swap hop in?
>
> As I implied earlier, I do not recommend researching swap-related
> details because I do not recommend using (or relying on) swap.
>
>>> 2. Removing swap (or at least not counting on it and treating
>>>  any non-trivial swap usage as a system misconfiguration).
>
>> What to search ? Key words, vm.swappines got us in problem, as we
>> tried to tune  ?

> I do not understand the question. I am sure you know how to turn swap
> off (i.e., completely disable it).

Question referenced at text inside  ( ).
So you suggest that we totally disable disk  swap  (or just for
debuging) ? That setting on production can be disaster.
We will try, we reduced number of auth helper, prolong ldap caching.
Little steps but at last every bit counts.

>> We have problems with squid.conf reconfiguration, because users lose
>> connection (they got squid error screen for moment) so it is nightmare
>> if we changing configuration in work time.
>
> Yes, this is a common complaint and a known Squid problem.
>
> http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
>
> Alex.

For now we collect requests, and push changes after hour work, so no
problems but we have delay in request, good enough.
Thanks for help.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Marcus Kool
On Thu, Nov 9, 2017 at 5:13 PM, Marcus Kool <[hidden email]> wrote:

>
>
> On 09/11/17 11:04, Bike dernikov1 wrote:
> [snip]
>>>>
>>>> Memory compsumption:squid use largest part of memory  (12GB now,
>>>> second proces use 300MB memory), 14GB used by all process. So squid
>>>> use over 80% of total used memory.
>>>> So no there are not any problematic process. But we changed swappiness
>>>> settings.
>>>
>>>
>>> Did you monitor Squid for growth (it can start with 12 GB and grow
>>> slowly) ?
>>
>>
>> Yes we are monitoring continuosly.
>> Now:
>> Output from free -m.
>>
>>             total       used    free   shared  buff/cache  available
>> Mem:  24101     20507  256    146      3337         3034
>> Swap: 24561      5040   19521
>>
>> vm.swappiness=40
>>
>> Memory by process:
>> squid  Virt       RES   SHR  MEM%
>>             22,9G  18.7   8164   79,6
>
>
> Hmm. Squid grew from 12 GB to 18.7 GB (23 GB virtual).

Today problem appeared again after logrotate at 2.56AM.
Used memory was at peek 23,7GB.
Before logrorate started, cached was at 2GB, buffer at 1,5GB.
After logrorate started cache jumped to 3.7GB and buffer unchanged at 1,5GB.

Fork errors stopped after 1 minute. At 2:57.
cache memory dropped by 500MB  to 3.2GB and continued at same level
till morning, buffer  same at 1.5GB.

After 4 at 3:00 minutes new WARNING appeared. external ACL queue
overload. Using stale results.

We have night shift and they told us that Internet worked ok.

After restart at around 7.00AM used memory dropped from 22 GB to 7GB,
cache and buffer remain at same levels.

> With vm.swappiness=40 Linux starts to page out parts of processes when they
> occupy more than 60% of the memory.
> This is a potential bottleneck and I would have also decreased vm.swappiness
> to 10 as you did.
>
> My guess is that Squid starts too many helpers in a short time frame and
> that because of paging there are too many forks in progress simultaneously
> which causes the memory exhaustion.

We are now testing with 100 helpers for negotiate_kerberos_auth.
vm.swappiness returned to 60.

> I suggest to reduce the memory cache of Squid by 50% and set vm.swappiness
> to 20.

Squid cache memory is set at 14GB reduced from 16GB from 20GB  in two turns.

> And then observe:
> - total memory use
> - total swap usage (should be lower than the 5 GB that you have now)
> - number of helper processes that are started in short time frames
> And then in small steps increase the memory cache and maybe further reduce
> vm.swappiness to 10.

If we survive with actual setup, we will continue with reducing as you suggest.
Last extreme will be swap disable swappof but just for test with 6
eyes on monitoring :)

>> squidguard two process  300MB boths,.
>>
>> CPU 0.33 0.37 0.43
>>
>>> Squid cannot fork and higher swappiness increases the amount of memory
>>> that
>>> the OS can use to copy processes.
>>> It makes me think that you have the memory overcommit set to 2 (no
>>> overcommit).
>>> What is the output of the following command ?
>>>     sysctl  -a | grep overcommit
>>
>>
>> Command output:
>>
>> vm.nr_overcommit_hugepages = 0
>> vm.overcommit_kbytes = 0
>> vm.overcommit_memory = 0
>> vm.overcommit_ratio = 50
>>
>> cat /proc/sys/vm/overcommit_memory
>> 0
>
>
> The overcommit settings look fine.

At least something right :)

>>
>>>> Advice for some settings:
>>>> We have absolute max peak of  2500 users which user squid (of 2800),
>>>> what are recomended settings for:
>>>> negotiate_kerberos_children start/idle
>>>> squidguard helpers.
>>>
>>>
>>>
>>> I have little experience with kerberos, but most likely this is not the
>>> issue.
>>> When Squid cannot fork the helpers, helper settings do not matter much.
>>
>>
>>> For 2500 users you probably need 32-64 squidguard helpers.
>>
>>
>> Can you confirm: For 2500 users:
>>
>> url_rewrite children X (squidguard)  32-64 will be ok ? We have set
>> much larger number.

Squidguard url_rewrite children was set to 64.

> Did I understand it correctly that earlier in this reply you said that there
> are two squidguard processes (300 MB each).

Yes (first two process in htop, two rewrite childrens) others was on 0.0%.

> ufdbGuard is faster than squidGuard and has multithreaded helpers.
> ufdbGuard needs less helpers than squidGuard.
> If you have a much larger number than 64 url rewrite helpers than I suggest
> to switch to ufdbGuard as soon as possible since the memory usage is then at
> least 600% less.

UfdbGuard have few strong features. Development, kerberos,
concurency/multitreading.
As i wrote, if we read documentation slower we wouldn't
Do ufdbGuard supoort ldap secure auth ? We tried ldap secure with
squidguard without success.

>> For  helper:
>> negotitate_kerberos_auth
>>
>> auth_param negotiate children X startup Y idle Z. What X, Y, Z are
>> best for our user number ?
>>
>> We disabled kerberos replay cache because of disk performance (4 SAS
>> DISK  15K, RAID 10) (iowait jumped high, and CPU load jumped to min
>> 40 max 200).
>> We don't use disk caching.
>>
>> Thanks for help,
>>
>>> Marcus
>>>
>>>
>>>> Thanks for help,
>>>>
>>>> On Wed, Nov 8, 2017 at 10:53 AM, Marcus Kool
>>>> <[hidden email]> wrote:
>>>>>
>>>>>
>>>>> There is definitely a problem with available memory because Squid
>>>>> cannot
>>>>> fork.
>>>>> So start with looking at how much memory Squid and its helpers use.
>>>>> Do do have other processes on this system that consume a lot of memory
>>>>> ?
>>>>>
>>>>> Also note that ufdbGuard uses less memory that squidGuard.
>>>>> If there are 30 helpers squidguard uses 300% more memory than
>>>>> ufdbGuard.
>>>>>
>>>>> Look at the wiki for more information about memory usage:
>>>>> https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an
>>>>> expired certificate but it is safe to go ahead)
>>>>>
>>>>> Marcus
>>>>>
>>>>>
>>>>>
>>>>> On 08/11/17 07:26, Bike dernikov1 wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi, I hope that someone can explain what happened, why squid stopped
>>>>>> working.
>>>>>> The problem is related to  memory/swap handling.
>>>>>>
>>>>>> After we changed vm.swappiness parameter from 60 to 10 (tuning
>>>>>> attempt, to lower a disk usage, because we have only 4 disks in a
>>>>>> RAID10, so disk subsystem  is a weak link), we got a lot of errors in
>>>>>> cache.log.
>>>>>> The problems started after scheduled logrotate after  2AM.
>>>>>> Squid ran out of memory, auth helpers stopped working.
>>>>>> It's weird because we didn't disable swap, but behavior is like we
>>>>>> did.
>>>>>> After an error, we increased parameter from 10 to 40.
>>>>>>
>>>>>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT
>>>>>> cores).
>>>>>> We have 2800 users, using  kerberos authentication, squidguard for
>>>>>> filtering, ldap authorization.
>>>>>> When problem appeared memory was still 3GB free (free column), ram
>>>>>> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>>>>>>
>>>>>> Thanks for help,
>>>>>>
>>>>>>
>>>>>> errors from cache.log.
>>>>>>
>>>>>> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
>>>>>> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
>>>>>> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
>>>>>> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
>>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>>> daemon:/var/log/squid/access.log
>>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>>> daemon:/var/log/squid/access.log
>>>>>> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
>>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
>>>>>> 'squidGuard' processes
>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard'
>>>>>> process.
>>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
>>>>>> 'negotiate_kerberos_auth' processes
>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>>
>>>>>> external ACL 'memberof' queue overload. Using stale result.
>>>>>> _______________________________________________
>>>>>> squid-users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>>>
>>>>> _______________________________________________
>>>>> squid-users mailing list
>>>>> [hidden email]
>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Alex Rousskov
In reply to this post by Bike dernikov1
On 11/10/2017 06:16 AM, Bike dernikov1 wrote:

> So you suggest that we totally disable disk swap (or just for
> debuging) ?

I would aim for totally disabling disk swap, but getting to that
configuration is easier if you keep swap enabled and consider any
non-trivial swap use as a problem that you need to fix. After all
non-trivial swap uses are eliminated, it does not really matter much
whether you keep swap enabled or not!

Please note that this suggestion is specific to performance-sensitive
Squid servers -- many other servers have very legitimate reasons to use
swap. YMMV.


> That setting on production can be disaster.

Squid swapping in production is an arguably worse disaster, as you have
learned. In many cases, it is better to deal with a lack of swap than to
rely on swap's magical effects that most humans poorly understand. YMMV.


> We will try, we reduced number of auth helper, prolong ldap caching.
> Little steps but at last every bit counts.

... unless those bits are actually hurting. You may need to understand
your server memory usage better to reduce the time spent on trying
random changes. For example, if auth helpers are close to being
overloaded, reducing their number may make things worse.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Marcus Kool
In reply to this post by Bike dernikov1


On 10/11/17 12:11, Bike dernikov1 wrote:

> On Thu, Nov 9, 2017 at 5:13 PM, Marcus Kool <[hidden email]> wrote:
>>
>>
>> On 09/11/17 11:04, Bike dernikov1 wrote:
>> [snip]
>>>>>
>>>>> Memory compsumption:squid use largest part of memory  (12GB now,
>>>>> second proces use 300MB memory), 14GB used by all process. So squid
>>>>> use over 80% of total used memory.
>>>>> So no there are not any problematic process. But we changed swappiness
>>>>> settings.
>>>>
>>>>
>>>> Did you monitor Squid for growth (it can start with 12 GB and grow
>>>> slowly) ?
>>>
>>>
>>> Yes we are monitoring continuosly.
>>> Now:
>>> Output from free -m.
>>>
>>>              total       used    free   shared  buff/cache  available
>>> Mem:  24101     20507  256    146      3337         3034
>>> Swap: 24561      5040   19521
>>>
>>> vm.swappiness=40
>>>
>>> Memory by process:
>>> squid  Virt       RES   SHR  MEM%
>>>              22,9G  18.7   8164   79,6
>>
>>
>> Hmm. Squid grew from 12 GB to 18.7 GB (23 GB virtual).
>
> Today problem appeared again after logrotate at 2.56AM.
> Used memory was at peek 23,7GB.

ok. it is clear that Squid grows too much.
On a 24GB system with many helpers and a URL filter I think the maximum size should be 14GB.

> Before logrorate started, cached was at 2GB, buffer at 1,5GB.
> After logrorate started cache jumped to 3.7GB and buffer unchanged at 1,5GB.
>
> Fork errors stopped after 1 minute. At 2:57.
> cache memory dropped by 500MB  to 3.2GB and continued at same level
> till morning, buffer  same at 1.5GB.
>
> After 4 at 3:00 minutes new WARNING appeared. external ACL queue
> overload. Using stale results.
>
> We have night shift and they told us that Internet worked ok.
>
> After restart at around 7.00AM used memory dropped from 22 GB to 7GB,
> cache and buffer remain at same levels.

How come Squid uses 7 GB at startup when there is no disk cache ?

>> With vm.swappiness=40 Linux starts to page out parts of processes when they
>> occupy more than 60% of the memory.
>> This is a potential bottleneck and I would have also decreased vm.swappiness
>> to 10 as you did.
>>
>> My guess is that Squid starts too many helpers in a short time frame and
>> that because of paging there are too many forks in progress simultaneously
>> which causes the memory exhaustion.
>
> We are now testing with 100 helpers for negotiate_kerberos_auth.
> vm.swappiness returned to 60.
>
>> I suggest to reduce the memory cache of Squid by 50% and set vm.swappiness
>> to 20.
>
> Squid cache memory is set at 14GB reduced from 16GB from 20GB  in two turns.

are you saying that you have
    cache_mem 14G
If yes, you should read the memory FAQ and reduce this.
'cache_mem 14G' explains that Squid starts 'small' and grows over time.

>> And then observe:
>> - total memory use
>> - total swap usage (should be lower than the 5 GB that you have now)
>> - number of helper processes that are started in short time frames
>> And then in small steps increase the memory cache and maybe further reduce
>> vm.swappiness to 10.
>
> If we survive with actual setup, we will continue with reducing as you suggest.
> Last extreme will be swap disable swappof but just for test with 6
> eyes on monitoring :)
>
>>> squidguard two process  300MB boths,.
>>>
>>> CPU 0.33 0.37 0.43
>>>
>>>> Squid cannot fork and higher swappiness increases the amount of memory
>>>> that
>>>> the OS can use to copy processes.
>>>> It makes me think that you have the memory overcommit set to 2 (no
>>>> overcommit).
>>>> What is the output of the following command ?
>>>>      sysctl  -a | grep overcommit
>>>
>>>
>>> Command output:
>>>
>>> vm.nr_overcommit_hugepages = 0
>>> vm.overcommit_kbytes = 0
>>> vm.overcommit_memory = 0
>>> vm.overcommit_ratio = 50
>>>
>>> cat /proc/sys/vm/overcommit_memory
>>> 0
>>
>>
>> The overcommit settings look fine.
>
> At least something right :)
>
>>>
>>>>> Advice for some settings:
>>>>> We have absolute max peak of  2500 users which user squid (of 2800),
>>>>> what are recomended settings for:
>>>>> negotiate_kerberos_children start/idle
>>>>> squidguard helpers.
>>>>
>>>>
>>>>
>>>> I have little experience with kerberos, but most likely this is not the
>>>> issue.
>>>> When Squid cannot fork the helpers, helper settings do not matter much.
>>>
>>>
>>>> For 2500 users you probably need 32-64 squidguard helpers.
>>>
>>>
>>> Can you confirm: For 2500 users:
>>>
>>> url_rewrite children X (squidguard)  32-64 will be ok ? We have set
>>> much larger number.
>
> Squidguard url_rewrite children was set to 64.
>
>> Did I understand it correctly that earlier in this reply you said that there
>> are two squidguard processes (300 MB each).
>
> Yes (first two process in htop, two rewrite childrens) others was on 0.0%.
>
>> ufdbGuard is faster than squidGuard and has multithreaded helpers.
>> ufdbGuard needs less helpers than squidGuard.
>> If you have a much larger number than 64 url rewrite helpers than I suggest
>> to switch to ufdbGuard as soon as possible since the memory usage is then at
>> least 600% less.
>
> UfdbGuard have few strong features. Development, kerberos,
> concurency/multitreading.
> As i wrote, if we read documentation slower we wouldn't
> Do ufdbGuard supoort ldap secure auth ? We tried ldap secure with
> squidguard without success.

ufdbGuard supports any user database with the "execuserlist" feature.
See the Reference Manual for details.

>>> For  helper:
>>> negotitate_kerberos_auth
>>>
>>> auth_param negotiate children X startup Y idle Z. What X, Y, Z are
>>> best for our user number ?
>>>
>>> We disabled kerberos replay cache because of disk performance (4 SAS
>>> DISK  15K, RAID 10) (iowait jumped high, and CPU load jumped to min
>>> 40 max 200).
>>> We don't use disk caching.
>>>
>>> Thanks for help,
>>>
>>>> Marcus
>>>>
>>>>
>>>>> Thanks for help,
>>>>>
>>>>> On Wed, Nov 8, 2017 at 10:53 AM, Marcus Kool
>>>>> <[hidden email]> wrote:
>>>>>>
>>>>>>
>>>>>> There is definitely a problem with available memory because Squid
>>>>>> cannot
>>>>>> fork.
>>>>>> So start with looking at how much memory Squid and its helpers use.
>>>>>> Do do have other processes on this system that consume a lot of memory
>>>>>> ?
>>>>>>
>>>>>> Also note that ufdbGuard uses less memory that squidGuard.
>>>>>> If there are 30 helpers squidguard uses 300% more memory than
>>>>>> ufdbGuard.
>>>>>>
>>>>>> Look at the wiki for more information about memory usage:
>>>>>> https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an
>>>>>> expired certificate but it is safe to go ahead)
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/11/17 07:26, Bike dernikov1 wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi, I hope that someone can explain what happened, why squid stopped
>>>>>>> working.
>>>>>>> The problem is related to  memory/swap handling.
>>>>>>>
>>>>>>> After we changed vm.swappiness parameter from 60 to 10 (tuning
>>>>>>> attempt, to lower a disk usage, because we have only 4 disks in a
>>>>>>> RAID10, so disk subsystem  is a weak link), we got a lot of errors in
>>>>>>> cache.log.
>>>>>>> The problems started after scheduled logrotate after  2AM.
>>>>>>> Squid ran out of memory, auth helpers stopped working.
>>>>>>> It's weird because we didn't disable swap, but behavior is like we
>>>>>>> did.
>>>>>>> After an error, we increased parameter from 10 to 40.
>>>>>>>
>>>>>>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT
>>>>>>> cores).
>>>>>>> We have 2800 users, using  kerberos authentication, squidguard for
>>>>>>> filtering, ldap authorization.
>>>>>>> When problem appeared memory was still 3GB free (free column), ram
>>>>>>> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>>>>>>>
>>>>>>> Thanks for help,
>>>>>>>
>>>>>>>
>>>>>>> errors from cache.log.
>>>>>>>
>>>>>>> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
>>>>>>> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
>>>>>>> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
>>>>>>> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
>>>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>>>> daemon:/var/log/squid/access.log
>>>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>>>> daemon:/var/log/squid/access.log
>>>>>>> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
>>>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
>>>>>>> 'squidGuard' processes
>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard'
>>>>>>> process.
>>>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
>>>>>>> 'negotiate_kerberos_auth' processes
>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>>>
>>>>>>> external ACL 'memberof' queue overload. Using stale result.
>>>>>>> _______________________________________________
>>>>>>> squid-users mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>>>>
>>>>>> _______________________________________________
>>>>>> squid-users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> [hidden email]
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
>>>
>>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Alex Rousskov
On Fri, Nov 10, 2017 at 4:43 PM, Alex Rousskov
<[hidden email]> wrote:

> On 11/10/2017 06:16 AM, Bike dernikov1 wrote:
>
>> So you suggest that we totally disable disk swap (or just for
>> debuging) ?
>
> I would aim for totally disabling disk swap, but getting to that
> configuration is easier if you keep swap enabled and consider any
> non-trivial swap use as a problem that you need to fix. After all
> non-trivial swap uses are eliminated, it does not really matter much
> whether you keep swap enabled or not!

That is also my goal. I have other scenarios in mind.
We have free older servers  IBM X3550 (8 cores), which can be upgrade
to 48GB ram, (from other old servers)
On new ones,we have only 24GB on primary, and 16GB ram on  second server.
For disabling swap, X3550  servers would be better, but they have only 8 cores.
That could be problem, but for now we have only 0.5 load on 12/HT 24  cores.
I know that core per core new cpu-s can process  4x-8x more, so more testing :)

> Please note that this suggestion is specific to performance-sensitive
> Squid servers -- many other servers have very legitimate reasons to use
> swap. YMMV.
>
Oracle can be good example.
Yes, we have one Oracle server, swap killed him, users was wild, load
average over 10 continuosly (8 core), 16GB mem.
We upgraded ram to 48GB. Now, max load is at 4-5 in peeks, 1-2  during
day. Swap enabled :) it use 9GB, but use all ram :).
Users don't call after upgrade.

>
>> That setting on production can be disaster.
>
> Squid swapping in production is an arguably worse disaster, as you have
> learned. In many cases, it is better to deal with a lack of swap than to
> rely on swap's magical effects that most humans poorly understand. YMMV.

In this scenario, swap is backup cache (as I understand) ? As TIER  in  SAN-s.
Swap could be used  to translate back data to mem if used, but it
stays on disk and purge after some time if not used ?
Or I am in delusion ?

>> We will try, we reduced number of auth helper, prolong ldap caching.
>> Little steps but at last every bit counts.

We reduced kerberos helpers to 100 on Friday, 20 of those 100
currently active, 13 of those have requests/replies different than
zero.
Now reduced to 50. No problem appear over weekend.
cache_mem 14G
swappines = 60
ldap cache set to 24h

Mem stats:

             total        used        free      shared  buff/cache   available
Mem:          24101       16032        1488         156        6580        7497
Swap:         24561        1907       22654

> ... unless those bits are actually hurting. You may need to understand
> your server memory usage better to reduce the time spent on trying
> random changes. For example, if auth helpers are close to being
> overloaded, reducing their number may make things worse.

No overload after changes. When we saw that we don't use so much
helpers it was logicaly to reduce that  number in little steps.
I agree, random testing can be painfully, on production even more.
Can you recommend best way to analyze memory, (except free -m, top/htop).
Squid-internal-mgr/mem have nice detail, i will start there with. Do
you have better way ?

> Alex.

Thanks for help,
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Marcus Kool
On Fri, Nov 10, 2017 at 11:11 PM, Marcus Kool
<[hidden email]> wrote:

>
>
> On 10/11/17 12:11, Bike dernikov1 wrote:
>>
>> On Thu, Nov 9, 2017 at 5:13 PM, Marcus Kool <[hidden email]>
>> wrote:
>>>
>>>
>>>
>>> On 09/11/17 11:04, Bike dernikov1 wrote:
>>> [snip]
>>>>>>
>>>>>>
>>>>>> Memory compsumption:squid use largest part of memory  (12GB now,
>>>>>> second proces use 300MB memory), 14GB used by all process. So squid
>>>>>> use over 80% of total used memory.
>>>>>> So no there are not any problematic process. But we changed swappiness
>>>>>> settings.
>>>>>
>>>>>
>>>>>
>>>>> Did you monitor Squid for growth (it can start with 12 GB and grow
>>>>> slowly) ?
>>>>
>>>>
>>>>
>>>> Yes we are monitoring continuosly.
>>>> Now:
>>>> Output from free -m.
>>>>
>>>>              total       used    free   shared  buff/cache  available
>>>> Mem:  24101     20507  256    146      3337         3034
>>>> Swap: 24561      5040   19521
>>>>
>>>> vm.swappiness=40
>>>>
>>>> Memory by process:
>>>> squid  Virt       RES   SHR  MEM%
>>>>              22,9G  18.7   8164   79,6
>>>
>>>
>>>
>>> Hmm. Squid grew from 12 GB to 18.7 GB (23 GB virtual).
>>
>>
>> Today problem appeared again after logrotate at 2.56AM.
>> Used memory was at peek 23,7GB.
>
>
> ok. it is clear that Squid grows too much.
> On a 24GB system with many helpers and a URL filter I think the maximum size
> should be 14GB.

We set cache_mem to 14GB. For now no problems appeared.
We failed with helpers totally. We had problems with keytab cache, so
we thought that increase number will help.
At first we setup 700 kerberos helpers (insane from now  50 / 13 Active)
Until we disabled cache we couldn't stabilize server.
Disk was too slow, IO wait exploded, CPU load was at one time over
200. Users weren't happy.

>> Before logrorate started, cached was at 2GB, buffer at 1,5GB.
>> After logrorate started cache jumped to 3.7GB and buffer unchanged at
>> 1,5GB.
>>
>> Fork errors stopped after 1 minute. At 2:57.
>> cache memory dropped by 500MB  to 3.2GB and continued at same level
>> till morning, buffer  same at 1.5GB.
>>
>> After 4 at 3:00 minutes new WARNING appeared. external ACL queue
>> overload. Using stale results.
>>
>> We have night shift and they told us that Internet worked ok.
>>
>> After restart at around 7.00AM used memory dropped from 22 GB to 7GB,
>> cache and buffer remain at same levels.
>
>
> How come Squid uses 7 GB at startup when there is no disk cache ?
>
>>> With vm.swappiness=40 Linux starts to page out parts of processes when
>>> they
>>> occupy more than 60% of the memory.
>>> This is a potential bottleneck and I would have also decreased
>>> vm.swappiness
>>> to 10 as you did.
>>>
>>> My guess is that Squid starts too many helpers in a short time frame and
>>> that because of paging there are too many forks in progress
>>> simultaneously
>>> which causes the memory exhaustion.
>>
>>
>> We are now testing with 100 helpers for negotiate_kerberos_auth.
>> vm.swappiness returned to 60.
>>
>>> I suggest to reduce the memory cache of Squid by 50% and set
>>> vm.swappiness
>>> to 20.
>>
>>
>> Squid cache memory is set at 14GB reduced from 16GB from 20GB  in two
>> turns.
>
>
> are you saying that you have
>    cache_mem 14G
> If yes, you should read the memory FAQ and reduce this.
> 'cache_mem 14G' explains that Squid starts 'small' and grows over time.

For our case, what do you recomend.  10GB or even lower ?
Plan reading today, i hope that I will have peace, to concentrate.

>>> And then observe:
>>> - total memory use
>>> - total swap usage (should be lower than the 5 GB that you have now)
>>> - number of helper processes that are started in short time frames
>>> And then in small steps increase the memory cache and maybe further
>>> reduce
>>> vm.swappiness to 10.
>>
>>
>> If we survive with actual setup, we will continue with reducing as you
>> suggest.
>> Last extreme will be swap disable swappof but just for test with 6
>> eyes on monitoring :)
>>
>>>> squidguard two process  300MB boths,.
>>>>
>>>> CPU 0.33 0.37 0.43
>>>>
>>>>> Squid cannot fork and higher swappiness increases the amount of memory
>>>>> that
>>>>> the OS can use to copy processes.
>>>>> It makes me think that you have the memory overcommit set to 2 (no
>>>>> overcommit).
>>>>> What is the output of the following command ?
>>>>>      sysctl  -a | grep overcommit
>>>>
>>>>
>>>>
>>>> Command output:
>>>>
>>>> vm.nr_overcommit_hugepages = 0
>>>> vm.overcommit_kbytes = 0
>>>> vm.overcommit_memory = 0
>>>> vm.overcommit_ratio = 50
>>>>
>>>> cat /proc/sys/vm/overcommit_memory
>>>> 0
>>>
>>>
>>>
>>> The overcommit settings look fine.
>>
>>
>> At least something right :)
>>
>>>>
>>>>>> Advice for some settings:
>>>>>> We have absolute max peak of  2500 users which user squid (of 2800),
>>>>>> what are recomended settings for:
>>>>>> negotiate_kerberos_children start/idle
>>>>>> squidguard helpers.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I have little experience with kerberos, but most likely this is not the
>>>>> issue.
>>>>> When Squid cannot fork the helpers, helper settings do not matter much.
>>>>
>>>>
>>>>
>>>>> For 2500 users you probably need 32-64 squidguard helpers.
>>>>
>>>>
>>>>
>>>> Can you confirm: For 2500 users:
>>>>
>>>> url_rewrite children X (squidguard)  32-64 will be ok ? We have set
>>>> much larger number.
>>
>>
>> Squidguard url_rewrite children was set to 64.
>>
>>> Did I understand it correctly that earlier in this reply you said that
>>> there
>>> are two squidguard processes (300 MB each).
>>
>>
>> Yes (first two process in htop, two rewrite childrens) others was on 0.0%.
>>
>>> ufdbGuard is faster than squidGuard and has multithreaded helpers.
>>> ufdbGuard needs less helpers than squidGuard.
>>> If you have a much larger number than 64 url rewrite helpers than I
>>> suggest
>>> to switch to ufdbGuard as soon as possible since the memory usage is then
>>> at
>>> least 600% less.
>>
>>
>> UfdbGuard have few strong features. Development, kerberos,
>> concurency/multitreading.
>> As i wrote, if we read documentation slower we wouldn't
>> Do ufdbGuard supoort ldap secure auth ? We tried ldap secure with
>> squidguard without success.
>
>
> ufdbGuard supports any user database with the "execuserlist" feature.
> See the Reference Manual for details.

As I can tell we will have much work in front of us.
But no price to high to get rid of TMG  :)
Thanks for help.

>>>> For  helper:
>>>> negotitate_kerberos_auth
>>>>
>>>> auth_param negotiate children X startup Y idle Z. What X, Y, Z are
>>>> best for our user number ?
>>>>
>>>> We disabled kerberos replay cache because of disk performance (4 SAS
>>>> DISK  15K, RAID 10) (iowait jumped high, and CPU load jumped to min
>>>> 40 max 200).
>>>> We don't use disk caching.
>>>>
>>>> Thanks for help,
>>>>
>>>>> Marcus
>>>>>
>>>>>
>>>>>> Thanks for help,
>>>>>>
>>>>>> On Wed, Nov 8, 2017 at 10:53 AM, Marcus Kool
>>>>>> <[hidden email]> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> There is definitely a problem with available memory because Squid
>>>>>>> cannot
>>>>>>> fork.
>>>>>>> So start with looking at how much memory Squid and its helpers use.
>>>>>>> Do do have other processes on this system that consume a lot of
>>>>>>> memory
>>>>>>> ?
>>>>>>>
>>>>>>> Also note that ufdbGuard uses less memory that squidGuard.
>>>>>>> If there are 30 helpers squidguard uses 300% more memory than
>>>>>>> ufdbGuard.
>>>>>>>
>>>>>>> Look at the wiki for more information about memory usage:
>>>>>>> https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an
>>>>>>> expired certificate but it is safe to go ahead)
>>>>>>>
>>>>>>> Marcus
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/11/17 07:26, Bike dernikov1 wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi, I hope that someone can explain what happened, why squid stopped
>>>>>>>> working.
>>>>>>>> The problem is related to  memory/swap handling.
>>>>>>>>
>>>>>>>> After we changed vm.swappiness parameter from 60 to 10 (tuning
>>>>>>>> attempt, to lower a disk usage, because we have only 4 disks in a
>>>>>>>> RAID10, so disk subsystem  is a weak link), we got a lot of errors
>>>>>>>> in
>>>>>>>> cache.log.
>>>>>>>> The problems started after scheduled logrotate after  2AM.
>>>>>>>> Squid ran out of memory, auth helpers stopped working.
>>>>>>>> It's weird because we didn't disable swap, but behavior is like we
>>>>>>>> did.
>>>>>>>> After an error, we increased parameter from 10 to 40.
>>>>>>>>
>>>>>>>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU
>>>>>>>> (24HT
>>>>>>>> cores).
>>>>>>>> We have 2800 users, using  kerberos authentication, squidguard for
>>>>>>>> filtering, ldap authorization.
>>>>>>>> When problem appeared memory was still 3GB free (free column), ram
>>>>>>>> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>>>>>>>>
>>>>>>>> Thanks for help,
>>>>>>>>
>>>>>>>>
>>>>>>>> errors from cache.log.
>>>>>>>>
>>>>>>>> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
>>>>>>>> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
>>>>>>>> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
>>>>>>>> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
>>>>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>>>>> daemon:/var/log/squid/access.log
>>>>>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>>>>>> daemon:/var/log/squid/access.log
>>>>>>>> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
>>>>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
>>>>>>>> 'squidGuard' processes
>>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate
>>>>>>>> memory
>>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard'
>>>>>>>> process.
>>>>>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
>>>>>>>> 'negotiate_kerberos_auth' processes
>>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate
>>>>>>>> memory
>>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate
>>>>>>>> memory
>>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate
>>>>>>>> memory
>>>>>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>>>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>>>>>
>>>>>>>> external ACL 'memberof' queue overload. Using stale result.
>>>>>>>> _______________________________________________
>>>>>>>> squid-users mailing list
>>>>>>>> [hidden email]
>>>>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> squid-users mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> squid-users mailing list
>>>>> [hidden email]
>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Marcus Kool


On 13/11/17 07:46, Bike dernikov1 wrote:

>> are you saying that you have
>>     cache_mem 14G
>> If yes, you should read the memory FAQ and reduce this.
>> 'cache_mem 14G' explains that Squid starts 'small' and grows over time.
>
> For our case, what do you recomend.  10GB or even lower ?
> Plan reading today, i hope that I will have peace, to concentrate.

cache_mem does NOT define the total memory use of Squid.
The FAQ explains it.
On a 24G system you can start with 7 GB and only after 3 days of running without issues and verifying that the cache is 100% utilised (if not, Squid can grow) and there is sufficient free memory, you
can increase it.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
On Mon, Nov 13, 2017 at 12:15 PM, Marcus Kool
<[hidden email]> wrote:

>
>
> On 13/11/17 07:46, Bike dernikov1 wrote:
>
>>> are you saying that you have
>>>     cache_mem 14G
>>> If yes, you should read the memory FAQ and reduce this.
>>> 'cache_mem 14G' explains that Squid starts 'small' and grows over time.
>>
>>
>> For our case, what do you recomend.  10GB or even lower ?
>> Plan reading today, i hope that I will have peace, to concentrate.
>
>
> cache_mem does NOT define the total memory use of Squid.
> The FAQ explains it.
> On a 24G system you can start with 7 GB and only after 3 days of running
> without issues and verifying that the cache is 100% utilised (if not, Squid
> can grow) and there is sufficient free memory, you can increase it.

Read FAQ, i am now trying to pass trough squid-internal-mgr/ reports/statistics.
For now we will stay at cache_mem 14GB, because we are modifying too
many settings at same time.


Cache information for squid:
Hits as % of all requests: 5min: 5.8%, 60min: 6.4%
Hits as % of bytes sent: 5min: 16.8%, 60min: 18.7%
Memory hits as % of hit requests: 5min: 64.1%, 60min: 64.9%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
Storage Swap size: 0 KB
Storage Swap capacity: 0.0% used,  0.0% free

Mean Object Size: 0.00 KB
Requests given to unlinkd: 0




> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Marcus Kool
On Mon, Nov 13, 2017 at 12:15 PM, Marcus Kool
<[hidden email]> wrote:

>
>
> On 13/11/17 07:46, Bike dernikov1 wrote:
>
>>> are you saying that you have
>>>     cache_mem 14G
>>> If yes, you should read the memory FAQ and reduce this.
>>> 'cache_mem 14G' explains that Squid starts 'small' and grows over time.
>>
>>
>> For our case, what do you recomend.  10GB or even lower ?
>> Plan reading today, i hope that I will have peace, to concentrate.
>
>
> cache_mem does NOT define the total memory use of Squid.
> The FAQ explains it.
> On a 24G system you can start with 7 GB and only after 3 days of running
> without issues and verifying that the cache is 100% utilised (if not, Squid
> can grow) and there is sufficient free memory, you can increase it.
>

Read FAQ.
Now  trying to pass trough squid-internal-mgr/ reports/statistics.
For now we will stay at cache_mem 14GB, because we are modifying too
many settings at same time. Now at 99% used at 14GB if i read correctly.
Thanks for sugestions and help.


squid-internal-mgr/info output:
Cache information for squid:
Hits as % of all requests: 5min: 5.8%, 60min: 6.4%
Hits as % of bytes sent: 5min: 16.8%, 60min: 18.7%
Memory hits as % of hit requests: 5min: 64.1%, 60min: 64.9%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
Storage Swap size: 0 KB
Storage Swap capacity: 0.0% used,  0.0% free
Storage Mem size: 14195856 KB
Storage Mem capacity: 99.0% used,  1.0% free
Mean Object Size: 0.00 KB
Requests given to unlinkd: 0


> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Marcus Kool


On 13/11/17 10:46, Bike dernikov1 wrote:

> On Mon, Nov 13, 2017 at 12:15 PM, Marcus Kool
> <[hidden email]> wrote:
>>
>>
>> On 13/11/17 07:46, Bike dernikov1 wrote:
>>
>>>> are you saying that you have
>>>>      cache_mem 14G
>>>> If yes, you should read the memory FAQ and reduce this.
>>>> 'cache_mem 14G' explains that Squid starts 'small' and grows over time.
>>>
>>>
>>> For our case, what do you recomend.  10GB or even lower ?
>>> Plan reading today, i hope that I will have peace, to concentrate.
>>
>>
>> cache_mem does NOT define the total memory use of Squid.
>> The FAQ explains it.
>> On a 24G system you can start with 7 GB and only after 3 days of running
>> without issues and verifying that the cache is 100% utilised (if not, Squid
>> can grow) and there is sufficient free memory, you can increase it.
>>
>
> Read FAQ.
> Now  trying to pass trough squid-internal-mgr/ reports/statistics.
> For now we will stay at cache_mem 14GB, because we are modifying too
> many settings at same time. Now at 99% used at 14GB if i read correctly.
> Thanks for sugestions and help.
>
>
> squid-internal-mgr/info output:
> Cache information for squid:
> Hits as % of all requests: 5min: 5.8%, 60min: 6.4%
> Hits as % of bytes sent: 5min: 16.8%, 60min: 18.7%
> Memory hits as % of hit requests: 5min: 64.1%, 60min: 64.9%
> Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
> Storage Swap size: 0 KB
> Storage Swap capacity: 0.0% used,  0.0% free
> Storage Mem size: 14195856 KB
> Storage Mem capacity: 99.0% used,  1.0% free
> Mean Object Size: 0.00 KB
> Requests given to unlinkd: 0

Beware that he storage mem size is not the same as total memory used.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Alex Rousskov
In reply to this post by Bike dernikov1
On 11/13/2017 02:34 AM, Bike dernikov1 wrote:
> On Fri, Nov 10, 2017 at 4:43 PM, Alex Rousskov wrote:
>> Squid swapping in production is an arguably worse disaster, as you have
>> learned. In many cases, it is better to deal with a lack of swap than to
>> rely on swap's magical effects that most humans poorly understand. YMMV.

> In this scenario, swap is backup cache (as I understand)?

In this scenario, swap is not a cache! In fact it is pretty much the
opposite:

* A cache is, by definition, an optional unreliable "fast" storage meant
to reduce the need to go to some "slow" storage.

* When in active use, swap is required reliable slow storage meant to
extend fast storage (RAM) capacity.

Do you see how almost every adjective in the first bullet is replaced
with an antonym in the second one?

Some services, including many databases, overallocate RAM to store
rarely used (computed and/or preloaded) data. When that data is swapped
out, the service often continues to operate normally because the data is
rarely accessed (and/or because swapping it in is still cheaper than
computing it from scratch).

With Squid, it is very difficult for the OS to correctly identify the
rarely used RAM areas to swap out. When the OS swaps out the wrong area,
Squid slows down (to access that area), which only increases the number
of concurrent transactions and, hence, the amount of RAM Squid needs to
operate, which triggers more wrong swap outs, creating a vicious cycle.


> Swap could be used  to translate back data to mem if used, but it
> stays on disk and purge after some time if not used ?

The purging bit is wrong. Think of swap as very very very slow RAM.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID memory error after vm.swappines changed from 60 to 10

Bike dernikov1
In reply to this post by Marcus Kool
On Mon, Nov 13, 2017 at 2:32 PM, Marcus Kool
<[hidden email]> wrote:

>
>
> On 13/11/17 10:46, Bike dernikov1 wrote:
>>
>> On Mon, Nov 13, 2017 at 12:15 PM, Marcus Kool
>> <[hidden email]> wrote:
>>>
>>>
>>>
>>> On 13/11/17 07:46, Bike dernikov1 wrote:
>>>
>>>>> are you saying that you have
>>>>>      cache_mem 14G
>>>>> If yes, you should read the memory FAQ and reduce this.
>>>>> 'cache_mem 14G' explains that Squid starts 'small' and grows over time.
>>>>
>>>>
>>>>
>>>> For our case, what do you recomend.  10GB or even lower ?
>>>> Plan reading today, i hope that I will have peace, to concentrate.
>>>
>>>
>>>
>>> cache_mem does NOT define the total memory use of Squid.
>>> The FAQ explains it.
>>> On a 24G system you can start with 7 GB and only after 3 days of running
>>> without issues and verifying that the cache is 100% utilised (if not,
>>> Squid
>>> can grow) and there is sufficient free memory, you can increase it.
>>>
>>
>> Read FAQ.
>> Now  trying to pass trough squid-internal-mgr/ reports/statistics.
>> For now we will stay at cache_mem 14GB, because we are modifying too
>> many settings at same time. Now at 99% used at 14GB if i read correctly.
>> Thanks for sugestions and help.
>>
>>
>> squid-internal-mgr/info output:
>> Cache information for squid:
>> Hits as % of all requests: 5min: 5.8%, 60min: 6.4%
>> Hits as % of bytes sent: 5min: 16.8%, 60min: 18.7%
>> Memory hits as % of hit requests: 5min: 64.1%, 60min: 64.9%
>> Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
>> Storage Swap size: 0 KB
>> Storage Swap capacity: 0.0% used,  0.0% free
>> Storage Mem size: 14195856 KB
>> Storage Mem capacity: 99.0% used,  1.0% free
>> Mean Object Size: 0.00 KB
>> Requests given to unlinkd: 0
>
>
> Beware that he storage mem size is not the same as total memory used.

It is not same but not too much difference. No problems so far today :)

              total        used        free      shared  buff/cache   available
Mem:          24101       17008         695         134        6397        6543
Swap:         24561         342       24219



> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
12