Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Cherukuri, Naresh

Hello,

 

I am new to squid. I am getting a problem every 19 hours squid takes all RAM memory, then started taking swap in  20 minutes my swap is full. Then server side (OOM) is activating and killing all squid child’s then finally killing squid parent. Can someone help me how to address this problem?

 

Why every 19 hours my memory is going to full?

 

How much Ram do I need for following squid version?

 

Squid Cache: Version 3.5.20

 

             total       used       free     shared    buffers     cached

Mem:         11845       2713       9132         14         71       1641

-/+ buffers/cache:       1000      10845

Swap:        25551        421      25130

 

Thanks,

Naresh


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Eliezer Croitoru
Hey Naresh,

The RAM you need would differ by the nature, hardware and couple other
things about the nature of the machine.
What you need is start from the bottom and move up.
List for yourself the machine specs and your goals.
It really helps to start from low ie the default 256MB ram cache and without
any cache_dire and then see how it all moves on from there.
For now you are getting to the 19 hours of up time and you need to overcome
this so just start with a dry run of no disk cache at all and stick to squid
defaults for the next 24-48 hours.
Also what OS are you running?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: squid-users [mailto:[hidden email]] On
Behalf Of Cherukuri, Naresh
Sent: Tuesday, August 8, 2017 16:28
To: [hidden email]
Subject: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill
process (squid)

Hello,

I am new to squid. I am getting a problem every 19 hours squid takes all RAM
memory, then started taking swap in  20 minutes my swap is full. Then server
side (OOM) is activating and killing all squid child’s then finally killing
squid parent. Can someone help me how to address this problem?

Why every 19 hours my memory is going to full?

How much Ram do I need for following squid version?

Squid Cache: Version 3.5.20

             total       used       free     shared    buffers     cached
Mem:         11845       2713       9132         14         71       1641
-/+ buffers/cache:       1000      10845
Swap:        25551        421      25130

Thanks,
Naresh

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Cherukuri, Naresh
Hello Eliezer,

We are using OS "Redhat 7" and squid version 3.5.20.

As of now we are not using any cache, we already commented out. You want me try using cache by uncommenting the following line.

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /cache/squid 10000 16 256

Thanks,
Naresh
-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Wednesday, August 9, 2017 8:08 PM
To: Cherukuri, Naresh; [hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Hey Naresh,

The RAM you need would differ by the nature, hardware and couple other things about the nature of the machine.
What you need is start from the bottom and move up.
List for yourself the machine specs and your goals.
It really helps to start from low ie the default 256MB ram cache and without any cache_dire and then see how it all moves on from there.
For now you are getting to the 19 hours of up time and you need to overcome this so just start with a dry run of no disk cache at all and stick to squid defaults for the next 24-48 hours.
Also what OS are you running?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: squid-users [mailto:[hidden email]] On Behalf Of Cherukuri, Naresh
Sent: Tuesday, August 8, 2017 16:28
To: [hidden email]
Subject: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Hello,

I am new to squid. I am getting a problem every 19 hours squid takes all RAM memory, then started taking swap in  20 minutes my swap is full. Then server side (OOM) is activating and killing all squid child's then finally killing squid parent. Can someone help me how to address this problem?

Why every 19 hours my memory is going to full?

How much Ram do I need for following squid version?

Squid Cache: Version 3.5.20

             total       used       free     shared    buffers     cached
Mem:         11845       2713       9132         14         71       1641 -/+ buffers/cache:       1000      10845
Swap:        25551        421      25130

Thanks,
Naresh

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Eliezer Croitoru
Hey,

Let start from 0.
How many CPU's and\or core's are on this machine?
Is it a VM?
How many users will be sitting behind this proxy?
Depends on the above we can decide on the right approach.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: Cherukuri, Naresh [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 16:03
To: Eliezer Croitoru <[hidden email]>;
[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hello Eliezer,

We are using OS "Redhat 7" and squid version 3.5.20.

As of now we are not using any cache, we already commented out. You want me
try using cache by uncommenting the following line.

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /cache/squid 10000 16 256

Thanks,
Naresh
-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Wednesday, August 9, 2017 8:08 PM
To: Cherukuri, Naresh; [hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hey Naresh,

The RAM you need would differ by the nature, hardware and couple other
things about the nature of the machine.
What you need is start from the bottom and move up.
List for yourself the machine specs and your goals.
It really helps to start from low ie the default 256MB ram cache and without
any cache_dire and then see how it all moves on from there.
For now you are getting to the 19 hours of up time and you need to overcome
this so just start with a dry run of no disk cache at all and stick to squid
defaults for the next 24-48 hours.
Also what OS are you running?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: squid-users [mailto:[hidden email]] On
Behalf Of Cherukuri, Naresh
Sent: Tuesday, August 8, 2017 16:28
To: [hidden email]
Subject: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill
process (squid)

Hello,

I am new to squid. I am getting a problem every 19 hours squid takes all RAM
memory, then started taking swap in  20 minutes my swap is full. Then server
side (OOM) is activating and killing all squid child's then finally killing
squid parent. Can someone help me how to address this problem?

Why every 19 hours my memory is going to full?

How much Ram do I need for following squid version?

Squid Cache: Version 3.5.20

             total       used       free     shared    buffers     cached
Mem:         11845       2713       9132         14         71       1641
-/+ buffers/cache:       1000      10845
Swap:        25551        421      25130

Thanks,
Naresh


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Cherukuri, Naresh

No this a physical box and we are using only for squid. We have 4 cpu’s and 16 cores. Please find below for reference

Accesslogs : redirected to /cache/squid

 

/dev/mapper/*****-root  351G  5.2G  328G   2% /

devtmpfs                          5.8G     0  5.8G   0% /dev

tmpfs                             5.8G  2.1M  5.8G   1% /dev/shm

tmpfs                             5.8G  385M  5.5G   7% /run

tmpfs                             5.8G     0  5.8G   0% /sys/fs/cgroup

/dev/sda1                         462M   62M  373M  15% /boot

/dev/mapper/*****-var   9.2G  616M  8.2G   7% /var

/dev/mapper/*****-home   20G   45M   19G   1% /home

tmpfs                             1.2G     0  1.2G   0% /run/user/537

tmpfs                             1.2G     0  1.2G   0% /run/user/536

 

[root@**** squid]# cat /proc/cpuinfo

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 44

model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz

stepping        : 2

microcode       : 0x14

cpu MHz         : 2133.000

cache size      : 8192 KB

physical id     : 0

siblings        : 4

core id         : 0

cpu cores       : 4

apicid          : 0

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 11

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid

bogomips        : 4267.00

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

 

processor       : 1

vendor_id       : GenuineIntel

cpu family      : 6

model           : 44

model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz

stepping        : 2

microcode       : 0x14

cpu MHz         : 2133.000

cache size      : 8192 KB

physical id     : 0

siblings        : 4

core id         : 1

cpu cores       : 4

apicid          : 2

initial apicid  : 2

fpu             : yes

fpu_exception   : yes

cpuid level     : 11

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid

bogomips        : 4267.00

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

 

processor       : 2

vendor_id       : GenuineIntel

cpu family      : 6

model           : 44

model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz

stepping        : 2

microcode       : 0x14

cpu MHz         : 2133.000

cache size      : 8192 KB

physical id     : 0

siblings        : 4

core id         : 9

cpu cores       : 4

apicid          : 18

initial apicid  : 18

fpu             : yes

fpu_exception   : yes

cpuid level     : 11

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid

bogomips        : 4267.00

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

 

processor       : 3

vendor_id       : GenuineIntel

cpu family      : 6

model           : 44

model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz

stepping        : 2

microcode       : 0x14

cpu MHz         : 2133.000

cache size      : 8192 KB

physical id     : 0

siblings        : 4

core id         : 10

cpu cores       : 4

apicid          : 20

initial apicid  : 20

fpu             : yes

fpu_exception   : yes

cpuid level     : 11

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid

bogomips        : 4267.00

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

 

-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 9:19 AM
To: Cherukuri, Naresh; [hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

 

Hey,

 

Let start from 0.

How many CPU's and\or core's are on this machine?

Is it a VM?

How many users will be sitting behind this proxy?

Depends on the above we can decide on the right approach.

 

Eliezer

 

----

Eliezer Croitoru

Linux System Administrator

Mobile: +972-5-28704261

Email: [hidden email]

 

 

 

-----Original Message-----

From: Cherukuri, Naresh [[hidden email]]

Sent: Thursday, August 10, 2017 16:03

To: Eliezer Croitoru <[hidden email]>; [hidden email]

Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:

Kill process (squid)

 

Hello Eliezer,

 

We are using OS "Redhat 7" and squid version 3.5.20.

 

As of now we are not using any cache, we already commented out. You want me try using cache by uncommenting the following line.

 

# Uncomment and adjust the following to add a disk cache directory.

#cache_dir ufs /cache/squid 10000 16 256

 

Thanks,

Naresh

-----Original Message-----

From: Eliezer Croitoru [[hidden email]]

Sent: Wednesday, August 9, 2017 8:08 PM

To: Cherukuri, Naresh; [hidden email]

Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:

Kill process (squid)

 

Hey Naresh,

 

The RAM you need would differ by the nature, hardware and couple other things about the nature of the machine.

What you need is start from the bottom and move up.

List for yourself the machine specs and your goals.

It really helps to start from low ie the default 256MB ram cache and without any cache_dire and then see how it all moves on from there.

For now you are getting to the 19 hours of up time and you need to overcome this so just start with a dry run of no disk cache at all and stick to squid defaults for the next 24-48 hours.

Also what OS are you running?

 

Eliezer

 

----

http://ngtech.co.il/lmgtfy/

Linux System Administrator

Mobile: +972-5-28704261

Email: [hidden email]

 

 

From: squid-users [[hidden email]] On Behalf Of Cherukuri, Naresh

Sent: Tuesday, August 8, 2017 16:28

To: [hidden email]

Subject: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

 

Hello,

 

I am new to squid. I am getting a problem every 19 hours squid takes all RAM memory, then started taking swap in  20 minutes my swap is full. Then server side (OOM) is activating and killing all squid child's then finally killing squid parent. Can someone help me how to address this problem?

 

Why every 19 hours my memory is going to full?

 

How much Ram do I need for following squid version?

 

Squid Cache: Version 3.5.20

 

             total       used       free     shared    buffers     cached

Mem:         11845       2713       9132         14         71       1641 -/+ buffers/cache:       1000      10845

Swap:        25551        421      25130

 

Thanks,

Naresh

 

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Eliezer Croitoru
For how many users this squid should act as a proxy?
Ie client, machines,networks?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: Cherukuri, Naresh [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 16:27
To: Eliezer Croitoru <[hidden email]>;
[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

No this a physical box and we are using only for squid. We have 4 cpu’s and
16 cores. Please find below for reference
Accesslogs : redirected to /cache/squid

/dev/mapper/*****-root  351G  5.2G  328G   2% /
devtmpfs                          5.8G     0  5.8G   0% /dev
tmpfs                             5.8G  2.1M  5.8G   1% /dev/shm
tmpfs                             5.8G  385M  5.5G   7% /run
tmpfs                             5.8G     0  5.8G   0% /sys/fs/cgroup
/dev/sda1                         462M   62M  373M  15% /boot
/dev/mapper/*****-var   9.2G  616M  8.2G   7% /var
/dev/mapper/*****-home   20G   45M   19G   1% /home
tmpfs                             1.2G     0  1.2G   0% /run/user/537
tmpfs                             1.2G     0  1.2G   0% /run/user/536

[root@**** squid]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz
stepping        : 2
microcode       : 0x14
cpu MHz         : 2133.000
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16
xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow
vnmi flexpriority ept vpid
bogomips        : 4267.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz
stepping        : 2
microcode       : 0x14
cpu MHz         : 2133.000
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16
xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow
vnmi flexpriority ept vpid
bogomips        : 4267.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz
stepping        : 2
microcode       : 0x14
cpu MHz         : 2133.000
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 9
cpu cores       : 4
apicid          : 18
initial apicid  : 18
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16
xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow
vnmi flexpriority ept vpid
bogomips        : 4267.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz
stepping        : 2
microcode       : 0x14
cpu MHz         : 2133.000
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 10
cpu cores       : 4
apicid          : 20
initial apicid  : 20
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16
xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow
vnmi flexpriority ept vpid
bogomips        : 4267.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 9:19 AM
To: Cherukuri, Naresh; mailto:[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hey,

Let start from 0.
How many CPU's and\or core's are on this machine?
Is it a VM?
How many users will be sitting behind this proxy?
Depends on the above we can decide on the right approach.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:[hidden email]



-----Original Message-----
From: Cherukuri, Naresh [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 16:03
To: Eliezer Croitoru <mailto:[hidden email]>;
mailto:[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hello Eliezer,

We are using OS "Redhat 7" and squid version 3.5.20.

As of now we are not using any cache, we already commented out. You want me
try using cache by uncommenting the following line.

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /cache/squid 10000 16 256

Thanks,
Naresh
-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Wednesday, August 9, 2017 8:08 PM
To: Cherukuri, Naresh; mailto:[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hey Naresh,

The RAM you need would differ by the nature, hardware and couple other
things about the nature of the machine.
What you need is start from the bottom and move up.
List for yourself the machine specs and your goals.
It really helps to start from low ie the default 256MB ram cache and without
any cache_dire and then see how it all moves on from there.
For now you are getting to the 19 hours of up time and you need to overcome
this so just start with a dry run of no disk cache at all and stick to squid
defaults for the next 24-48 hours.
Also what OS are you running?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:[hidden email]


From: squid-users [mailto:[hidden email]] On
Behalf Of Cherukuri, Naresh
Sent: Tuesday, August 8, 2017 16:28
To: mailto:[hidden email]
Subject: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill
process (squid)

Hello,

I am new to squid. I am getting a problem every 19 hours squid takes all RAM
memory, then started taking swap in  20 minutes my swap is full. Then server
side (OOM) is activating and killing all squid child's then finally killing
squid parent. Can someone help me how to address this problem?

Why every 19 hours my memory is going to full?

How much Ram do I need for following squid version?

Squid Cache: Version 3.5.20

             total       used       free     shared    buffers     cached
Mem:         11845       2713       9132         14         71       1641
-/+ buffers/cache:       1000      10845
Swap:        25551        421      25130

Thanks,
Naresh



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Cherukuri, Naresh
Eliezer,

I cannot say by client or network. But, for sure I can say we have around 7000 computers using squid as a proxy.

Thanks,
Naresh

-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 1:43 PM
To: Cherukuri, Naresh; [hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

For how many users this squid should act as a proxy?
Ie client, machines,networks?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: Cherukuri, Naresh [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 16:27
To: Eliezer Croitoru <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

No this a physical box and we are using only for squid. We have 4 cpu's and
16 cores. Please find below for reference Accesslogs : redirected to /cache/squid

/dev/mapper/*****-root  351G  5.2G  328G   2% / devtmpfs                          5.8G     0  5.8G   0% /dev tmpfs                             5.8G  2.1M  5.8G   1% /dev/shm tmpfs                             5.8G  385M  5.5G   7% /run tmpfs                             5.8G     0  5.8G   0% /sys/fs/cgroup
/dev/sda1                         462M   62M  373M  15% /boot /dev/mapper/*****-var   9.2G  616M  8.2G   7% /var /dev/mapper/*****-home   20G   45M   19G   1% /home tmpfs                             1.2G     0  1.2G   0% /run/user/537 tmpfs                             1.2G     0  1.2G   0% /run/user/536

[root@**** squid]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz stepping        : 2 microcode       : 0x14 cpu MHz         : 2133.000 cache size      : 8192 KB physical id     : 0 siblings        : 4 core id         : 0 cpu cores       : 4 apicid          : 0 initial apicid  : 0 fpu             : yes fpu_exception   : yes cpuid level     : 11 wp              : yes flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid bogomips        : 4267.00 clflush size    : 64 cache_alignment : 64 address sizes   : 40 bits physical, 48 bits virtual power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz stepping        : 2 microcode       : 0x14 cpu MHz         : 2133.000 cache size      : 8192 KB physical id     : 0 siblings        : 4 core id         : 1 cpu cores       : 4 apicid          : 2 initial apicid  : 2 fpu             : yes fpu_exception   : yes cpuid level     : 11 wp              : yes flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid bogomips        : 4267.00 clflush size    : 64 cache_alignment : 64 address sizes   : 40 bits physical, 48 bits virtual power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz stepping        : 2 microcode       : 0x14 cpu MHz         : 2133.000 cache size      : 8192 KB physical id     : 0 siblings        : 4 core id         : 9 cpu cores       : 4 apicid          : 18 initial apicid  : 18 fpu             : yes fpu_exception   : yes cpuid level     : 11 wp              : yes flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid bogomips        : 4267.00 clflush size    : 64 cache_alignment : 64 address sizes   : 40 bits physical, 48 bits virtual power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz stepping        : 2 microcode       : 0x14 cpu MHz         : 2133.000 cache size      : 8192 KB physical id     : 0 siblings        : 4 core id         : 10 cpu cores       : 4 apicid          : 20 initial apicid  : 20 fpu             : yes fpu_exception   : yes cpuid level     : 11 wp              : yes flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid bogomips        : 4267.00 clflush size    : 64 cache_alignment : 64 address sizes   : 40 bits physical, 48 bits virtual power management:

-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 9:19 AM
To: Cherukuri, Naresh; mailto:[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hey,

Let start from 0.
How many CPU's and\or core's are on this machine?
Is it a VM?
How many users will be sitting behind this proxy?
Depends on the above we can decide on the right approach.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:[hidden email]



-----Original Message-----
From: Cherukuri, Naresh [mailto:[hidden email]]
Sent: Thursday, August 10, 2017 16:03
To: Eliezer Croitoru <mailto:[hidden email]>; mailto:[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hello Eliezer,

We are using OS "Redhat 7" and squid version 3.5.20.

As of now we are not using any cache, we already commented out. You want me try using cache by uncommenting the following line.

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /cache/squid 10000 16 256

Thanks,
Naresh
-----Original Message-----
From: Eliezer Croitoru [mailto:[hidden email]]
Sent: Wednesday, August 9, 2017 8:08 PM
To: Cherukuri, Naresh; mailto:[hidden email]
Subject: RE: [squid-users] Crash: every 19 hours: kernel: Out of memory:
Kill process (squid)

Hey Naresh,

The RAM you need would differ by the nature, hardware and couple other things about the nature of the machine.
What you need is start from the bottom and move up.
List for yourself the machine specs and your goals.
It really helps to start from low ie the default 256MB ram cache and without any cache_dire and then see how it all moves on from there.
For now you are getting to the 19 hours of up time and you need to overcome this so just start with a dry run of no disk cache at all and stick to squid defaults for the next 24-48 hours.
Also what OS are you running?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:[hidden email]


From: squid-users [mailto:[hidden email]] On Behalf Of Cherukuri, Naresh
Sent: Tuesday, August 8, 2017 16:28
To: mailto:[hidden email]
Subject: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Hello,

I am new to squid. I am getting a problem every 19 hours squid takes all RAM memory, then started taking swap in  20 minutes my swap is full. Then server side (OOM) is activating and killing all squid child's then finally killing squid parent. Can someone help me how to address this problem?

Why every 19 hours my memory is going to full?

How much Ram do I need for following squid version?

Squid Cache: Version 3.5.20

             total       used       free     shared    buffers     cached
Mem:         11845       2713       9132         14         71       1641 -/+ buffers/cache:       1000      10845
Swap:        25551        421      25130

Thanks,
Naresh



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Amos Jeffries
Administrator
On 11/08/17 06:36, Cherukuri, Naresh wrote:
> Eliezer,
>
> I cannot say by client or network. But, for sure I can say we have around 7000 computers using squid as a proxy.
>

FYI: that number means peak load may be as high as 70K RPS (~100
req/client), a quad-core machine might be able to handle that but the
2.13 GHz CPU speed makes me doubtful. I'd plan for up to 4 machines of
this type to be built in the medium-long term.


>
> From: Cherukuri, Naresh
> Sent: Thursday, August 10, 2017 16:27
>
> No this a physical box and we are using only for squid. We have 4 cpu's and
> 16 cores. Please find below for reference Accesslogs : redirected to /cache/squid
>

NP: I don't see any access logs details in your post. Just CPU specs and
disk FS mappings.


> -----Original Message-----
> From: Cherukuri, Naresh
> Sent: Thursday, August 10, 2017 16:03
>
> Hello Eliezer,
>
> We are using OS "Redhat 7" and squid version 3.5.20.
>
> As of now we are not using any cache, we already commented out. You want me try using cache by uncommenting the following line.
>
> # Uncomment and adjust the following to add a disk cache directory.
> #cache_dir ufs /cache/squid 10000 16 256
>

No, suggestion was for the default memory-only cache. Which means no
cache_mem or cache_dir entries in your squid.conf (letting Squid use its
defaults). It should still be caching, just much less.


> From: squid-users On Behalf Of Cherukuri, Naresh
> Sent: Tuesday, August 8, 2017 16:28
>
> Hello,
>
> I am new to squid. I am getting a problem every 19 hours squid takes all RAM memory, then started taking swap in  20 minutes my swap is full. Then server side (OOM) is activating and killing all squid child's then finally killing squid parent. Can someone help me how to address this problem?
>

FYI: My brief understanding of the OOM is that when the sum total of all
processes on the machine start consuming too mush RAM it kills off the
largest user. So it may be that Squid is using some large (but
reasonable) amount of RAM and something else entirely pushes it over the
edge - OOM just killing Squid because it has the most memory at that time.

So, if you have any record of the machines memory usage by process over
time it would be good to know for certain whether it is Squid alone, or
squid + something else that is the problem. Something along the lines of
a log processor that kicks in once a day and uses lots of RAM briefly
may exist.


> Why every 19 hours my memory is going to full?
>
> How much Ram do I need for following squid version?
>
> Squid Cache: Version 3.5.20
>
>               total       used       free     shared    buffers     cached
> Mem:         11845       2713       9132         14         71       1641 -/+ buffers/cache:       1000      10845
> Swap:        25551        421      25130


The biggest problem I see here is that there are no units. These could
be KB or GB for all we know.


Please clarify that missing info.


FWIW: Squid with some minimal tuning runs fine on embeded devices with
32MB of RAM and has not had a memory leak in a long time. So it is
usually a matter of some feature misconfigured. That said there is an
issue with OpenSSL objects that can look like a memory leak in 3.x if
one is not careful.

To see if anything is misconfigured please post your squid.conf (without
the #commented out lines) so we can review it for problems.



Cheers
Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Matus UHLAR - fantomas
In reply to this post by Cherukuri, Naresh
On 10.08.17 18:36, Cherukuri, Naresh wrote:
>I cannot say by client or network. But, for sure I can say we have around
> 7000 computers using squid as a proxy.

>I am new to squid. I am getting a problem every 19 hours squid takes all
> RAM memory, then started taking swap in  20 minutes my swap is full.  Then
> server side (OOM) is activating and killing all squid child's then finally
> killing squid parent.  Can someone help me how to address this problem?

>Why every 19 hours my memory is going to full?

is the memory usage slowly growing up during those 19 hours, or does the memory
usage jump at some times?


--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I drive way too fast to worry about cholesterol.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Cherukuri, Naresh
In reply to this post by Amos Jeffries

Amos,

 

Please find below my squid conf and access logs and memory output in MB. Appreciate any help.

 

Memory Info:

[root@******prod ~]# free -m

             total       used       free     shared    buffers     cached

Mem:         11845       4194       7651         41        190       1418

-/+ buffers/cache:       2585       9260

Swap:        25551        408      25143

 

Squidconf:

[root@******prod squid]# more squid.conf

#

# Recommended minimum configuration:

#

max_filedesc 4096

acl manager proto cache_object

visible_hostname ******prod

logfile_rotate 10

 

access_log /cache/access.log

 

acl localnet src 172.16.0.0/16

acl backoffice_users src 10.136.0.0/13

acl h****_backoffice_users src 10.142.0.0/15

acl re****_users src 10.128.0.0/13

acl hcity_r*****_users src 10.134.0.0/15

acl par**** url_regex par****

 

acl SSL_ports port 443

acl Safe_ports port 80          # http

#acl Safe_ports port 21         # ftp

acl Safe_ports port 443         # https

#acl Safe_ports port 70         # gopher

#acl Safe_ports port 210                # wais

#acl Safe_ports port 1025-65535 # unregistered ports

#acl Safe_ports port 280                # http-mgmt

#acl Safe_ports port 488                # gss-http

#acl Safe_ports port 591                # filemaker

#acl Safe_ports port 777                # multiling http

acl CONNECT method CONNECT

acl backoffice_allowed_sites url_regex "/etc/squid/backoffice_allowed_sites"

acl h***_backoffice_allowed_sites url_regex "/etc/squid/backoffice_allowed_sites"

acl backoffice_blocked_sites url_regex "/etc/squid/backoffice_blocklist"

acl h***_backoffice_blocked_sites url_regex "/etc/squid/backoffice_blocklist"

acl re****_allowed_sites url_regex "/etc/squid/re****_allowed_sites"

acl h****_reg****_allowed_sites url_regex "/etc/squid/h***_reg*****_allowed_sites"

#

 

http_access allow localnet reg***_allowed_sites

http_access deny backoffice_users backoffice_blocked_sites

http_access deny h***_backoffice_users backoffice_blocked_sites

http_access allow backoffice_users backoffice_allowed_sites

http_access allow h***_backoffice_users backoffice_allowed_sites

http_access allow reg****_users reg****_allowed_sites

http_access allow h***_reg****_users h***_reg****_allowed_sites

no_cache deny par****

http_access deny all

 

#http_access allow manager localhost

#http_access deny manager

 

# Deny requests to certain unsafe ports

http_access deny !Safe_ports

 

# Deny CONNECT to other than secure SSL ports

#http_access deny CONNECT !SSL_ports

http_access  allow CONNECT SSL_ports

# We strongly recommend the following be uncommented to protect innocent

# web applications running on the proxy server who think the only

# one who can access services on "localhost" is a local user

http_access deny to_localhost

 

# Example rule allowing access from your local networks.

# Adapt localnet in the ACL section to list your (internal) IP networks

# from where browsing should be allowed

#http_access allow localnet

http_access allow localhost

 

# And finally deny all other access to this proxy

http_access deny all

 

# Squid normally listens to port 3128

http_port 3128 ssl-bump \

key=/etc/squid/pc****sslcerts/pc*****prod.pkey \

cert=/etc/squid/pc******sslcerts/pc*****prod.crt \

generate-host-certificates=on dynamic_cert_mem_cache_size=4MB

 

acl step1 at_step SslBump1

ssl_bump peek step1

#ssl_bump bump all

ssl_bump bump backoffice_users !localnet !h***_backoffice_users !reg****_users !h***_reg***_users !par***

 

#sslproxy_capath /etc/ssl/certs

sslproxy_cert_error allow all

always_direct allow all

sslproxy_flags DONT_VERIFY_PEER

 

sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB sslcrtd_children 8 startup=1 idle=1

 

# Uncomment and adjust the following to add a disk cache directory.

#cache_dir ufs /cache/squid 10000 16 256

 

# Leave coredumps in the first cache dir

#rdescoredump_dir /var/spool/squid

#coredump_dir /var/log/squid/squid

coredump_dir /cache/squid

 

# Add any of your own refresh_pattern entries above these.

refresh_pattern ^ftp:           1440    20%     10080

refresh_pattern ^gopher:        1440    0%      1440

refresh_pattern -i (/cgi-bin/|\?) 0     0%      0

refresh_pattern .               0       20%     4320

 

#url_rewrite_access allow all

#url_rewrite_program /usr/bin/squidGuard -c /etc/squid/squidguard.conf

 

Accesslogs:

1502424001.504      0 10.138.142.6 TCP_DENIED/403 4175 GET http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html

1502424001.533    329 10.140.230.6 TAG_NONE/200 0 CONNECT watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -

1502424001.543      0 10.141.80.6 TCP_DENIED/403 4167 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424001.546    331 10.140.230.6 TAG_NONE/200 0 CONNECT watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -

1502424001.551  29923 10.130.27.24 TCP_MISS_ABORTED/000 0 GET http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.571      0 10.141.108.6 TCP_DENIED/403 4269 GET http://update.scansoft.com/GetMessages.asp? - HIER_NONE/- text/html

1502424001.572      0 10.138.142.6 TCP_DENIED/403 4175 GET http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html

1502424001.579      0 10.140.167.6 TCP_DENIED/403 4168 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424001.590  27992 10.140.248.7 TCP_MISS_ABORTED/000 0 GET http://pc-sep.pcwhq.par*****.net:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.631      0 10.141.108.6 TCP_DENIED/403 4269 GET http://update.scansoft.com/GetMessages.asp? - HIER_NONE/- text/html

1502424001.643      0 10.138.142.6 TCP_DENIED/403 4175 GET http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html

1502424001.646      0 10.136.76.7 TCP_MISS_ABORTED/000 0 POST https://watson.telemetry.microsoft.com/Telemetry.Request - HIER_DIRECT/65.55.252.202 -

1502424001.654  29864 10.133.222.25 TCP_MISS_ABORTED/000 0 GET http://10.1.2.35:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.670      1 10.140.167.6 TCP_DENIED/403 4168 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424001.676      0 10.141.215.6 TCP_DENIED/403 3998 OPTIONS http://172.16.4.19/PrintQueue/completed/ - HIER_NONE/- text/html

1502424001.678  29927 10.132.157.21 TCP_MISS_ABORTED/000 0 GET http://10.1.2.35:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.688      0 10.141.108.6 TCP_DENIED/403 4269 GET http://update.scansoft.com/GetMessages.asp? - HIER_NONE/- text/html

1502424001.700    363 10.136.171.6 TAG_NONE/200 0 CONNECT watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -

1502424001.702    365 10.138.31.10 TAG_NONE/200 0 CONNECT ent-shasta-rrs.symantec.com:443 - HIER_DIRECT/104.40.50.196 -

1502424001.716      0 10.138.142.6 TCP_DENIED/403 4175 GET http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html

1502424001.756      0 10.141.108.6 TCP_DENIED/403 4269 GET http://update.scansoft.com/GetMessages.asp? - HIER_NONE/- text/html

1502424001.782      0 10.140.230.6 TCP_MISS_ABORTED/000 0 POST https://watson.telemetry.microsoft.com/Telemetry.Request - HIER_DIRECT/65.55.252.202 -

1502424001.782      0 10.140.230.6 TCP_MISS_ABORTED/000 0 POST https://watson.telemetry.microsoft.com/Telemetry.Request - HIER_DIRECT/65.55.252.202 -

1502424001.787  29983 10.132.141.21 TCP_MISS_ABORTED/000 0 GET http://pc-sep.pcwhq.par***.net:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.792      2 10.138.142.6 TCP_DENIED/403 4175 GET http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html

1502424001.792  29928 10.128.101.24 TCP_MISS_ABORTED/000 0 GET http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.815      0 10.141.108.6 TCP_DENIED/403 4269 GET http://update.scansoft.com/GetMessages.asp? - HIER_NONE/- text/html

1502424001.841      0 10.141.160.6 TCP_DENIED/403 4168 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424001.843      0 10.141.215.6 TCP_DENIED/403 3998 OPTIONS http://172.16.4.19/PrintQueue/completed/ - HIER_NONE/- text/html

1502424001.873      0 10.141.108.6 TCP_DENIED/403 4269 GET http://update.scansoft.com/GetMessages.asp? - HIER_NONE/- text/html

1502424001.892  29805 10.141.82.6 TCP_MISS_ABORTED/000 0 GET http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.907      0 10.141.160.6 TCP_DENIED/403 4168 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424001.912  29990 10.128.10.24 TCP_MISS_ABORTED/000 0 GET http://10.1.2.35:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.927      0 10.136.147.6 TCP_DENIED/403 4168 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424001.938     77 10.138.31.10 TCP_MISS/200 514 POST https://ent-shasta-rrs.symantec.com/mrclean? - HIER_DIRECT/104.40.50.196 -

1502424001.946  29949 10.130.45.23 TCP_MISS_ABORTED/000 0 GET http://pc-sep.pcwhq.par*****.net:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.974      0 10.141.160.6 TCP_DENIED/403 4168 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424001.976  28002 10.136.84.6 TCP_MISS_ABORTED/000 0 GET http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? - HIER_DIRECT/10.1.2.35 -

1502424001.997      0 10.136.171.6 TCP_MISS_ABORTED/000 0 POST https://watson.telemetry.microsoft.com/Telemetry.Request - HIER_DIRECT/65.55.252.202 -

1502424002.003      1 10.136.147.6 TCP_DENIED/403 4168 GET http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html

1502424002.013      0 10.138.33.6 TCP_DENIED/403 4268 GET http://update.scansoft.com/GetMessages.asp? - HIER_NONE/- text/html

 

Cachelog errors I am seeing daily:

 

Error negotiating SSL connection on FD 26: error:140A1175:SSL routines:SSL_BYTES_TO_CIPHER_LIST:inappropriate fallback (1/-1)

Error negotiating SSL connection on FD 1175: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown (1/0)

2017/08/02 09:01:02 kid1| Error negotiating SSL on FD 989: error:00000000:lib(0):func(0):reason(0) (5/-1/104) ##Very rare i found few not frequently

2017/08/02 09:01:43 kid1| Queue overload, rejecting # too many times

2017/08/02 09:01:45 kid1| Error negotiating SSL connection on FD 1749: (104) Connection reset by peer ## too many times

2017/08/02 10:12:58 kid1| WARNING: Closing client connection due to lifetime timeout ## only one

2017/08/07 22:37:56 kid1| comm_open: socket failure: (24) Too many open files

2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en

2017/08/07 22:39:37 kid1| '/usr/share/squid/errors/en-us/ERR_DNS_FAIL': (24) Too many open files

2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en-us

2017/08/07 22:01:42 kid1| WARNING: All 32/32 ssl_crtd processes are busy.

2017/08/07 22:01:42 kid1| WARNING: 32 pending requests queued

2017/08/07 22:01:42 kid1| WARNING: Consider increasing the number of ssl_crtd processes in your config file.

2017/08/11 00:58:56 kid1| WARNING: Closing client connection due to lifetime timeout

2017/08/09 12:55:45 kid1| WARNING! Your cache is running out of filedescriptors

 

Thanks,

Naresh

-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
Sent: Friday, August 11, 2017 12:51 AM
To: [hidden email]
Subject: Re: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

 

On 11/08/17 06:36, Cherukuri, Naresh wrote:

> Eliezer,

>

> I cannot say by client or network. But, for sure I can say we have around 7000 computers using squid as a proxy.

>

 

FYI: that number means peak load may be as high as 70K RPS (~100 req/client), a quad-core machine might be able to handle that but the

2.13 GHz CPU speed makes me doubtful. I'd plan for up to 4 machines of this type to be built in the medium-long term.

 

 

>

> From: Cherukuri, Naresh

> Sent: Thursday, August 10, 2017 16:27

>

> No this a physical box and we are using only for squid. We have 4

> cpu's and

> 16 cores. Please find below for reference Accesslogs : redirected to

> /cache/squid

>

 

NP: I don't see any access logs details in your post. Just CPU specs and disk FS mappings.

 

 

> -----Original Message-----

> From: Cherukuri, Naresh

> Sent: Thursday, August 10, 2017 16:03

>

> Hello Eliezer,

>

> We are using OS "Redhat 7" and squid version 3.5.20.

>

> As of now we are not using any cache, we already commented out. You want me try using cache by uncommenting the following line.

>

> # Uncomment and adjust the following to add a disk cache directory.

> #cache_dir ufs /cache/squid 10000 16 256

>

 

No, suggestion was for the default memory-only cache. Which means no

cache_mem or cache_dir entries in your squid.conf (letting Squid use its

defaults). It should still be caching, just much less.

 

 

> From: squid-users On Behalf Of Cherukuri, Naresh

> Sent: Tuesday, August 8, 2017 16:28

>

> Hello,

>

> I am new to squid. I am getting a problem every 19 hours squid takes all RAM memory, then started taking swap in  20 minutes my swap is full. Then server side (OOM) is activating and killing all squid child's then finally killing squid parent. Can someone help me how to address this problem?

>

 

FYI: My brief understanding of the OOM is that when the sum total of all

processes on the machine start consuming too mush RAM it kills off the

largest user. So it may be that Squid is using some large (but

reasonable) amount of RAM and something else entirely pushes it over the

edge - OOM just killing Squid because it has the most memory at that time.

 

So, if you have any record of the machines memory usage by process over

time it would be good to know for certain whether it is Squid alone, or

squid + something else that is the problem. Something along the lines of

a log processor that kicks in once a day and uses lots of RAM briefly

may exist.

 

 

> Why every 19 hours my memory is going to full?

>

> How much Ram do I need for following squid version?

>

> Squid Cache: Version 3.5.20

>

>               total       used       free     shared    buffers     cached

> Mem:         11845       2713       9132         14         71       1641 -/+ buffers/cache:       1000      10845

> Swap:        25551        421      25130

 

 

The biggest problem I see here is that there are no units. These could

be KB or GB for all we know.

 

 

Please clarify that missing info.

 

 

FWIW: Squid with some minimal tuning runs fine on embeded devices with

32MB of RAM and has not had a memory leak in a long time. So it is

usually a matter of some feature misconfigured. That said there is an

issue with OpenSSL objects that can look like a memory leak in 3.x if

one is not careful.

 

To see if anything is misconfigured please post your squid.conf (without

the #commented out lines) so we can review it for problems.

 

 

 

Cheers

Amos

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Amos Jeffries
Administrator
On 12/08/17 01:13, Cherukuri, Naresh wrote:

> Amos,
>
> Please find below my squid conf and access logs and memory output in MB.
> Appreciate any help.
>
> Memory Info:
>
> [root@******prod ~]# free -m
>
>               total       used       free     shared    buffers     cached
>
> Mem:         11845       4194       7651         41        190       1418
>
> -/+ buffers/cache:       2585       9260
>
> Swap:        25551        408      25143
>
> Squidconf:
>
> [root@******prod squid]# more squid.conf
>
> #
>
> # Recommended minimum configuration:
>
> #
>
> max_filedesc 4096

Ouch. Squid requires between 2 and 6 sockets (FD) per client connection
and clients tend to make upwards of 8 connections per domain being
contacted (dozens per web page loaded). While HTTP/1.1 improvements can
reduce that average a lot 4K FD cannot serve 7K clients well.

The above number should be at least 4x the expected client count to cope
with load, at least 8x would be better for peak times.


>
> acl manager proto cache_object
>
> visible_hostname ******prod
>
> logfile_rotate 10
>
> access_log /cache/access.log
>
> acl localnet src 172.16.0.0/16
>
> acl backoffice_users src 10.136.0.0/13
>
> acl h****_backoffice_users src 10.142.0.0/15
>
> acl re****_users src 10.128.0.0/13
>
> acl hcity_r*****_users src 10.134.0.0/15
>

So you are immediately confusing "users" with "clients". They are
different, and at the Squid layer of networking the difference starts to
matter.

For example; are you aware that automatic software and devices without
any user logged in can still perform network transactions through the
proxy? usually it is not a problem, but if you are thinking of only
*users* utilizing the proxy you could be in for a major surprise.


> acl par**** url_regex par****
>
> acl SSL_ports port 443
>
> acl Safe_ports port 80          # http
>
> #acl Safe_ports port 21         # ftp
>
> acl Safe_ports port 443         # https
>
> #acl Safe_ports port 70         # gopher
>
> #acl Safe_ports port 210                # wais
>
> #acl Safe_ports port 1025-65535 # unregistered ports
> > #acl Safe_ports port 280                # http-mgmt
>
> #acl Safe_ports port 488                # gss-http
>
> #acl Safe_ports port 591                # filemaker
>
> #acl Safe_ports port 777                # multiling http
>

NP: "Safe_ports" is meant to block HTTP connections made to ports whose
defined protocols are known to be dangerous for HTTP messages to go to.
Those protocols are so similar they can be confused with HTTP messages
and things go badly wrong.

By doing the above you are making the default security rules (if you
re-enable them) decide that any web API using a non-80 port is
prohibited to all your clients.
eg. services like http://pc-sep.pcwhq.par****.net:8014/ in your logs.


The above change to the default http_access security rules behaviour is
probably why you had to move your custom config to the top of the
http_access list - thus bypassing everything making use of the above
ports definitions.


> acl CONNECT method CONNECT
>
> acl backoffice_allowed_sites url_regex "/etc/squid/backoffice_allowed_sites"
>
> acl h***_backoffice_allowed_sites url_regex
> "/etc/squid/backoffice_allowed_sites"
>
> acl backoffice_blocked_sites url_regex "/etc/squid/backoffice_blocklist"
>
> acl h***_backoffice_blocked_sites url_regex
> "/etc/squid/backoffice_blocklist"
>
> acl re****_allowed_sites url_regex "/etc/squid/re****_allowed_sites"
>
> acl h****_reg****_allowed_sites url_regex
> "/etc/squid/h***_reg*****_allowed_sites"

Are all these URLs *actually* needing regex? regex is the second slowest
and most memory consuming type of ACL Squid provides.

If you can reduce that to just domain names (which can have wildcard
subdomains), the dstdomain ACL type would improve both memory and
performance a fair bit.


>
> #
>
> http_access allow localnet reg***_allowed_sites
>
> http_access deny backoffice_users backoffice_blocked_sites
>
> http_access deny h***_backoffice_users backoffice_blocked_sites
>
> http_access allow backoffice_users backoffice_allowed_sites
>
> http_access allow h***_backoffice_users backoffice_allowed_sites
>
> http_access allow reg****_users reg****_allowed_sites
>
> http_access allow h***_reg****_users h***_reg****_allowed_sites
>
> no_cache deny par****
>

"no_cache" has not existed for many years. Even "cache" directive is now
deprecated.

Most likely you want to use "store_miss deny ..." to prevent caching of
objects.



> http_access deny all
>

"deny all" ... so none of the below security protections will work.

Your custom http_access lines should be down ....

> #http_access allow manager localhost
>
> #http_access deny manager
>

(sure, manager report access can be moved or removed. In fact current
recommendation is to place it after the CONNECT rule below.

BUT, be aware that removing it *allows* all clients to access the Squid
management interfaces. Probably not what you intended with the above.
Only sysadmin should need that access.
)

> # Deny requests to certain unsafe ports
>
> http_access deny !Safe_ports
>
> # Deny CONNECT to other than secure SSL ports
>
> #http_access deny CONNECT !SSL_ports
>
> http_access  allow CONNECT SSL_ports

Oh boy. You are lucky these were disabled by the above "deny all".

The default config rule was specifically crafted to prevent anonymous
tunneling to send arbitrary bytes to arbitrary ports with no proxy
control possible. Only HTTPS (port 443) is safe enough to deliver by
default.

Other ports can be permitted if you need by adding to the SSL_Ports ACL.
Absolutely DO NOT change that to an "allow CONNECT" like above.

>
> # We strongly recommend the following be uncommented to protect innocent
>
> # web applications running on the proxy server who think the only
>
> # one who can access services on "localhost" is a local user
>
> http_access deny to_localhost
>

... Your custom http_access lines should be down here. After the basic
security protections.


> # Example rule allowing access from your local networks.
>
> # Adapt localnet in the ACL section to list your (internal) IP networks
>
> # from where browsing should be allowed
>
> #http_access allow localnet
>
> http_access allow localhost
>
> # And finally deny all other access to this proxy
>
> http_access deny all
>
> # Squid normally listens to port 3128
>
> http_port 3128 ssl-bump \
>
> key=/etc/squid/pc****sslcerts/pc*****prod.pkey \
>
> cert=/etc/squid/pc******sslcerts/pc*****prod.crt \
>

cert= before key=. The most recent Squid will complain loudly, and may
not start if there is no cert to associate with the key.

> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
>

You should also add the option sslflags=NO_DEFAULT_CA here.

This lack is probably a bit part of your memory problem as OpenSSL grabs
a huge amount of memory per client connection (ouch!) to store the
"globally trusted CA" certificates - which are pointless on that type of
connection and never used in your setup.


> acl step1 at_step SslBump1
>
> ssl_bump peek step1
>
> #ssl_bump bump all
>
> ssl_bump bump backoffice_users !localnet !h***_backoffice_users
> !reg****_users !h***_reg***_users !par***
>

Outwardly that may look reasonable, but it can be simplified a fair bit.

eg. A message sent by client whose IP is in the range 10.136.0.0/13
cannot simultaneously be sent using an IP in the range 172.16.0.0/16 or
10.128.0.0/13.

You only need to exclude (with '!') when the excluded ACL can match
things that would otherwise be caught by the other ACLs in the list. For
example the h****_backoffice_users is a subset of backoffice_users, and
the par*** ACL matches a wholly different criteria than src-IP.



> #sslproxy_capath /etc/ssl/certs
>
> sslproxy_cert_error allow all
>
> always_direct allow all
>
> sslproxy_flags DONT_VERIFY_PEER
>

Just no. Remove the above three lines. Then fix any TLS/SSL issues you
find in a proper way - which will be different per-problem, and
worthwhile fixing.

Since 3.2 Squid has been able to mimic errors in the fake certs it
generates so silencing them is more harmful than good.

If there are errors that actually cannot be fixed AND happen to be safe
for a proxy to hide from the end-user [be very sure of that first], then
set only that error into an "sslproxy_cert_error allow ..." rule. Leave
all other errors going on through to the client software, often they are
better able to cope with it cleanly than the proxy can.


and for the log;

All these 403 look like the relevant URLs are not in your *_sites ACL
definitions.

>
> 1502424001.504      0 10.138.142.6 TCP_DENIED/403 4175 GET
> http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html
>
> 1502424001.533    329 10.140.230.6 TAG_NONE/200 0 CONNECT
> watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -
>
> 1502424001.543      0 10.141.80.6 TCP_DENIED/403 4167 GET
> http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html
>
> 1502424001.546    331 10.140.230.6 TAG_NONE/200 0 CONNECT
> watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -
>
> 1502424001.551  29923 10.130.27.24 TCP_MISS_ABORTED/000 0 GET
> http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? -
> HIER_DIRECT/10.1.2.35 -
>

That is a little more worrying, your web server at 10.1.2.35 appears not
to be producing a response for at least 30sec.



>
> Cachelog errors I am seeing daily:
>
> Error negotiating SSL connection on FD 26: error:140A1175:SSL
> routines:SSL_BYTES_TO_CIPHER_LIST:inappropriate fallback (1/-1)
>
> Error negotiating SSL connection on FD 1175: error:14094416:SSL
> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown (1/0)
>

First thing to do is ensure that your OpenSSL library is up to date.
That should resolve most cipher and TLS protocol related issues.


Second thing is to ensure that the qa-certificates package (or whatever
your OS calls it) is kept up to date. It changes every few weeks.
Squid-3.x cannot download CA certs on demand so this is particularly
important, and you may need to configure the
sslproxy_foreign_intermediate_certs directive with additional and
intermediate CA - only as needed and after checking them though.




> 2017/08/02 09:01:02 kid1| Error negotiating SSL on FD 989:
> error:00000000:lib(0):func(0):reason(0) (5/-1/104) ##Very rare i found
> few not frequently
>

These opaque error codes tends to be non-TLS being pushed into port 443.
It varies based on what software your clients are running.



> 2017/08/02 09:01:43 kid1| Queue overload, rejecting # too many times
>
> 2017/08/02 09:01:45 kid1| Error negotiating SSL connection on FD 1749:
> (104) Connection reset by peer ## too many times
>
> 2017/08/02 10:12:58 kid1| WARNING: Closing client connection due to
> lifetime timeout ## only one
>
> 2017/08/07 22:37:56 kid1| comm_open: socket failure: (24) Too many open
> files

Either lack of FD or your OS has set a per-process open FD limit far too
low for what Squid requires. Either HDD disk files or network socket
limits result in the above message.

>
> 2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en
>
> 2017/08/07 22:39:37 kid1| '/usr/share/squid/errors/en-us/ERR_DNS_FAIL':
> (24) Too many open files
>
> 2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en-us
>

Your system appears to be missing the Squid langpack for error page
localization. Or if you have one it needs updating to match the error
pages your Squid version uses.
<http://www.squid-cache.org/Versions/langpack/>


> 2017/08/07 22:01:42 kid1| WARNING: All 32/32 ssl_crtd processes are busy.
>
> 2017/08/07 22:01:42 kid1| WARNING: 32 pending requests queued
>
> 2017/08/07 22:01:42 kid1| WARNING: Consider increasing the number of
> ssl_crtd processes in your config file.

As Squid said, you can try increasing the crtd processes being run.

Though the above says 32 running and your squid.conf sslcrtd_children
directive says maximum of 8 to be run.


Alternatively, since you are bumping you may want to try Squid-4. The
SSL-Bump code and behaviour is quite a bit better in the latest version
even though it is in beta still.


>
> 2017/08/11 00:58:56 kid1| WARNING: Closing client connection due to
> lifetime timeout
>
> 2017/08/09 12:55:45 kid1| WARNING! Your cache is running out of
> filedescriptors
>


Yes, well. I covered that already, the above just confirms it.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

Cherukuri, Naresh
Thank You Amos. Appreciate your help!

Thanks & Regards,
Naresh

-----Original Message-----
From: Amos Jeffries [mailto:[hidden email]]
Sent: Friday, August 11, 2017 12:50 PM
To: [hidden email]
Cc: Cherukuri, Naresh
Subject: Re: [squid-users] Crash: every 19 hours: kernel: Out of memory: Kill process (squid)

On 12/08/17 01:13, Cherukuri, Naresh wrote:

> Amos,
>
> Please find below my squid conf and access logs and memory output in MB.
> Appreciate any help.
>
> Memory Info:
>
> [root@******prod ~]# free -m
>
>               total       used       free     shared    buffers     cached
>
> Mem:         11845       4194       7651         41        190       1418
>
> -/+ buffers/cache:       2585       9260
>
> Swap:        25551        408      25143
>
> Squidconf:
>
> [root@******prod squid]# more squid.conf
>
> #
>
> # Recommended minimum configuration:
>
> #
>
> max_filedesc 4096

Ouch. Squid requires between 2 and 6 sockets (FD) per client connection and clients tend to make upwards of 8 connections per domain being contacted (dozens per web page loaded). While HTTP/1.1 improvements can reduce that average a lot 4K FD cannot serve 7K clients well.

The above number should be at least 4x the expected client count to cope with load, at least 8x would be better for peak times.


>
> acl manager proto cache_object
>
> visible_hostname ******prod
>
> logfile_rotate 10
>
> access_log /cache/access.log
>
> acl localnet src 172.16.0.0/16
>
> acl backoffice_users src 10.136.0.0/13
>
> acl h****_backoffice_users src 10.142.0.0/15
>
> acl re****_users src 10.128.0.0/13
>
> acl hcity_r*****_users src 10.134.0.0/15
>

So you are immediately confusing "users" with "clients". They are
different, and at the Squid layer of networking the difference starts to
matter.

For example; are you aware that automatic software and devices without
any user logged in can still perform network transactions through the
proxy? usually it is not a problem, but if you are thinking of only
*users* utilizing the proxy you could be in for a major surprise.


> acl par**** url_regex par****
>
> acl SSL_ports port 443
>
> acl Safe_ports port 80          # http
>
> #acl Safe_ports port 21         # ftp
>
> acl Safe_ports port 443         # https
>
> #acl Safe_ports port 70         # gopher
>
> #acl Safe_ports port 210                # wais
>
> #acl Safe_ports port 1025-65535 # unregistered ports
> > #acl Safe_ports port 280                # http-mgmt
>
> #acl Safe_ports port 488                # gss-http
>
> #acl Safe_ports port 591                # filemaker
>
> #acl Safe_ports port 777                # multiling http
>

NP: "Safe_ports" is meant to block HTTP connections made to ports whose
defined protocols are known to be dangerous for HTTP messages to go to.
Those protocols are so similar they can be confused with HTTP messages
and things go badly wrong.

By doing the above you are making the default security rules (if you
re-enable them) decide that any web API using a non-80 port is
prohibited to all your clients.
eg. services like http://pc-sep.pcwhq.par****.net:8014/ in your logs.


The above change to the default http_access security rules behaviour is
probably why you had to move your custom config to the top of the
http_access list - thus bypassing everything making use of the above
ports definitions.


> acl CONNECT method CONNECT
>
> acl backoffice_allowed_sites url_regex "/etc/squid/backoffice_allowed_sites"
>
> acl h***_backoffice_allowed_sites url_regex
> "/etc/squid/backoffice_allowed_sites"
>
> acl backoffice_blocked_sites url_regex "/etc/squid/backoffice_blocklist"
>
> acl h***_backoffice_blocked_sites url_regex
> "/etc/squid/backoffice_blocklist"
>
> acl re****_allowed_sites url_regex "/etc/squid/re****_allowed_sites"
>
> acl h****_reg****_allowed_sites url_regex
> "/etc/squid/h***_reg*****_allowed_sites"

Are all these URLs *actually* needing regex? regex is the second slowest
and most memory consuming type of ACL Squid provides.

If you can reduce that to just domain names (which can have wildcard
subdomains), the dstdomain ACL type would improve both memory and
performance a fair bit.


>
> #
>
> http_access allow localnet reg***_allowed_sites
>
> http_access deny backoffice_users backoffice_blocked_sites
>
> http_access deny h***_backoffice_users backoffice_blocked_sites
>
> http_access allow backoffice_users backoffice_allowed_sites
>
> http_access allow h***_backoffice_users backoffice_allowed_sites
>
> http_access allow reg****_users reg****_allowed_sites
>
> http_access allow h***_reg****_users h***_reg****_allowed_sites
>
> no_cache deny par****
>

"no_cache" has not existed for many years. Even "cache" directive is now
deprecated.

Most likely you want to use "store_miss deny ..." to prevent caching of
objects.



> http_access deny all
>

"deny all" ... so none of the below security protections will work.

Your custom http_access lines should be down ....

> #http_access allow manager localhost
>
> #http_access deny manager
>

(sure, manager report access can be moved or removed. In fact current
recommendation is to place it after the CONNECT rule below.

BUT, be aware that removing it *allows* all clients to access the Squid
management interfaces. Probably not what you intended with the above.
Only sysadmin should need that access.
)

> # Deny requests to certain unsafe ports
>
> http_access deny !Safe_ports
>
> # Deny CONNECT to other than secure SSL ports
>
> #http_access deny CONNECT !SSL_ports
>
> http_access  allow CONNECT SSL_ports

Oh boy. You are lucky these were disabled by the above "deny all".

The default config rule was specifically crafted to prevent anonymous
tunneling to send arbitrary bytes to arbitrary ports with no proxy
control possible. Only HTTPS (port 443) is safe enough to deliver by
default.

Other ports can be permitted if you need by adding to the SSL_Ports ACL.
Absolutely DO NOT change that to an "allow CONNECT" like above.

>
> # We strongly recommend the following be uncommented to protect innocent
>
> # web applications running on the proxy server who think the only
>
> # one who can access services on "localhost" is a local user
>
> http_access deny to_localhost
>

... Your custom http_access lines should be down here. After the basic
security protections.


> # Example rule allowing access from your local networks.
>
> # Adapt localnet in the ACL section to list your (internal) IP networks
>
> # from where browsing should be allowed
>
> #http_access allow localnet
>
> http_access allow localhost
>
> # And finally deny all other access to this proxy
>
> http_access deny all
>
> # Squid normally listens to port 3128
>
> http_port 3128 ssl-bump \
>
> key=/etc/squid/pc****sslcerts/pc*****prod.pkey \
>
> cert=/etc/squid/pc******sslcerts/pc*****prod.crt \
>

cert= before key=. The most recent Squid will complain loudly, and may
not start if there is no cert to associate with the key.

> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
>

You should also add the option sslflags=NO_DEFAULT_CA here.

This lack is probably a bit part of your memory problem as OpenSSL grabs
a huge amount of memory per client connection (ouch!) to store the
"globally trusted CA" certificates - which are pointless on that type of
connection and never used in your setup.


> acl step1 at_step SslBump1
>
> ssl_bump peek step1
>
> #ssl_bump bump all
>
> ssl_bump bump backoffice_users !localnet !h***_backoffice_users
> !reg****_users !h***_reg***_users !par***
>

Outwardly that may look reasonable, but it can be simplified a fair bit.

eg. A message sent by client whose IP is in the range 10.136.0.0/13
cannot simultaneously be sent using an IP in the range 172.16.0.0/16 or
10.128.0.0/13.

You only need to exclude (with '!') when the excluded ACL can match
things that would otherwise be caught by the other ACLs in the list. For
example the h****_backoffice_users is a subset of backoffice_users, and
the par*** ACL matches a wholly different criteria than src-IP.



> #sslproxy_capath /etc/ssl/certs
>
> sslproxy_cert_error allow all
>
> always_direct allow all
>
> sslproxy_flags DONT_VERIFY_PEER
>

Just no. Remove the above three lines. Then fix any TLS/SSL issues you
find in a proper way - which will be different per-problem, and
worthwhile fixing.

Since 3.2 Squid has been able to mimic errors in the fake certs it
generates so silencing them is more harmful than good.

If there are errors that actually cannot be fixed AND happen to be safe
for a proxy to hide from the end-user [be very sure of that first], then
set only that error into an "sslproxy_cert_error allow ..." rule. Leave
all other errors going on through to the client software, often they are
better able to cope with it cleanly than the proxy can.


and for the log;

All these 403 look like the relevant URLs are not in your *_sites ACL
definitions.

>
> 1502424001.504      0 10.138.142.6 TCP_DENIED/403 4175 GET
> http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html
>
> 1502424001.533    329 10.140.230.6 TAG_NONE/200 0 CONNECT
> watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -
>
> 1502424001.543      0 10.141.80.6 TCP_DENIED/403 4167 GET
> http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html
>
> 1502424001.546    331 10.140.230.6 TAG_NONE/200 0 CONNECT
> watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -
>
> 1502424001.551  29923 10.130.27.24 TCP_MISS_ABORTED/000 0 GET
> http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? -
> HIER_DIRECT/10.1.2.35 -
>

That is a little more worrying, your web server at 10.1.2.35 appears not
to be producing a response for at least 30sec.



>
> Cachelog errors I am seeing daily:
>
> Error negotiating SSL connection on FD 26: error:140A1175:SSL
> routines:SSL_BYTES_TO_CIPHER_LIST:inappropriate fallback (1/-1)
>
> Error negotiating SSL connection on FD 1175: error:14094416:SSL
> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown (1/0)
>

First thing to do is ensure that your OpenSSL library is up to date.
That should resolve most cipher and TLS protocol related issues.


Second thing is to ensure that the qa-certificates package (or whatever
your OS calls it) is kept up to date. It changes every few weeks.
Squid-3.x cannot download CA certs on demand so this is particularly
important, and you may need to configure the
sslproxy_foreign_intermediate_certs directive with additional and
intermediate CA - only as needed and after checking them though.




> 2017/08/02 09:01:02 kid1| Error negotiating SSL on FD 989:
> error:00000000:lib(0):func(0):reason(0) (5/-1/104) ##Very rare i found
> few not frequently
>

These opaque error codes tends to be non-TLS being pushed into port 443.
It varies based on what software your clients are running.



> 2017/08/02 09:01:43 kid1| Queue overload, rejecting # too many times
>
> 2017/08/02 09:01:45 kid1| Error negotiating SSL connection on FD 1749:
> (104) Connection reset by peer ## too many times
>
> 2017/08/02 10:12:58 kid1| WARNING: Closing client connection due to
> lifetime timeout ## only one
>
> 2017/08/07 22:37:56 kid1| comm_open: socket failure: (24) Too many open
> files

Either lack of FD or your OS has set a per-process open FD limit far too
low for what Squid requires. Either HDD disk files or network socket
limits result in the above message.

>
> 2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en
>
> 2017/08/07 22:39:37 kid1| '/usr/share/squid/errors/en-us/ERR_DNS_FAIL':
> (24) Too many open files
>
> 2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en-us
>

Your system appears to be missing the Squid langpack for error page
localization. Or if you have one it needs updating to match the error
pages your Squid version uses.
<http://www.squid-cache.org/Versions/langpack/>


> 2017/08/07 22:01:42 kid1| WARNING: All 32/32 ssl_crtd processes are busy.
>
> 2017/08/07 22:01:42 kid1| WARNING: 32 pending requests queued
>
> 2017/08/07 22:01:42 kid1| WARNING: Consider increasing the number of
> ssl_crtd processes in your config file.

As Squid said, you can try increasing the crtd processes being run.

Though the above says 32 running and your squid.conf sslcrtd_children
directive says maximum of 8 to be run.


Alternatively, since you are bumping you may want to try Squid-4. The
SSL-Bump code and behaviour is quite a bit better in the latest version
even though it is in beta still.


>
> 2017/08/11 00:58:56 kid1| WARNING: Closing client connection due to
> lifetime timeout
>
> 2017/08/09 12:55:45 kid1| WARNING! Your cache is running out of
> filedescriptors
>


Yes, well. I covered that already, the above just confirms it.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users