3.5.20 run out of my memory.

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

3.5.20 run out of my memory.

minh hưng đỗ hoàng
Dear all, i use squid 3.5.20 on ubuntu14 in TPROXY mode.
With basic config in squid.conf, but squid is run out of my server's memory.
Here is my configure option :

'--prefix=/usr' '--includedir=/usr/include' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/lib/squid' '--srcdir=.' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=24' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-gnuregex' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-http-violations' '--enable-ssl-crtd' '--enable-linux-netfilter' '--enable-ltdl-install' '--enable-ltdl-convenience' '--enable-x-accelerator-vary' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--disable-translation' '--disable-ipv6' '--disable-ident-lookups' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-aufs-threads=24' '--with-filedescriptors=65536' '--with-large-files' '--with-maxfd=65536' '--with-openssl' '--with-default-user=proxy' '--with-included-ltdl'
--------------------------------------

And i apply this patch before compile for disabling host forgery checks :

+diff -ur squid-3.5.20-orig/src/client_side_request.cc squid-3.5.20/src/client_side_request.cc
+--- squid-3.5.20-orig/src/client_side_request.cc    2016-07-01 13:37:50.000000000 +0200
++++ squid-3.5.20/src/client_side_request.cc    2017-03-10 16:48:08.920084072 +0100
+@@ -530,6 +530,10 @@
+             }
+             debugs(85, 3, HERE << "validate IP " << clientConn->local << " non-match from Host: IP " << ia->in_addrs[i]);
+         }
++    // disable fogery check. See https://code.nethesis.it/Nethesis/dev/issues/5088
++        http->request->flags.hostVerified = true;
++        http->doCallouts();
++        return;
+     }
+     debugs(85, 3, HERE << "FAIL: validate IP " << clientConn->local << " possible from Host:");
+     hostHeaderVerifyFailed("local IP", "any domain IP");

And here is my squid.conf ( i don't post my http_access for clearly view :()

###############################################################################
# Squid normally listens to port 3128
###############################################################################

https_port 3130 tproxy ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl/e1f19c0494badc8dc14e8c4c56a8b97a.dyn
http_port 3129 tproxy
http_port 3128

###############################################################################
# squid ssl_bump option
###############################################################################
acl step1 at_step SslBump1
acl block ssl::server_name "/etc/squid/block_domain.txt"
ssl_bump peek step1
ssl_bump terminate block
ssl_bump splice all
sslproxy_options NO_SSLv2,NO_SSLv3,No_Compression
sslproxy_cipher  ALL:!SSLv2:!SSLv3:!ADH:!DSS:!MD5:!EXP:!DES:!PSK:!SRP:!RC4:!IDEA:!SEED:!aNULL:!eNULL
sslproxy_cert_error deny all
sslproxy_foreign_intermediate_certs /etc/squid/intermediate_ca.pem

sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB
sslcrtd_children 8 startup=1 idle=1

###############################################################################
## LOGFILE OPTIONS
###############################################################################

mime_table /etc/squid/mime.conf
pid_filename /var/run/squid.pid

include /etc/squid/logging.conf
###############################################################################
## OPTIONS FOR TROUBLESHOOTING
###############################################################################

coredump_dir /var/spool/squid
debug_options ALL,1
cache_effective_user squid
cache_effective_group squid
###############################################################################
## PERSISTENT CONNECTION HANDLING
###############################################################################
 
detect_broken_pconn off
client_persistent_connections off
server_persistent_connections on

###############################################################################
## ERROR PAGE OPTIONS
###############################################################################
error_directory /usr/share/squid/errors/en
error_log_languages off

###############################################################################
## DNS OPTIONS
###############################################################################
check_hostnames off
hosts_file /etc/hosts
connect_retries 2
ipcache_low 90
ipcache_size 5024       # Maximum number of DNS IP cache entries.
fqdncache_size 3024     # Maximum number of FQDN cache entries.
pipeline_prefetch 100

###############################################################################
##  MISCELLANEOUS
###############################################################################

max_filedescriptors 65536

------------------------------------------------------------------------

The problem is my squid spent alot of memory. I have about 200 user, and my server is 4gb dram with 8gb swap dram but not enough !
             total       used       free     shared    buffers     cached
Mem:          3.8G       3.4G       503M       736K       181M       1.7G
-/+ buffers/cache:       1.5G       2.4G
Swap:         8.1G       9.3M       8.1G

There is any issue with my squid ?? How can i fix it ?

I have attach files for detail (squid.conf and squid-3.5.20-ssl-forgery.patch)

--
Thanks & Best Regards,
--------------
Đỗ Hoàng Minh Hưng
Gmail : [hidden email]
SĐT : 01234454115

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

squid.conf (3K) Download Attachment
squid-3.5.20-ssl-forgery.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 3.5.20 run out of my memory.

Vacheslav

I cron those for memory, try it.
0                             */1 *   *   *          root                                       /usr/sbin/sysctl -w vm.drop_caches=3

0                             */1 *   *   *          root                                       /bin/sync && /bin/echo 3 | /usr/bin/tee /proc/sys/vm/drop_cache

 

From: squid-users [mailto:[hidden email]] On Behalf Of minh hung d? hoang
Sent: Wednesday, February 7, 2018 9:35 AM
To: [hidden email]
Subject: [squid-users] 3.5.20 run out of my memory.

 

Dear all, i use squid 3.5.20 on ubuntu14 in TPROXY mode.

With basic config in squid.conf, but squid is run out of my server's memory.

Here is my configure option :

'--prefix=/usr' '--includedir=/usr/include' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/lib/squid' '--srcdir=.' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=24' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-gnuregex' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-http-violations' '--enable-ssl-crtd' '--enable-linux-netfilter' '--enable-ltdl-install' '--enable-ltdl-convenience' '--enable-x-accelerator-vary' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--disable-translation' '--disable-ipv6' '--disable-ident-lookups' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-aufs-threads=24' '--with-filedescriptors=65536' '--with-large-files' '--with-maxfd=65536' '--with-openssl' '--with-default-user=proxy' '--with-included-ltdl'
--------------------------------------

And i apply this patch before compile for disabling host forgery checks :

+diff -ur squid-3.5.20-orig/src/client_side_request.cc squid-3.5.20/src/client_side_request.cc
+--- squid-3.5.20-orig/src/client_side_request.cc    2016-07-01 13:37:50.000000000 +0200
++++ squid-3.5.20/src/client_side_request.cc    2017-03-10 16:48:08.920084072 +0100
+@@ -530,6 +530,10 @@
+             }
+             debugs(85, 3, HERE << "validate IP " << clientConn->local << " non-match from Host: IP " << ia->in_addrs[i]);
+         }
++    // disable fogery check. See https://code.nethesis.it/Nethesis/dev/issues/5088
++        http->request->flags.hostVerified = true;
++        http->doCallouts();
++        return;
+     }
+     debugs(85, 3, HERE << "FAIL: validate IP " << clientConn->local << " possible from Host:");
+     hostHeaderVerifyFailed("local IP", "any domain IP");

 

And here is my squid.conf ( i don't post my http_access for clearly view :()

###############################################################################
# Squid normally listens to port 3128
###############################################################################

https_port 3130 tproxy ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl/e1f19c0494badc8dc14e8c4c56a8b97a.dyn
http_port 3129 tproxy
http_port 3128

###############################################################################
# squid ssl_bump option
###############################################################################
acl step1 at_step SslBump1
acl block ssl::server_name "/etc/squid/block_domain.txt"
ssl_bump peek step1
ssl_bump terminate block
ssl_bump splice all
sslproxy_options NO_SSLv2,NO_SSLv3,No_Compression
sslproxy_cipher  ALL:!SSLv2:!SSLv3:!ADH:!DSS:!MD5:!EXP:!DES:!PSK:!SRP:!RC4:!IDEA:!SEED:!aNULL:!eNULL
sslproxy_cert_error deny all
sslproxy_foreign_intermediate_certs /etc/squid/intermediate_ca.pem

sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB
sslcrtd_children 8 startup=1 idle=1

###############################################################################
## LOGFILE OPTIONS
###############################################################################

mime_table /etc/squid/mime.conf
pid_filename /var/run/squid.pid

include /etc/squid/logging.conf
###############################################################################
## OPTIONS FOR TROUBLESHOOTING
###############################################################################

coredump_dir /var/spool/squid
debug_options ALL,1
cache_effective_user squid
cache_effective_group squid
###############################################################################
## PERSISTENT CONNECTION HANDLING
###############################################################################
 
detect_broken_pconn off
client_persistent_connections off
server_persistent_connections on

###############################################################################
## ERROR PAGE OPTIONS
###############################################################################
error_directory /usr/share/squid/errors/en
error_log_languages off

###############################################################################
## DNS OPTIONS
###############################################################################
check_hostnames off
hosts_file /etc/hosts
connect_retries 2
ipcache_low 90
ipcache_size 5024       # Maximum number of DNS IP cache entries.
fqdncache_size 3024     # Maximum number of FQDN cache entries.
pipeline_prefetch 100

###############################################################################
##  MISCELLANEOUS
###############################################################################

max_filedescriptors 65536

------------------------------------------------------------------------

 

The problem is my squid spent alot of memory. I have about 200 user, and my server is 4gb dram with 8gb swap dram but not enough !
             total       used       free     shared    buffers     cached
Mem:          3.8G       3.4G       503M       736K       181M       1.7G
-/+ buffers/cache:       1.5G       2.4G
Swap:         8.1G       9.3M       8.1G

There is any issue with my squid ?? How can i fix it ?

I have attach files for detail (squid.conf and squid-3.5.20-ssl-forgery.patch)

--

Thanks & Best Regards,
--------------

Đỗ Hoàng Minh Hưng

Gmail : [hidden email]

SĐT : 01234454115


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 3.5.20 run out of my memory.

Amos Jeffries
Administrator
In reply to this post by minh hưng đỗ hoàng

On 07/02/18 19:34, minh hưng đỗ hoàng wrote:
> Dear all, i use squid 3.5.20 on ubuntu14 in TPROXY mode.
> With basic config in squid.conf, but squid is run out of my server's memory.
> Here is my configure option :
...
>
> https_port 3130 tproxy ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB
> cert=/etc/squid/ssl/e1f19c0494badc8dc14e8c4c56a8b97a.dyn

Please add sslflags=NO_DEFAULT_CA to the above config line. That should
reduce the memory usage a lot.

If the problems remains please try:
 a) removing that patch. It makes your Squid vulnerable to the worst
security issues Squid has faced this century.
 (One of the MANY effects of that vulnerability is ability of remote
attackers to consume large amounts of your network resources without any
traceability or visibility.)

  b) upgrade to Squid-4. The version is still in beta due to a few
issues, but overall MUCH better for SSL-Bump than Squid-3.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 3.5.20 run out of my memory.

minh hưng đỗ hoàng
Thanks alot for your help,

> https_port 3130 tproxy ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB
> cert=/etc/squid/ssl/e1f19c0494badc8dc14e8c4c56a8b97a.dyn

Please add sslflags=NO_DEFAULT_CA to the above config line. That should
reduce the memory usage a lot.

 
I have tried this command, but my squid still used alot of my memory for cache .

KiB Mem:   4037016 total,  3729152 used,   307864 free,   120508 buffers
KiB Swap:  8511484 total,        0 used,  8511484 free.  2213580 cached Mem

Squid only use about 1.2Gb dram to run, but use alot of memory for cached ( 2213580 cached Mem )
What was cached by my squid with my squid.conf ? Can i reduce or set life-time for this cache ?

--
Thanks & Best Regards,
--------------
Đỗ Hoàng Minh Hưng
Gmail : [hidden email]
SĐT : 01234454115

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 3.5.20 run out of my memory.

Sticher, Jascha
Hi,

> KiB Mem:   4037016 total,  3729152 used,   307864 free,   120508 buffers
> KiB Swap:  8511484 total,        0 used,  8511484 free.  2213580 cached Mem

this is normal behaviour in Linux - everything that's once read from disk is cached in RAM, as long as there is free memory.
If the RAM is needed in another way, the cache in memory will be reduced. See also: https://www.linuxatemyram.com/


Kind regards,

Jascha Sticher


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 3.5.20 run out of my memory.

Amos Jeffries
Administrator
In reply to this post by minh hưng đỗ hoàng
On 09/02/18 14:12, minh hưng đỗ hoàng wrote:

> Thanks alot for your help,
>
>     > https_port 3130 tproxy ssl-bump generate-host-certificates=on
>     > dynamic_cert_mem_cache_size=4MB
>     > cert=/etc/squid/ssl/e1f19c0494badc8dc14e8c4c56a8b97a.dyn
>
>     Please add sslflags=NO_DEFAULT_CA to the above config line. That should
>     reduce the memory usage a lot.
>
>  
> I have tried this command, but my squid still used alot of my memory for
> cache .
>
> KiB Mem:   4037016 total,  3729152 used,   307864 free,   120508 buffers
> KiB Swap:  8511484 total,        0 used,  8511484 free.  2213580 cached Mem
>
> Squid only use about 1.2Gb dram to run, but use alot of memory for
> cached ( 2213580 cached Mem )
> What was cached by my squid with my squid.conf ? Can i reduce or set
> life-time for this cache ?
>

"cached Mem" is not Squid memory. It is Operating System memory and
nothing to worry about so long as the used and free values are reasonable.

You have "0 used" in Swap, and lots of "free" in main memory. Those are
the most important things.

These pages may be of help understanding what all the numbers mean:

<https://www.networkworld.com/article/2722141/it-management/making-sense-of-memory-usage-on-linux.html>

<https://wiki.squid-cache.org/SquidFaq/SquidMemory>

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 3.5.20 run out of my memory.

Amos Jeffries
Administrator
In reply to this post by Sticher, Jascha
On 09/02/18 20:30, Sticher, Jascha wrote:

> Hi,
>
>> KiB Mem:   4037016 total,  3729152 used,   307864 free,   120508 buffers
>> KiB Swap:  8511484 total,        0 used,  8511484 free.  2213580 cached Mem
>
> this is normal behaviour in Linux - everything that's once read from disk is cached in RAM, as long as there is free memory.
> If the RAM is needed in another way, the cache in memory will be reduced. See also: https://www.linuxatemyram.com/
>
>
> Kind regards,
>
> Jascha Sticher
>

Nice way to say it. Do you mind If I quote you for this in the Squid FAQ
pages?

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 3.5.20 run out of my memory.

Sticher, Jascha
> Von: squid-users [mailto:[hidden email]] Im
> Auftrag von Amos Jeffries
> Gesendet: Freitag, 9. Februar 2018 08:37
>
> On 09/02/18 20:30, Sticher, Jascha wrote:
> > Hi,
> >
> >> KiB Mem:   4037016 total,  3729152 used,   307864 free,   120508 buffers
> >> KiB Swap:  8511484 total,        0 used,  8511484 free.  2213580 cached Mem
> >
> > this is normal behaviour in Linux - everything that's once read from disk is
> cached in RAM, as long as there is free memory.
> > If the RAM is needed in another way, the cache in memory will be reduced.
> See also: https://www.linuxatemyram.com/
> >
> >
> > Kind regards,
> >
> > Jascha Sticher
> >
>
> Nice way to say it. Do you mind If I quote you for this in the Squid FAQ
> pages?
>
> Amos

I don't mind - go ahead.

I'm glad to help!

Kind regards,

Jascha Sticher
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users