squid logging disable based on ACL & kernel: Out of memory

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

squid logging disable based on ACL & kernel: Out of memory

Akshay Hegde
Dear All,
I got 2 query which are as follows

1. How to disable logging of few ACLs ? for example I have below ACL which I trust and I do not enable caching


# and in squid.conf
acl no-cache-domains dstdomain "/etc/squid/no_cache_domain.txt"
cache deny no-cache-domains

2. Kernel Out of Memory ( I currently have logrotate on weekly basis ), how this issue can be fixed it runs nearly 12 hours without any error

Log error is as follows:

May  1 08:45:20 proxy kernel: Out of memory: Kill process 24384 (squid) score 961 or sacrifice child
May  1 08:45:20 proxy kernel: Killed process 24384, UID 0, (squid) total-vm:23620544kB, anon-rss:15381672kB, file-rss:292kB
May  1 08:45:22 proxy squid[24382]: Squid Parent: child process 24384 exited due to signal 9 with status 0

[root@proxy squid]# cat /proc/swaps
Filename  Type Size Used Priority
/dev/sda3                               partition 8191992 38984 -1

[root@proxy squid]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)

[root@proxy squid]# squid -v
Squid Cache: Version 3.1.10

[root@proxy squid]# ll -sh /var/log/squid/access.log
37G -rw-r----- 1 squid squid 37G May  1 10:47 /var/log/squid/access.log

[root@proxy squid]# ll -sh /var/log/squid/store.log
52G -rw-r----- 1 squid squid 52G May  1 10:47 /var/log/squid/store.log

[root@proxy squid]# du -sh /cache/
23G /cache/

[root@proxy squid]# grep '^cache_dir' /etc/squid/squid.conf
cache_dir ufs /cache 25000 16 256

[root@proxy squid]# vmstat -s
     16334568  total memory
     16165900  used memory
       916616  active memory
     14824916  inactive memory
       168668  free memory
       111912  buffer memory
     14773484  swap cache
      8191992  total swap
        38984  used swap
      8153008  free swap
    150918525 non-nice user cpu ticks
      1039325 nice user cpu ticks
     88617581 system cpu ticks
   3955184009 idle cpu ticks
      8975352 IO-wait cpu ticks
      6237581 IRQ cpu ticks
     35714514 softirq cpu ticks
            0 stolen cpu ticks
   3099830316 pages paged in
    546843845 pages paged out
      2155651 pages swapped in
     11728162 pages swapped out
     78924125 interrupts
   3426967839 CPU context switches
   1577614862 boot time
       547399 forks




_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Alex Rousskov
On 5/1/20 1:20 AM, Akshay Hegde wrote:

> *1. How to disable logging of few ACLs ?

Use "access_log none aclX" to prevent creation of access.log records for
transactions matching aclX. See
http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html for
some related caveats.


> *2. Kernel Out of Memory

This problem is most likely unrelated to logging. If your Squid is
gradually leaking memory (rather than just being overwhelmed with
traffic), then the first step towards removing those memory leaks would
be to upgrade your Squid from the unsupported and buggy v3.1.10.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Akshay Hegde
Dear Alex,

Thanks a lot, I started installing new squid on centos8 as you suggested.

I got one more doubt its about logging.

I have below option globally, which I don't want to make "off"
strip_query_terms on

and my ACL as follows:
logformat squid_custom %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt
acl track dstdomain "/etc/squid/sites_track.txt"
access_log /var/log/squid/full_site_links.log squid_custom track

however for specific ACL I would like to log full URL with query parameters, how this can be done ?

- Akshay


On Fri, May 1, 2020 at 7:05 PM Alex Rousskov <[hidden email]> wrote:
On 5/1/20 1:20 AM, Akshay Hegde wrote:

> *1. How to disable logging of few ACLs ?

Use "access_log none aclX" to prevent creation of access.log records for
transactions matching aclX. See
http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html for
some related caveats.


> *2. Kernel Out of Memory

This problem is most likely unrelated to logging. If your Squid is
gradually leaking memory (rather than just being overwhelmed with
traffic), then the first step towards removing those memory leaks would
be to upgrade your Squid from the unsupported and buggy v3.1.10.


HTH,

Alex.


--

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Alex Rousskov
On 5/1/20 12:43 PM, Akshay Hegde wrote:

> I have below option globally, which I don't want to make "off"
> strip_query_terms on

> acl track dstdomain "/etc/squid/sites_track.txt"
> access_log /var/log/squid/full_site_links.log squid_custom track

> however for specific ACL I would like to log full URL with query
> parameters, how this can be done ?

I have not tested this, and the results may be version-dependent, but
according to logformat documentation[1], %ru honors strip_query_terms
while %>ru does not:

    logformat strippedFormat %ts... %ru ...
    access_log ... strippedFormat track !specific_ACL

    logformat detailedFormat %ts... %>ru ...
    access_log ... detailedFormat track specific_ACL

[1] http://www.squid-cache.org/Doc/config/logformat/


HTH,

Alex.

> On Fri, May 1, 2020 at 7:05 PM Alex Rousskov wrote:
>
>     On 5/1/20 1:20 AM, Akshay Hegde wrote:
>
>     > *1. How to disable logging of few ACLs ?
>
>     Use "access_log none aclX" to prevent creation of access.log records for
>     transactions matching aclX. See
>     http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html
>     for
>     some related caveats.
>
>
>     > *2. Kernel Out of Memory
>
>     This problem is most likely unrelated to logging. If your Squid is
>     gradually leaking memory (rather than just being overwhelmed with
>     traffic), then the first step towards removing those memory leaks would
>     be to upgrade your Squid from the unsupported and buggy v3.1.10.
>
>
>     HTH,
>
>     Alex.
>
>
>
> --
> <https://about.me/akshay.k.hegde?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
>
> Akshay Hegde
> about.me/akshay.k.hegde
> <https://about.me/akshay.k.hegde?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
>
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Amos Jeffries
Administrator
In reply to this post by Akshay Hegde
On 2/05/20 4:43 am, Akshay Hegde wrote:

> Dear Alex,
>
> Thanks a lot, I started installing new squid on centos8 as you suggested.
>
> I got one more doubt its about logging.
>
> I have below option globally, which I don't want to make "off"
> strip_query_terms on
>
> and my ACL as follows:
> logformat squid_custom %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un
> %Sh/%<A %mt
> acl track dstdomain "/etc/squid/sites_track.txt"
> access_log /var/log/squid/full_site_links.log squid_custom track
>
> however for specific ACL I would like to log full URL with query
> parameters, how this can be done ?
>

If you are upgrading to a Squid with annotation support you can use an
external ACL helper to do the URL mangling you want for a custom log
%note column.
 Otherwise there is only that global on/off setting.


NP: stripping query-string is a very weak workaround for
security+privacy flaws. Any details hidden are being published elsewhere
anyway. All it does is prevent local detection of important information
leaks.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Akshay Hegde
In reply to this post by Alex Rousskov
Dear Alex,

Thank you so much, I will test on new squid,  on old 3.1 it didn't work as you said.

-Akshay

On Sat, May 2, 2020, 1:00 AM Alex Rousskov <[hidden email]> wrote:
On 5/1/20 12:43 PM, Akshay Hegde wrote:

> I have below option globally, which I don't want to make "off"
> strip_query_terms on

> acl track dstdomain "/etc/squid/sites_track.txt"
> access_log /var/log/squid/full_site_links.log squid_custom track

> however for specific ACL I would like to log full URL with query
> parameters, how this can be done ?

I have not tested this, and the results may be version-dependent, but
according to logformat documentation[1], %ru honors strip_query_terms
while %>ru does not:

    logformat strippedFormat %ts... %ru ...
    access_log ... strippedFormat track !specific_ACL

    logformat detailedFormat %ts... %>ru ...
    access_log ... detailedFormat track specific_ACL

[1] http://www.squid-cache.org/Doc/config/logformat/


HTH,

Alex.

> On Fri, May 1, 2020 at 7:05 PM Alex Rousskov wrote:
>
>     On 5/1/20 1:20 AM, Akshay Hegde wrote:
>
>     > *1. How to disable logging of few ACLs ?
>
>     Use "access_log none aclX" to prevent creation of access.log records for
>     transactions matching aclX. See
>     http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html
>     for
>     some related caveats.
>
>
>     > *2. Kernel Out of Memory
>
>     This problem is most likely unrelated to logging. If your Squid is
>     gradually leaking memory (rather than just being overwhelmed with
>     traffic), then the first step towards removing those memory leaks would
>     be to upgrade your Squid from the unsupported and buggy v3.1.10.
>
>
>     HTH,
>
>     Alex.
>
>
>
> --
> <https://about.me/akshay.k.hegde?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
>       
> Akshay Hegde
> about.me/akshay.k.hegde
> <https://about.me/akshay.k.hegde?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
>
>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Akshay Hegde
In reply to this post by Amos Jeffries
Dear Amos,

Can you please elaborate, I didnt understand. If possible can you explain with one example ? I mean behaviour of security and privacy flaws when 
strip_query_terms is on and when strip_query_terms is off.

- Akshay

On Sat, May 2, 2020, 1:03 AM Amos Jeffries <[hidden email]> wrote:
On 2/05/20 4:43 am, Akshay Hegde wrote:
> Dear Alex,
>
> Thanks a lot, I started installing new squid on centos8 as you suggested.
>
> I got one more doubt its about logging.
>
> I have below option globally, which I don't want to make "off"
> strip_query_terms on
>
> and my ACL as follows:
> logformat squid_custom %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un
> %Sh/%<A %mt
> acl track dstdomain "/etc/squid/sites_track.txt"
> access_log /var/log/squid/full_site_links.log squid_custom track
>
> however for specific ACL I would like to log full URL with query
> parameters, how this can be done ?
>

If you are upgrading to a Squid with annotation support you can use an
external ACL helper to do the URL mangling you want for a custom log
%note column.
 Otherwise there is only that global on/off setting.


NP: stripping query-string is a very weak workaround for
security+privacy flaws. Any details hidden are being published elsewhere
anyway. All it does is prevent local detection of important information
leaks.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Amos Jeffries
Administrator
On 3/05/20 12:58 am, Akshay Hegde wrote:
> Dear Amos,
>
> Can you please elaborate, I didnt understand. If possible can you
> explain with one example ? I mean behaviour of security and privacy
> flaws when 
> strip_query_terms is on and when strip_query_terms is off.
>

That directive only affects the URLs visible in your logs etc. on the
proxy machine. It's main purpose is to prevent security/privacy
information leaks when site store sensitive info in the query-string of
the URL. The benefit is that your service is not a vector for those leaks.

On the other hand, it also prevents you being able to troubleshoot a lot
of types of issue with any site using query strings. Both allowing a
range of security attacks to hide themselves, and preventing you being
aware when sensitive info is wrongly placed in the URL.

It is up to you to decide which type of security/privacy issue is the
most important to prevent.


I bring this up because there have recently been several high-profile
services caught for major credential leaks - noticed only because some
people paid attention to their query-string's.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Akshay Hegde
In reply to this post by Alex Rousskov
Hi Alex,

I updated to latest squid as you suggested, and I tried SSL-Bump using below config (which filters URLs which are in 443 too), however I have 600 users (windows, linux, Mac, mobile OS like Androd, Windows etc), so asking them to import CA certificate in browser is not feasible.

1. Is there any way to filter HTTPS URLs without importing CA certificates on client side? if available can you share config snippet
2. for 16GB RAM, 4 core CPU, 8GB Swap, expected to have 10GB cache,  how to calculate configurations parameters, is there any thumb rule ? please share how you usually calculate.

# config
cache_mgr webmaster
cache deny QUERY
cache_mem 256 MB
cache_swap_low 90
cache_swap_high 95
maximum_object_size 4 MB
minimum_object_size 0 KB
maximum_object_size_in_memory 512 kB
ipcache_size 2048
ipcache_low 90
ipcache_high 95
fqdncache_size 1024
cache_replacement_policy lru
memory_replacement_policy lru
cache_dir ufs /var/spool/squid 10000 16 256
cache_effective_user squid
cache_effective_group squid
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
memory_pools on
memory_pools_limit 5 MB

# SSL-Bump -working but not feasible.
http_port 3128 ssl-bump cert=/etc/squid/sslcert/proxyCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/lib64/squid/security_file_certgen -s  /var/spool/squid/ssl_db -M 4MB
sslcrtd_children 5
acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump bump all

------------------------------------ My New Environment --------------------
# squid -v
Squid Cache: Version 4.4
Service Name: squid

# cat /etc/redhat-release
CentOS Linux release 8.1.1911 (Core)


# Tested ACLs
logformat test_log %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %>ru %[un %Sh/%<a %mt
acl test_sites dstdomain "/etc/squid/acls/test_sites.acl"
access_log /var/log/squid/test_site.log test_log test_sites

# tail -f /var/log/squid/test_site.log
1588678050.178   3247 10.0.2.15 TCP_TUNNEL/200 28073 CONNECT nav.sciencedirect.com:443 akshay HIER_DIRECT/91.235.133.74 -
1588678050.189   3942 10.0.2.15 TCP_TUNNEL/200 24000 CONNECT nav.sciencedirect.com:443 akshay HIER_DIRECT/91.235.133.74 -
1588678050.355   2552 10.0.2.15 TCP_TUNNEL/200 788 CONNECT nav.sciencedirect.com:443 akshay HIER_DIRECT/91.235.133.74 -
1588681419.635    647 10.0.2.15 TCP_MISS/200 402 POST http://scratchpads.eu/modules/statistics/statistics.php akshay HIER_DIRECT/157.140.2.32 text/html
1588681420.055   1069 10.0.2.15 TCP_MISS/200 46772 GET http://scratchpads.eu/sites/all/themes/scratchpads_eu/images/shrimp-202px.png akshay HIER_DIRECT/157.140.2.32 image/png




On Sat, May 2, 2020 at 1:00 AM Alex Rousskov <[hidden email]> wrote:
On 5/1/20 12:43 PM, Akshay Hegde wrote:

> I have below option globally, which I don't want to make "off"
> strip_query_terms on

> acl track dstdomain "/etc/squid/sites_track.txt"
> access_log /var/log/squid/full_site_links.log squid_custom track

> however for specific ACL I would like to log full URL with query
> parameters, how this can be done ?

I have not tested this, and the results may be version-dependent, but
according to logformat documentation[1], %ru honors strip_query_terms
while %>ru does not:

    logformat strippedFormat %ts... %ru ...
    access_log ... strippedFormat track !specific_ACL

    logformat detailedFormat %ts... %>ru ...
    access_log ... detailedFormat track specific_ACL

[1] http://www.squid-cache.org/Doc/config/logformat/


HTH,

Alex.

> On Fri, May 1, 2020 at 7:05 PM Alex Rousskov wrote:
>
>     On 5/1/20 1:20 AM, Akshay Hegde wrote:
>
>     > *1. How to disable logging of few ACLs ?
>
>     Use "access_log none aclX" to prevent creation of access.log records for
>     transactions matching aclX. See
>     http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html
>     for
>     some related caveats.
>
>
>     > *2. Kernel Out of Memory
>
>     This problem is most likely unrelated to logging. If your Squid is
>     gradually leaking memory (rather than just being overwhelmed with
>     traffic), then the first step towards removing those memory leaks would
>     be to upgrade your Squid from the unsupported and buggy v3.1.10.
>
>
>     HTH,
>
>     Alex.
>
>
>
> --
> <https://about.me/akshay.k.hegde?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
>       
> Akshay Hegde
> about.me/akshay.k.hegde
> <https://about.me/akshay.k.hegde?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
>
>



--

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Alex Rousskov
On 5/6/20 8:58 AM, Akshay Hegde wrote:

> 1. Is there any way to filter HTTPS URLs without importing CA
> certificates on client side?

No, there is no way for a proxy to look at request URLs without the
browser trusting the proxy certificate. There are other ways to police
traffic (e.g., browser plugins), but they all require fiddling with the
client environment.


> 2. for 16GB RAM, 4 core CPU, 8GB Swap, expected to have 10GB cache,  how
> to calculate configurations parameters, is there any thumb rule ?

I believe there is some related advice on Squid wiki:
https://wiki.squid-cache.org/SquidFaq/SquidMemory

HTH,

Alex.


> # config
> cache_mgr webmaster
> cache deny QUERY
> cache_mem 256 MB
> cache_swap_low 90
> cache_swap_high 95
> maximum_object_size 4 MB
> minimum_object_size 0 KB
> maximum_object_size_in_memory 512 kB
> ipcache_size 2048
> ipcache_low 90
> ipcache_high 95
> fqdncache_size 1024
> cache_replacement_policy lru
> memory_replacement_policy lru
> cache_dir ufs /var/spool/squid 10000 16 256
> cache_effective_user squid
> cache_effective_group squid
> cache_log /var/log/squid/cache.log
> cache_store_log /var/log/squid/store.log
> memory_pools on
> memory_pools_limit 5 MB
>
> # SSL-Bump -working but not feasible.
> http_port 3128 ssl-bump cert=/etc/squid/sslcert/proxyCA.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> sslcrtd_program /usr/lib64/squid/security_file_certgen -s
>  /var/spool/squid/ssl_db -M 4MB
> sslcrtd_children 5
> acl step1 at_step SslBump1
> ssl_bump peek step1
> ssl_bump bump all
>
> ------------------------------------ My New Environment --------------------
> # squid -v
> Squid Cache: Version 4.4
> Service Name: squid
>
> # cat /etc/redhat-release
> CentOS Linux release 8.1.1911 (Core)
>
>
> # Tested ACLs
> logformat test_log %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %>ru %[un
> %Sh/%<a %mt
> acl test_sites dstdomain "/etc/squid/acls/test_sites.acl"
> access_log /var/log/squid/test_site.log test_log test_sites
>
> # tail -f /var/log/squid/test_site.log
> 1588678050.178   3247 10.0.2.15 TCP_TUNNEL/200 28073 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588678050.189   3942 10.0.2.15 TCP_TUNNEL/200 24000 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588678050.355   2552 10.0.2.15 TCP_TUNNEL/200 788 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588681419.635    647 10.0.2.15 TCP_MISS/200 402 POST
> http://scratchpads.eu/modules/statistics/statistics.php akshay
> HIER_DIRECT/157.140.2.32 <http://157.140.2.32> text/html
> 1588681420.055   1069 10.0.2.15 TCP_MISS/200 46772 GET
> http://scratchpads.eu/sites/all/themes/scratchpads_eu/images/shrimp-202px.png
> akshay HIER_DIRECT/157.140.2.32 <http://157.140.2.32> image/png
>
>
>
>
> On Sat, May 2, 2020 at 1:00 AM Alex Rousskov
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 5/1/20 12:43 PM, Akshay Hegde wrote:
>
>     > I have below option globally, which I don't want to make "off"
>     > strip_query_terms on
>
>     > acl track dstdomain "/etc/squid/sites_track.txt"
>     > access_log /var/log/squid/full_site_links.log squid_custom track
>
>     > however for specific ACL I would like to log full URL with query
>     > parameters, how this can be done ?
>
>     I have not tested this, and the results may be version-dependent, but
>     according to logformat documentation[1], %ru honors strip_query_terms
>     while %>ru does not:
>
>         logformat strippedFormat %ts... %ru ...
>         access_log ... strippedFormat track !specific_ACL
>
>         logformat detailedFormat %ts... %>ru ...
>         access_log ... detailedFormat track specific_ACL
>
>     [1] http://www.squid-cache.org/Doc/config/logformat/
>
>
>     HTH,
>
>     Alex.
>
>     > On Fri, May 1, 2020 at 7:05 PM Alex Rousskov wrote:
>     >
>     >     On 5/1/20 1:20 AM, Akshay Hegde wrote:
>     >
>     >     > *1. How to disable logging of few ACLs ?
>     >
>     >     Use "access_log none aclX" to prevent creation of access.log
>     records for
>     >     transactions matching aclX. See
>     >   
>      http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html
>     >     for
>     >     some related caveats.
>     >
>     >
>     >     > *2. Kernel Out of Memory
>     >
>     >     This problem is most likely unrelated to logging. If your Squid is
>     >     gradually leaking memory (rather than just being overwhelmed with
>     >     traffic), then the first step towards removing those memory
>     leaks would
>     >     be to upgrade your Squid from the unsupported and buggy v3.1.10.
>     >
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Akshay Hegde
Hi Alex, 

Thanks for confirming, I lost hope. Can you share some link or details about below

> There are other ways to police
traffic (e.g., browser plugins), but they all require fiddling with the
client environment.

On Wed, May 6, 2020, 7:56 PM Alex Rousskov <[hidden email]> wrote:
On 5/6/20 8:58 AM, Akshay Hegde wrote:

> 1. Is there any way to filter HTTPS URLs without importing CA
> certificates on client side?

No, there is no way for a proxy to look at request URLs without the
browser trusting the proxy certificate. There are other ways to police
traffic (e.g., browser plugins), but they all require fiddling with the
client environment.


> 2. for 16GB RAM, 4 core CPU, 8GB Swap, expected to have 10GB cache,  how
> to calculate configurations parameters, is there any thumb rule ?

I believe there is some related advice on Squid wiki:
https://wiki.squid-cache.org/SquidFaq/SquidMemory

HTH,

Alex.


> # config
> cache_mgr webmaster
> cache deny QUERY
> cache_mem 256 MB
> cache_swap_low 90
> cache_swap_high 95
> maximum_object_size 4 MB
> minimum_object_size 0 KB
> maximum_object_size_in_memory 512 kB
> ipcache_size 2048
> ipcache_low 90
> ipcache_high 95
> fqdncache_size 1024
> cache_replacement_policy lru
> memory_replacement_policy lru
> cache_dir ufs /var/spool/squid 10000 16 256
> cache_effective_user squid
> cache_effective_group squid
> cache_log /var/log/squid/cache.log
> cache_store_log /var/log/squid/store.log
> memory_pools on
> memory_pools_limit 5 MB
>
> # SSL-Bump -working but not feasible.
> http_port 3128 ssl-bump cert=/etc/squid/sslcert/proxyCA.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> sslcrtd_program /usr/lib64/squid/security_file_certgen -s
>  /var/spool/squid/ssl_db -M 4MB
> sslcrtd_children 5
> acl step1 at_step SslBump1
> ssl_bump peek step1
> ssl_bump bump all
>
> ------------------------------------ My New Environment --------------------
> # squid -v
> Squid Cache: Version 4.4
> Service Name: squid
>
> # cat /etc/redhat-release
> CentOS Linux release 8.1.1911 (Core)
>
>
> # Tested ACLs
> logformat test_log %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %>ru %[un
> %Sh/%<a %mt
> acl test_sites dstdomain "/etc/squid/acls/test_sites.acl"
> access_log /var/log/squid/test_site.log test_log test_sites
>
> # tail -f /var/log/squid/test_site.log
> 1588678050.178   3247 10.0.2.15 TCP_TUNNEL/200 28073 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588678050.189   3942 10.0.2.15 TCP_TUNNEL/200 24000 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588678050.355   2552 10.0.2.15 TCP_TUNNEL/200 788 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588681419.635    647 10.0.2.15 TCP_MISS/200 402 POST
> http://scratchpads.eu/modules/statistics/statistics.php akshay
> HIER_DIRECT/157.140.2.32 <http://157.140.2.32> text/html
> 1588681420.055   1069 10.0.2.15 TCP_MISS/200 46772 GET
> http://scratchpads.eu/sites/all/themes/scratchpads_eu/images/shrimp-202px.png
> akshay HIER_DIRECT/157.140.2.32 <http://157.140.2.32> image/png
>
>
>
>
> On Sat, May 2, 2020 at 1:00 AM Alex Rousskov
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 5/1/20 12:43 PM, Akshay Hegde wrote:
>
>     > I have below option globally, which I don't want to make "off"
>     > strip_query_terms on
>
>     > acl track dstdomain "/etc/squid/sites_track.txt"
>     > access_log /var/log/squid/full_site_links.log squid_custom track
>
>     > however for specific ACL I would like to log full URL with query
>     > parameters, how this can be done ?
>
>     I have not tested this, and the results may be version-dependent, but
>     according to logformat documentation[1], %ru honors strip_query_terms
>     while %>ru does not:
>
>         logformat strippedFormat %ts... %ru ...
>         access_log ... strippedFormat track !specific_ACL
>
>         logformat detailedFormat %ts... %>ru ...
>         access_log ... detailedFormat track specific_ACL
>
>     [1] http://www.squid-cache.org/Doc/config/logformat/
>
>
>     HTH,
>
>     Alex.
>
>     > On Fri, May 1, 2020 at 7:05 PM Alex Rousskov wrote:
>     >
>     >     On 5/1/20 1:20 AM, Akshay Hegde wrote:
>     >
>     >     > *1. How to disable logging of few ACLs ?
>     >
>     >     Use "access_log none aclX" to prevent creation of access.log
>     records for
>     >     transactions matching aclX. See
>     >   
>      http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html
>     >     for
>     >     some related caveats.
>     >
>     >
>     >     > *2. Kernel Out of Memory
>     >
>     >     This problem is most likely unrelated to logging. If your Squid is
>     >     gradually leaking memory (rather than just being overwhelmed with
>     >     traffic), then the first step towards removing those memory
>     leaks would
>     >     be to upgrade your Squid from the unsupported and buggy v3.1.10.
>     >

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid logging disable based on ACL & kernel: Out of memory

Alex Rousskov
On 5/6/20 10:45 AM, Akshay Hegde wrote:

> Can you share some link or details about below

Sorry, I cannot -- it has been many years since I worked on browser
plugins, and I have heard that there were significant changes in
APIs/rights since then. Perhaps others on the mailing list can help you.
If not, most of the related information should be publicly available.

Alex.

>> There are other ways to police
> traffic (e.g., browser plugins), but they all require fiddling with the
> client environment.
>
> On Wed, May 6, 2020, 7:56 PM Alex Rousskov wrote:
>
>     On 5/6/20 8:58 AM, Akshay Hegde wrote:
>
>     > 1. Is there any way to filter HTTPS URLs without importing CA
>     > certificates on client side?
>
>     No, there is no way for a proxy to look at request URLs without the
>     browser trusting the proxy certificate. There are other ways to police
>     traffic (e.g., browser plugins), but they all require fiddling with the
>     client environment.
>
>
>     > 2. for 16GB RAM, 4 core CPU, 8GB Swap, expected to have 10GB
>     cache,  how
>     > to calculate configurations parameters, is there any thumb rule ?
>
>     I believe there is some related advice on Squid wiki:
>     https://wiki.squid-cache.org/SquidFaq/SquidMemory
>
>     HTH,
>
>     Alex.
>
>
>     > # config
>     > cache_mgr webmaster
>     > cache deny QUERY
>     > cache_mem 256 MB
>     > cache_swap_low 90
>     > cache_swap_high 95
>     > maximum_object_size 4 MB
>     > minimum_object_size 0 KB
>     > maximum_object_size_in_memory 512 kB
>     > ipcache_size 2048
>     > ipcache_low 90
>     > ipcache_high 95
>     > fqdncache_size 1024
>     > cache_replacement_policy lru
>     > memory_replacement_policy lru
>     > cache_dir ufs /var/spool/squid 10000 16 256
>     > cache_effective_user squid
>     > cache_effective_group squid
>     > cache_log /var/log/squid/cache.log
>     > cache_store_log /var/log/squid/store.log
>     > memory_pools on
>     > memory_pools_limit 5 MB
>     >
>     > # SSL-Bump -working but not feasible.
>     > http_port 3128 ssl-bump cert=/etc/squid/sslcert/proxyCA.pem
>     > generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
>     > sslcrtd_program /usr/lib64/squid/security_file_certgen -s
>     >  /var/spool/squid/ssl_db -M 4MB
>     > sslcrtd_children 5
>     > acl step1 at_step SslBump1
>     > ssl_bump peek step1
>     > ssl_bump bump all
>     >
>     > ------------------------------------ My New Environment
>     --------------------
>     > # squid -v
>     > Squid Cache: Version 4.4
>     > Service Name: squid
>     >
>     > # cat /etc/redhat-release
>     > CentOS Linux release 8.1.1911 (Core)
>     >
>     >
>     > # Tested ACLs
>     > logformat test_log %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %>ru %[un
>     > %Sh/%<a %mt
>     > acl test_sites dstdomain "/etc/squid/acls/test_sites.acl"
>     > access_log /var/log/squid/test_site.log test_log test_sites
>     >
>     > # tail -f /var/log/squid/test_site.log
>     > 1588678050.178   3247 10.0.2.15 TCP_TUNNEL/200 28073 CONNECT
>     > nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443>
>     <http://nav.sciencedirect.com:443> akshay
>     > HIER_DIRECT/91.235.133.74 <http://91.235.133.74>
>     <http://91.235.133.74> -
>     > 1588678050.189   3942 10.0.2.15 TCP_TUNNEL/200 24000 CONNECT
>     > nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443>
>     <http://nav.sciencedirect.com:443> akshay
>     > HIER_DIRECT/91.235.133.74 <http://91.235.133.74>
>     <http://91.235.133.74> -
>     > 1588678050.355   2552 10.0.2.15 TCP_TUNNEL/200 788 CONNECT
>     > nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443>
>     <http://nav.sciencedirect.com:443> akshay
>     > HIER_DIRECT/91.235.133.74 <http://91.235.133.74>
>     <http://91.235.133.74> -
>     > 1588681419.635    647 10.0.2.15 TCP_MISS/200 402 POST
>     > http://scratchpads.eu/modules/statistics/statistics.php akshay
>     > HIER_DIRECT/157.140.2.32 <http://157.140.2.32>
>     <http://157.140.2.32> text/html
>     > 1588681420.055   1069 10.0.2.15 TCP_MISS/200 46772 GET
>     >
>     http://scratchpads.eu/sites/all/themes/scratchpads_eu/images/shrimp-202px.png
>     > akshay HIER_DIRECT/157.140.2.32 <http://157.140.2.32>
>     <http://157.140.2.32> image/png
>     >
>     >
>     >
>     >
>     > On Sat, May 2, 2020 at 1:00 AM Alex Rousskov
>     > <[hidden email]
>     <mailto:[hidden email]>
>     > <mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >
>     >     On 5/1/20 12:43 PM, Akshay Hegde wrote:
>     >
>     >     > I have below option globally, which I don't want to make "off"
>     >     > strip_query_terms on
>     >
>     >     > acl track dstdomain "/etc/squid/sites_track.txt"
>     >     > access_log /var/log/squid/full_site_links.log squid_custom track
>     >
>     >     > however for specific ACL I would like to log full URL with query
>     >     > parameters, how this can be done ?
>     >
>     >     I have not tested this, and the results may be
>     version-dependent, but
>     >     according to logformat documentation[1], %ru honors
>     strip_query_terms
>     >     while %>ru does not:
>     >
>     >         logformat strippedFormat %ts... %ru ...
>     >         access_log ... strippedFormat track !specific_ACL
>     >
>     >         logformat detailedFormat %ts... %>ru ...
>     >         access_log ... detailedFormat track specific_ACL
>     >
>     >     [1] http://www.squid-cache.org/Doc/config/logformat/
>     >
>     >
>     >     HTH,
>     >
>     >     Alex.
>     >
>     >     > On Fri, May 1, 2020 at 7:05 PM Alex Rousskov wrote:
>     >     >
>     >     >     On 5/1/20 1:20 AM, Akshay Hegde wrote:
>     >     >
>     >     >     > *1. How to disable logging of few ACLs ?
>     >     >
>     >     >     Use "access_log none aclX" to prevent creation of access.log
>     >     records for
>     >     >     transactions matching aclX. See
>     >     >   
>     >   
>       http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html
>     >     >     for
>     >     >     some related caveats.
>     >     >
>     >     >
>     >     >     > *2. Kernel Out of Memory
>     >     >
>     >     >     This problem is most likely unrelated to logging. If
>     your Squid is
>     >     >     gradually leaking memory (rather than just being
>     overwhelmed with
>     >     >     traffic), then the first step towards removing those memory
>     >     leaks would
>     >     >     be to upgrade your Squid from the unsupported and buggy
>     v3.1.10.
>     >     >
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users