url_rewrite_program and ACLs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

url_rewrite_program and ACLs

Vieri
Thanks. I defined the following, and it worked as expected:

url_rewrite_access deny allowed_domains
url_rewrite_access deny allowed_ips
url_rewrite_program /usr/bin/squidGuard
url_rewrite_children 80 startup=10 idle=3


How can I rewrite a URL in squid without a helper such as SG?
ie. how can emulate SG's "rew" in squid.conf?

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Amos Jeffries
Administrator
On 09/11/17 03:12, Vieri wrote:

> Thanks. I defined the following, and it worked as expected:
>
> url_rewrite_access deny allowed_domains
> url_rewrite_access deny allowed_ips
> url_rewrite_program /usr/bin/squidGuard
> url_rewrite_children 80 startup=10 idle=3
>
>
> How can I rewrite a URL in squid without a helper such as SG?
> ie. how can emulate SG's "rew" in squid.conf?

That depends on the rew(rite) substitutions being made, and more
specifically what your intended end-goal behind having it was.

Usually what it is used for is emulate the denial of a request, or to
redirect it somewhere specific.


* Denial is better done by "http_access deny ...". No rewrite/redirect
necessary at all.

* Redirect is better done by using deny_info to change what action
'deny' means for a particular ACL. Like so:

  acl foo ...
  http_access deny foo
  deny_info 302:http://example.com/ foo

In Squid-3.2+ the deny_info URL portion can use logformat macros for
dynamic redirection - like the "rew" substitutions only changing
portions of the URL.

Time constraints are added by using a time ACL on the original *_access
line to limit when the foo ACL gets checked (aka. takes effect).


NP: The SG documented example for use of "rew" (diverting traffic to a
local server during work hours) is better performed by a cache_peer
directing traffic to a local server, and cache_peer_access ACLs
determining what and when traffic gets delivered there. No denials,
redirects, or rewrites necessary.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri

________________________________
From: Amos Jeffries <[hidden email]>
>
> acl foo ...
>  http_access deny foo
>  deny_info 302:http://example.com/ foo
>
> In Squid-3.2+ the deny_info URL portion can use logformat macros for
> dynamic redirection - like the "rew" substitutions only changing
> portions of the URL.


I was already using deny_info like this:

deny_info <a href="http://squidserver/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_domains">http://squidserver/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_domains denied_domains

I was wondering how to do an immediate redirect without doing it from my custom php script. Now I see.

I guess I'll have trouble redirecting https sites though... (TLS/SSL trust issues)

I don't know if I can "cleanly" redirect from an HTTPS to an HTTP site (ie. so the user's browser doesn't show a "can't open page" message of some sort...).

You mention that I can avoid using SG, ufdbGuard, or any other redirector/helper for access control. The problem I see when trying to use huge plain text blacklists within Squid directly is that it takes a LOT of time for the proxy cache to start up as it populates the ACLs.
I can't afford to wait for Squid to do that before serving client requests. I'd rather "allow everything" until the ACLs are populated than have users wait for so long.
Am I missing something? Is there a way to tell Squid to process the ACLs "in the background", but start handling requests immediately. If so, is it also possible to tell squid in which order to process the ACLs, ie. first process the allowed_domains ACL, and then the denied_domains ACLs? (is ordering in squid.conf enough?)

I'm saying this because I've sometimes had the need to restart Squid during the worst time of the day (peak working hours). That usually happens when I have an issue with too many open file descriptors (working on it). Stopping it cleanly takes up to 2-3 minutes. If I had to wait several more minutes for Squid to start again because it re-populates huge blacklist ACLs then I think they'd hang me for it.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Amos Jeffries
Administrator
On 09/11/17 20:51, Vieri wrote:

>
> ________________________________
> From: Amos Jeffries
>>
>> acl foo ...
>>   http_access deny foo
>>   deny_info 302:http://example.com/ foo
>>
>> In Squid-3.2+ the deny_info URL portion can use logformat macros for
>> dynamic redirection - like the "rew" substitutions only changing
>> portions of the URL.
>
>
> I was already using deny_info like this:
>
> deny_info <a href="http://squidserver/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_domains">http://squidserver/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_domains denied_domains
>
> I was wondering how to do an immediate redirect without doing it from my custom php script. Now I see.
>
> I guess I'll have trouble redirecting https sites though... (TLS/SSL trust issues)

The proper HTTP *redirect* with 3xx statsu code has no problems there.
Since the redirect is simply telling the client to connect elsewhere.

It is re-writing the URL which encounters cert problems because the
server gets secretly changed from what the client "knows" it is
connected to mid-way along the connection.

If you are redirecting https:// URLs to a http:// location they may
still warn about insecure content. But that is not a cert issue, just
the browser scare tactics for pushing its "TLS-Everywhere" policy on
your users.


>
> I don't know if I can "cleanly" redirect from an HTTPS to an HTTP site (ie. so the user's browser doesn't show a "can't open page" message of some sort...).
>
> You mention that I can avoid using SG, ufdbGuard, or any other redirector/helper for access control. The problem I see when trying to use huge plain text blacklists within Squid directly is that it takes a LOT of time for the proxy cache to start up as it populates the ACLs.
> I can't afford to wait for Squid to do that before serving client requests. I'd rather "allow everything" until the ACLs are populated than have users wait for so long.
> Am I missing something? Is there a way to tell Squid to process the ACLs "in the background", but start handling requests immediately. If so, is it also possible to tell squid in which order to process the ACLs, ie. first process the allowed_domains ACL, and then the denied_domains ACLs? (is ordering in squid.conf enough?)

Darn. You have the one case that calls for keeping the helper :-(

You can still move the ACLs that load in a reasonable times into
squid.conf and leave the others in SG/ufdbguard. Using
url_rewrite_access to restrict which transactions the helper gets
involved with. That will reduce its latency impact on lie traffic, but
still cause much the same memory related (non-)issues as now.

>
> I'm saying this because I've sometimes had the need to restart Squid during the worst time of the day (peak working hours). That usually happens when I have an issue with too many open file descriptors (working on it). Stopping it cleanly takes up to 2-3 minutes. If I had to wait several more minutes for Squid to start again because it re-populates huge blacklist ACLs then I think they'd hang me for it.
>

One thing that may also help you there while you figure out what those
FD issues are caused by:

If you run "squid -k shutdown" once it signals a slow graceful shutdown
of Squid. Which will take a *minimum* of the time configured in
shutdown_lifetime directive (default 30sec) to wait for existing clients
to finish up and disconnect.

Running "squid -k shutdown" a _second_ time sends the running proxy a
signal to immediately skip to the processing as if the shutdown_lifetime
had already been reached.

The shutdown time with two sequential calls should be only the time
needed to close all the open FD connections, shutdown helpers and
release however much memory Squid is using. All that happens in just a
few seconds normally, but I'm not sure if "normal" matches your FD
overload case. Will definitely be faster than waiting for the graceful
shutdown anyhow.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri

________________________________
From: Amos Jeffries <[hidden email]>
>
> Darn. You have the one case that calls for keeping the helper :-(
>
> You can still move the ACLs that load in a reasonable times into
> squid.conf and leave the others in SG/ufdbguard. Using
> url_rewrite_access to restrict which transactions the helper gets
> involved with. That will reduce its latency impact on lie traffic, but
> still cause much the same memory related (non-)issues as now.


That's exactly what I'm doing right now...
Thanks.

> Running "squid -k shutdown" a _second_ time sends the running proxy a
> signal to immediately skip to the processing as if the shutdown_lifetime
> had already been reached.


Thanks for that double-shutdown signal trick. I'll have to try that asap.

I'm making progress (sort of) on the FD (non-)issues I'm having.

I'll try to post back to Alex asap.

I have a custom perl script that does MySQL lookups for blacklisted sites (lots of them - so I can't use ACLs within squid.conf). I define that helper with external_acl_type.

Yesterday I changed my squid.conf by disabling this helper, and used squidGuard instead.
I noticed a huge improvement.

I took this snapshot yesterday:

15:25 08/11/2017:

File descriptor usage for squid:
Maximum number of file descriptors: 65536
Largest file desc currently in use: 2730
Number of file desc currently in use: 1838
Files queued for open: 0
Available number of file descriptors: 63698
Reserved number of file descriptors: 100
Store Disk files open: 0

Today I took another peak and found:

Thu Nov 9 12:19:05 CET 2017:

File descriptor usage for squid:
Maximum number of file descriptors: 65536
Largest file desc currently in use: 6980
Number of file desc currently in use: 6627
Files queued for open: 0
Available number of file descriptors: 58909
Reserved number of file descriptors: 100
Store Disk files open: 0

The FDs are still increasing steadily, but a LOT less.

On the other hand, the "free" RAM went from 2GB yesterday to just 275MB today:

# free --mega
total used free shared buff/cache available
Mem: 32865 8685 275 157 23904 23683
Swap: 37036 286 36750

Used swap is still low enough (unchanged actually), so I guess I don't need to worry about it.

However, I'm bound to have issues when the "free" mem reaches 0... and I bet it will eventually.
That's when the double-shutdown trick will kick in.

I'll review the perl helper code, or maybe just switch to ufdbGuard.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri
In reply to this post by Amos Jeffries

________________________________
From: Amos Jeffries <[hidden email]>
>
> The shutdown time with two sequential calls should be only the time
> needed to close all the open FD connections, shutdown helpers and
> release however much memory Squid is using. All that happens in just a

> few seconds normally

I updated to version 3.5.27-20171101-re69e56c, and restarted squid with the double-shutdown trick (I didn't NEED to do it - I just couldn't wait to try it out ;-) ).

The result was acceptable. It didn't take a "few seconds", but almost 30 seconds. Still, it's a LOT better than before when I had to wait 2 or 3 minutes.

It's fair to say that there weren't as many open FDs as in my previous cases when shutting down Squid was very slow. Now, open FDs were around 7000.
A lot of mem was freed up after the restart.

Making progress.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Amos Jeffries
Administrator
In reply to this post by Vieri
On 10/11/17 00:39, Vieri wrote:

>
> ________________________________
> From: Amos Jeffries <[hidden email]>
>>
>> Darn. You have the one case that calls for keeping the helper :-(
>>
>> You can still move the ACLs that load in a reasonable times into
>> squid.conf and leave the others in SG/ufdbguard. Using
>> url_rewrite_access to restrict which transactions the helper gets
>> involved with. That will reduce its latency impact on lie traffic, but
>> still cause much the same memory related (non-)issues as now.
>
>
> That's exactly what I'm doing right now...
> Thanks.
>
>> Running "squid -k shutdown" a _second_ time sends the running proxy a
>> signal to immediately skip to the processing as if the shutdown_lifetime
>> had already been reached.
>
>
> Thanks for that double-shutdown signal trick. I'll have to try that asap.
>
> I'm making progress (sort of) on the FD (non-)issues I'm having.
>
> I'll try to post back to Alex asap.
>
> I have a custom perl script that does MySQL lookups for blacklisted sites (lots of them - so I can't use ACLs within squid.conf). I define that helper with external_acl_type.
>
> Yesterday I changed my squid.conf by disabling this helper, and used squidGuard instead.
> I noticed a huge improvement.

Hmm that is suspicious. AFAIK SquidGuards' one remaining useful feature
is how it loads large lists into memory in big chunks then processes
after its technically already processing traffic from Squid - whereas
Squid loads the files line by line, which is slower initially. Once
loaded there is no difference in the lookup algorithms, and the SQL DB
storage should be no different to how SG does it.

I would compare your custom script to the ext_sql_session_acl.pl.in
script we bundle with current Squid.
  If yours lacks concurrency channel-ID I highly recommend adding that
behaviour.
  If the DB is designed to store the protocol scheme, domain[:port] and
path?query portions of URLs in separate columns it will be more
efficient to pass those parameters as separate (%PROTO %DST %PORT %PATH)
to the helper instead of just %URI.

The overheads in Squid of using external_acl_type helper interface
should be slightly less than the url_rewrite_program for SG. The SQL DB
data loading is about the same or better than what SG does AFAIK.


>
> I took this snapshot yesterday:
>
> 15:25 08/11/2017:
>
> File descriptor usage for squid:
> Maximum number of file descriptors: 65536
> Largest file desc currently in use: 2730
> Number of file desc currently in use: 1838
> Files queued for open: 0
> Available number of file descriptors: 63698
> Reserved number of file descriptors: 100
> Store Disk files open: 0
>
> Today I took another peak and found:
>
> Thu Nov 9 12:19:05 CET 2017:
>
> File descriptor usage for squid:
> Maximum number of file descriptors: 65536
> Largest file desc currently in use: 6980
> Number of file desc currently in use: 6627
> Files queued for open: 0
> Available number of file descriptors: 58909
> Reserved number of file descriptors: 100
> Store Disk files open: 0
>
> The FDs are still increasing steadily, but a LOT less.
>
> On the other hand, the "free" RAM went from 2GB yesterday to just 275MB today:
>

Ouch, but kind of expected with those FD number increase. Each client
connection will use about 3 FD, and two of them will use ~70KB each and
the third will use some multiple of your avg object size.

Which reminds me ... Are you using SSL-Bump? if so ensure that you have
configured "sslflags=NO_DEFAULT_CA" on the port lines. The default
Trusted CA set can add a huge amount of useless memory to each client
connection, which can add up to many GB quite quickly.


> # free --mega
> total used free shared buff/cache available
> Mem: 32865 8685 275 157 23904 23683
> Swap: 37036 286 36750
>
> Used swap is still low enough (unchanged actually), so I guess I don't need to worry about it.

Nod, until the RAM runs out entirely, then problems are definitely to be
expected and that sounds like it is your problem now.

FYI: The first side-effect of RAM swapping is that Squid starts slowing
down on completed transactions - memory cache hits slow by 3-6 orders of
magnitude when swapping in/out from disk, and any I/O buffers swapped
out get 1+ speed penalties for the swap I/O time.
  That all leads to more active client connections overlapping their
active periods (thus more FD usage total), and also clients start
opening more connections to get better service from parallel fetches. So
FD usage gets growth from two directions simultaneously.
  Which is all in a feedback loop since that extra memory pressure from
more FD slows all transactions even further ... until the machine goes
crunch. Maybe a literal crunch - I actually physically blew up a test
machine (fire and smoke puring out the back!) measuring the effects of
RAID on a overloaded proxy about a decade ago.


>
> However, I'm bound to have issues when the "free" mem reaches 0... and I bet it will eventually.
> That's when the double-shutdown trick will kick in.
>
> I'll review the perl helper code, or maybe just switch to ufdbGuard.
>
> Thanks,
>
> Vieri
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri
BTW the docs say:

#         %PATH         Requested URL path
#         %METHOD       Request method
#         %MYADDR       Squid interface address
#         %MYPORT       Squid http_port number
#         %PATH         Requested URL-path (including query-string if any)


There's no such thing as %QUERY, right?
Squid cannot send "just the path" in one variable, and "just the query after ?" in another variable, right?
%PATH is the only thing I can read from to get both path and query (eg. "/some/path/?a=1&b=2")?

squid 3.5.27

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Amos Jeffries
Administrator
On 11/11/17 01:01, Vieri wrote:

> BTW the docs say:
>
> #         %PATH         Requested URL path
> #         %METHOD       Request method
> #         %MYADDR       Squid interface address
> #         %MYPORT       Squid http_port number
> #         %PATH         Requested URL-path (including query-string if any)
>
>
> There's no such thing as %QUERY, right?
> Squid cannot send "just the path" in one variable, and "just the query after ?" in another variable, right?
> %PATH is the only thing I can read from to get both path and query (eg. "/some/path/?a=1&b=2")?
>

Correct.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri
In reply to this post by Amos Jeffries

________________________________
From: Amos Jeffries <[hidden email]>

>
> I would compare your custom script to the ext_sql_session_acl.pl.in
> script we bundle with current Squid.
>  If yours lacks concurrency channel-ID I highly recommend adding that
> behaviour.
>  If the DB is designed to store the protocol scheme, domain[:port] and
> path?query portions of URLs in separate columns it will be more
> efficient to pass those parameters as separate (%PROTO %DST %PORT %PATH)
> to the helper instead of just %URI.
>
> The overheads in Squid of using external_acl_type helper interface
> should be slightly less than the url_rewrite_program for SG. The SQL DB
> data loading is about the same or better than what SG does AFAIK.


Thanks for the info. The script I was running was actually using concurrency with channel IDs. I also think it was correctly closing all file handles, DB connections, etc. However, I'm now merging your script which looks tidier into the one I was using. I'll see how it behaves over a few days.


> Ouch, but kind of expected with those FD number increase.

I'll have to find out why I'm still seeing this number rise albeit slower.
> Which reminds me ... Are you using SSL-Bump? if so ensure that you have
> configured "sslflags=NO_DEFAULT_CA" on the port lines. The default
> Trusted CA set can add a huge amount of useless memory to each client
> connection, which can add up to many GB quite quickly.


Many thanks. Applied. I also noticed that Squid 4 defaults to tls_default_ca=off. Will keep that in mind when migrating to v.4 (actually, I'll just need to remove sslflags=NO_DEFAULT_CA).

> Nod, until the RAM runs out entirely, then problems are definitely to be
> expected and that sounds like it is your problem now.


I really don't know how Linux manages memory, but despite my open FDs growing steadily and my "free" RAM slowly decreasing, at some point I noticed that the FDs kept growing slowly while the free mem suddenly went back up a bit (not a whole lot, but significantly -- around 0.5-0.7GB sudden increase).
> I actually physically blew up a test
> machine (fire and smoke puring out the back!) measuring the effects of
> RAID on a overloaded proxy about a decade ago.


I don't need this kind of horror stories... :-) Not yet at least. Let me first get a grip on my installation.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri
In reply to this post by Amos Jeffries

________________________________
From: Amos Jeffries <[hidden email]>

>>
>> File descriptor usage for squid:
>> Maximum number of file descriptors: 65536
>> Largest file desc currently in use: 6980
>> Number of file desc currently in use: 6627
>> Files queued for open: 0
>> Available number of file descriptors: 58909
>> Reserved number of file descriptors: 100
>> Store Disk files open: 0
>>
>> The FDs are still increasing steadily, but a LOT less.
>>
>> On the other hand, the "free" RAM went from 2GB yesterday to just 275MB today:
>
> Which reminds me ... Are you using SSL-Bump? if so ensure that you have

> configured "sslflags=NO_DEFAULT_CA"

FYI, this looks promising... I set "sslflags=NO_DEFAULT_CA". Today I had these readings:

File descriptor usage for squid:
Maximum number of file descriptors:   65536
Largest file desc currently in use:   7712
Number of file desc currently in use: 7220
Files queued for open:                   0
Available number of file descriptors: 58316
Reserved number of file descriptors:   100
Store Disk files open:                   0

# free --mega
total        used        free      shared  buff/cache   available
Mem:          32865        8235        3409         157       21221       24133
Swap:         37036         247       36789


So I'm having more open FDs than last time, but now I have more "free RAM" (3409MB instead of 275MB).


Thanks!

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri
In reply to this post by Amos Jeffries
________________________________
From: Amos Jeffries <[hidden email]>
>
> I would compare your custom script to the ext_sql_session_acl.pl.in
> script we bundle with current Squid.


I've rewritten my perl script, and have been running it for a full week now without any issues. Free RAM drops down to alarming values, but then rises back up again. In any case, "used swap" is always the same. The only thing that keeps be edgy is the fact that the open FDs keep growing (slowly but steadily). After a few days the value is around 6000, but after a week (today) it's:

Squid Object Cache: Version 3.5.27-20171101-re69e56c
Build Info:
Service Name: squid
Start Time:     Mon, 13 Nov 2017 11:06:36 GMT
Current Time:   Mon, 20 Nov 2017 08:48:00 GMT
Connection information for squid:
Number of clients accessing cache:      582
Number of HTTP requests received:       6435251
Number of ICP messages received:        0
Number of ICP messages sent:    0
Number of queued ICP replies:   0
Number of HTCP messages received:       0
Number of HTCP messages sent:   0
Request failure ratio:   0.00
Average HTTP requests per minute since start:   647.3
Average ICP messages per minute since start:    0.0
Select loop called: 246503925 times, 2.420 ms avg
Cache information for squid:
Hits as % of all requests:      5min: 4.4%, 60min: 4.3%
Hits as % of bytes sent:        5min: -0.7%, 60min: -6.0%
Memory hits as % of hit requests:       5min: 75.4%, 60min: 67.9%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
Storage Swap size:      29848 KB
Storage Swap capacity:  91.1% used,  8.9% free
Storage Mem size:       29120 KB
Storage Mem capacity:   88.9% used, 11.1% free
Mean Object Size:       13.19 KB
Requests given to unlinkd:      97921
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   0.18699  0.19742
Cache Misses:          0.19742  0.20843
Cache Hits:            0.00000  0.00000
Near Hits:             0.00000  0.27332
Not-Modified Replies:  0.00000  0.00000
DNS Lookups:           0.08334  0.07618
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time:        596484.490 seconds
CPU Time:       15823.550 seconds
CPU Usage:      2.65%
CPU Usage, 5 minute avg:        4.38%
CPU Usage, 60 minute avg:       4.86%
Maximum Resident Size: 14493888 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted:       -862888 KB
memPoolAlloc calls: 2199430697
memPoolFree calls:  2241183896
File descriptor usage for squid:
Maximum number of file descriptors:   65536
Largest file desc currently in use:   12714
Number of file desc currently in use: 11998
Files queued for open:                   0
Available number of file descriptors: 53538
Reserved number of file descriptors:   100
Store Disk files open:                   0
Internal Data Structures:
2520 StoreEntries
2519 StoreEntries with MemObjects
2314 Hot Object Cache Items
2263 on-disk objects


mgr:filedescriptors shows a great deal of these:

Remote Address        Description
--------------------- ------------------------------
127.0.0.1:1344        127.0.0.1


# squidclient mgr:filedescriptors | grep -c "127.0.0.1:1344"
11578


Port 1344 is where the c-icap daemon listens on.
This is the relevant part in squid.conf:

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024
icap_service squidclamav respmod_precache bypass=0 icap://127.0.0.1:1344/clamav
adaptation_access squidclamav allow all
icap_service_failure_limit -1


The number of connections to this port fluctuates over time (it also decreases), but overall it clearly increases day by day.
I could have an issue with either c-icap itself or one of its modules.
I'll keep an eye on it.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Amos Jeffries
Administrator
On 20/11/17 22:15, Vieri wrote:
> ________________________________
> From: Amos Jeffries <[hidden email]>
>>
>> I would compare your custom script to the ext_sql_session_acl.pl.in
>> script we bundle with current Squid.
>
>
> I've rewritten my perl script, and have been running it for a full week now without any issues. Free RAM drops down to alarming values, but then rises back up again. In any case, "used swap" is always the same. The only thing that keeps be edgy is the fact that the open FDs keep growing (slowly but steadily). After a few days the value is around 6000, but after a week (today) it's:
>


> Squid Object Cache: Version 3.5.27-20171101-re69e56c
> Build Info:
> Service Name: squid
> Start Time:     Mon, 13 Nov 2017 11:06:36 GMT
> Current Time:   Mon, 20 Nov 2017 08:48:00 GMT

> Average HTTP requests per minute since start:   647.3
...
> File descriptor usage for squid:
> Maximum number of file descriptors:   65536
> Largest file desc currently in use:   12714
> Number of file desc currently in use: 11998

So ~100 RPS, I would expect the open FDs used by ICAP to be around
299-300. Not the 11K+ you are seeing.

If we assume that each request opens a new connection and they are not
closed until TCP times out on the socket we do get numbers much more
like that 11K+ you are seeing.

That implies that ICAP transactions are probably not finishing
completely. Is the ICAP service finishing the ICAP reply messages
properly? (eg with a 0-size chunk)

Or maybe the error that led to you deciding to configure
"icap_service_failure_limit -1" was actually a real problem.


>
> mgr:filedescriptors shows a great deal of these:
>
> Remote Address        Description
> --------------------- ------------------------------
> 127.0.0.1:1344        127.0.0.1
>
>
> # squidclient mgr:filedescriptors | grep -c "127.0.0.1:1344"
> 11578
>
>
> Port 1344 is where the c-icap daemon listens on.
> This is the relevant part in squid.conf:
>
> icap_enable on
> icap_send_client_ip on
> icap_send_client_username on
> icap_client_username_encode off
> icap_client_username_header X-Authenticated-User
> icap_preview_enable on
> icap_preview_size 1024
> icap_service squidclamav respmod_precache bypass=0 icap://127.0.0.1:1344/clamav
> adaptation_access squidclamav allow all
> icap_service_failure_limit -1
>
>
> The number of connections to this port fluctuates over time (it also decreases), but overall it clearly increases day by day.
> I could have an issue with either c-icap itself or one of its modules.
> I'll keep an eye on it.

I think those 11K open FDs is sufficient evidence to say something is
wrong with the ICAP transactions. If you can get a packet trace of some
of some ICAP connections it might give better clues to what is happening
there.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Vieri

________________________________
From: Amos Jeffries <[hidden email]>
>
> If we assume that each request opens a new connection and they are not
> closed until TCP times out on the socket we do get numbers much more
> like that 11K+ you are seeing.
>
> That implies that ICAP transactions are probably not finishing

> completely.

I'll have to look into this asap. Quick question: if I restart c-icap shouldn't I see a drop in open FD numbers if it were c-icap's "fault"?

I restarted c-icap (stop+start), but the open FDs are the same.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: url_rewrite_program and ACLs

Amos Jeffries
Administrator
On 23/11/17 03:33, Vieri wrote:

>
> ________________________________
> From: Amos Jeffries <[hidden email]>
>>
>> If we assume that each request opens a new connection and they are not
>> closed until TCP times out on the socket we do get numbers much more
>> like that 11K+ you are seeing.
>>
>> That implies that ICAP transactions are probably not finishing
>
>> completely.
>
> I'll have to look into this asap. Quick question: if I restart c-icap shouldn't I see a drop in open FD numbers if it were c-icap's "fault"?
>
> I restarted c-icap (stop+start), but the open FDs are the same.
>

Maybe, but there have been a few bugs where Squid being tricked into
waiting on something arriving that was never coming left sockets in the
TCP half-open CLOSE_WAIT state indefinitely.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users