squid with quota limit using external helper problem !

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

squid with quota limit using external helper problem !

--Ahmad--
Hello squid folks .

I’m trying to use squid external helper to get quote to ips or users.

I’m following the wiki :


i have done everything my side on squid .

i have tested the connection :
root@localhost:~# /usr/local/bin/bandwidth_calculate /etc/squid/bandwidth_rules
root@localhost:~# 

no errors above !

#######################################

the issue I’m not sure if I’m using squid config file integration correctly .

here is my squid.conf file :

dns_v4_first on
acl localnet src all
auth_param basic program /lib/squid/basic_ncsa_auth  /etc/squid/squid_user
acl ncsa_users proxy_auth REQUIRED
auth_param basic children 1000
external_acl_type bandwidth_check ttl=60 %SRC /usr/local/bin/bandwidth_check
acl bandwidth_auth external bandwidth_check
http_access allow localnet bandwidth_auth
http_access deny  localnet !bandwidth_auth
###################################################
cache_effective_user squid
cache_effective_group squid
###########################################
http_access allow ncsa_users
############################
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_access deny all
http_port 3128
coredump_dir /var/spool/squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern . 

















here is errors i get :


2017/09/03 19:32:38 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '11.13.209.12'.
2017/09/03 19:38:31 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '11.13.209.12'.
2017/09/03 19:44:46 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '148.161.111.42'.
2017/09/03 19:44:47 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '148.161.111.42’.



but I’m sure 100 % that the ips above not blacklisted bec i check them over the helper :



root@localhost:~/squid-3.5.22# /usr/local/bin/bandwidth_check
11.13.209.12
OK
11.13.209.12
OK





root@localhost:~# cat /etc/squid/bandwidth_rules 
# A subnet
192.168.1.0/24        100mb/d 500mb/w    10gb/m
# A range
10.0.0.100-200        200mb/m
# A single IP
192.168.2.105        1gb/w 20gb/m
# A username
mike                 5gb/w
as1  10mb/d
hola    10mb/d
11.13.209.12           10mb/d





here is squid when it run :

root@localhost:~# tailf /var/log/squid/cache.log
2017/09/03 19:32:33 kid1| ERROR: Failed to create helper child read FD: TCP [::1]
2017/09/03 19:32:33 kid1| WARNING: Cannot run '/usr/local/bin/bandwidth_check' process.
2017/09/03 19:32:33 kid1| HTCP Disabled.
2017/09/03 19:32:33 kid1| Finished loading MIME types and icons.
2017/09/03 19:32:33 kid1| Accepting HTTP Socket connections at local=44.33.95.148:10001 remote=[::] FD 36 flags=9
2017/09/03 19:32:33 kid1| Accepting HTTP Socket connections at local=44.33.95.148:10002 remote=[::] FD 37 flags=9
2017/09/03 19:32:38 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '11.13.209.12'.
2017/09/03 19:38:31 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '11.13.209.12'.
2017/09/03 19:44:46 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '148.161.111.42'.
2017/09/03 19:44:47 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '148.161.111.42'.
2017/09/03 19:46:14 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '11.13.209.12’.




Guys i know this is not squid 100 % question .

plz don’t put me down and just guide me where to troubleshoot to figure out this issue .


many thanks !








_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid with quota limit using external helper problem !

Amos Jeffries
Administrator
On 04/09/17 07:49, --Ahmad-- wrote:

> Hello squid folks .
>
> I’m trying to use squid external helper to get quote to ips or users.
>
> I’m following the wiki :
>
> http://www.mikealeonetti.com/wiki/index.php?title=Squid_Arms_and_Tentacles:_Bandwidth_quotas
>
> i have done everything my side on squid .
>
> i have tested the connection :
> root@localhost:~# /usr/local/bin/bandwidth_calculate
> /etc/squid/bandwidth_rules
> root@localhost:~#
>
> no errors above !
>
> #######################################
>
> the issue I’m not sure if I’m using squid config file integration
> correctly .
>
> here is my squid.conf file :
>
> dns_v4_first on
> acl localnet src all

You have defined your LAN to be the entire Internet. Don't do that.

Define localnet to be your actual network ranges.

Use the provided 'all' ACL to refer to things that are allowed/denied to
everyone online. Most of the time 'all' is unnecessary.

If you expect clients from the general web to access your proxy and some
access control to apply to them, then simply do not limit those access
controls with the 'localnet' ACL.


> auth_param basic program /lib/squid/basic_ncsa_auth  /etc/squid/squid_user
> acl ncsa_users proxy_auth REQUIRED
> auth_param basic children 1000

How many users do expect exactly?

Squid de-duplicated overlapping Basic auth logins so one user can login
multiple times at once (ie login bursts when a Browser starts up) with
only one query sent to the auth helper. NCSA is also extremely fast lookups.

If you bumped that up because of the WARNING logged, then please change
your practices to fix ERRORs before WARNINGs.
* WARNINGs are logged for things Squid can workaround but needs help to
fix properly,
* ERRORs are things Squid cannot do anything about and need your attention,
* FATALs are things that are absolutely critical to fix if you are going
to use Squid at all.


> external_acl_type bandwidth_check ttl=60 %SRC /usr/local/bin/bandwidth_check

The ttl= parameter needs to be 0 for accurate bandwidth results. With
the above the helper is only checked once per minute, not on every request.
Keep in mind that you are only controlling whether new requests can
start, and once started they will complete. So regular re-checking is
required to minimize overages.

NP: negative_ttl= control how often Squid re-checks results from the
helper once users go over their quota. This is the option that you will
want to tune with non-0 values to reduce helper load, but also keep it
low enough not to block clients for too long after their quota renews.


> acl bandwidth_auth external bandwidth_check
> http_access allow localnet bandwidth_auth
> http_access deny  localnet !bandwidth_auth

The wiki is documenting the above two rules as *alternatives*. I suggest
you go back and read their descriptions, then pick the one that does
what you need.


> ###################################################
> cache_effective_user squid
> cache_effective_group squid
> ###########################################
> http_access allow ncsa_users

This will only login users that broadcast their credentials. It will not
require credentials from clients, and none of your below rules require
login to have happened.

Best practice for authentication is to place the rules applying to
non-authenticate clients first, then have:

   http_access deny !ncsa_users

... then to follow that with any rules applying to authenticated clients.


> ############################
> acl SSL_ports port 443
> acl Safe_ports port 80          # http
> acl Safe_ports port 21          # ftp
> acl Safe_ports port 443         # https
> acl Safe_ports port 70          # gopher
> acl Safe_ports port 210         # wais
> acl Safe_ports port 1025-65535  # unregistered ports
> acl Safe_ports port 280         # http-mgmt
> acl Safe_ports port 488         # gss-http
> acl Safe_ports port 591         # filemaker
> acl Safe_ports port 777         # multiling http
> acl CONNECT method CONNECT
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports

These Safe_ports and CONNECT rule need to be *above* all of your custom
rules. Otherwise they will have zero ability to protect your proxy
against the DoS and hijacking attacks they are supposed to prevent.

<snip>

>
> here is errors i get :
>
>
> 2017/09/03 19:32:38 kid1| WARNING: external ACL 'bandwidth_check' queue
> overload. Request rejected '11.13.209.12'.
> 2017/09/03 19:38:31 kid1| WARNING: external ACL 'bandwidth_check' queue
> overload. Request rejected '11.13.209.12'.
> 2017/09/03 19:44:46 kid1| WARNING: external ACL 'bandwidth_check' queue
> overload. Request rejected '148.161.111.42'.
> 2017/09/03 19:44:47 kid1| WARNING: external ACL 'bandwidth_check' queue
> overload. Request rejected '148.161.111.42’.
>
>
>
> but I’m sure 100 % that the ips above not blacklisted bec i check them
> over the helper :

Please re-read the WARNING message.

IPs are *not* being rejected because they are listed. They are being
rejected because the helper lookup queue is overloaded and no OK is
received.

>
> here is squid when it run :
>
> root@localhost:~# tailf /var/log/squid/cache.log
> 2017/09/03 19:32:33 kid1| ERROR: Failed to create helper child read FD:
> TCP [::1]

Fix that ERROR. The WARNING's about the helper and ACL checking are all
side effects of there not actually being a helper running.

There are several ways to do that:

1) fix the helpers IPv6 support. It seems not to have any, or if it does
is somehow still only using the IPv4-only address of localhost. Squid is
trying to contact it over an IPv6-v4-mapped address for localhost.


2) add the 'ipv4' option to your external_acl_type, to make Squid
temporarily be IPv4-only when talking to this helper.

While (2) is very tempting and easy, you will probably find that an
IPv4-only helper like this has errors when it gets told the IP address
of an IPv6 client. So (1) is the better option and I see the wiki page
author goes on about being happy to fix problem with their helper - just
get in touch.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid with quota limit using external helper problem !

--Ahmad--
Hi amos , thanks for the kind response .

i denied to rebuild squid without IPV6 support and seems now no error in helper .


i just curious to know about the auth directors in squid how should i arrange it :

acl localnet src all

auth_param basic program /lib/squid/basic_ncsa_auth  /etc/squid/squid_user
acl ncsa_users proxy_auth REQUIRED
auth_param basic children 1000

external_acl_type bandwidth_check ttl=0 %SRC /usr/local/bin/bandwidth_check
acl bandwidth_auth external bandwidth_check
http_access allow localnet bandwidth_auth
http_access deny  localnet !bandwidth_auth
###################################################
http_access allow ncsa_users


is above correct sequence to block any user exceeded quota ?
also should i use  
external_acl_type bandwidth_check ttl=0 %SRC /usr/local/bin/bandwidth_check

or

external_acl_type bandwidth_check ttl=0 %SRC %LOGIN /usr/local/bin/bandwidth_check

or 

external_acl_type bandwidth_check ttl=0  %EXT_USER /usr/local/bin/bandwidth_check


thanks amos in advance 
On Sep 4, 2017, at 8:10 AM, Amos Jeffries <[hidden email]> wrote:

On 04/09/17 07:49, --Ahmad-- wrote:
Hello squid folks .
I’m trying to use squid external helper to get quote to ips or users.
I’m following the wiki :
http://www.mikealeonetti.com/wiki/index.php?title=Squid_Arms_and_Tentacles:_Bandwidth_quotas
i have done everything my side on squid .
i have tested the connection :
root@localhost:~# /usr/local/bin/bandwidth_calculate /etc/squid/bandwidth_rules
root@localhost:~#
no errors above !
#######################################
the issue I’m not sure if I’m using squid config file integration correctly .
here is my squid.conf file :
dns_v4_first on
acl localnet src all

You have defined your LAN to be the entire Internet. Don't do that.

Define localnet to be your actual network ranges.

Use the provided 'all' ACL to refer to things that are allowed/denied to everyone online. Most of the time 'all' is unnecessary.

If you expect clients from the general web to access your proxy and some access control to apply to them, then simply do not limit those access controls with the 'localnet' ACL.


auth_param basic program /lib/squid/basic_ncsa_auth  /etc/squid/squid_user
acl ncsa_users proxy_auth REQUIRED
auth_param basic children 1000

How many users do expect exactly?

Squid de-duplicated overlapping Basic auth logins so one user can login multiple times at once (ie login bursts when a Browser starts up) with only one query sent to the auth helper. NCSA is also extremely fast lookups.

If you bumped that up because of the WARNING logged, then please change your practices to fix ERRORs before WARNINGs.
* WARNINGs are logged for things Squid can workaround but needs help to fix properly,
* ERRORs are things Squid cannot do anything about and need your attention,
* FATALs are things that are absolutely critical to fix if you are going to use Squid at all.


external_acl_type bandwidth_check ttl=60 %SRC /usr/local/bin/bandwidth_check

The ttl= parameter needs to be 0 for accurate bandwidth results. With the above the helper is only checked once per minute, not on every request.
Keep in mind that you are only controlling whether new requests can start, and once started they will complete. So regular re-checking is required to minimize overages.

NP: negative_ttl= control how often Squid re-checks results from the helper once users go over their quota. This is the option that you will want to tune with non-0 values to reduce helper load, but also keep it low enough not to block clients for too long after their quota renews.


acl bandwidth_auth external bandwidth_check
http_access allow localnet bandwidth_auth
http_access deny  localnet !bandwidth_auth

The wiki is documenting the above two rules as *alternatives*. I suggest you go back and read their descriptions, then pick the one that does what you need.


###################################################
cache_effective_user squid
cache_effective_group squid
###########################################
http_access allow ncsa_users

This will only login users that broadcast their credentials. It will not require credentials from clients, and none of your below rules require login to have happened.

Best practice for authentication is to place the rules applying to non-authenticate clients first, then have:

 http_access deny !ncsa_users

... then to follow that with any rules applying to authenticated clients.


############################
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

These Safe_ports and CONNECT rule need to be *above* all of your custom rules. Otherwise they will have zero ability to protect your proxy against the DoS and hijacking attacks they are supposed to prevent.

<snip>
here is errors i get :
2017/09/03 19:32:38 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '11.13.209.12'.
2017/09/03 19:38:31 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '11.13.209.12'.
2017/09/03 19:44:46 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '148.161.111.42'.
2017/09/03 19:44:47 kid1| WARNING: external ACL 'bandwidth_check' queue overload. Request rejected '148.161.111.42’.
but I’m sure 100 % that the ips above not blacklisted bec i check them over the helper :

Please re-read the WARNING message.

IPs are *not* being rejected because they are listed. They are being rejected because the helper lookup queue is overloaded and no OK is received.

here is squid when it run :
root@localhost:~# tailf /var/log/squid/cache.log
2017/09/03 19:32:33 kid1| ERROR: Failed to create helper child read FD: TCP [::1]

Fix that ERROR. The WARNING's about the helper and ACL checking are all side effects of there not actually being a helper running.

There are several ways to do that:

1) fix the helpers IPv6 support. It seems not to have any, or if it does is somehow still only using the IPv4-only address of localhost. Squid is trying to contact it over an IPv6-v4-mapped address for localhost.


2) add the 'ipv4' option to your external_acl_type, to make Squid temporarily be IPv4-only when talking to this helper.

While (2) is very tempting and easy, you will probably find that an IPv4-only helper like this has errors when it gets told the IP address of an IPv6 client. So (1) is the better option and I see the wiki page author goes on about being happy to fix problem with their helper - just get in touch.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid with quota limit using external helper problem !

Amos Jeffries
Administrator
In reply to this post by Amos Jeffries
On 04/09/17 21:09, Ahmed Alzaeem wrote:

> Hi amos , thanks for the kind response .
>
> i denied to rebuild squid without IPV6 support and seems now no error in
> helper .
>
>
> i just curious to know about the auth directors in squid how should i
> arrange it :
>
> acl localnet src all
>
> auth_param basic program /lib/squid/basic_ncsa_auth  /etc/squid/squid_user
> acl ncsa_users proxy_auth REQUIRED
> auth_param basic children 1000
>
> external_acl_type bandwidth_check ttl=0 %SRC /usr/local/bin/bandwidth_check
> acl bandwidth_auth external bandwidth_check
> http_access allow localnet bandwidth_auth
> http_access deny  localnet !bandwidth_auth
> ###################################################
> http_access allow ncsa_users
>
>
> is above correct sequence to block any user exceeded quota ?

I put comments under each problematic line in my last post about the
problems in that http_access sequence. The config has not changed, so
they are all still occuring.


> also should i use
> external_acl_type bandwidth_check ttl=0 %SRC /usr/local/bin/bandwidth_check
>
> or
>
> external_acl_type bandwidth_check ttl=0 %SRC %LOGIN
> /usr/local/bin/bandwidth_check
>
> or
>
> external_acl_type bandwidth_check ttl=0
> *%EXT_USER* /usr/local/bin/bandwidth_check
>

That is up to you, and depends on what you want the helper to be checking.

%LOGIN supplies the HTTP authentication login. It will trigger a full
authentication sequence if there are no credentials, so place all uses
of ACLs involving this after your ncsa_users login check.

%EXT_USER supplies the user= value some earlier external_acl_type helper
produced. You do not seem to have any other external ACL helpers - so
this is probably not for you.

If you have a mix of authentication methods happening you might want the
%un code.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users