Best practice for adding or removing ACLs dynamically ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Best practice for adding or removing ACLs dynamically ?

Roeeklinger60

Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002
http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to change it in squid.conf and also in htpassword and then I run "squid -k reconfigure", if I don't reconfigure the old user still has access to the proxy and the new one doesn't for about 30 minutes.

I am expecting to have 100s of users soon that will change credentials often, and also I would like to blacklist websites often and on the fly, so I was searching for a better way to manage this without reconfiguring every time, since sometimes a reconfigure can take up to 10-15 seconds.

I am new to Squid and wasn't able to find any info on this, am I doing this currently or there is a better way to change users/ACLs on the fly without reloading Squid?

Thanks,
Roee Klinger
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Best practice for adding or removing ACLs dynamically ?

Amos Jeffries
Administrator
On 31/10/20 1:34 pm, roee klinger wrote:
> 
> Hey,
> I have Squid configured to send users to different outgoing interface like so:
>
> ..
> auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
> acl acl_for_user3002 proxy_auth user2
> tcp_outgoing_address 192.168.8.12 acl_for_user3002
> http_port 3002 name=3002


No need to name a *_port like this. The default name is the first
parameter string ("3002" on this line).


> http_access allow authenticated
> ..
>
> When I wanted to change the username:password for user2, I run a bash script to change it in squid.conf and also in htpassword and then I run "squid -k reconfigure", if I don't reconfigure the old user still has access to the proxy and the new one doesn't for about 30 minutes.
>

No need to restart for that change. The helper you have there will
automatically detect changes to the htpassword file and reload it.

It is a little odd that the new user was not able to authenticate. Check
that your test did not lookup and cache a non-existence result for them
prior to being added.


The delay is due to the credentials being valid for a period of time. To
reduce workload on the auth system Squid caches credential details for a
while.

Set "auth_param basic credentialsttl " to shorter values to reduce the
delay (default is 2hrs).


> I am expecting to have 100s of users soon that will change credentials often, and also I would like to blacklist websites often and on the fly, so I was searching for a better way to manage this without reconfiguring every time, since sometimes a reconfigure can take up to 10-15 seconds.
>

This helper does not need a reconfigure at all as far as I can tell from
the code.

All the reconfigure was doing for you previously was triggering an early
prune of the records in the credentials cache. Probably why you saw
about 30min delay instead of about 2hrs.


> I am new to Squid and wasn't able to find any info on this, am I doing this currently or there is a better way to change users/ACLs on the fly without reloading Squid?
>

Config changes in squid.conf itself needs a reconfigure or sometimes a
restart.


For auth and ACLs whose values that come into Squid from a helper it
depends on the helper itself. Most can auto-detect changes to their
background databases and not need anything from Squid to update the
outputs. All helpers do have some form of caching of their results by
Squid, so there are settings in squid.conf to tune that to your needs -
as you can see from the auth issue above.


For ACLs with values that are expected to change often it is best to use
an external_acl_type helper that manages the updates or fetches from
somewhere the updates are handled without a reload.



Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Best practice for adding or removing ACLs dynamically ?

Roeeklinger60
Thanks Amos!

I updated "auth_param basic credentialsttl" according to your advice and it is working great.

I am still having issues with the "tcp_outgoing_address 192.168.8.12 acl_for_user3002" part, you mentioned:
> For ACLs with values that are expected to change often it is best to use
> an external_acl_type helper that manages the updates or fetches from
> somewhere the updates are handled without a reload.

My script updates the authenticator successfully, but when I update "acl acl_for_user3002 proxy_auth user2" to the new username I have to reconfigure to take effect.
I read online for hours but to my best understanding external_acl_type are for auth and access control, but they don't work for my needs I believe.

Is there any way to use external_acl_type in a way I don't understand to solve this problem? Do I have to reconfigure every time I make changes to an ACL in squid.conf?

Thanks again for your help.

On Sat, Oct 31, 2020 at 5:48 PM Amos Jeffries <[hidden email]> wrote:
On 31/10/20 1:34 pm, roee klinger wrote:
> 
> Hey,
> I have Squid configured to send users to different outgoing interface like so:
>
> ..
> auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
> acl acl_for_user3002 proxy_auth user2
> tcp_outgoing_address 192.168.8.12 acl_for_user3002
> http_port 3002 name=3002


No need to name a *_port like this. The default name is the first
parameter string ("3002" on this line).


> http_access allow authenticated
> ..
>
> When I wanted to change the username:password for user2, I run a bash script to change it in squid.conf and also in htpassword and then I run "squid -k reconfigure", if I don't reconfigure the old user still has access to the proxy and the new one doesn't for about 30 minutes.
>

No need to restart for that change. The helper you have there will
automatically detect changes to the htpassword file and reload it.

It is a little odd that the new user was not able to authenticate. Check
that your test did not lookup and cache a non-existence result for them
prior to being added.


The delay is due to the credentials being valid for a period of time. To
reduce workload on the auth system Squid caches credential details for a
while.

Set "auth_param basic credentialsttl " to shorter values to reduce the
delay (default is 2hrs).


> I am expecting to have 100s of users soon that will change credentials often, and also I would like to blacklist websites often and on the fly, so I was searching for a better way to manage this without reconfiguring every time, since sometimes a reconfigure can take up to 10-15 seconds.
>

This helper does not need a reconfigure at all as far as I can tell from
the code.

All the reconfigure was doing for you previously was triggering an early
prune of the records in the credentials cache. Probably why you saw
about 30min delay instead of about 2hrs.


> I am new to Squid and wasn't able to find any info on this, am I doing this currently or there is a better way to change users/ACLs on the fly without reloading Squid?
>

Config changes in squid.conf itself needs a reconfigure or sometimes a
restart.


For auth and ACLs whose values that come into Squid from a helper it
depends on the helper itself. Most can auto-detect changes to their
background databases and not need anything from Squid to update the
outputs. All helpers do have some form of caching of their results by
Squid, so there are settings in squid.conf to tune that to your needs -
as you can see from the auth issue above.


For ACLs with values that are expected to change often it is best to use
an external_acl_type helper that manages the updates or fetches from
somewhere the updates are handled without a reload.



Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Best practice for adding or removing ACLs dynamically ?

Eliezer Croitoru-3
In reply to this post by Roeeklinger60
Hey Roee,

I am trying to understand what part of squid.conf you want to be able to change without a reconfigure/reload?
If you have many users, ie above 50 you should probably not use a simple ncsa_auth although  it's possible and in more then one case is preferable.
You could probably write your own basic auth helper that will interact with a DB which will probably simplify your whole setup.
(You can use existing basic auth helpers with mysql or ldap)

As for the tcp_outgoing_address, it’s a whole different story.
Since it's a "fast" acl type the options to do something dynamic with it are an issue.
(Maybe eCAP/ICAP service or a "pre-cooked" note or other factor to the acl can be used)

I am pretty sure that if an authentication service can reply with a note ie connection annotation then it can be used for the address selection.
One issue with it is that It will be valid for the next X ttl seconds/minutes/hours.

I do believe that there should be a way to allow something like external_acl helper to affect this squid feature.
I was thinking that an eCAP or an ICAP service or an external_acl helper can add a note for a connection based on couple other factors like:
* src ip
* auth username
* request domain or request sni
* ...

So let say the proxy will have a set of 100 addresses, each will have a single specific matching acl for a request header or connection annotation/note.
This way the selection of a tcp_outgoing_address would be a little less complex the it is today.

I have couple other ideas for implementations which I have experimented with but the proxy admin need to learn how these work which might be
a bit complicated some times.

Eliezer
----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]

-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of roee klinger
Sent: Saturday, October 31, 2020 2:35 AM
To: [hidden email]
Subject: [squid-users] Best practice for adding or removing ACLs dynamically ?


Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002
http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to change it in squid.conf and also in htpassword and then I run "squid -k reconfigure", if I don't reconfigure the old user still has access to the proxy and the new one doesn't for about 30 minutes.

I am expecting to have 100s of users soon that will change credentials often, and also I would like to blacklist websites often and on the fly, so I was searching for a better way to manage this without reconfiguring every time, since sometimes a reconfigure can take up to 10-15 seconds.

I am new to Squid and wasn't able to find any info on this, am I doing this currently or there is a better way to change users/ACLs on the fly without reloading Squid?

Thanks,
Roee Klinger
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Best practice for adding or removing ACLs dynamically ?

Amos Jeffries
Administrator
In reply to this post by Roeeklinger60
On 1/11/20 12:27 pm, roee klinger wrote:

> Thanks Amos!
>
> I updated "auth_param basic credentialsttl" according to your advice and
> it is working great.
>
> I am still having issues with the "tcp_outgoing_address 192.168.8.12
> acl_for_user3002" part, you mentioned:
>  > For ACLs with values that are expected to change often it is best to use
>  > an external_acl_type helper that manages the updates or fetches from
>  > somewhere the updates are handled without a reload.
>
> My script updates the authenticator successfully, but when I update "acl
> acl_for_user3002 proxy_auth user2" to the new username I have to
> reconfigure to take effect.
> I read online for hours but to my best understanding external_acl_type
> are for auth and access control, but they don't work for my needs I believe.
>
> Is there any way to use external_acl_type in a way I don't understand to
> solve this problem? Do I have to reconfigure every time I make changes
> to an ACL in squid.conf?


Some directives have to produce allow/deny result immediately, without
waiting for a helper to respond. The details are documented here:
  <https://wiki.squid-cache.org/SquidFaq/SquidAcl>

In modern Squid you can use a helper to set annotations which are
checked with the "note" ACL type in the fast checks.



It sounds a bit like you are trying to tie IPs to individual users.
Please be aware that breaks the multiplexing and persistence features of
HTTP, which is a major performance loss.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Best practice for adding or removing ACLs dynamically ?

Eliezer Croitoru-3
In reply to this post by Eliezer Croitoru-3
Following up.

Well, github and google is full of horrible ways of implementing the tcp_outgoing_address selection so..
I have written a sketch of the concept I was writing about based on ncsa_auth which I must admit is far more .. simple then using a DB like mysql or LDAP.

The config I found was using either:
- multiple inbound squid ports -> mapped each to a single tcp_outgoing_addrees
- multiple static usernames mapped to a specific address(based on either username or usernamer+dst-domain.. else ACLs types)
- Others which are not relevant since these break the internet

What I am offering is an example which offloads the actual selection code/algorithm and config into an external_acl_type helper.
The result would be that squid will only listen(when possible and required) only on one port.
(which means that adding more ports can multiply/power up the service config "capacity", like vlans ontop of vlans)
Also the authentication mechanism would be ncsa_auth which is good for up to 1000s++ of users since the helpers store the htaccess file In RAM.
Even if the file will contain 10k users and the auth helpers in use would be let say 40 and up, the proxy would still consume less Memory on the
storage of the htaccess file in RAM compared to a MySQL or LDAP DB.

To change an address for a specific existing user you can use a simple sed command:
sed -i -E "s@user80[\ \t]+[0-9]+@user80 1@g" /etc/squid/user-to-ip.txt

and to change password for a user you can use a simple htaccess command:
htpasswd -b /etc/squid/htpasswd  user80 1234


Within my external_acl_helper I have used a similar approach to the one the ncsa_auth helper uses to reload modified files from disk.
(If the mtime stamp is being changed from the old one the helper would reload the mapping config file from disk)
* https://github.com/elico/vagrant-squid-outgoing-addresses/blob/master/shared/note.rb

To demonstrate this piece of code I have used Vagrant on-top of Virtualbox.
(I have verified that on the current Windows 10 PRO installations both Virtualbox and Hyper-v can be used simultaneously with CPU Virtualization support ON)

The example lab is at:
https://github.com/elico/vagrant-squid-outgoing-addresses

The test is composed of two nodes:
* Squid
* Web

The Squid instance will be populated with more then 20 IP addresses on the same subnet.
For simplicity I have added a simple /32 address to the Squid+Web private/closed network interface.
There are other ways which can be used in production when Routing daemons are in the picture.

To start the demo/test you can use:
vagrant up
vagrant ssh squid --command /vagrant/test-random-ips-assignment.sh

The user to ip config mapping file is at:
/etc/squid/user-to-ip.txt

I have used a simple "<user> <tab or space> <ip integer id>" mapping syntax.

Technically speaking this setup can work with scales of 1000s and up of addresses and clients.
I am pretty sure it's good enough for proxy services which uses multiple routing tunnels and/or BGP feeds.

If anyone wants to ask about this setup I will be more than happy to receive emails about it here in the list.

All The Bests,

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]

-----Original Message-----
From: Eliezer Croitor <[hidden email]>
Sent: Sunday, November 1, 2020 2:41 AM
To: [hidden email]
Cc: 'roee klinger' <[hidden email]>
Subject: RE: [squid-users] Best practice for adding or removing ACLs dynamically ?

Hey Roee,

I am trying to understand what part of squid.conf you want to be able to change without a reconfigure/reload?
If you have many users, ie above 50 you should probably not use a simple ncsa_auth although  it's possible and in more then one case is preferable.
You could probably write your own basic auth helper that will interact with a DB which will probably simplify your whole setup.
(You can use existing basic auth helpers with mysql or ldap)

As for the tcp_outgoing_address, it’s a whole different story.
Since it's a "fast" acl type the options to do something dynamic with it are an issue.
(Maybe eCAP/ICAP service or a "pre-cooked" note or other factor to the acl can be used)

I am pretty sure that if an authentication service can reply with a note ie connection annotation then it can be used for the address selection.
One issue with it is that It will be valid for the next X ttl seconds/minutes/hours.

I do believe that there should be a way to allow something like external_acl helper to affect this squid feature.
I was thinking that an eCAP or an ICAP service or an external_acl helper can add a note for a connection based on couple other factors like:
* src ip
* auth username
* request domain or request sni
* ...

So let say the proxy will have a set of 100 addresses, each will have a single specific matching acl for a request header or connection annotation/note.
This way the selection of a tcp_outgoing_address would be a little less complex the it is today.

I have couple other ideas for implementations which I have experimented with but the proxy admin need to learn how these work which might be
a bit complicated some times.

Eliezer
----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]

-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of roee klinger
Sent: Saturday, October 31, 2020 2:35 AM
To: [hidden email]
Subject: [squid-users] Best practice for adding or removing ACLs dynamically ?


Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002
http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to change it in squid.conf and also in htpassword and then I run "squid -k reconfigure", if I don't reconfigure the old user still has access to the proxy and the new one doesn't for about 30 minutes.

I am expecting to have 100s of users soon that will change credentials often, and also I would like to blacklist websites often and on the fly, so I was searching for a better way to manage this without reconfiguring every time, since sometimes a reconfigure can take up to 10-15 seconds.

I am new to Squid and wasn't able to find any info on this, am I doing this currently or there is a better way to change users/ACLs on the fly without reloading Squid?

Thanks,
Roee Klinger
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users