Help with UA filtering in https connections

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with UA filtering in https connections

squidnoob
Hi there,

I'm a squid noob. I have been trying to configure squid for the past 3 days
looking high and low on the interwebs and have not found exactly what i'm
looking for.

Here's the context:
- the squid server is running in a server environment. It will not serve
end-users, but servers.
- privacy in regards to ssl interception is not a concern in this
environment.
- running squid: 3.5.23 on Ubuntu 16.04


Here are my goals:
- whitelist approach for domains. i.e. i only want a handful of domains to
be accessible.
- i want to allow certain UA's to bypass the whitelist rules. I know that
user agents are easy to spoof, but in this context and environment, it
doesn't matter.


I've pieced together the following configuration and have not been able to
figure this out. Any help is greatly appreciated!

---------------------------------------------squid.conf-------------------------------------------------
visible_hostname squid

acl CONNECT method CONNECT

access_log daemon:/var/log/squid/access.log combined

#Handling HTTP requests
http_port 3129 intercept
acl allowed_http_sites dstdomain "/etc/squid/http_allow_domains.txt"
http_access allow allowed_http_sites

#Handling HTTPS requests
https_port 3130 ssl-bump intercept cert=/etc/squid/ssl_cert/myCA.pem
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB


acl SSL_port port 443
http_access allow SSL_port

## This route does not work with UA processing below, but properly
terminates non-whitelisted sites
# The ssl::server_name ACL will not work outside of the ssl_bump directive.
acl allowed_https_sites ssl::server_name "/etc/squid/http_allow_domains.txt"
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump peek step2 allowed_https_sites
ssl_bump splice step3 allowed_https_sites
ssl_bump terminate step2 all
##


## This route does not work at all at preventing non-whitelisted sites
#acl allowed_https_sites ssl::server_name
"/etc/squid/http_allow_domains.txt"
#acl step1 at_step SslBump1
#acl step2 at_step SslBump2
#acl step3 at_step SslBump3
#ssl_bump peek step1 all
#ssl_bump peek step2 allowed_https_sites
#ssl_bump splice step3 allowed_https_sites
#ssl_bump bump all
##
 

## Bypass the proxy by UA
acl proxy_bypass_ua browser ^python-requests.*$
http_access allow proxy_bypass_ua



# And finally deny all other access to this proxy
http_access deny all







--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Alex Rousskov
On 12/28/2017 03:59 PM, squidnoob wrote:

> Here are my goals:
> - i only want a handful of domains to be accessible.
> - i want to allow certain UA's to bypass the whitelist rules.

Since you appear to have full control over the environment, have you
tried bumping everything and applying your access rules to bumped (or
plain) traffic?


  # bump everything
  ssl_bump stare all
  ssl_bump bump all

  # delay filtering decisions until we get to bumped requests
  http_access allow CONNECT toSafePorts
  http_access deny CONNECT

  # filter plain and bumped requests
  http_access allow certainUserAgents
  http_access allow handfulOfDomains
  http_access deny all


The above allows all (safe) CONNECTs in case some CONNECT requests do
not have User-Agent headers or lack other details important for your
certainUserAgents and handfulOfDomains ACLs. Since you are bumping all
those allowed CONNECTs and validating all "real" requests inside bumped
tunnels, allowing all (safe) CONNECTs does not contradict your goals AFAICT.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Amos Jeffries
Administrator
In reply to this post by squidnoob
On 2017-12-29 11:59, squidnoob wrote:

> Hi there,
>
> I'm a squid noob. I have been trying to configure squid for the past 3
> days
> looking high and low on the interwebs and have not found exactly what
> i'm
> looking for.
>
> Here's the context:
> - the squid server is running in a server environment. It will not
> serve
> end-users, but servers.

Is that server<->server traffic you are talking about here?
or the proxy acting as CDN frontend for some servers?

The difference will determine what config is applicable.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

squidnoob
In reply to this post by Alex Rousskov
Ahh that's it! Thank you for your help!

For anyone interested, i'm posting the working config i'm using. Hopefully
this helps someone.


#
# Working on squid version: 3.5.23
#
# The general purpose of this configuration is:
# - only allow a set of whitelisted domains through the proxy
# - option to allow specific browser user agents to bypass the domains
whitelist
# - option to allow specific hosts to bypass the domains whitelist
# - option to allow speicfic host + user agent to bypass the domains
whitelist
#
# Useful in a restricted environment, like a server environment with
restricted egress requirements.
#
# Requirements for this to work properly
#
# On proxy host:
#   iptables rules to support redirection to appropriate ports
#     iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port
3129
#     iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port
3130
#
#   Self-signed cert route:
#     openssl req -newkey rsa:4096 -x509 -keyout
/etc/squid/ssl_cert/mySquidCA.pem -out /etc/squid/mySquidCA.pem -days 1825
-nodes
#
# On clients
#   For self-signed cert route:
#     Add public key of mySquidCA cert to appropriate stores
#     e.g. Ubuntu 16.04, add public key of the .pem file to:
/usr/local/share/ca-certificates/mySquidCA.crt and then run sudo
update-ca-certs
#
#     If running python, may need to update appropriate package cert stores:
#     e.g. /usr/local/lib/python2.7/dist-packages/requests/cacert.pem
#
# Refs
#   - install 3.5.23:
https://docs.diladele.com/howtos/build_squid_ubuntu16/index.html
#   - example:
https://aws.amazon.com/blogs/security/how-to-add-dns-filtering-to-your-nat-instance-with-squid/ 
#   - http://www.squid-cache.org/Doc/
#

visible_hostname squid

# The default log formats available (which do not need re-defining) are:
#logformat combined   %>a %[ui %[un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st
"%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
access_log daemon:/var/log/squid/access.log combined


# acls for ports allowed
acl safe_ports port 80          # http
acl safe_ports port 443         # https

# acl for whitelisting domains
acl whitelist_domains dstdomain "/etc/squid/whitelist_domains.txt"

# acl for browser user agents
acl useragent_bypass browser "/etc/squid/useragents_bypass_regex.txt"

# acl for hosts
acl host_bypass src "/etc/squid/hosts_bypass.txt"

# acls for use with host AND user agent combo rule
acl host_and_useragent_ualist_bypass browser
"/etc/squid/host_AND_useragent_useragentlist_bypass.txt"
acl host_and_useragent_hostlist_bypass src
"/etc/squid/host_AND_useragent_hostlist_bypass.txt"


acl CONNECT method CONNECT



#Handling HTTP requests
#http_port 3128         # will need this live for squid v4
http_port 3129 intercept

#Handling HTTPS requests
# transparent proxy option
#https_port 3130 cert=/etc/squid/ssl/squid.pem ssl-bump intercept

# full ssl intercept option
https_port 3130 ssl-bump intercept cert=/etc/squid/ssl_cert/mySquidCA.pem
generate-host-certificates=on dynamic_cert_mem_cache_size=10MB
sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 10MB

# for ver 4.x
#sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M
10MB


# bump everything
ssl_bump stare all
ssl_bump bump all

# delay filtering decisions until we get to bumped requests
http_access allow CONNECT safe_ports
http_access deny CONNECT

# filter plain and bumped requests
# allow specified hosts to bypass
http_access allow host_bypass

# allow specified useragents to bypass
http_access allow useragent_bypass

# allow combo of host + useragent to bypass
http_access allow host_and_useragent_ualist_bypass
host_and_useragent_hostlist_bypass

# allow only whitelisted domains if above rules haven't bypassed it yet
http_access allow whitelist_domains

# finally, deny all other access to this proxy
http_access deny all




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

squidnoob
In reply to this post by Amos Jeffries
Yes to clarify, this is basically trying to filter server egress traffic to
the internet. It's not for internal server to other internal server traffic.




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Amos Jeffries
Administrator
In reply to this post by squidnoob
On 30/12/17 05:32, squidnoob wrote:
> Ahh that's it! Thank you for your help!
>
> For anyone interested, i'm posting the working config i'm using. Hopefully
> this helps someone.
>

This config allows clients to tunnel arbitrary traffic through your
proxy to another one listening on port 80 without any controls.

Please restore the default security settings as your first http_access
lines:

   http_access deny !Safe_Ports
   http_access deny CONNECT !SSL_Ports

... your existing config should still work fine when placed after those.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

squidnoob
In my existing config, i have:

# delay filtering decisions until we get to bumped requests
http_access allow CONNECT safe_ports
http_access deny CONNECT


I understand adding this line that you suggested as it's not already there.
http_access deny !safe_ports

However, i don't understand why i would need to add this (http_access deny
CONNECT !SSL_Ports ) given the two lines above in the existing config. I'm
probably just misunderstanding how this works.

Thank you for the help!



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Matus UHLAR - fantomas
On 02.01.18 06:04, squidnoob wrote:

>In my existing config, i have:
>
># delay filtering decisions until we get to bumped requests
>http_access allow CONNECT safe_ports
>http_access deny CONNECT
>
>
>I understand adding this line that you suggested as it's not already there.
>http_access deny !safe_ports
>
>However, i don't understand why i would need to add this (http_access deny
>CONNECT !SSL_Ports ) given the two lines above in the existing config. I'm
>probably just misunderstanding how this works.

the two lines above unconditionally allow CONNECT anywhere, you can't deny
it further because no further checking is done.

when using:

http_access deny CONNECT !SSL_ports

you deny CONNECT request to non-SSL ports and can deny them further.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
99 percent of lawyers give the rest a bad name.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Alex Rousskov
On 01/02/2018 07:08 AM, Matus UHLAR - fantomas wrote:
> On 02.01.18 06:04, squidnoob wrote:
>> http_access allow CONNECT safe_ports
>> http_access deny CONNECT


>> I understand adding this line that you suggested as it's not already
>> there.
>> http_access deny !safe_ports

Yes, this or similar line (and possibly other lines) is needed, provided
it matches your proxying environment. My sketch only dealt with your
original/specific problem, not general proxying protections...


>> However, i don't understand why i would need to add this (http_access
>> deny CONNECT !SSL_Ports ) given the two lines above in the existing config.

You do not need to add it AFAICT.


> the two lines above unconditionally allow CONNECT anywhere,

This is incorrect. The lines deny CONNECT to unsafe ports. What Amos
correctly pointed out is that *non-CONNECT* transactions may go to
unsafe ports as well, and it is considered best practice to block such
traffic by default.

Please note that denying CONNECTs to unsafe ports at step1 may not work
well because the generated by Squid certificate will be rejected by the
browser in many cases. You may decide to simply terminate such CONNECT
transactions instead:

  # terminate malicious tunnels and bump everything else
  ssl_bump terminate !safe_ports
  ssl_bump stare all
  ssl_bump bump all


Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Matus UHLAR - fantomas
On 02.01.18 09:06, Alex Rousskov wrote:
>On 01/02/2018 07:08 AM, Matus UHLAR - fantomas wrote:
>> On 02.01.18 06:04, squidnoob wrote:
>>> http_access allow CONNECT safe_ports
>>> http_access deny CONNECT

>> the two lines above unconditionally allow CONNECT anywhere,
>
>This is incorrect. The lines deny CONNECT to unsafe ports.

You miss something.

Those lines unconditionally allow CONNECT requests to safe ports ANYWHERE,
which is apparently not what was wanted/expected.

the first line ALLOWS all CONNECT requests to safe ports in the way they
CAN NOT BE DISABLED later.

the second line denies connect to unsafe ports.

the difference between lines above and the following one:

http_access deny CONNECT !safe_ports

is, that in this case you can deny the connect request later, unlike the
previous example, where the CONNECT was allowed and further checks are done.

However, what Amos proposed and what is in the default config is:

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

which denies all access to unsafe ports, and denies CONNECT to non-SSL
ports, but does not allow access anywhere, so it must be allowed further.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I wonder how much deeper the ocean would be without sponges.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Matus UHLAR - fantomas
On 03.01.18 13:52, Matus UHLAR - fantomas wrote:
>http_access deny CONNECT !safe_ports
>
>... in this case you can deny the connect request later, unlike the
>previous example, where the CONNECT was allowed and further checks are done.

corrected: _no_ futher checks are done.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Nothing is fool-proof to a talented fool.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Alex Rousskov
In reply to this post by Matus UHLAR - fantomas
On 01/03/2018 05:52 AM, Matus UHLAR - fantomas wrote:
> On 02.01.18 09:06, Alex Rousskov wrote:
>> On 01/02/2018 07:08 AM, Matus UHLAR - fantomas wrote:
>>> On 02.01.18 06:04, squidnoob wrote:
>>>> http_access allow CONNECT safe_ports
>>>> http_access deny CONNECT

>>> the two lines above unconditionally allow CONNECT anywhere,

>> This is incorrect. The lines deny CONNECT to unsafe ports.

> Those lines unconditionally allow CONNECT requests to safe ports ANYWHERE,

Yes, or, to be more precise, they (together with ssl_bump rules) allow
fetching of any server certificate from a reasonable(*) port. They do
not allow HTTP requests to arbitrary safe ports. Only Squid-generated
TLS handshakes.


> which is apparently not what was wanted/expected.

Why not?


> that in this case you can[not] deny the connect request later,

Denying CONNECTs at step1 does not really work well in a general case
because, during SslBump step1, Squid does not have enough information to
generate the right certificate for the access denial error page.

In a general case, the admin has to pick between two evils:

* Allow TLS handshakes with arbitrary servers on TLS ports (my sketch)

* or tell Squid to respond with error pages that the user cannot see
  (without bypassing browser security warnings).

Which evil is lesser is up to the admin to decide. Needless to say,
there are environments where both strategies should be used, depending
on the transaction parameters.


(*) We should allow CONNECTs to SSL_ports, not Safe_ports. I hope my
sketch did not use those ACLs.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Matus UHLAR - fantomas
>On 01/03/2018 05:52 AM, Matus UHLAR - fantomas wrote:
>> On 02.01.18 09:06, Alex Rousskov wrote:
>>> On 01/02/2018 07:08 AM, Matus UHLAR - fantomas wrote:
>>>> On 02.01.18 06:04, squidnoob wrote:
>>>>> http_access allow CONNECT safe_ports
>>>>> http_access deny CONNECT
>
>>>> the two lines above unconditionally allow CONNECT anywhere,
>
>>> This is incorrect. The lines deny CONNECT to unsafe ports.
>
>> Those lines unconditionally allow CONNECT requests to safe ports ANYWHERE,

On 03.01.18 08:55, Alex Rousskov wrote:
>Yes, or, to be more precise, they (together with ssl_bump rules) allow
>fetching of any server certificate from a reasonable(*) port. They do
>not allow HTTP requests to arbitrary safe ports. Only Squid-generated
>TLS handshakes.


>> which is apparently not what was wanted/expected.
>
>Why not?

because there can be many reasons to deny CONNECT request for example
destined to localhost or internal network.

in the default config, these directives are at the beginning, before
checking for allowed clients and destinations is done.

in the provided config:
http://lists.squid-cache.org/pipermail/squid-users/2017-December/017267.html

there are no deny directives before "http_access allow SSL_port"
and so it's quite possible that all clients that should not have access will
be allowed.

of course, there MAY be other directives or measures to avoid that
but I really think it's better to put "deny CONNECT !SSL_ports"
than allow CONNECT and later wonder why some requests are not


>> that in this case you can[not] deny the connect request later,
>
>Denying CONNECTs at step1 does not really work well in a general case
>because, during SslBump step1, Squid does not have enough information to
>generate the right certificate for the access denial error page.

I don't think this matters when we do have "http_access deny CONNECT" in
both cases.

>In a general case, the admin has to pick between two evils:
>
>* Allow TLS handshakes with arbitrary servers on TLS ports (my sketch)
>
>* or tell Squid to respond with error pages that the user cannot see
>  (without bypassing browser security warnings).
>
>Which evil is lesser is up to the admin to decide. Needless to say,
>there are environments where both strategies should be used, depending
>on the transaction parameters.
>
>
>(*) We should allow CONNECTs to SSL_ports, not Safe_ports. I hope my
>sketch did not use those ACLs.

I'm afraid you did.
+and I'm also afraid that your proposal also prevents us from disabling
CONNECTs later:

http://lists.squid-cache.org/pipermail/squid-users/2017-December/017268.html

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Atheism is a non-prophet organization.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Help with UA filtering in https connections

Alex Rousskov
On 01/03/2018 10:38 AM, Matus UHLAR - fantomas wrote:

>> In a general case, the admin has to pick between two evils:
>>
>> * Allow TLS handshakes with arbitrary servers on TLS ports (my sketch)
>>
>> * or tell Squid to respond with error pages that the user cannot see
>>  (without bypassing browser security warnings).
>>
>> Which evil is lesser is up to the admin to decide.


>> (*) We should allow CONNECTs to SSL_ports, not Safe_ports. I hope my
>> sketch did not use those ACLs.

> I'm afraid you did.

I did not:
http://lists.squid-cache.org/pipermail/squid-users/2017-December/017268.html

I used toSafePorts which is not one of the default ACLs (but may contain
them). The admin should define the ACLs left out of the sketch
correctly, of course. Moreover, I would rename toSafePorts to
toConnectableDestinations or similar to emphasize that this is the right
place to ban CONNECTs to wrong/dangerous/etc. addresses.


> I'm also afraid that your proposal also prevents us from disabling
> CONNECTs later

If you are saying that my simple sketch does not address all possible
use cases, then I certainly agree! I believe it addressed what OP
requested, but if I misinterpreted his or her desires, I apologize. I
hope the general description quoted at the start of this email combined
with Amos and yours warnings about undesirable CONNECT destinations will
allow them to fix their configuration as needed.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users