A patch for intercepted/WCCP HTTPS and 409 errors

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

A patch for intercepted/WCCP HTTPS and 409 errors

Scott
Hi,

I understand that squid does some security checking that the SNI of an
intercepted/WCCP HTTPS requests matches the reverse DNS of the IP of the
connection.  Or something like that.

However with the prevalence of CDNs and badly configured DNSs and geographic
DNSs, this breaks lots of connections (eg, I can't watch the NHL).

I run Squid on a trusted network and use it primarily for caching and
logging, and so I while I need to run WCCP for some non-proxy capable
devices, I don't need that security check.

It stops all of those 409 errors occurring.

Because of that I've created some patches that add a new option
"host_verify_strict_intercepted" which is off by default.  They are
for Squid 4.9.  As this is disabling a security feature of Squid do
not apply this patch unless you are prepared for any and all consequences.

--- cf.data.pre 2019-12-11 12:56:37.263665000 +1100
+++ cf.data.pre.orig    2019-12-11 11:25:20.708044000 +1100
@@ -2632,15 +2632,6 @@
        See http://wiki.squid-cache.org/SquidFaq/SquidAcl for details.
 DOC_END

-NAME: host_verify_strict_intercepted
-TYPE: onoff
-DEFAULT: on
-LOC: Config.onoff.hostStrictVerifyIntercepted
-DOC_START
-       To be completed.
-
-DOC_END
-
 NAME: host_verify_strict
 TYPE: onoff
 DEFAULT: off

--- client_side_request.cc      2019-12-11 12:52:04.552556000 +1100
+++ client_side_request.cc.orig 2019-11-06 06:14:40.000000000 +1100
@@ -642,21 +642,16 @@

     debugs(85, 3, "validate host=" << host << ", port=" << port << ", portStr=" << (portStr?portStr:"NULL"));
     if (http->request->flags.intercepted || http->request->flags.interceptTproxy) {
-        if (Config.onoff.hostStrictVerifyIntercepted) {
-            // verify the Host: port (if any) matches the apparent destination
-            if (portStr && port != http->getConn()->clientConnection->local.port()) {
-                debugs(85, 3, "FAIL on validate port " << http->getConn()->clientConnection->local.port() <<
-                       " matches Host: port " << port << " (" << portStr << ")");
-                hostHeaderVerifyFailed("intercepted port", portStr);
-            } else {
-                // XXX: match the scheme default port against the apparent destination
-
-                // verify the destination DNS is one of the Host: headers IPs
-                ipcache_nbgethostbyname(host, hostHeaderIpVerifyWrapper, this);
-            }
+        // verify the Host: port (if any) matches the apparent destination
+        if (portStr && port != http->getConn()->clientConnection->local.port()) {
+            debugs(85, 3, "FAIL on validate port " << http->getConn()->clientConnection->local.port() <<
+                   " matches Host: port " << port << " (" << portStr << ")");
+            hostHeaderVerifyFailed("intercepted port", portStr);
         } else {
-            debugs(85, 3, "validate intercept skipped.");
-            http->doCallouts();
+            // XXX: match the scheme default port against the apparent destination
+
+            // verify the destination DNS is one of the Host: headers IPs
+            ipcache_nbgethostbyname(host, hostHeaderIpVerifyWrapper, this);
         }
     } else if (!Config.onoff.hostStrictVerify) {
         debugs(85, 3, "validate skipped.");

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: A patch for intercepted/WCCP HTTPS and 409 errors

Amos Jeffries
Administrator
On 11/12/19 8:51 pm, Scott wrote:
> Hi,
>
> I understand that squid does some security checking that the SNI of an
> intercepted/WCCP HTTPS requests matches the reverse DNS of the IP of the
> connection.  Or something like that.

Not being able to say precisely what Squid is actually doing shows that
you are lacking understanding of the processes taking place.

The security check you are posting about has many secondary consequences
and side effects to be taken into account. Quite a few people have taken
a stab at solving these rejections and what we have in Squid right now
is the best that can be done without significant redesign work (which is
underway - just very slowly, help welcome).

This is why we have the squid-dev mailing list for code change
discussion. If you want to actually help solving false-positives in this
security check please post there and we who have been working on this
issue for 10+ years now can discuss what we know about the situation,
the "gotcha" side effects we have to avoid and ideas for improvement.


>
> However with the prevalence of CDNs and badly configured DNSs and geographic
> DNSs, this breaks lots of connections (eg, I can't watch the NHL).
>
> I run Squid on a trusted network and use it primarily for caching and
> logging, and so I while I need to run WCCP for some non-proxy capable
> devices, I don't need that security check.

Without that check you cannot call your network a "secure network"
anymore. The absence of the check opens a nest of security holes for
attackers to walk right in past all those other protections.


>
> It stops all of those 409 errors occurring.
>
> Because of that I've created some patches that add a new option
> "host_verify_strict_intercepted" which is off by default.  They are
> for Squid 4.9.  As this is disabling a security feature of Squid do
> not apply this patch unless you are prepared for any and all consequences.
>

Please do not spread this around. People who want to really insist on
allowing virus/malware to spread unchecked around their networks can
make smaller patches.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: A patch for intercepted/WCCP HTTPS and 409 errors

Scott
In reply to this post by Scott
> On 11/12/19 8:51 pm, Scott wrote:
> > Hi,
> >
> > I understand that squid does some security checking that the SNI of an
> > intercepted/WCCP HTTPS requests matches the reverse DNS of the IP of the
> > connection.  Or something like that.
>
> Not being able to say precisely what Squid is actually doing shows that
> you are lacking understanding of the processes taking place.
>
> The security check you are posting about has many secondary consequences
> and side effects to be taken into account. Quite a few people have taken
> a stab at solving these rejections and what we have in Squid right now
> is the best that can be done without significant redesign work (which is
> underway - just very slowly, help welcome).
>
> This is why we have the squid-dev mailing list for code change
> discussion. If you want to actually help solving false-positives in this
> security check please post there and we who have been working on this
> issue for 10+ years now can discuss what we know about the situation,
> the "gotcha" side effects we have to avoid and ideas for improvement.
>
>
> >
> > However with the prevalence of CDNs and badly configured DNSs and geographic
> > DNSs, this breaks lots of connections (eg, I can't watch the NHL).
> >
> > I run Squid on a trusted network and use it primarily for caching and
> > logging, and so I while I need to run WCCP for some non-proxy capable
> > devices, I don't need that security check.
>
> Without that check you cannot call your network a "secure network"
> anymore. The absence of the check opens a nest of security holes for
> attackers to walk right in past all those other protections.
>
>
> >
> > It stops all of those 409 errors occurring.
> >
> > Because of that I've created some patches that add a new option
> > "host_verify_strict_intercepted" which is off by default.  They are
> > for Squid 4.9.  As this is disabling a security feature of Squid do
> > not apply this patch unless you are prepared for any and all consequences.
> >
>
> Please do not spread this around. People who want to really insist on
> allowing virus/malware to spread unchecked around their networks can
> make smaller patches.
>
> Amos
>
Hi Amos,

sorry for posting in the wrong forum.  While you're here: I've seen a handful
of posts about the 409s and the response has been "security".  Fair enough.  

Can you please provide a concrete example of

a) why host_verify_strict is available as a toggle for non-intercepted
requests, and
b) why intercepted requests don't have this option at all?

I'm suffering from a lack of imagination and I've yet to see any example
given (and ok, I may have missed one somewhere) and would like one brought to
my (and other reader's) attention.

Thanks,
Scott
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: A patch for intercepted/WCCP HTTPS and 409 errors

Amos Jeffries
Administrator
On 12/12/19 1:49 am, Scott wrote:

>> On 11/12/19 8:51 pm, Scott wrote:
>>> Hi,
>>>
>>> I understand that squid does some security checking that the SNI of an
>>> intercepted/WCCP HTTPS requests matches the reverse DNS of the IP of the
>>> connection.  Or something like that.
>>
>> Not being able to say precisely what Squid is actually doing shows that
>> you are lacking understanding of the processes taking place.
>>
>> The security check you are posting about has many secondary consequences
>> and side effects to be taken into account. Quite a few people have taken
>> a stab at solving these rejections and what we have in Squid right now
>> is the best that can be done without significant redesign work (which is
>> underway - just very slowly, help welcome).
>>
>> This is why we have the squid-dev mailing list for code change
>> discussion. If you want to actually help solving false-positives in this
>> security check please post there and we who have been working on this
>> issue for 10+ years now can discuss what we know about the situation,
>> the "gotcha" side effects we have to avoid and ideas for improvement.
>>
>>
>>>
>>> However with the prevalence of CDNs and badly configured DNSs and geographic
>>> DNSs, this breaks lots of connections (eg, I can't watch the NHL).
>>>
>>> I run Squid on a trusted network and use it primarily for caching and
>>> logging, and so I while I need to run WCCP for some non-proxy capable
>>> devices, I don't need that security check.
>>
>> Without that check you cannot call your network a "secure network"
>> anymore. The absence of the check opens a nest of security holes for
>> attackers to walk right in past all those other protections.
>>
>>
>>>
>>> It stops all of those 409 errors occurring.
>>>
>>> Because of that I've created some patches that add a new option
>>> "host_verify_strict_intercepted" which is off by default.  They are
>>> for Squid 4.9.  As this is disabling a security feature of Squid do
>>> not apply this patch unless you are prepared for any and all consequences.
>>>
>>
>> Please do not spread this around. People who want to really insist on
>> allowing virus/malware to spread unchecked around their networks can
>> make smaller patches.
>>
>> Amos
>>
> Hi Amos,
>
> sorry for posting in the wrong forum.  While you're here: I've seen a handful
> of posts about the 409s and the response has been "security".  Fair enough.  
>
> Can you please provide a concrete example of
>
> a) why host_verify_strict is available as a toggle for non-intercepted
> requests, and
> b) why intercepted requests don't have this option at all?
>

a) Because the root vulnerability (CVE-2009-0801) which opens up the
nest of effects has no external effect on non-intercepted HTTP syntax
traffic.
 If you have a paranoid security policy or want to track down broken
software it can be useful to validate everything. So its there, but off
by default.


b) While CVE-2009-0801 appears simply a Browser issue the other side
effects that are possible include seeding the proxy cache with arbitrary
content at arbitrary URLs, using that to spread infections, to
exfiltrate data, and use that data to bypass other security mechanisms
like the network border firewalls.

The worst part is that the Squid access.log are also corrupted with
false information from the Host header (and/or SNI in HTTPS) on the
transactions where these things happen. So the attacker is hidden from
admin and security forensics. That is going too far into unsafe
territory IMO.


Other useful reading:
 <http://www.squid-cache.org/Advisories/SQUID-2011_1.txt>
 <https://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery>


> I'm suffering from a lack of imagination and I've yet to see any example
> given (and ok, I may have missed one somewhere) and would like one brought to
> my (and other reader's) attention.
>
> Thanks,
> Scott
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: A patch for intercepted/WCCP HTTPS and 409 errors

Scott
In reply to this post by Scott
> On 12/12/19 1:49 am, Scott wrote:
> >> On 11/12/19 8:51 pm, Scott wrote:
> >>> Hi,
> >>>
> >>> I understand that squid does some security checking that the SNI of an
> >>> intercepted/WCCP HTTPS requests matches the reverse DNS of the IP of the
> >>> connection.  Or something like that.
> >>
> >> Not being able to say precisely what Squid is actually doing shows that
> >> you are lacking understanding of the processes taking place.
> >>
> >> The security check you are posting about has many secondary consequences
> >> and side effects to be taken into account. Quite a few people have taken
> >> a stab at solving these rejections and what we have in Squid right now
> >> is the best that can be done without significant redesign work (which is
> >> underway - just very slowly, help welcome).
> >>
> >> This is why we have the squid-dev mailing list for code change
> >> discussion. If you want to actually help solving false-positives in this
> >> security check please post there and we who have been working on this
> >> issue for 10+ years now can discuss what we know about the situation,
> >> the "gotcha" side effects we have to avoid and ideas for improvement.
> >>
> >>
> >>>
> >>> However with the prevalence of CDNs and badly configured DNSs and geographic
> >>> DNSs, this breaks lots of connections (eg, I can't watch the NHL).
> >>>
> >>> I run Squid on a trusted network and use it primarily for caching and
> >>> logging, and so I while I need to run WCCP for some non-proxy capable
> >>> devices, I don't need that security check.
> >>
> >> Without that check you cannot call your network a "secure network"
> >> anymore. The absence of the check opens a nest of security holes for
> >> attackers to walk right in past all those other protections.
> >>
> >>
> >>>
> >>> It stops all of those 409 errors occurring.
> >>>
> >>> Because of that I've created some patches that add a new option
> >>> "host_verify_strict_intercepted" which is off by default.  They are
> >>> for Squid 4.9.  As this is disabling a security feature of Squid do
> >>> not apply this patch unless you are prepared for any and all consequences.
> >>>
> >>
> >> Please do not spread this around. People who want to really insist on
> >> allowing virus/malware to spread unchecked around their networks can
> >> make smaller patches.
> >>
> >> Amos
> >>
> > Hi Amos,
> >
> > sorry for posting in the wrong forum.  While you're here: I've seen a handful
> > of posts about the 409s and the response has been "security".  Fair enough.  
> >
> > Can you please provide a concrete example of
> >
> > a) why host_verify_strict is available as a toggle for non-intercepted
> > requests, and
> > b) why intercepted requests don't have this option at all?
> >
>
> a) Because the root vulnerability (CVE-2009-0801) which opens up the
> nest of effects has no external effect on non-intercepted HTTP syntax
> traffic.
>  If you have a paranoid security policy or want to track down broken
> software it can be useful to validate everything. So its there, but off
> by default.
>
>
> b) While CVE-2009-0801 appears simply a Browser issue the other side
> effects that are possible include seeding the proxy cache with arbitrary
> content at arbitrary URLs, using that to spread infections, to
> exfiltrate data, and use that data to bypass other security mechanisms
> like the network border firewalls.
>
> The worst part is that the Squid access.log are also corrupted with
> false information from the Host header (and/or SNI in HTTPS) on the
> transactions where these things happen. So the attacker is hidden from
> admin and security forensics. That is going too far into unsafe
> territory IMO.
>
>
> Other useful reading:
>  <http://www.squid-cache.org/Advisories/SQUID-2011_1.txt>
>  <https://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery>
>
>
> > I'm suffering from a lack of imagination and I've yet to see any example
> > given (and ok, I may have missed one somewhere) and would like one brought to
> > my (and other reader's) attention.
> >
> > Thanks,
> > Scott
Thanks Amos,

I found
https://www.thesecuritypractice.com/the_security_practice/TransparentProxyAbuse.pdf 
extremely useful.  Unfortunately the documentation around Squid's fix doesn't
get into the side-effects in great detail.

The biggest side-effect is accessing non-malicious sites whose full list of A
and AAAA records is incomplete.  This is becoming increasingly common
especially given the preponderance of CDNs (and poor administration).

For a malicious client it's easy to circumvent Squid's fix (just don't send
an SNI at all) - this applies to my network as I peek and splice (no MITM -
so no HTTP headers).  But the main point of the CVE seems to be a general
user visiting a malicious site.

My reading of the PDF is that Squid could have implemented an option to
connect to the IP of the intercepted SYN rather than use the SNI/Host:, which
wouldn't cause these annoying 409s.

That said, do you think the implementation of an acl of permitted dstdomains
that aren't validated to be insecure?  I'd be happy to have a prescriptive
list of domain names which are NOT to be verified but known to be broken from
a DNS perspective.

eg:
acl safe_domains google.com
host_verify_strict !safe_domains

Scott
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users