Squid returns 400 to GET / HTTP/1.1 with Host Header

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid returns 400 to GET / HTTP/1.1 with Host Header

Stephen Nelson-Smith
Hello,

I need to demonstrate and test a Squid setup, which should blacklist
by default, and allow requests only to whitelisted URLs from known
networks. This is currently running in my staging environment, and is
working as expected, but I want to test and demo it on demand, with
nicer feedback than with curl.

I've deployed Redbot (https://github.com/mnot/redbot), which I've set
up to send all HTTP requests via the Squid proxy

Using curl -x from the Redbot machine, all my tests pass, but using
the application, Squid returns a 400 whatever happens. All requests go
to Squid, and I see every request, but instead of returning a 403 and
X-Squid-Error: ERR_ACCESS_DENIED 0, or allowing the request, every
request gets a 400, and X-Squid-Error: ERR_INVALID_URL 0.

Digging into it - logs and tcpdump - the key difference I see is that
Redbot sends a request of the form:

GET / HTTP/1.1
Host: chess.com

Curl sends:

GET http://chess.com/ HTTP/1.1
Host: chess.com

From the RFC it seems like Redbot's request is perfectly valid, and so
I feel like Squid should do the right thing and deduce from the host
header what Redbot wants, and go through its ACLs. However, it just
errors with:

HTTP/1.1 400 Bad Request
Server: squid/3.5.27
Mime-Version: 1.0
Date: Mon, 23 Apr 2018 11:50:23 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3465
X-Squid-Error: ERR_INVALID_URL 0
X-Cache: MISS from proxy.redaction.com
Via: 1.1 proxy.redaction.com (squid/3.5.27)

Does this seem like a Squid config issue? Or do I need to make Redbot
make a request like Curl does?
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid returns 400 to GET / HTTP/1.1 with Host Header

Amos Jeffries
Administrator
On 24/04/18 03:15, Stephen Nelson-Smith wrote:

> Hello,
>
> I need to demonstrate and test a Squid setup, which should blacklist
> by default, and allow requests only to whitelisted URLs from known
> networks. This is currently running in my staging environment, and is
> working as expected, but I want to test and demo it on demand, with
> nicer feedback than with curl.
>
> I've deployed Redbot (https://github.com/mnot/redbot), which I've set
> up to send all HTTP requests via the Squid proxy
>
> Using curl -x from the Redbot machine, all my tests pass, but using
> the application, Squid returns a 400 whatever happens. All requests go
> to Squid, and I see every request, but instead of returning a 403 and
> X-Squid-Error: ERR_ACCESS_DENIED 0, or allowing the request, every
> request gets a 400, and X-Squid-Error: ERR_INVALID_URL 0.
>

ERR_INVALID_URL --> the URL is the invalid part, not the Host header.

> Digging into it - logs and tcpdump - the key difference I see is that
> Redbot sends a request of the form:
>
> GET / HTTP/1.1
> Host: chess.com
>
> Curl sends:
>
> GET http://chess.com/ HTTP/1.1
> Host: chess.com
>
> From the RFC it seems like Redbot's request is perfectly valid, and so
> I feel like Squid should do the right thing and deduce from the host
> header what Redbot wants, and go through its ACLs. However, it just
> errors with:

You missed the part where it says which type of recipient the various
URL forms are valid.

The redbot example is a origin-form URL - valid only when sent to origin
servers (or reverse-proxy). The curl one is an absolute-form URL - valid
when sent to proxies and gateways.

...
>
> Does this seem like a Squid config issue? Or do I need to make Redbot
> make a request like Curl does?

Redbot is designed primarily for debugging HTTP problems with origin
servers to check why their output is not caching in a proxy or browser
properly. If you can find an option to inform it that it is operating
through a proxy, turn that on.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid returns 400 to GET / HTTP/1.1 with Host Header

Stephen Nelson-Smith
Hi Amos,


On Mon, Apr 23, 2018 at 4:31 PM, Amos Jeffries <[hidden email]> wrote:

>> From the RFC it seems like Redbot's request is perfectly valid, and so
>> I feel like Squid should do the right thing and deduce from the host
>> header what Redbot wants, and go through its ACLs. However, it just
>> errors with:
>
> You missed the part where it says which type of recipient the various
> URL forms are valid.
>
> The redbot example is a origin-form URL - valid only when sent to origin
> servers (or reverse-proxy). The curl one is an absolute-form URL - valid
> when sent to proxies and gateways.

Thanks - that's a helpful distinction.

>> Does this seem like a Squid config issue? Or do I need to make Redbot
>> make a request like Curl does?
>
> Redbot is designed primarily for debugging HTTP problems with origin
> servers to check why their output is not caching in a proxy or browser
> properly. If you can find an option to inform it that it is operating
> through a proxy, turn that on.

There's no such option, and I had to modify RedFetcher to instantiate
with a proxy.  The constructor does support it, but there's no login
in Thor or Redbot to behave differently if going through a proxy.

Adding that functionality would be an option, but am I right in
thinking squid should be able to infer the destination from the host
header?

Just looking at the documentation for http_port, would adding
'intercept' help, or is that explicitly for interception caching in
conjunction with a traffic filter?

S.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid returns 400 to GET / HTTP/1.1 with Host Header

Stephen Nelson-Smith
Hi,

On Mon, Apr 23, 2018 at 4:48 PM, Stephen Nelson-Smith
<[hidden email]> wrote:

> Adding that functionality would be an option, but am I right in
> thinking squid should be able to infer the destination from the host
> header?
>
> Just looking at the documentation for http_port, would adding
> 'intercept' help, or is that explicitly for interception caching in
> conjunction with a traffic filter?

Adding `intercept` to `http_port` has resulted in the host header
appearing as the URL in the request.

Squid is now giving a 403... which it shouldn't... I think:

1524498993.558      0 10.8.0.33 TCP_MISS/403 3985 GET
http://www.openstreetmap.com/ - HIER_NONE/- text/html
1524498993.559      0 10.8.2.19 TCP_MISS/403 4077 GET
http://www.openstreetmap.com/ - ORIGINAL_DST/10.8.0.33 text/html

# Source ACLs

acl cluster src 10.8.0.0/16 # Kubernetes Cluster

# Destination ACLs

acl google dstdomain google.com
http_access allow cluster google

acl streetmap dstdomain .openstreetmap.com
http_access allow cluster streetmap

# and finally deny all other access to this proxy

http_access deny all

S.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid returns 400 to GET / HTTP/1.1 with Host Header

Amos Jeffries
Administrator
On 24/04/18 04:03, Stephen Nelson-Smith wrote:
> Hi,
>
> On Mon, Apr 23, 2018 at 4:48 PM, Stephen Nelson-Smith wrote:
>
>> Adding that functionality would be an option,

I think that is worth asking Mark Nottingham about adding that
functionality.


>> but am I right in
>> thinking squid should be able to infer the destination from the host
>> header?

No, that is rather dangerous. The CVE-2009-0801 and related nest of
vulnerabilities are opened up if Host header is trusted by a proxy.


>>
>> Just looking at the documentation for http_port, would adding
>> 'intercept' help, or is that explicitly for interception caching in
>> conjunction with a traffic filter?
>
> Adding `intercept` to `http_port` has resulted in the host header
> appearing as the URL in the request.
>
> Squid is now giving a 403... which it shouldn't... I think:
>
> 1524498993.558      0 10.8.0.33 TCP_MISS/403 3985 GET
> http://www.openstreetmap.com/ - HIER_NONE/- text/html
> 1524498993.559      0 10.8.2.19 TCP_MISS/403 4077 GET
> http://www.openstreetmap.com/ - ORIGINAL_DST/10.8.0.33 text/html
>

That is the CVE-2009-0801 protections doing their thing for intercept'ed
traffic (second log line). The 10.8.0.33 IP is where the client was
apparently going before MITM'd into the proxy, so the server there MUST
be able to handle whatever the client is expecting back regardless of
whether the proxy trusts it for caching purposes.

But 10.8.0.33 is your Squid, so the traffic loops (first log line).
Squid detects the loop and rejects it to prevent infinite memory and TCP
port numbers being consumed.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid returns 400 to GET / HTTP/1.1 with Host Header

Stephen Nelson-Smith
Hi,

On Mon, Apr 23, 2018 at 5:58 PM, Amos Jeffries <[hidden email]> wrote:
> On 24/04/18 04:03, Stephen Nelson-Smith wrote:
>> Hi,
>>
>> On Mon, Apr 23, 2018 at 4:48 PM, Stephen Nelson-Smith wrote:
>>
>>> Adding that functionality would be an option,
>
> I think that is worth asking Mark Nottingham about adding that
> functionality.

I'll open an issue on the repo.  I did already fork it to add the 'use
a proxy' functionality, and would be happy to contribute such
functionality.  Would be a good way to get my head around it all
anyway.

>>> but am I right in
>>> thinking squid should be able to infer the destination from the host
>>> header?
>
> No, that is rather dangerous. The CVE-2009-0801 and related nest of
> vulnerabilities are opened up if Host header is trusted by a proxy.

Thanks for explaining - I'll look into that, but I can see what you mean.

>
>>>
>>> Just looking at the documentation for http_port, would adding
>>> 'intercept' help, or is that explicitly for interception caching in
>>> conjunction with a traffic filter?
>>
>> Adding `intercept` to `http_port` has resulted in the host header
>> appearing as the URL in the request.
>>
>> Squid is now giving a 403... which it shouldn't... I think:
>
> That is the CVE-2009-0801 protections doing their thing for intercept'ed
> traffic (second log line). The 10.8.0.33 IP is where the client was
> apparently going before MITM'd into the proxy, so the server there MUST
> be able to handle whatever the client is expecting back regardless of
> whether the proxy trusts it for caching purposes.
>
> But 10.8.0.33 is your Squid, so the traffic loops (first log line).
> Squid detects the loop and rejects it to prevent infinite memory and TCP
> port numbers being consumed.

Right.  I understand in a traditional transparent proxy environment
we'd handle this with iptables/pf.

Short of making Redbot behave better, through a proxy, is there a
solution I can use that will get me through my demo (I have to demo
this tomorrow) without resorting to a bunch of curls?  As long as
Redbot can make requests to a bunch of URLs and I can show the results
in the browser (failures and successes), I can worry about doing it
properly later - this is a throw-away environment, and won't exist
after the demo.  The point of the demo is to show the proxy working
and being used by a web app.

Thanks for your time and insight - this is all tremendously useful and
informative.

S.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users