Does squid generates/adds additional HTTP headers?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Does squid generates/adds additional HTTP headers?

avi_h
Hi,

Does squid generates/adds additional HTTP headers be default?
If so, are they being sent to the Web server or only to the client?

Thanks in advance,
Avi
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

Amos Jeffries
Administrator
On 11/07/17 07:25, avi_h wrote:
> Hi,
>
> Does squid generates/adds additional HTTP headers be default?

Squid generates all outgoing message headers. They may or may not be
based on (or identical in many cases) to the received message headers.

Why do you ask?


> If so, are they being sent to the Web server or only to the client?
>

Yes. More specific depends on how you configured Squid, eg which
features are being used.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

avi_h
Hi Amos,

Thanks for the prompt reply.
I'm trying to get squid to use the same headers as it received (to make it more transparent) so I would like to understand how it works so I can figure out how to configure it that way.
After looking up online I configured the following:

request_header_access Authorization allow all
request_header_access Proxy-Authorization allow all
request_header_access Cache-Control deny all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Connection allow all
request_header_access All deny all

I also have the following configs in addition:

via off
forwarded_for delete

Is there any way to have squid generate the exact same headers as it received?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

Alex Rousskov
On 07/10/2017 05:52 PM, avi_h wrote:

> I'm trying to get squid to use the same headers as it received (to make it
> more transparent)

To improve your chances of arriving at a usable solution (or quickly
abandoning a futile search), I suggest detailing/narrowing your goal:
"Make it more transparent" to whom and in what way? What specific
problems are you trying to solve? For example:

* Do you want to configure Squid to become invisible to a knowledgeable
human observer?

* Do you want to configure Squid to make some server(s) think that the
request is coming directly from a user agent (e.g., browser)? What do
those servers use to detect proxies now?

Etc. etc. You probably know exactly what you need to achieve. Narrow it
down for us as much as possible.


> After looking up online I configured the following:
>
> request_header_access Cache-Control deny all

Denying general-purpose headers makes no sense if you want Squid to
forward as many original headers as possible. Yes, Squid may generate
new Cache-Control headers but that does not mean that Squid does not
forward client Cache-Control headers as well. The task of preserving
original headers may feel trivial to you, but the actual complexity of
what you are asking is one of the reasons we need you to come up with a
narrower goal.


> Is there any way to have squid generate the exact same headers as it
> received?

No. However, you probably do not actually need that. You probably need
something else that Squid may be able to do (with or without code
modifications).

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

Sonya Roy
In reply to this post by avi_h
As Alex mentioned its not possible to do with squid. I modified the squid source code to do this a month ago. Its not hard to do, you will only need to modify http.cc and client_side.cc a bit.

On Tue, Jul 11, 2017 at 9:42 PM, Alex Rousskov <[hidden email]> wrote:
On 07/10/2017 05:52 PM, avi_h wrote:

> I'm trying to get squid to use the same headers as it received (to make it
> more transparent)

To improve your chances of arriving at a usable solution (or quickly
abandoning a futile search), I suggest detailing/narrowing your goal:
"Make it more transparent" to whom and in what way? What specific
problems are you trying to solve? For example:

* Do you want to configure Squid to become invisible to a knowledgeable
human observer?

* Do you want to configure Squid to make some server(s) think that the
request is coming directly from a user agent (e.g., browser)? What do
those servers use to detect proxies now?

Etc. etc. You probably know exactly what you need to achieve. Narrow it
down for us as much as possible.


> After looking up online I configured the following:
>
> request_header_access Cache-Control deny all

Denying general-purpose headers makes no sense if you want Squid to
forward as many original headers as possible. Yes, Squid may generate
new Cache-Control headers but that does not mean that Squid does not
forward client Cache-Control headers as well. The task of preserving
original headers may feel trivial to you, but the actual complexity of
what you are asking is one of the reasons we need you to come up with a
narrower goal.


> Is there any way to have squid generate the exact same headers as it
> received?

No. However, you probably do not actually need that. You probably need
something else that Squid may be able to do (with or without code
modifications).

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

avi_h
In reply to this post by Alex Rousskov
Hi Alex,

Thanks for this.
I can narrow it down, as you mentioned, I want to configure Squid to make some server(s) think that the
request is coming directly from a user agent (e.g., browser).
However, I don't know what those servers use to detect proxies, any idea on how I can figure it out?
Ideally, I would like to find a solution that would fit all (or the most popular) detection mechanisms, if that's possible.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

Alex Rousskov
On 07/11/2017 01:00 PM, avi_h wrote:

> I want to configure Squid to make some server(s) think that the
> request is coming directly from a user agent (e.g., browser).
> However, I don't know what those servers use to detect proxies,

Understood. We are still where we used to be then: Without a known
detection vector, it is impossible to recommend a specific solution (or
to declare the problem unsolvable).


> any idea on how I can figure it out?

I can suggest two complementary approaches:

* Experimentation: Send HTTP requests using some highly-configurable
client tool and vary header composition to move from a "browser request"
to "Squid request", one header (or group of headers) at a time. One may
be able to reverse engineer (parts of) the server algorithm this way.

* Asking specific questions: Ask about specific servers your Squid does
not work with. Perhaps others (on this or other mailing lists) know more
about those servers. You already got a response from Sonya Roy
indicating that some Squid modifications helped them. Perhaps Sonya Roy
and/or others know about your servers as well.


> Ideally, I would like to find a solution that would fit all (or the most
> popular) detection mechanisms, if that's possible.

I do not know what all (or the most popular) detection mechanisms are so
I cannot answer this question, but perhaps others on this list can.
Needless to say, if you succeed, then once those servers discover that
their detection mechanism stopped working, they are likely to change it.

This is why we can't have nice things,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

Amos Jeffries
Administrator
In reply to this post by avi_h
On 12/07/17 07:00, avi_h wrote:
> Hi Alex,
>
> Thanks for this.
> I can narrow it down, as you mentioned, I want to configure Squid to make
> some server(s) think that the
> request is coming directly from a user agent (e.g., browser).
> However, I don't know what those servers use to detect proxies, any idea on
> how I can figure it out?

I start with a copy of the HTTP traffic. Both the messages coming from
the client and the ones going to the serve. For both the proxied and
non-proxy traffic.

I then run a set of scripts I've built up over the years to see how the
server responds to various of the header changes the proxy does, and
some possible alternatives. That usually leads to identifying what the
server will accept and what makes it crash or produce errors. Usually
server crashing with uncommon normal inputs are the real problem, not
the proxy "breaking" traffic.

If you will list the websites you are having trouble with and what
behaviour you want to see happen vs what they currently do that would be
a good start.


> Ideally, I would like to find a solution that would fit all (or the most
> popular) detection mechanisms, if that's possible.

The solution is usually to accept that there is not actually a problem.

The average web object goes through something like 4-6 intermediaries
(ie. proxy) before it arrives at an end users Browser. You have a proxy,
everybody else does too. Nothing to gain by hiding.

In fact, hiding the proxy means literally revealing its users and some
of your internal network structure to any web server they visit. Loosing
privacy and a bit of security too. It is a lose-lose situation.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does squid generates/adds additional HTTP headers?

Amos Jeffries
Administrator
In reply to this post by Sonya Roy
On 12/07/17 04:36, Sonya Roy wrote:
> As Alex mentioned its not possible to do with squid. I modified the
> squid source code to do this a month ago. Its not hard to do, you will
> only need to modify http.cc and client_side.cc a bit.
>

What you did and what avi_h is asking about does not match what avi_h
says they want to happen.


On 12/07/17 07:00, avi_h wrote:
 > make
 > some server(s) think that the
 > request is coming directly from a user agent



Even sending the entire HTTP headers as-is through to the server cannot
prevent proxy detection if the server is actively trying to detect it.
Some naive services look only at the headers, others inject code into
the client to scan the Browsers view of the network environment and send
that back to the server for comparison of what the server environment
contains - yelling "proxy" if anything appears different, regardless of
whether a proxy actually exists.


So as Alex hinted but did not state - what would help is info about the
specific websites/services one is trying to work around. Narrowing the
problem down to certain sites, and what behaviour you want to stop them
having would be a great first step.


FWIW; in my experience most of the real traffic problems are not caused
by proxy detection at all. That seems to be purely users/admin getting
thrown off by other equally broken problem-detection websites, or
blaming the proxy when something else is causing problems.

The real problem is usually servers dying in horrible ways when
unexpected HTTP headers are given to them - even fully standardized
headers like Via (RFC 2068, 2616, 7231) with standard values is beyond
some server scripts ability to parse.

Not many web dev seems to understand that HTTP headers can contain
arbitrary-length comments. "via off" is not so much preventing the proxy
causing problems, but preventing clients behind the proxy injecting
bogus XSS code into the server script through it - by granting any
attacker more complete anonimity to do other attacks.

(sorry for the rant - I'm just tired of people thinking that hiding
their proxy actually helps).

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...