extract http headers from CONNECT / bumped ssl?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

extract http headers from CONNECT / bumped ssl?

Aaron Turner
So I've deployed squid in forward mode, installed the CA in my web
clients, etc and have squid working fine for both http and https
traffic.

One thing I need to do is be able to extract a http request header
into an external_acl_type:

external_acl_type client_ip_map_0 %>{My-Custom-Client-Id}
/usr/lib64/squid/user_loadbalance.py 0 4

This works fine for standard HTTP requests, but doesn't work for https
queries via CONNECT.  Is there some way to configure Squid to parse
them?  I need to load balance outbound requests via multiple IP
addresses based on this header and probably 50% of my traffic is
https.

Thanks!

--
Aaron Turner
https://synfin.net/         Twitter: @synfinatic
My father once told me that respect for the truth comes close to being
the basis for all morality.  "Something cannot emerge from nothing,"
he said.  This is profound thinking if you understand how unstable
"the truth" can be.  -- Frank Herbert, Dune
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Alex Rousskov
On 08/24/2017 06:00 PM, Aaron Turner wrote:
> So I've deployed squid in forward mode, installed the CA in my web
> clients, etc and have squid working fine for both http and https
> traffic.

Forgive me for double checking, but is SSL bumping actually working? For
example, do you see individual decrypted HTTPS requests in access.log?

What is your Squid version?


> One thing I need to do is be able to extract a http request header
> into an external_acl_type:
>
> external_acl_type client_ip_map_0 %>{My-Custom-Client-Id}
> /usr/lib64/squid/user_loadbalance.py 0 4

That is not your actual external_acl_type line, I hope. The %>h part
looks malformed.


> This works fine for standard HTTP requests, but doesn't work for https
> queries via CONNECT.  Is there some way to configure Squid to parse
> them?

Do you need to extract My-Custom-Client-Id header field value from

* the CONNECT request itself,
* the HTTP requests inside the (bumped) CONNECT tunnel,
* or all of the above?

Is that header field actually _present_ in the request(s) you want to
extract it from? You can answer this question by analyzing packet dumps
(wireshark can decrypt SSL for you) and/or by looking at cache.log with
debug_options set to ALL,2.

If you omit the parameter and simply use %>h, does the helper get any
headers?

If you see a request with the desired header and %>h expansion lacks it,
consider filing a bug report with the relevant information.


Thank you,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Aaron Turner
On Thu, Aug 24, 2017 at 5:16 PM, Alex Rousskov
<[hidden email]> wrote:
> On 08/24/2017 06:00 PM, Aaron Turner wrote:
>> So I've deployed squid in forward mode, installed the CA in my web
>> clients, etc and have squid working fine for both http and https
>> traffic.
>
> Forgive me for double checking, but is SSL bumping actually working? For
> example, do you see individual decrypted HTTPS requests in access.log?

Actually, looks like I was misunderstanding the access.log, it was working:

1503620688.280      0 10.93.3.85 TAG_NONE/200 0 CONNECT synfin.net:443
- HIER_NONE/- - ip_index=0,client=-
1503620689.241    947 10.93.3.85 TCP_MISS/200 57810 GET
https://synfin.net/sock_stream/ - HIER_DIRECT/45.79.73.39 text/html
ip_index=2,client=foobar1

I didn't initially understand that each CONNECT then generates a
second entry.  As you can see the second line has both the full URI
(indicating the SSL got bumped) and decoded my client id (foobar1).

> What is your Squid version?

3.5.26


>> One thing I need to do is be able to extract a http request header
>> into an external_acl_type:
>>
>> external_acl_type client_ip_map_0 %>{My-Custom-Client-Id}
>> /usr/lib64/squid/user_loadbalance.py 0 4
>
> That is not your actual external_acl_type line, I hope. The %>h part
> looks malformed.

Really?  Works and seems to match the instructions indicating "%>{Header}"

Thanks,
Aaron
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Alex Rousskov
On 08/24/2017 06:31 PM, Aaron Turner wrote:

> Actually, looks like I was misunderstanding the access.log, it was working:
>
> 1503620688.280      0 10.93.3.85 TAG_NONE/200 0 CONNECT synfin.net:443
> - HIER_NONE/- - ip_index=0,client=-
> 1503620689.241    947 10.93.3.85 TCP_MISS/200 57810 GET
> https://synfin.net/sock_stream/ - HIER_DIRECT/45.79.73.39 text/html
> ip_index=2,client=foobar1
>
> I didn't initially understand that each CONNECT then generates a
> second entry.

Each bumped CONNECT tunnel generates one or two CONNECT entries
(depending on the configuration) followed by zero or more HTTP requests
found inside the decrypted tunnel.


>>> external_acl_type client_ip_map_0 %>{My-Custom-Client-Id}
>>> /usr/lib64/squid/user_loadbalance.py 0 4

>> That is not your actual external_acl_type line, I hope. The %>h part
>> looks malformed.

> Really?  Works and seems to match the instructions indicating "%>{Header}"

If some instructions imply that omitting "h" from "%>h" is a good idea,
then I do not recommend following them, even if omiting "h" works.

The {header-field-name} parameter is fine. It is the missing "h" that I
would worry about.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Amos Jeffries
Administrator
On 25/08/17 15:37, Alex Rousskov wrote:

> On 08/24/2017 06:31 PM, Aaron Turner wrote:
>
>> Actually, looks like I was misunderstanding the access.log, it was working:
>>
>> 1503620688.280      0 10.93.3.85 TAG_NONE/200 0 CONNECT synfin.net:443
>> - HIER_NONE/- - ip_index=0,client=-
>> 1503620689.241    947 10.93.3.85 TCP_MISS/200 57810 GET
>> https://synfin.net/sock_stream/ - HIER_DIRECT/45.79.73.39 text/html
>> ip_index=2,client=foobar1
>>
>> I didn't initially understand that each CONNECT then generates a
>> second entry.
>
> Each bumped CONNECT tunnel generates one or two CONNECT entries
> (depending on the configuration) followed by zero or more HTTP requests
> found inside the decrypted tunnel.
>
>
>>>> external_acl_type client_ip_map_0 %>{My-Custom-Client-Id}
>>>> /usr/lib64/squid/user_loadbalance.py 0 4
>
>>> That is not your actual external_acl_type line, I hope. The %>h part
>>> looks malformed.
>
>> Really?  Works and seems to match the instructions indicating "%>{Header}"
>
> If some instructions imply that omitting "h" from "%>h" is a good idea,
> then I do not recommend following them, even if omiting "h" works.
>
> The {header-field-name} parameter is fine. It is the missing "h" that I
> would worry about.


FWIW: The non-h forms are only accepted by current Squid-3 for backward
compatibility and should be producing a high level WARNING on use. That
has been removed with Squid-4.
  (thanks for the reminder I'm going to have to mention that in the
release notes).


Please run "squid -k parse" and fix any config problems it highlights.
This command should be used after upgrades and when editing the config
to make sure it will actually do what you want in production.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Aaron Turner
Fyi, the 3.5.x docs is where I learned that format:

http://www.squid-cache.org/Versions/v3/3.5/cfgman/external_acl_type.html


--
Aaron Turner
https://synfin.net/         Twitter: @synfinatic
My father once told me that respect for the truth comes close to being
the basis for all morality.  "Something cannot emerge from nothing,"
he said.  This is profound thinking if you understand how unstable
"the truth" can be.  -- Frank Herbert, Dune


On Fri, Aug 25, 2017 at 1:35 AM, Amos Jeffries <[hidden email]> wrote:

> On 25/08/17 15:37, Alex Rousskov wrote:
>>
>> On 08/24/2017 06:31 PM, Aaron Turner wrote:
>>
>>> Actually, looks like I was misunderstanding the access.log, it was
>>> working:
>>>
>>> 1503620688.280      0 10.93.3.85 TAG_NONE/200 0 CONNECT synfin.net:443
>>> - HIER_NONE/- - ip_index=0,client=-
>>> 1503620689.241    947 10.93.3.85 TCP_MISS/200 57810 GET
>>> https://synfin.net/sock_stream/ - HIER_DIRECT/45.79.73.39 text/html
>>> ip_index=2,client=foobar1
>>>
>>> I didn't initially understand that each CONNECT then generates a
>>> second entry.
>>
>>
>> Each bumped CONNECT tunnel generates one or two CONNECT entries
>> (depending on the configuration) followed by zero or more HTTP requests
>> found inside the decrypted tunnel.
>>
>>
>>>>> external_acl_type client_ip_map_0 %>{My-Custom-Client-Id}
>>>>> /usr/lib64/squid/user_loadbalance.py 0 4
>>
>>
>>>> That is not your actual external_acl_type line, I hope. The %>h part
>>>> looks malformed.
>>
>>
>>> Really?  Works and seems to match the instructions indicating
>>> "%>{Header}"
>>
>>
>> If some instructions imply that omitting "h" from "%>h" is a good idea,
>> then I do not recommend following them, even if omiting "h" works.
>>
>> The {header-field-name} parameter is fine. It is the missing "h" that I
>> would worry about.
>
>
>
> FWIW: The non-h forms are only accepted by current Squid-3 for backward
> compatibility and should be producing a high level WARNING on use. That has
> been removed with Squid-4.
>  (thanks for the reminder I'm going to have to mention that in the release
> notes).
>
>
> Please run "squid -k parse" and fix any config problems it highlights. This
> command should be used after upgrades and when editing the config to make
> sure it will actually do what you want in production.
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Aaron Turner
Followup: I tried %{My-Custom-Client-Id}>h with 3.5.26 and squid
errors out.  Looking at the 3.5.x docs
(http://www.squid-cache.org/Versions/v3/3.5/cfgman/external_acl_type.html),
nothing there indicates it supports the logformat method?  Looks like
that's a 4.0+ feature?
--
Aaron Turner
https://synfin.net/         Twitter: @synfinatic
My father once told me that respect for the truth comes close to being
the basis for all morality.  "Something cannot emerge from nothing,"
he said.  This is profound thinking if you understand how unstable
"the truth" can be.  -- Frank Herbert, Dune


On Fri, Aug 25, 2017 at 8:09 AM, Aaron Turner <[hidden email]> wrote:

> Fyi, the 3.5.x docs is where I learned that format:
>
> http://www.squid-cache.org/Versions/v3/3.5/cfgman/external_acl_type.html
>
>
> --
> Aaron Turner
> https://synfin.net/         Twitter: @synfinatic
> My father once told me that respect for the truth comes close to being
> the basis for all morality.  "Something cannot emerge from nothing,"
> he said.  This is profound thinking if you understand how unstable
> "the truth" can be.  -- Frank Herbert, Dune
>
>
> On Fri, Aug 25, 2017 at 1:35 AM, Amos Jeffries <[hidden email]> wrote:
>> On 25/08/17 15:37, Alex Rousskov wrote:
>>>
>>> On 08/24/2017 06:31 PM, Aaron Turner wrote:
>>>
>>>> Actually, looks like I was misunderstanding the access.log, it was
>>>> working:
>>>>
>>>> 1503620688.280      0 10.93.3.85 TAG_NONE/200 0 CONNECT synfin.net:443
>>>> - HIER_NONE/- - ip_index=0,client=-
>>>> 1503620689.241    947 10.93.3.85 TCP_MISS/200 57810 GET
>>>> https://synfin.net/sock_stream/ - HIER_DIRECT/45.79.73.39 text/html
>>>> ip_index=2,client=foobar1
>>>>
>>>> I didn't initially understand that each CONNECT then generates a
>>>> second entry.
>>>
>>>
>>> Each bumped CONNECT tunnel generates one or two CONNECT entries
>>> (depending on the configuration) followed by zero or more HTTP requests
>>> found inside the decrypted tunnel.
>>>
>>>
>>>>>> external_acl_type client_ip_map_0 %>{My-Custom-Client-Id}
>>>>>> /usr/lib64/squid/user_loadbalance.py 0 4
>>>
>>>
>>>>> That is not your actual external_acl_type line, I hope. The %>h part
>>>>> looks malformed.
>>>
>>>
>>>> Really?  Works and seems to match the instructions indicating
>>>> "%>{Header}"
>>>
>>>
>>> If some instructions imply that omitting "h" from "%>h" is a good idea,
>>> then I do not recommend following them, even if omiting "h" works.
>>>
>>> The {header-field-name} parameter is fine. It is the missing "h" that I
>>> would worry about.
>>
>>
>>
>> FWIW: The non-h forms are only accepted by current Squid-3 for backward
>> compatibility and should be producing a high level WARNING on use. That has
>> been removed with Squid-4.
>>  (thanks for the reminder I'm going to have to mention that in the release
>> notes).
>>
>>
>> Please run "squid -k parse" and fix any config problems it highlights. This
>> command should be used after upgrades and when editing the config to make
>> sure it will actually do what you want in production.
>>
>>
>> Amos
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Alex Rousskov
On 08/25/2017 10:37 AM, Aaron Turner wrote:
> Followup: I tried %{My-Custom-Client-Id}>h with 3.5.26 and squid
> errors out.  Looking at the 3.5.x docs
> (http://www.squid-cache.org/Versions/v3/3.5/cfgman/external_acl_type.html),
> nothing there indicates it supports the logformat method?  Looks like
> that's a 4.0+ feature?

Apparently v3.5 is still using those custom format codes for external
ACLs (i.e., codes that differ from "standard" logformat codes). I did
not realize that. Sorry.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Amos Jeffries
Administrator
On 26/08/17 09:42, Alex Rousskov wrote:
> On 08/25/2017 10:37 AM, Aaron Turner wrote:
>> Followup: I tried %{My-Custom-Client-Id}>h with 3.5.26 and squid
>> errors out.  Looking at the 3.5.x docs
>> (http://www.squid-cache.org/Versions/v3/3.5/cfgman/external_acl_type.html),
>> nothing there indicates it supports the logformat method?  Looks like
>> that's a 4.0+ feature?

It seems I forgot to do the documentation changes in 3.4 or 3.5 to
follow the code changes in 3.3.

As recompense I shall dedicate some time into adding back-compat code
for Squid-4+ so that it accepts the 3.5 documented syntax. The Squid-2
syntax is not possible to do in the new code. Which IIRC was a big
reason for the slow staged rollout plan, hoping not to need back-compat
here. Sorry for the muckup.

>
> Apparently v3.5 is still using those custom format codes for external
> ACLs (i.e., codes that differ from "standard" logformat codes). I did
> not realize that. Sorry.
>

The custom parser back in Squid-3.3.7 was prepared for logformat codes
when the logformat syntax was only %code{parameters}.


squid -k parse for me with the old options produces:
   WARNING: external_acl_type format %{...} is being replaced by
%>ha{...} for %{test}
   WARNING: external_acl_type format %>{...} is being replaced by
%>ha{...} for %>{test}

and accepts %>ha{test}


Notice that the old %{...} and %>{...} send the *adapted* request
headers to the helper, not the original client headers. To distinguish
the real client headers from adapted ones you do need the proper
logformat codes in Squid-4+.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: extract http headers from CONNECT / bumped ssl?

Aaron Turner
Thanks Amos.  I didn't realize that %>ha{} was a valid format.
--
Aaron Turner
https://synfin.net/         Twitter: @synfinatic
My father once told me that respect for the truth comes close to being
the basis for all morality.  "Something cannot emerge from nothing,"
he said.  This is profound thinking if you understand how unstable
"the truth" can be.  -- Frank Herbert, Dune


On Fri, Aug 25, 2017 at 10:14 PM, Amos Jeffries <[hidden email]> wrote:

> On 26/08/17 09:42, Alex Rousskov wrote:
>>
>> On 08/25/2017 10:37 AM, Aaron Turner wrote:
>>>
>>> Followup: I tried %{My-Custom-Client-Id}>h with 3.5.26 and squid
>>> errors out.  Looking at the 3.5.x docs
>>>
>>> (http://www.squid-cache.org/Versions/v3/3.5/cfgman/external_acl_type.html),
>>> nothing there indicates it supports the logformat method?  Looks like
>>> that's a 4.0+ feature?
>
>
> It seems I forgot to do the documentation changes in 3.4 or 3.5 to follow
> the code changes in 3.3.
>
> As recompense I shall dedicate some time into adding back-compat code for
> Squid-4+ so that it accepts the 3.5 documented syntax. The Squid-2 syntax is
> not possible to do in the new code. Which IIRC was a big reason for the slow
> staged rollout plan, hoping not to need back-compat here. Sorry for the
> muckup.
>
>>
>> Apparently v3.5 is still using those custom format codes for external
>> ACLs (i.e., codes that differ from "standard" logformat codes). I did
>> not realize that. Sorry.
>>
>
> The custom parser back in Squid-3.3.7 was prepared for logformat codes when
> the logformat syntax was only %code{parameters}.
>
>
> squid -k parse for me with the old options produces:
>   WARNING: external_acl_type format %{...} is being replaced by %>ha{...}
> for %{test}
>   WARNING: external_acl_type format %>{...} is being replaced by %>ha{...}
> for %>{test}
>
> and accepts %>ha{test}
>
>
> Notice that the old %{...} and %>{...} send the *adapted* request headers to
> the helper, not the original client headers. To distinguish the real client
> headers from adapted ones you do need the proper logformat codes in
> Squid-4+.
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users