websockets through Squid

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

websockets through Squid

Vieri
I'm compiling on a Gentoo Linux system the tarball taken from http://www.squid-cache.org/Versions/v5/squid-5.0.4.tar.gz.

The build log (failed) is here (notice the call to make -j1):

https://drive.google.com/file/d/1no0uV3Ti1ILZavAaiOyFIY9W0eLRv87q/view?usp=sharing

If I build from git f4ade36 all's well:

https://drive.google.com/file/d/1y-3wlDT_OrwSp7epvDq63xpkYv8gu9Pq/view?usp=sharing

So now I'm just going to have to spot the difference.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: websockets through Squid

Alex Rousskov
On 10/12/20 7:35 PM, Vieri wrote:

> I'm compiling on a Gentoo Linux system the tarball taken from
> http://www.squid-cache.org/Versions/v5/squid-5.0.4.tar.gz.

> The build log (failed) is here (notice the call to make -j1):

> https://drive.google.com/file/d/1no0uV3Ti1ILZavAaiOyFIY9W0eLRv87q/view?usp=sharing

The beginning of the above log appears to show some unofficial
bootstrapping steps.

* If you do not have to bootstrap official tarballed sources, then start
with the ./configure step.

* If you have to bootstrap tarballed sources, then use ./bootstrap.sh to
do that.

In most cases, the first bullet is applicable if you have not modified
any auto-generated source files.


> If I build from git f4ade36 all's well:

> https://drive.google.com/file/d/1y-3wlDT_OrwSp7epvDq63xpkYv8gu9Pq/view?usp=sharing

FWIW, this log appears to show the official bootstrapping sequence
(bootstrap.sh).

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: websockets through Squid

Vieri

On Tuesday, October 13, 2020, 3:55:56 PM GMT+2, Alex Rousskov <[hidden email]> wrote:

> The beginning of the above log appears to show some unofficial bootstrapping steps.


Yes, I was looking into this today and I saw that the actual difference between a manual build and a Gentoo Linux build is with the following:

1) the build fails as mentioned earlier in this thread when running Gentoo-specific "configure" scripts. Bootstrapping makes no real difference.

econf: updating squid-5.0.4-20200825-rf4ade365f/cfgaux/config.sub with /usr/share/gnuconfig/config.sub
econf: updating squid-5.0.4-20200825-rf4ade365f/cfgaux/config.guess with /usr/share/gnuconfig/config.guess
./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --disable-dependency-tracking --disable-silent-rules --docdir=/usr/share/doc/squid-5.0.4 --htmldir=/usr/share/doc/squid-5.0.4/html --with-sysroot=/ --libdir=/usr/lib64

Correct me if I'm wrong, but I don't see anything wrong with the third line and the parameters passed to configure (unless disable-dependency-tracking could have some side-effects).
So I guess the problem might be with the first and second lines where some config scripts seem to be replaced.
The timestamps in /usr/share/gnuconfig/config.{sub,guess} are more recent than the ones distributed in the Squid tarball.

2) the build succeeds even when using the Gentoo build environment just as long as I do not run the Gentoo-specific econf (configure) script but "./configure" instead.

I guess I will need to bring this up on the Gentoo forum to see what's going on. I am not instructing the build system to "patch" cfgaux so I guess "econf" automatically detects something in the squid tarball that makes it patch the config.* files.

Thanks for your time.

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: websockets through Squid

Alex Rousskov
On 10/13/20 11:21 AM, Vieri wrote:

> I saw that the actual difference between a manual build and a Gentoo Linux build is with the following:

> 1) the build fails as mentioned earlier in this thread when running
> Gentoo-specific "configure" scripts. Bootstrapping makes no real
> difference.

> econf: updating squid-5.0.4-20200825-rf4ade365f/cfgaux/config.sub with /usr/share/gnuconfig/config.sub
> econf: updating squid-5.0.4-20200825-rf4ade365f/cfgaux/config.guess with /usr/share/gnuconfig/config.guess
> ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --disable-dependency-tracking --disable-silent-rules --docdir=/usr/share/doc/squid-5.0.4 --htmldir=/usr/share/doc/squid-5.0.4/html --with-sysroot=/ --libdir=/usr/lib64

> I don't see anything wrong with the third line and the parameters
> passed to configure (unless disable-dependency-tracking could have
> some side-effects).

* I speculate that disabling dependency tracking results in such
dependencies as the target creating the "tests" directory being skipped
or even not-generated, but I do not know the actual details.

* Customizing various installation directories should not affect the build.

* I assume the host settings are correct.

* Disabling silent rules is a bad idea during build triage, but should
not affect the build outcome AFAICT.


> So I guess the problem might be with the first and second lines where
> some config scripts seem to be replaced.

You can easily test this theory. I cannot, but my first bet would be on
--disable-dependency-tracking.


> 2) the build succeeds even when using the Gentoo build environment just as long as I do not run the Gentoo-specific econf (configure) script but "./configure" instead.

Glad we could identify the primary suspect. You should probably follow
up with Gentoo folks responsible for this Squid customization.


Cheers,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: websockets through Squid

Vieri
 On Tuesday, October 13, 2020, 6:14:18 PM GMT+2, Alex Rousskov <[hidden email]> wrote:

> You should probably follow up with Gentoo folks responsible for this Squid customization.

Squid 5 now builds and installs perfectly on Gentoo Linux with a few custom changes to the distro's package installation script. I hope the devs will include these changes so Squid 5 can be readily available to everyone.
BTW it "makes" in parallel fine with -jx where x > 1, so no issues there either.

So, coming back to the original post: websockets.

I added this to Squid 5:

http_upgrade_request_protocols OTHER allow all

Am I right if I state that this should allow any protocol not just WebSockets?
In other words, I do not need to be specific with 'http_upgrade_request_protocols WebSocket allow all' unless I want to, right?

Unfortunately, after reloading Squid 5 the client browser still states the same:

The connection to wss://ed1lncb65702.webex.com/direct?type=websocket&dtype=binary&rand=1602769907574&uuidtag=9E73C14G-1580-43B4-B8D4-91453FCF1939&gatewayip=MY_IP_ADDR was interrupted while the page was loading.

And in access.log I can see this:

[Thu Oct 15 15:52:27 2020].411  29846 10.215.144.48 TCP_TUNNEL/101 0 GET https://ed1lncb65702.webex.com/direct? - ORIGINAL_DST/62.109.225.174 -
[Thu Oct 15 15:52:27 2020].831    125 10.215.144.48 NONE_NONE/000 0 CONNECT 62.109.225.174:443 - ORIGINAL_DST/62.109.225.174 -
[Thu Oct 15 15:52:28 2020].786     11 10.215.111.210 NONE_NONE_ABORTED/000 0 CONNECT 44.233.111.149:443 - HIER_NONE/- -
[Thu Oct 15 15:52:37 2020].414  29870 10.215.144.48 TCP_TUNNEL/101 0 GET https://ed1lncb65702.webex.com/direct? - ORIGINAL_DST/62.109.225.174 -
[Thu Oct 15 15:52:37 2020].919    107 10.215.144.48 NONE_NONE/000 0 CONNECT 62.109.225.174:443 - ORIGINAL_DST/62.109.225.174 -

What does NONE_NONE/000 mean?

Where can I go from here?
What can I try to debug this further?

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: websockets through Squid

Alex Rousskov
On 10/15/20 10:08 AM, Vieri wrote:
>  On Tuesday, October 13, 2020, 6:14:18 PM GMT+2, Alex Rousskov wrote:
>> You should probably follow up with Gentoo folks responsible for this Squid customization.

> Squid 5 now builds and installs perfectly on Gentoo Linux with a few
> custom changes to the distro's package installation script. I hope
> the devs will include these changes so Squid 5 can be readily
> available to everyone.

Thank you for reporting your fixes to Gentoo.


> I added this to Squid 5:

> http_upgrade_request_protocols OTHER allow all

> Am I right if I state that this should allow any protocol not just WebSockets?

Yes.

> In other words, I do not need to be specific with
> 'http_upgrade_request_protocols WebSocket allow all' unless I want
> to, right?

Just in case somebody else starts copy-pasting the above rule into their
configurations: The standard (RFC 6455) WebSocket protocol name in HTTP
Upgrade requests is "websocket". Squid uses case-sensitive comparison
for those names so you should use "websocket" in squid.conf.

Other than that correction, you are right -- a bare OTHER rule is more
risky in general but is sufficient for allowing WebSocket upgrades.


> Unfortunately, after reloading Squid 5 the client browser still states the same:
>
> The connection to wss://ed1lncb65702.webex.com/direct?type=websocket&dtype=binary&rand=1602769907574&uuidtag=9E73C14G-1580-43B4-B8D4-91453FCF1939&gatewayip=MY_IP_ADDR was interrupted while the page was loading.
>
> And in access.log I can see this:
>
> [Thu Oct 15 15:52:27 2020].411  29846 10.215.144.48 TCP_TUNNEL/101 0 GET https://ed1lncb65702.webex.com/direct? - ORIGINAL_DST/62.109.225.174 -
> [Thu Oct 15 15:52:27 2020].831    125 10.215.144.48 NONE_NONE/000 0 CONNECT 62.109.225.174:443 - ORIGINAL_DST/62.109.225.174 -
> [Thu Oct 15 15:52:28 2020].786     11 10.215.111.210 NONE_NONE_ABORTED/000 0 CONNECT 44.233.111.149:443 - HIER_NONE/- -
> [Thu Oct 15 15:52:37 2020].414  29870 10.215.144.48 TCP_TUNNEL/101 0 GET https://ed1lncb65702.webex.com/direct? - ORIGINAL_DST/62.109.225.174 -
> [Thu Oct 15 15:52:37 2020].919    107 10.215.144.48 NONE_NONE/000 0 CONNECT 62.109.225.174:443 - ORIGINAL_DST/62.109.225.174 -

> What does NONE_NONE/000 mean?

It is a (relatively minor) bug in Squid. Squid probably forgot to set
the exact outcome of the transaction at transaction logging time and
probably logged absent status code as 000. We already have a project
that should fix the status code handling in such cases.

The important part here is the existence of those extra transactions.
They may be related to SslBump if you are bumbing this traffic, but then
I would expect a slightly different access.log composition.


> Where can I go from here?
> What can I try to debug this further?

The log shows successful tunnel negotiation (two TCP_TUNNEL/101 entries)
and potentially problematic transactions. It is not clear (to me)
whether they all correspond to the same client interaction. It is
possible that the client successfully tunneled some websocket
transactions but failed to tunnel others. It is also possible, although
less likely, that the client has detected a proxy presence and refused
to tunnel despite receiving a successful 101 response from Squid.

IMO, further triage requires analyzing debugging cache.log. In my
experience, such analysis usually requires developer expertise. You can
try posting a link to compressed cache.log containing just the
corresponding transactions/interactions with debug_options set to
"ALL,9" (this may log passwords, keys, and such so be careful). Somebody
here may volunteer to analyze your log for you.

https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: websockets through Squid

Vieri

 On Thursday, October 15, 2020, 5:28:03 PM GMT+2, Alex Rousskov <[hidden email]> wrote:

>> In other words, I do not need to be specific with
>> 'http_upgrade_request_protocols WebSocket allow all' unless I want
>> to, right?
>
> Just in case somebody else starts copy-pasting the above rule into their
> configurations: The standard (RFC 6455) WebSocket protocol name in HTTP
> Upgrade requests is "websocket". Squid uses case-sensitive comparison
> for those names so you should use "websocket" in squid.conf.

OK, good to know because:

squid-5.0.4-20200825-rf4ade365f/src/cf.data.pre contains:
        Usage: http_upgrade_request_protocols <protocol> allow|deny [!]acl ...

        The required "protocol" parameter is either an all-caps word OTHER or an
        explicit protocol name (e.g. "WebSocket") optionally followed by a slash
        and a version token (e.g. "HTTP/3"). Explicit protocol names and
        versions are case sensitive.

That's why I used "WebSocket" instead of "websocket" in my example. To avoid confusion, cf.data.pre could be updated to be more clear.


> The important part here is the existence of those extra transactions.
> They may be related to SslBump if you are bumbing this traffic, but then
> I would expect a slightly different access.log composition.

Hmm, I'm supposed to be sslbumping, yes. I can share my full squid config & iptables redirection entries if you wish.

> https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction

 I enabled debugging on a test system where I was the only client (one Firefox instance).

The access log is here:

https://drive.google.com/file/d/1jryX5BW4yxLTSBe0QDavPSiKLBpOvtnV/view?usp=sharing

The only odd thing I see is a few ABORTED but they are all WOFF fonts which should be unimportant except for https://join-test.webex.com/mw3300/mywebex/header.do which is only a TCP refresh "abort".

The overwhelming cache log is here (I've sed'ed a few strings for privacy reasons):

https://drive.google.com/file/d/1QYRr-0F-DGnCZtyuuAw8RsEgcHICN_0c/view?usp=sharing

I can see the upgrade messages are parsed:

HttpHeader.cc(1548) parse: parsed HttpHeaderEntry: 'Upgrade: WebSocket'

I suppose that adding the "Upgrade[66]" entry is as expected.

Then, I get lost. I can see that Squid is trying to open ed1lncb62801.webex.com with https, but it is unclear to me why the ciient complains that the connection to the wss:// site is being interrupted:

The connection to wss://ed1lncb62801.webex.com/direct?type=websocket&dtype=binary&rand=1602830016480&uuidtag=5659FGE6-DF29-47A7-859A-G4D5FDC937A2&gatewayip=PUB_IPv4_ADDR_2 was interrupted while the page was loading.

Thanks for all the help you can give me.

Vieri

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: websockets through Squid

Alex Rousskov
On 10/16/20 3:35 AM, Vieri wrote:
> squid-5.0.4-20200825-rf4ade365f/src/cf.data.pre contains:
>         Usage: http_upgrade_request_protocols <protocol> allow|deny [!]acl ...
>
>         The required "protocol" parameter is either an all-caps word OTHER or an
>         explicit protocol name (e.g. "WebSocket") optionally followed by a slash
>         and a version token (e.g. "HTTP/3"). Explicit protocol names and
>         versions are case sensitive.

> That's why I used "WebSocket" instead of "websocket" in my example.
> To avoid confusion, cf.data.pre could be updated to be more clear.

My email saying "websocket" was based on an actual traffic sample that I
have seen. I agree that the text should be changed, but I would prefer
that this change is done by somebody more knowledgeable about the
prevailing WebSocket spelling(s) in Upgrade headers (than I am).


> https://drive.google.com/file/d/1jryX5BW4yxLTSBe0QDavPSiKLBpOvtnV/view?usp=sharing
> https://drive.google.com/file/d/1QYRr-0F-DGnCZtyuuAw8RsEgcHICN_0c/view?usp=sharing

> The connection to wss://ed1lncb62801.webex.com/direct?type=websocket&dtype=binary&rand=1602830016480&uuidtag=5659FGE6-DF29-47A7-859A-G4D5FDC937A2&gatewayip=PUB_IPv4_ADDR_2 was interrupted while the page was loading.

AFAICT, Squid did not see this particular HTTP Upgrade request. While
there are approximately 6 requests mentioning
5659FGE6-DF29-47A7-859A-G4D5FDC937A2, none of them have
rand=1602830016480. It is possible that the random number changes in the
middle of a connection, I guess, but that does not help with
identification of the relevant HTTP transaction.


> Thanks for all the help you can give me.

Debugging logs are meant for Squid developers so do not feel bad about
being unable to use them in this triage. Unfortunately, I cannot help
you with interpreting this particular log right now because the log
contains too many transactions -- I do not know which transaction
corresponds to the client complaint, and I do not have enough time to
look through all 50+ HTTP requests in the log.

One way you can help is to use browser (or client-side) debugging tools
to identify the client port (or at least the millisecond-precision
timestamps) of the problematic transaction. This can help narrow down
the suspects in the new set of logs.


FWIW, I see 467 TLS server handshake parsing "state < atHelloReceived"
failures in your log, but I did not investigate further. They may be
normal/expected or problematic but unrelated to the problem at hand.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users