Microsoft store issues with ssl-bump

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Microsoft store issues with ssl-bump

Eliezer Croitoru-3

I am trying to implement a full SSL-BUMP and I am having trouble with MS Store.

The Windows 10 MS Store tries to connect the domains:

storeedgefd.dsx.mp.microsoft.com

 

which is bypassed from SSL BUMP with a regex and server-name.

For some reason the store claims that there is an issue with the connection.

To resolve this issue what I am dong is resolving the domain: storeedgefd.dsx.mp.microsoft.com

And bypassing it from squid interception which is done with an nftables/iptables REDIRECT.

 

I believe I am not the first to encounter such an issue and looking for a way to resolve this issue without
the dns+fw level bypass.

 

Any hints might help to find and resolve this issue

 

Thanks,

Eliezer

 

  • Squid 5.0.4 on Fedora 33.
  • I can generate a “support file” which contains all squid conf for reproduction of the issue by the dev team.

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

Zoom: Coming soon

 

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Lorenzo Marcantonio
On Tue, Jan 12, 2021 at 10:33:00AM +0200, Eliezer Croitoru wrote:
>
> Any hints might help to find and resolve this issue

From my experience MS Update and probably the store too use custom root
certificates; check if that's the case. It's also possible that that
connection is so hardwired that it doesn't accept a redirect. So it sees
that and become suspicious (Windows Update is extremely suspicious :D)

For some antivirus (avast maybe? I don't remember) the updater actually
checks the server certificate fingerprint so you can't bump it and you
need a special NAT rule for all the fscking IPs it uses (if you set a
proxy it does a connect BY IP and not by name, and the IPs are hardcoded
and not resolved by DNS).

So it is possible you can't bump a store connection (remember that
technically a bump is a MITM intrusion that TLS is explicitely design to
detect!)

--
Lorenzo Marcantonio

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Eliezer Croitoru-3
Even with splice???
This is a weird way of MS Store of handling things

I was sure that when I am using SPLICE it is expected to work.
Maybe there is a way to handle these IP addresses before even peeking, which
should work.
I think that there is some level of a BUMP happening when it shouldn't.
I will try to test it with another proxy which only looks at the SNI.

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]
Zoom: Coming soon


-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of
Lorenzo Marcantonio
Sent: Tuesday, January 12, 2021 10:58 AM
To: [hidden email]
Subject: Re: [squid-users] Microsoft store issues with ssl-bump

On Tue, Jan 12, 2021 at 10:33:00AM +0200, Eliezer Croitoru wrote:
>
> Any hints might help to find and resolve this issue

From my experience MS Update and probably the store too use custom root
certificates; check if that's the case. It's also possible that that
connection is so hardwired that it doesn't accept a redirect. So it sees
that and become suspicious (Windows Update is extremely suspicious :D)

For some antivirus (avast maybe? I don't remember) the updater actually
checks the server certificate fingerprint so you can't bump it and you
need a special NAT rule for all the fscking IPs it uses (if you set a
proxy it does a connect BY IP and not by name, and the IPs are hardcoded
and not resolved by DNS).

So it is possible you can't bump a store connection (remember that
technically a bump is a MITM intrusion that TLS is explicitely design to
detect!)

--
Lorenzo Marcantonio

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Eliezer Croitoru-3
In reply to this post by Lorenzo Marcantonio
This works in another proxy which looks at the SNI only without any bump
involved.
Remember that Squid should splice the connection based on regex and
server-name dst.

On the other proxy this is what I have:
Jan 12 11:12:46 ndpi-fw proxy[497]: 2021/01/12 11:12:46 conn
192.168.189.X:64632 - 104.79.221.20:443 released
[storeedgefd.dsx.mp.microsoft.com:443]
Jan 12 11:12:46 ndpi-fw proxy[497]: 2021/01/12 11:12:46 conn
192.168.189.X:64633 - 104.79.221.20:443 released
[storeedgefd.dsx.mp.microsoft.com:443]
Jan 12 11:12:46 ndpi-fw proxy[497]: 2021/01/12 11:12:46 conn
192.168.189.X:64634 - 104.79.221.20:443 released
[storeedgefd.dsx.mp.microsoft.com:443]
Jan 12 11:12:46 ndpi-fw proxy[497]: 2021/01/12 11:12:46 conn
192.168.189.X:64630 - 104.79.221.20:443 released
[storeedgefd.dsx.mp.microsoft.com:443]
Jan 12 11:12:46 ndpi-fw proxy[497]: 2021/01/12 11:12:46 conn
192.168.189.X:64631 - 104.79.221.20:443 released
[storeedgefd.dsx.mp.microsoft.com:443]
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54
SNI:https://storeedgefd.dsx.mp.microsoft.com:443
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54 use parent : false,
storeedgefd.dsx.mp.microsoft.com:443
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54 ip 192.168.189.X
rate, current: 1/s, max: 20/s
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54 conn
192.168.189.X:64667 - 104.79.221.20:443 connected
[storeedgefd.dsx.mp.microsoft.com:443]
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54
SNI:https://storeedgefd.dsx.mp.microsoft.com:443
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54 use parent : false,
storeedgefd.dsx.mp.microsoft.com:443
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54 ip 192.168.189.X
rate, current: 2/s, max: 20/s
Jan 12 11:12:54 ndpi-fw proxy[497]: 2021/01/12 11:12:54 conn
192.168.189.X:64669 - 104.79.221.20:443 connected
[storeedgefd.dsx.mp.microsoft.com:443]

So the regex:
storeedgefd\.dsx\.mp\.microsoft\.com

should work.

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]
Zoom: Coming soon


-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of
Lorenzo Marcantonio
Sent: Tuesday, January 12, 2021 10:58 AM
To: [hidden email]
Subject: Re: [squid-users] Microsoft store issues with ssl-bump

On Tue, Jan 12, 2021 at 10:33:00AM +0200, Eliezer Croitoru wrote:
>
> Any hints might help to find and resolve this issue

From my experience MS Update and probably the store too use custom root
certificates; check if that's the case. It's also possible that that
connection is so hardwired that it doesn't accept a redirect. So it sees
that and become suspicious (Windows Update is extremely suspicious :D)

For some antivirus (avast maybe? I don't remember) the updater actually
checks the server certificate fingerprint so you can't bump it and you
need a special NAT rule for all the fscking IPs it uses (if you set a
proxy it does a connect BY IP and not by name, and the IPs are hardcoded
and not resolved by DNS).

So it is possible you can't bump a store connection (remember that
technically a bump is a MITM intrusion that TLS is explicitely design to
detect!)

--
Lorenzo Marcantonio

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Amos Jeffries
Administrator
On 12/01/21 10:15 pm, Eliezer Croitoru wrote:
> This works in another proxy which looks at the SNI only without any bump
> involved.

So you are saying you find a bug with Squid?
   or .. ??


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Eliezer Croitoru-3
Im saying that my config might be wrong and I will send you a full config save which can show you the whole setup like most vendors has.
I have upgraded squid in production.

Let me verify first before shouting "bug".

Eliezer

On Tue, Jan 12, 2021, 12:15 Amos Jeffries <[hidden email]> wrote:
On 12/01/21 10:15 pm, Eliezer Croitoru wrote:
> This works in another proxy which looks at the SNI only without any bump
> involved.

So you are saying you find a bug with Squid?
   or .. ??


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Amos Jeffries
Administrator
On 12/01/21 11:32 pm, NgTech LTD wrote:
> Im saying that my config might be wrong and I will send you a full
> config save which can show you the whole setup like most vendors has.
> I have upgraded squid in production.
>
> Let me verify first before shouting "bug".
>
> Eliezer
>

Okay. I see a few things to follow up on.


The other proxy logs show SNI as being
"https://storeedgefd.dsx.mp.microsoft.com:443". SNI should be only a
name, not a full URL. So if we assume that log is correct the client is
producing invalid SNI. This may be an issue for Squid, causing it to
ignore the SNI value entirely.

The openssl tool connecting to the same IP address the other proxy
claims to be going to gets "sfdataservice.microsoft.com" as the server
name. In absence of valid SNI to work with that is the name your Squid
will be trying to match against to decide splice vs bump.


The server prefers to use TLS/1.3 unless explicitly connected to with
TLS/1.2 immediately. IIRC latest Squid force the client to TLS/1.2 when
preparing to bump, but may not for spliceand stare. So YMMV.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Eliezer Croitoru-3
-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Amos Jeffries
Sent: Tuesday, January 12, 2021 2:42 PM
To: Squid Users <[hidden email]>
Subject: Re: [squid-users] Microsoft store issues with ssl-bump

On 12/01/21 11:32 pm, NgTech LTD wrote:
> Im saying that my config might be wrong and I will send you a full
> config save which can show you the whole setup like most vendors has.
> I have upgraded squid in production.
>
> Let me verify first before shouting "bug".
>
> Eliezer
>

> The other proxy logs show SNI as being
> "https://storeedgefd.dsx.mp.microsoft.com:443". SNI should be only a
>name, not a full URL. So if we assume that log is correct the client is
>producing invalid SNI. This may be an issue for Squid, causing it to
> ignore the SNI value entirely.

It’s only fprint the does this with <a href="https://XYZ:port">https://XYZ:port
It sees only the ip + domain(plain SNI) + port


> The openssl tool connecting to the same IP address the other proxy
> claims to be going to gets "sfdataservice.microsoft.com" as the server
> name. In absence of valid SNI to work with that is the name your Squid
> will be trying to match against to decide splice vs bump.

So squid tried to match only the certificate and not the SNI?
From what I see the SNI is ok with the certificate version 3 extensions ie DNS=XYZ
(it should, I cannot verify this against the server at the moment.)


> The server prefers to use TLS/1.3 unless explicitly connected to with
> TLS/1.2 immediately. IIRC latest Squid force the client to TLS/1.2 when
> preparing to bump, but may not for spliceand stare. So YMMV.
OK

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]
Zoom: Coming soon




_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Alex Rousskov
In reply to this post by Amos Jeffries
On 1/12/21 7:42 AM, Amos Jeffries wrote:
> IIRC latest Squid force the client to TLS/1.2 when
> preparing to bump, but may not for spliceand stare. So YMMV.

FTR: Bugs notwithstanding, modern Squid changes nothing on TLS level
when peeking, splicing, and/or terminating. Squid changes TLS bytes when
staring and/or bumping.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Alex Rousskov
In reply to this post by Eliezer Croitoru-3
On 1/12/21 3:33 AM, Eliezer Croitoru wrote:

> The Windows 10 MS Store tries to connect the domains:
> storeedgefd.dsx.mp.microsoft.com

> which is bypassed from SSL BUMP with a regex and server-name.

>   * Squid 5.0.4 on Fedora 33.

It sounds like you have tried to configure Squid to splice traffic
matching some criteria. So does Squid actually splice traffic matching
those criteria? That is the first question I would ask myself when
trying to triage this problem.

Assuming you can create test traffic, there are many ways to answer that
question, including:

1. Checking whether Squid signs Squid-to-client traffic with its own
certificate.

2. After skipping any CONNECT exchanges, comparing to-Squid TCP payload
with from-Squid TCP payload. If the answer to the question is "yes",
then that payload should be identical, in both client-server and
server-client directions.

3. Sharing Squid debugging logs containing an isolated test transaction.

Testing with other proxies and speculating about the magical possibility
of client detection of TLS splicing is a waste of time _if_ your Squid
configuration is incorrect (i.e. if Squid correctly follows its
configuration, but that configuration contradicts your goals). Thus, I
recommend starting by validating that splicing is happening, as
discussed above.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Eliezer Croitoru-3
In reply to this post by Alex Rousskov
Alex,

I am using the next stare rule:
acl tls_s1_connect at_step SslBump1
acl tls_s2_client_hello at_step SslBump2
acl tls_s3_server_hello at_step SslBump3
ssl_bump stare tls_s2_client_hello

Which I am not sure about.
For now this issue seems to be gone.
I don't know why or how but it seems that some IP rotation is happening as we speak/write.
The IP address my service was accessing is different then the one now so I think what Amos
wrote is probably the real reason, ie that the service certificate was for another service CN/DNS Name.
While it's ok for the windows client it's not OK for Squid and any other SNI based certificate validator.

Thanks Helped and Helps,
Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]
Zoom: Coming soon


-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Alex Rousskov
Sent: Tuesday, January 12, 2021 5:15 PM
To: Squid Users <[hidden email]>
Subject: Re: [squid-users] Microsoft store issues with ssl-bump

On 1/12/21 7:42 AM, Amos Jeffries wrote:
> IIRC latest Squid force the client to TLS/1.2 when
> preparing to bump, but may not for spliceand stare. So YMMV.

FTR: Bugs notwithstanding, modern Squid changes nothing on TLS level
when peeking, splicing, and/or terminating. Squid changes TLS bytes when
staring and/or bumping.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Microsoft store issues with ssl-bump

Alex Rousskov
On 1/12/21 10:46 AM, Eliezer Croitoru wrote:

> I am using the next stare rule:
> acl tls_s1_connect at_step SslBump1
> acl tls_s2_client_hello at_step SslBump2
> acl tls_s3_server_hello at_step SslBump3
> ssl_bump stare tls_s2_client_hello

I do not know what you are trying to acheive, but if the above is your
entire ssl_bump configuration, then, bugs notwithstanding, it should be
equivalent to a much simpler one:

  # splice at step1, without looking at SNI
  ssl_bump splice all

Alex.


> -----Original Message-----
> From: squid-users <[hidden email]> On Behalf Of Alex Rousskov
> Sent: Tuesday, January 12, 2021 5:15 PM
> To: Squid Users <[hidden email]>
> Subject: Re: [squid-users] Microsoft store issues with ssl-bump
>
> On 1/12/21 7:42 AM, Amos Jeffries wrote:
>> IIRC latest Squid force the client to TLS/1.2 when
>> preparing to bump, but may not for spliceand stare. So YMMV.
>
> FTR: Bugs notwithstanding, modern Squid changes nothing on TLS level
> when peeking, splicing, and/or terminating. Squid changes TLS bytes when
> staring and/or bumping.
>
> Alex.
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users