tcp_outgoing_address and HTTPS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

tcp_outgoing_address and HTTPS

Michael Pro
squid-5 master branch, not have  personal/private repository changes,
not use  cache_peer's ability, (if it's matters - not use transparent
proxying ability).

We have a set of rules (ACL's with url regex) for content, depending
on which we make a decision for the outgoing address, for example,
from 10.10.1.xx to 10.10.6.xx
-----log 1part {{{ -----
Acl.cc(151) matches: checked: tcp_outgoing_address 10.10.5.11 = 1
Checklist.cc(63) markFinished: 0x7fffffffe2b8 answer ALLOWED for match
FilledChecklist.cc(67) ~ACLFilledChecklist: ACLFilledChecklist
destroyed 0x7fffffffe2b8
Checklist.cc(197) ~ACLChecklist: ACLChecklist::~ACLChecklist:
destroyed 0x7fffffffe2b8
peer_select.cc(1026) handlePath: PeerSelector3438 found
local=10.10.5.11 remote=17.253.37.204:80 HIER_DIRECT flags=1,
destination #2 for http://iosapps.itunes.apple.com/...xxx...ipa
...
peer_select.cc(1002) interestedInitiator: PeerSelector3438
peer_select.cc(112) ~PeerSelector: http://iosapps.itunes.apple.com/...xxx...ipa
store.cc(464) unlock: peerSelect unlocking key
60080000000000001C0E000001000000 e:=p2IWV/0x815c09500*3
AsyncCallQueue.cc(55) fireNext: entering AsyncJob::start()
AsyncCall.cc(38) make: make call AsyncJob::start [call195753]
AsyncJob.cc(123) callStart: Comm::ConnOpener status in: [ job10909]
comm.cc(348) comm_openex: comm_openex: Attempt open socket for: 10.10.5.11
comm.cc(391) comm_openex: comm_openex: Opened socket local=10.10.5.11
remote=[::] FD 114 flags=1 : family=2, type=1, protocol=6
-----log 1part }}} -----
In the case of normal traffic (http), everything works fine, as shuld.

In the case of HTTPS with traffic analysis (ssl_bump) we have such a picture:
-----log 2part {{{ ------
Acl.cc(151) matches: checked: tcp_outgoing_address 10.10.5.120 = 1
Checklist.cc(63) markFinished: 0x7fffffffe2b8 answer ALLOWED for match
FilledChecklist.cc(67) ~ACLFilledChecklist: ACLFilledChecklist
destroyed 0x7fffffffe2b8
Checklist.cc(197) ~ACLChecklist: ACLChecklist::~ACLChecklist:
destroyed 0x7fffffffe2b8
peer_select.cc(1026) handlePath: PeerSelector569 found
local=10.10.5.120 remote=23.16.9.11:443 PINNED flags=1, destination #1
for https://some.https.com/...xxx...zip
peer_select.cc(1027) handlePath: always_direct = DENIED
peer_select.cc(1028) handlePath: never_direct = DENIED
peer_select.cc(1029) handlePath: timedout = 0
peer_select.cc(1002) interestedInitiator: PeerSelector569
FwdState.cc(443) startConnectionOrFail: https://some.https.com/...xxx...zip
HttpRequest.cc(472) clearError: old error details: 0/0
FwdState.cc(886) connectStart: fwdConnectStart:
https://some.https.com/...xxx...zip
FwdState.cc(905) connectStart: pinned peer connection: 0x812c51018
client_side.cc(4082) borrowPinnedConnection: local=10.10.2.120:47901
remote=23.16.9.11:443 HIER_DIRECT FD 28 flags=1
client_side.cc(4057) validatePinnedConnection: local=10.10.2.120:47901
remote=23.16.9.11:443 HIER_DIRECT FD 28 flags=1
AsyncCall.cc(56) cancel: will not call
ConnStateData::clientPinnedConnectionRead [call20129] because
comm_read_cancel
AsyncCall.cc(56) cancel: will not call
ConnStateData::clientPinnedConnectionRead [call20129] also because
comm_read_cancel
ModKqueue.cc(174) SetSelect: FD 28, type=1, handler=0,
client_data=0x0, timeout=0
comm.cc(964) comm_add_close_handler: comm_add_close_handler: FD 28,
handler=1, data=0x8028bf398
AsyncCall.cc(26) AsyncCall: The AsyncCall SomeCloseHandler
constructed, this=0x802a456d0 [call20139]
comm.cc(975) comm_add_close_handler: comm_add_close_handler: FD 28,
AsyncCall=0x802a456d01
FwdState.cc(987) dispatch: local=127.0.0.1:20990
remote=127.0.0.120:59799 FD 26 flags=1: Fetching GET
https://some.https.com/...xxx...zip
AsyncJob.cc(34) AsyncJob: AsyncJob constructed, this=0x81258fe38
type=HttpStateData [job1763]
store.cc(439) lock: Client locked key 3F020000000000001C0E000001000000
e:=p2IWV/0x812b2df004
...
peer_select.cc(112) ~PeerSelector: https://some.https.com/...xxx...zip
store.cc(464) unlock: peerSelect unlocking key
3F020000000000001C0E000001000000 e:=p2IWV/0x812b2df004
AsyncCallQueue.cc(55) fireNext: entering AsyncJob::start()
AsyncCall.cc(38) make: make call AsyncJob::start [call20141]
AsyncJob.cc(123) callStart: HttpStateData status in: [ job1763]
http.cc(2838) sendRequest: local=10.10.2.120:47901
remote=23.16.9.11:443 HIER_DIRECT FD 28 flags=1, request 0x8125e88005,
this 0x81258fd18.
AsyncCall.cc(26) AsyncCall: The AsyncCall HttpStateData::httpTimeout
constructed, this=0x812492f80 [call20142]
comm.cc(554) commSetConnTimeout: local=10.10.2.120:47901
remote=23.16.9.11:443 HIER_DIRECT FD 28 flags=1 timeout 86400
http.cc(2204) maybeMakeSpaceAvailable: may read up to 131072 bytes
info buf(0/131072) from local=10.10.2.120:47901
remote=213.156.90.131:443 HIER_DIRECT FD 28 flags=1
AsyncCall.cc(26) AsyncCall: The AsyncCall HttpStateData::readReply
constructed, this=0x812493020 [call20143]
Read.cc(57) comm_read_base: comm_read, queueing read for
local=10.10.2.120:47901 remote=23.16.9.11:443 HIER_DIRECT FD 28
flags=1; asynCall 0x812493020*1
ModKqueue.cc(174) SetSelect: FD 28, type=1, handler=1,
client_data=0x80ce04728, timeout=0
AsyncCall.cc(26) AsyncCall: The AsyncCall HttpStateData::wroteLast
constructed, this=0x812493160 [call20144]
-----log 2part }}} -----

I understand that without analyzing the traffic and not knowing the
final goal for the beginning, we can not manage the process further.
Question: how can we break the established channel (unpinn it) along
the old route and establish a new channel along the new route, when we
already know how.

IN 127.0.0.1:443 (22.33.44.55:443 ???) ---> OUT 10.10.1.1 ---> (Catch
22.33.44.55:443/this/is/it.zip) ---> Kill IN ... ??? OUT 10.10.1.1
---> Establish OUT 10.10.5.1 ---> 22.33.44.55:443/this/is/it.zip

I'm willing to pay a large price for traffic congestion in this case,
since the goal justifies it.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: tcp_outgoing_address and HTTPS

Amos Jeffries
Administrator
On 21/03/18 00:11, Michael Pro wrote:

> squid-5 master branch, not have  personal/private repository changes,
> not use  cache_peer's ability, (if it's matters - not use transparent
> proxying ability).
>
> We have a set of rules (ACL's with url regex) for content, depending
> on which we make a decision for the outgoing address, for example,
> from 10.10.1.xx to 10.10.6.xx
> -----log 1part {{{ -----
> Acl.cc(151) matches: checked: tcp_outgoing_address 10.10.5.11 = 1
> Checklist.cc(63) markFinished: 0x7fffffffe2b8 answer ALLOWED for match
> FilledChecklist.cc(67) ~ACLFilledChecklist: ACLFilledChecklist
> destroyed 0x7fffffffe2b8
> Checklist.cc(197) ~ACLChecklist: ACLChecklist::~ACLChecklist:
> destroyed 0x7fffffffe2b8
> peer_select.cc(1026) handlePath: PeerSelector3438 found
> local=10.10.5.11 remote=17.253.37.204:80 HIER_DIRECT flags=1,
> destination #2 for http://iosapps.itunes.apple.com/...xxx...ipa
> ...
> peer_select.cc(1002) interestedInitiator: PeerSelector3438
> peer_select.cc(112) ~PeerSelector: http://iosapps.itunes.apple.com/...xxx...ipa
> store.cc(464) unlock: peerSelect unlocking key
> 60080000000000001C0E000001000000 e:=p2IWV/0x815c09500*3
> AsyncCallQueue.cc(55) fireNext: entering AsyncJob::start()
> AsyncCall.cc(38) make: make call AsyncJob::start [call195753]
> AsyncJob.cc(123) callStart: Comm::ConnOpener status in: [ job10909]
> comm.cc(348) comm_openex: comm_openex: Attempt open socket for: 10.10.5.11
> comm.cc(391) comm_openex: comm_openex: Opened socket local=10.10.5.11
> remote=[::] FD 114 flags=1 : family=2, type=1, protocol=6
> -----log 1part }}} -----
> In the case of normal traffic (http), everything works fine, as shuld.
>

The difference to be aware of is that there is zero security on this
type of HTTP. So while it is better not to play with destinations, and
Squid default is to go where the client wanted - it is permitted to go
elsewhere if a better source is found.


> In the case of HTTPS with traffic analysis (ssl_bump) we have such a picture:
> -----log 2part {{{ ------
> Acl.cc(151) matches: checked: tcp_outgoing_address 10.10.5.120 = 1
> Checklist.cc(63) markFinished: 0x7fffffffe2b8 answer ALLOWED for match
> FilledChecklist.cc(67) ~ACLFilledChecklist: ACLFilledChecklist
> destroyed 0x7fffffffe2b8
> Checklist.cc(197) ~ACLChecklist: ACLChecklist::~ACLChecklist:
> destroyed 0x7fffffffe2b8
> peer_select.cc(1026) handlePath: PeerSelector569 found
> local=10.10.5.120 remote=23.16.9.11:443 PINNED flags=1, destination #1
> for https://some.https.com/...xxx...zip

What PINNED means to Squid is that the client TCP connection is tied up
with some details related to some specific TCP server connection.

In this case the TLS crypto used during the bumping process took crypto
details from the client connection and gave them to the server, then
from the server and gave them to the client. Resulting in a forced
end-to-end relationship between the clinet and server for all traffic
over both those connections.
 The only thing Squid can do is to server some content from cache as
normal HITs, or if you specifically configure ICAP/eCAP service they can
modify the messages as they flow. Delivering the traffic to another
server is not permissable because the HTTP messages can (and often are)
tied to the TLS crypto details as well in ways that are not visible to
Squid.

For example; it is becoming very popular for the endpoints to crypto
sign messages or embed a hash signature which can only be verified valid
using details the server and client exchanged up front. No other server
would be able to send valid values and the client breaks if it is wrong.
 This kind of thing survives even when SSL-Bump'ing because of Squid
pinning, but does add the restrictions you found.


>
> I understand that without analyzing the traffic and not knowing the
> final goal for the beginning, we can not manage the process further.
> Question: how can we break the established channel (unpinn it) along
> the old route and establish a new channel along the new route, when we
> already know how.

There are three possibilities that I am aware of - in no particular order:

1) An ICAP service can do whatever it pleases with requests it receives.
We hold no responsibility for anything happening there and I publicly
advise against playing with the crypto that way - the above issues are
the least of the problems to be faced.


2) It is technically possible to make Squid open a CONNECT tunnel
through an HTTP peer proxy to the origin instead of going there
directly. The only thing preventing this is nobody writing the necessary
code.

It has been on my (and many others) wishlist for a long while but still
nobody has been able to work on it. Any assistance towards getting that
coded is very, very welcome.


3) The client-first type of bumping does not involve any server crypto.
This is *highly* unsafe and often encounters problems like the ICAP
approach and for all the same reasons.

BUT that said, if you are sufficiently in control of the traffic to be
sure of its safety then Squid can do it by simply using the bump action
at "step 1" of the SSL-Bump process - rather than the recommended step 2
or 3.

We have err'ed on the side of security with these things so it may not
work right now. But it is not technically required to pin for this
specific type of bumped connection - so the case could be made that it
is a bug to fix if Squid does force pinning on that traffic.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: tcp_outgoing_address and HTTPS

Alex Rousskov
In reply to this post by Michael Pro
On 03/20/2018 05:11 AM, Michael Pro wrote:

> Question: how can we break the established channel (unpinn it) along
> the old route and establish a new channel along the new route, when we
> already know how.

Squid supports using multiple sequential connections for the same
from-client request, but not under the conditions you describe.
Moreover, the already supported cases are limited to simpler HTTP/TCP
failures. Complex code modifications would be required to support what
you want for HTTPS, but it is doable, and others have wanted a similar
"peek and then start from scratch" feature.


> I'm willing to pay a large price for traffic congestion in this case,
> since the goal justifies it.
Please note that some origin servers may have a different opinion about
this trade off: Some might view (repeatedly) terminated innocent
sessions as an attack and block all your traffic.


Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: [squid-dev] tcp_outgoing_address and HTTPS

Alex Rousskov
In reply to this post by Amos Jeffries
On 03/20/2018 07:55 AM, Amos Jeffries wrote:
> 2) It is technically possible to make Squid open a CONNECT tunnel
> through an HTTP peer proxy to the origin instead of going there
> directly. The only thing preventing this is nobody writing the necessary
> code.
>
> It has been on my (and many others) wishlist for a long while but still
> nobody has been able to work on it.

FWIW, Factory is working on it now.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: tcp_outgoing_address and HTTPS

Michael Pro
In reply to this post by Michael Pro
Totally agree with you, and at the same time - do not agree. But,
consider the following situation. There is https://site.net/ where
there is 1.jpg and 2.jpg. If I download from this site 1.jpg from the
address 1.1.1.1 and 2.jpg from the address 2.2.2.2. Even more. There
are situations when you need to release a certain connection to the
Internet through a single provider (for example, mobile), but you need
to download the largest file that is never physically downloaded by
this connection. On another no way. Squid theoretically these are
several computers (by the number of incoming connections) and, what
prevents us as different computers from using different outgoing
interfaces even for the same origin address?

I'm not saying that you need to push the unbroken. Look at the problem
from the other side.

For example, in Chrome, I set up a proxy 1.1.1.1 and download
https://site.net/1.jpg. At the same time in Mozilla I set up a proxy
2.2.2.2 and download https://site.net/2.jpg. What's the difference if
you set up the same one squid?
acl 1s-jpg url_regex ...1.jpg
acl 2s-jpg url_regex ... 2.jpg
tcp_outgoing_address 1.1.1.1 1s-jpg
tcp_outgoing_address 2.2.2.2 2s-jpg

Where is the entrance here?

Why do we try to shove it into one hole, if we can divide it into
separate processes. It may even need some new key or ACL to determine
that for these connections to create always new tunnels (TLS, ssl,
certs, ...)
acl separate_this CreateNewTunnelForNewLink

>> Question: how can we break the established channel (unpinn it) along
>> the old route and establish a new channel along the new route, when we
>> already know how.

> Squid supports using multiple sequential connections for the same
> from-client request, but not under the conditions you describe.
> Moreover, the already supported cases are limited to simpler HTTP/TCP
> failures. Complex code modifications would be required to support what
> you want for HTTPS, but it is doable, and others have wanted a similar
> "peek and then start from scratch" feature.


>> I'm willing to pay a large price for traffic congestion in this case,
>> since the goal justifies it.
> Please note that some origin servers may have a different opinion about
> this trade off: Some might view (repeatedly) terminated innocent
> sessions as an attack and block all your traffic.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: tcp_outgoing_address and HTTPS

Alex Rousskov
On 03/20/2018 01:12 PM, Michael Pro wrote:
> Totally agree with you, and at the same time - do not agree.

AFAICT, I only stated facts, not opinions. There is nothing to agree or
disagree with in my response.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: tcp_outgoing_address and HTTPS

Amos Jeffries
Administrator
In reply to this post by Michael Pro
On 21/03/18 08:12, Michael Pro wrote:
> Totally agree with you, and at the same time - do not agree. But,
> consider the following situation. There is https://site.net/ where
> there is 1.jpg and 2.jpg. If I download from this site 1.jpg from the
> address 1.1.1.1 and 2.jpg from the address 2.2.2.2.

There is no such concept as "site" in TLS. It is a point-to-point protocol.
The client opened a single connection and sent two requests to what it
perceives to be a *single* server. If the interception proxy were not
there the content would have been served by that same server anyway.
There is no loss for the proxy to mimic the exact behaviour of the real
client.
 Also the responses have to be served to the client sequentially. So
there is little gain from fetching them any way but sequentially, AND
going to all the trouble of multiple TCP + TLS handshakes adds CPU + RAM
+ socket + time costs to the transaction. So it is a net negative to do
as proposed.


> Even more. There
> are situations when you need to release a certain connection to the
> Internet through a single provider (for example, mobile), but you need
> to download the largest file that is never physically downloaded by
> this connection. On another no way. Squid theoretically these are
> several computers (by the number of incoming connections) and, what
> prevents us as different computers from using different outgoing
> interfaces even for the same origin address?

The machine-specific TLS crypto keys. In RSA it was possible to copy
these keys between machines (but considered very bad practice). In DH
and ECDH new secret keys are generated for every individual TLS
handshake. They cannot be shared. Once those keys are started being used
the data inside (particular signed items) is locked to them.

To stop an HTTPS transfer mid-delivery requires the proxy to abort both
client and server TLS (and TCP) connections. Which is what the pinning does.

>
> I'm not saying that you need to push the unbroken.

That sentence does not compute for me.

> Look at the problem
> from the other side.
>
> For example, in Chrome, I set up a proxy 1.1.1.1 and download
> https://site.net/1.jpg. At the same time in Mozilla I set up a proxy
> 2.2.2.2 and download https://site.net/2.jpg. What's the difference if
> you set up the same one squid?

The differences are:
1) Squid is not a browser.
2) Squid is not the TLS "end-client".
2) Squid is not the TLS origin server.
3) different TLS sessions
4) different client TLS security keys
5) different server TLS security keys

Overall there is a 3-way TLS "origin":

  Chrome + 1.1.1.1:port + the specific IP:443 address of "site.net" that
Chrome chose to connect to.

  Mozilla + 2.2.2.2:port + the specific IP:443 address of "site.net"
that Mozilla chose to connect to.


> acl 1s-jpg url_regex ...1.jpg
> acl 2s-jpg url_regex ... 2.jpg
> tcp_outgoing_address 1.1.1.1 1s-jpg
> tcp_outgoing_address 2.2.2.2 2s-jpg
>
> Where is the entrance here?

Please explain this word "entrance" as you mean it?

1.1.1.1 does not hold the dynamically created TLS state inside 2.2.2.2.

2.2.2.2 does not hold the dynamically created TLS state inside 1.1.1.1.

Squid does not hold the TLS state inside the client.

What Squid can do is deliver content in its cache (if caching is
permitted by the origin who generated it). Or deliver the encrypted
traffic to the origin server the client accepted TLS handshake with.


You are perhapse still thinking in traditional caching terms where Squid
is a client independent of the Browser and where TCP connections can be
freely disconnected and rewritten per HTTP request. In plain-text HTTP
that would be true.

When intercepting TLS / HTTPS that is false. The true end-client /
Browser is maintaining client-to-origin state in its TLS properties. For
*precisely* and intentionally the purpose of preventing exactly this
type of traffic rewriting by a proxy.


IF, and *only* if, the client is using "TLS explicit" as defined by TLS
to the proxy and sending regular HTTP requests over that secured
connection can the proxy do its own origin-server choosing freely. Some
people have been calling that type of setup "HTTPS", but really the
existence of proxy choice makes it a lot different from traditional
HTTPS on port 443.


>
> Why do we try to shove it into one hole, if we can divide it into
> separate processes. It may even need some new key or ACL to determine
> that for these connections to create always new tunnels (TLS, ssl,
> certs, ...)
> acl separate_this CreateNewTunnelForNewLink

When ability to generate CONNECTs is added that should not be necessary.
Whether the cache_peer can handle the CONNECT attempt will automatically
determine whether a tunnel is possible as an alternative to
DIRECT/PINNED connection.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users