seeking assistance for home users wanting to cache https contents

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

seeking assistance for home users wanting to cache https contents

Michael Davis
okay, so I'm working on making a public github repository for others like me out there that are having such a hard time with this, given the state of the web being almost completely run via SSL websites, who want to use squid for bandwidth easing in these times of everyone being stuck in home isolation, but given that this literally constitutes making what is by design a man in the middle attack, I am finding it more difficult than learning to do brain surgery (I am not a surgeon).

my goal is to set up squid so that it can properly decrypt SSL traffic for my own local devices, I own everything on this network, so this is not an ethical problem for me given I am the sole user of everything on my own network, and I want to PROPERLY be able to cache contents that are otherwise delivered by SSL (nvidia graphics driver updates, Microsoft updates (if I can do so without WSUS, verdict on this one still highly fuzzy) web content, such as for example twitter contents, facebook posts/videos/pictures, images on image sites like Photobucket and others) given most web content is delivered by SSL these days, NOT having it work using MITM setup is kind of impossible to actually cache data in this day in age.

I have done extensive research, and even after having another member of the pfsense community join my attempts at this, we both are at a loss on how to correctly set up peek and splice to do the job were after here, that being decrypting SSL traffic for local lans (yes we both know the implications, but they are in both of our cases our own property and networks and we are both the only people using them individually respectively) and yes we both have also installed our local certificate authority certificates on our devices to let it work properly, we just don't seem to understand enough on how peek and splice is supposed to work, to implement it properly, and thus, the guy I'm working with on this suggested we reach out to this mailing list and ask those here that understand it more than we do.

so, could we kindly request some assistance in understanding this and how to implement it please? I will admit, the guy I'm working with understands this far better than I do myself, however I figured I'd reach out on my end given that I'm the guy that's publishing the information into the public github repo I made for this.

I am NOT the most knowledgeable on networking, I will straight up admit that, I learn by trial and error and am almost completely self taught on what I know, so please bear with me if it takes me a little bit to understand a given term or other item if I'm a little bit slow to grasp it.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: seeking assistance for home users wanting to cache https contents

Amos Jeffries
Administrator
On 31/08/20 7:51 am, Michael Davis wrote:
> okay, so I'm working on making a public github repository for others
> like me out there that are having such a hard time with this, given the
> state of the web being almost completely run via SSL websites, who want
> to use squid for bandwidth easing in these times of everyone being stuck
> in home isolation, but given that this literally constitutes making what
> is by design a man in the middle attack, I am finding it more difficult
> than learning to do brain surgery (I am not a surgeon).

FYI, a github repository is not always the right answer. Everyones needs
are slightly different, so what we have in the Squid Project is a wiki
of examples with enough explanation that people should be able to make
the small changes necessary for their needs.

 <https://wiki.squid-cache.org/ConfigExamples/>

I have updated
<https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpWithIntermediateCA>
to clarify the squid.conf port lines you need.

<https://wiki.squid-cache.org/Features/SslPeekAndSplice> documents the
ssl_bump access controls.


>
> my goal is to set up squid so that it can properly decrypt SSL traffic
> for my own local devices, I own everything on this network, so this is
> not an ethical problem for me given I am the sole user of everything on
> my own network, and I want to PROPERLY be able to cache contents that
> are otherwise delivered by SSL (nvidia graphics driver updates,

FYI, "proper" caching has nothing to do with SSL-Bump.

Separate any changes you want in relation to caching from the SSL-Bump
changes. Test each set of changes independently to get one feature going
before you move on to the other.


>
> I have done extensive research, and even after having another member of

Unfortunately TLS is one topic where things have been very volatile. So
the more research you do may expose you to outdated and/or irrelevant
details that just add confusion.

If you have two confusing sources of information (including archived
mailing list replies) go with the official wiki page as authoritative.
Or ask here, that is what this mailing list is for.


> the pfsense community join my attempts at this, we both are at a loss on

FYI, pfsense should not be relevant to SSL-Bump. Like caching the two
features can be used, but are not directly related to each other. So
setup, test and get each working separately.


> how to correctly set up peek and splice to do the job were after here,
> that being decrypting SSL traffic for local lans (yes we both know the
> implications, but they are in both of our cases our own property and
> networks and we are both the only people using them individually
> respectively) and yes we both have also installed our local certificate
> authority certificates on our devices to let it work properly, we just
> don't seem to understand enough on how peek and splice is supposed to
> work, to implement it properly, and thus, the guy I'm working with on
> this suggested we reach out to this mailing list and ask those here that
> understand it more than we do.
>

So what I am understanding from your description is that you are trying to:
 A) intercept traffic with pfsense
 B) SSL-Bump the TLS which arrives at the proxy
 C) cache the decrypted HTTP messages

Is that correct?


The pfsense portion I cannot help much with right now, it has been too
long since I worked with that software.

All I can say is:

 1) the config examples we have in the wiki for setting up the
interception part should work fine, and

 2) test it *without* worrying about SSL-Bump or caching. Make sure it
works before going on to the other features, and

 3) "working" for the port 443 intercept (no bumping) can take the form
of HTTP error messages from Squid or client rejecting TLS handshake
details from Squid. Both of these mean the traffic is reaching Squid and
client getting whatever Squid produces.


For SSL-Bump when (after pfsense intercept is working) you want to
follow
<https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpWithIntermediateCA>
to get a successful TLS handshake happening.

You can use these bare-bones ssl_bump settings to pass the traffic
through Squid without decrypt to start with:

  acl step1 at_step SslBump1
  ssl_bump peek step1
  ssl_bump splice all


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: seeking assistance for home users wanting to cache https contents

Amos Jeffries
Administrator
On 1/09/20 6:52 am, Michael Davis wrote:

> •The github repo is actually for refresh patterns specifically
>
> •I extensively read over almost every squid wiki page, but unfortunately
> my attempts to understand them felt kind of useless, i did read likely
> well over fifty different attempts guides, and found myself realizing to
> your point, most info IS highly outdated
>
> •my config for everything is fine so far, so we can ignore pfsense snd
> things unrelated to squid safely, thank heavens for that one
>
> • as far as i can tell, squid is properly?? Caching non ssl delivered
> contents *disclaimer, i am fully guessing on this one* it shows bulk
> access logs of tcp_miss/200 | tcp_tunnel/200 | tcp_inm_hit/200 |
> none/200 real time results


Okay. So some HTTP(S) is happening.

You can possibly do some finer grain tests to see which of these are for
regular http:// requests by the Browser and which are for https://
traffic. That is, checking to see which of port 80 and 443 got
intercepted and how the log looks from that.


>
> • "FYI, "proper" caching has nothing to do with SSL-Bump.
>
> Separate any changes you want in relation to caching from the SSL-Bump
> changes. Test each set of changes independently to get one feature going
> before you move on to the other." ...... much as i don't want to admit
> this,  THAT should have been obvious, but i think i might have over
> looked that fact....
>
> ● "So what I am understanding from your description is that you are
> trying to:
>  A) intercept traffic with pfsense
>  B) SSL-Bump the TLS which arrives at the proxy
>  C) cache the decrypted HTTP messages
>
> Is that correct?"  THAT is exactly what my goals are, yes.
>
> • as for that bare bones config example, that's exactly what im using
> right now, but admittedly im not knowledgeable enough to fully and
> confidently tell if it's working at this point, i feel it isnt however
>


IIRC SSL-Bump traffic from that minimal test ssl_bump config should log
as one NONE/200 [if I remember wrong this might be TCP_TUNNEL/200] with
"CONNECT raw-IP:port" (the "peek" happening) followed by a
TCP_TUNNEL/200 (the "splice") with "CONNECT domain:port" - depending on
client sending TLS SNI.

Squid does not have any part in the TLS handshakes for that minimal
config, so any TLS errors are between the client and origin server.


If that all checks out as what you are seeing. Adding a "stare step2"
instead of splice, and a "bump all" final line starts Squid decrypting.
You should see the two CONNECT transactions mentioned above will still
be logged, maybe a third one as well from the "stare" with updated
domain name from the server cert. But the main sign of bump happening is
that log now also contains normal GET/POST etc lines where the URL
starts with "https://" scheme.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users