Large memory leak with ssl_peek (now partly understood)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Large memory leak with ssl_peek (now partly understood)

Steve Hill

I've been suffering from a significant memory leak on multiple servers
running Squid 3.5 for months, but was unable to reproduce it in a test
environment.  I've now figured out how to reproduce it and have done
some investigation:

When using TPROXY, Squid generates fake "CONNECT 192.0.2.1:443"
requests, using the IP address that the client connected to.  At
ssl_bump step 1, we peek and Squid generates another fake "CONNECT
example.com:443" request containing the SNI from the client's SSL handshake.

At ssl_bump step 2 we splice the connection and Squid does verification
to make sure that example.com does actually resolve to 192.0.2.1.  If it
doesn't, Squid is supposed to reject the connection in
ClientRequestContext::hostHeaderVerifyFailed() to prevent clients from
manipulating the SNI to bypass ACLs.

Unfortunately, when verification fails, rather than actually dropping
the client's connection, Squid just leaves the client hanging.
Eventually the client (hopefully) times out and drops the connection
itself, but the associated ClientRequestContext is never destroyed.

This is testable by repeatedly executing:
openssl s_client -connect 17.252.76.30:443 -servername
courier.push.apple.com

That is a traffic pattern that we see in the real world and is now
clearly what is triggering the leak: Apple devices make connections to
addresses within the 17.0.0.0/8 network with an SNI of
"courier.push.apple.com".  courier.push.apple.com resolves to a CNAME
pointing to courier-push-apple.com.akadns.net, but
courier-push-apple.com.akadns.net doesn't exist.  Since Squid can't
verify the connection, it won't allow it and after 30 seconds the client
times out.  Each Apple device keeps retrying the connection, leaking a
ClientRequestContext each time, and before long we've leaked several
gigabytes of memory (on some networks I'm seeing 16GB or more of leaked
RAM over 24 hours!).

Unfortunately I'm a bit lost in the Squid code and can't quite figure
out how to gracefully terminate the connection and destroy the context.

--
  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Sales / enquiries:
    Email:            [hidden email]
    Phone:            +44-1792-824568 / sip:[hidden email]

Support:
    Email:            [hidden email]
    Phone:            +44-1792-825748 / sip:[hidden email]
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Large memory leak with ssl_peek (now partly understood)

Alex Rousskov
On 08/11/2016 10:56 AM, Steve Hill wrote:

> At ssl_bump step 2 we splice the connection and Squid does verification
...
> Unfortunately, when verification fails, rather than actually dropping
> the client's connection, Squid just leaves the client hanging.

Hi Steve,

    This sounds very similar to Squid bug 4508. Factory proposed a fix
for that bug, but the patch is for Squid v4. You may be able to adapt it
to v3. Testing (with any version) is very welcomed, of course:

  http://bugs.squid-cache.org/show_bug.cgi?id=4508

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Large memory leak with ssl_peek (now partly understood)

djch
Pretty sure this is affecting our 3.5.x systems as well — we use a very similar splicing implementation.

I'll keep an eye out in hope someone adapts that patch!

Dan

On 12 August 2016 at 06:22, Alex Rousskov <[hidden email]> wrote:
On 08/11/2016 10:56 AM, Steve Hill wrote:

> At ssl_bump step 2 we splice the connection and Squid does verification
...
> Unfortunately, when verification fails, rather than actually dropping
> the client's connection, Squid just leaves the client hanging.

Hi Steve,

    This sounds very similar to Squid bug 4508. Factory proposed a fix
for that bug, but the patch is for Squid v4. You may be able to adapt it
to v3. Testing (with any version) is very welcomed, of course:

  http://bugs.squid-cache.org/show_bug.cgi?id=4508

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Large memory leak with ssl_peek (now partly understood)

Steve Hill

>         This sounds very similar to Squid bug 4508. Factory proposed a fix
>     for that bug, but the patch is for Squid v4. You may be able to adapt it
>     to v3. Testing (with any version) is very welcomed, of course:

Thanks for that - I'll look into adapting and testing it.

(been chasing this bug off and on for months - hadn't spotted that there
was a bug report open for it :)


--
  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Sales / enquiries:
    Email:            [hidden email]
    Phone:            +44-1792-824568 / sip:[hidden email]

Support:
    Email:            [hidden email]
    Phone:            +44-1792-825748 / sip:[hidden email]
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Large memory leak with ssl_peek (now partly understood)

djch
Hey Steve,

Deployed a 3.5.20 build with both of those patches and have noticed a big improvement in memory consumption of squid processes at a couple of splice-heavy sites.

Thank you, sir!

Dan

> On 12 Aug 2016, at 7:05 PM, Steve Hill <[hidden email]> wrote:
>
>
>>        This sounds very similar to Squid bug 4508. Factory proposed a fix
>>    for that bug, but the patch is for Squid v4. You may be able to adapt it
>>    to v3. Testing (with any version) is very welcomed, of course:
>
> Thanks for that - I'll look into adapting and testing it.
>
> (been chasing this bug off and on for months - hadn't spotted that there was a bug report open for it :)
>
>
> --
> - Steve Hill
>   Technical Director
>   Opendium Limited     http://www.opendium.com
>
> Sales / enquiries:
>   Email:            [hidden email]
>   Phone:            +44-1792-824568 / sip:[hidden email]
>
> Support:
>   Email:            [hidden email]
>   Phone:            +44-1792-825748 / sip:[hidden email]
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Large memory leak with ssl_peek (now partly understood)

Steve Hill
On 17/08/16 06:22, Dan Charlesworth wrote:

> Deployed a 3.5.20 build with both of those patches and have noticed a big improvement in memory consumption of squid processes at a couple of splice-heavy sites.
>
> Thank you, sir!

We've now started tentatively rolling this out to a few production sites
too and are seeing good results so far.


--
  - Steve Hill
    Technical Director
    Opendium    Online Safety / Web Filtering    http://www.opendium.com

    Enquiries                 Support
    ---------                 -------
    [hidden email]        [hidden email]
    +44-1792-824568           +44-1792-825748
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users