Caching https data

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Caching https data

Darwin O'Connor
I run a transit prediction web app <https://www.transsee.ca/>. It
connects to a variety of web APIs to collect the real time data it
needs. The apps activities are split among many processes. They
currently uses libcurl to connect to squid for caching (often for as
little as 10-30 seconds) and benefits of connection sharing, but some of
the APIs use https, so in that case the data passes through squid
without the benefits of caching or connection sharing.

I would like to configure squid to connect to these servers securely and
pass it unencrypted to clients. Security isn't really an issue since
this step is all within the one server. I'll have to configure libcurl
to allow unencrypted data.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Caching https data

Amos Jeffries
Administrator
On 30/07/20 10:34 am, Darwin O'Connor wrote:

> I run a transit prediction web app <https://www.transsee.ca/>. It
> connects to a variety of web APIs to collect the real time data it
> needs. The apps activities are split among many processes. They
> currently uses libcurl to connect to squid for caching (often for as
> little as 10-30 seconds) and benefits of connection sharing, but some of
> the APIs use https, so in that case the data passes through squid
> without the benefits of caching or connection sharing.
>
> I would like to configure squid to connect to these servers securely and
> pass it unencrypted to clients. Security isn't really an issue since
> this step is all within the one server. I'll have to configure libcurl
> to allow unencrypted data.
>

There are several approaches you can take;

1) configure libcurl and/or the apps to send https:// URLs to Squid in
regular HTTP requests. Leaving Squid to handle all the HTTPS portion.
  <>


2) configure libcurl and/or the apps to send https:// URLs to a Squid
secure listening port (https_port) in regular HTTP requests. Leaving
Squid to handle all the HTTPS portion with servers. I'm not sure about
libcurl, but some clients allow this when they prohibit (1) because it
is somewhat more secure.

<https://wiki.squid-cache.org/Features/HTTPS#Encrypted_browser-Squid_connection>


3) SSL-Bump to decrypt the CONNECT tunnels clients tend to prefer when
using an HTTP proxy.
  <https://wiki.squid-cache.org/Features/SslPeekAndSplice>


Current Squid (v4 or later) support (1) and (2) with either GnuTLS or
OpenSSL. (3) requires OpenSSL.

Once decrypted the https:// URLs are subject to normal HTTP caching rules.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Caching https data

Alex Rousskov
On 7/30/20 5:11 AM, Amos Jeffries wrote:

> On 30/07/20 10:34 am, Darwin O'Connor wrote:
>> I run a transit prediction web app <https://www.transsee.ca/>. It
>> connects to a variety of web APIs to collect the real time data it
>> needs. The apps activities are split among many processes. They
>> currently uses libcurl to connect to squid for caching (often for as
>> little as 10-30 seconds) and benefits of connection sharing, but some of
>> the APIs use https, so in that case the data passes through squid
>> without the benefits of caching or connection sharing.
>>
>> I would like to configure squid to connect to these servers securely and
>> pass it unencrypted to clients. Security isn't really an issue since
>> this step is all within the one server. I'll have to configure libcurl
>> to allow unencrypted data.


> There are several approaches you can take;
>
> 1) configure libcurl and/or the apps to send https:// URLs to Squid in
> regular HTTP requests. Leaving Squid to handle all the HTTPS portion.

In 2017, curl did not support "GET https" requests:
https://curl.haxx.se/mail/lib-2017-12/0019.html

AFAICT from the curl v7.68 man page, curl still does not support "GET
https" requests: The https scheme in the request URI implies CONNECT
through the proxy (including through the HTTPS proxy discussed below).

Perhaps there is an API trick to force libcurl into sending "GET https"
requests to proxies. If not, you would have to use SslBump (item 3 on
Amos' list).


> 2) configure libcurl and/or the apps to send https:// URLs to a Squid
> secure listening port (https_port) in regular HTTP requests. Leaving
> Squid to handle all the HTTPS portion with servers. I'm not sure about
> libcurl

Modern curl versions support HTTPS proxies -- a Factory project added
that support to curl. I am pretty sure the library has the same level of
HTTPS proxies support as the command line client. The problem is with
convincing libcurl to send "GET https" requests.


HTH,

Alex.


> <https://wiki.squid-cache.org/Features/HTTPS#Encrypted_browser-Squid_connection>
>
>
> 3) SSL-Bump to decrypt the CONNECT tunnels clients tend to prefer when
> using an HTTP proxy.
>   <https://wiki.squid-cache.org/Features/SslPeekAndSplice>
>
>
> Current Squid (v4 or later) support (1) and (2) with either GnuTLS or
> OpenSSL. (3) requires OpenSSL.
>
> Once decrypted the https:// URLs are subject to normal HTTP caching rules.
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Caching https data

Darwin O'Connor
On 2020-07-30 12:08 p.m., Alex Rousskov wrote:

> On 7/30/20 5:11 AM, Amos Jeffries wrote:
>> On 30/07/20 10:34 am, Darwin O'Connor wrote:
>>> I run a transit prediction web app <https://www.transsee.ca/>. It
>>> connects to a variety of web APIs to collect the real time data it
>>> needs. The apps activities are split among many processes. They
>>> currently uses libcurl to connect to squid for caching (often for as
>>> little as 10-30 seconds) and benefits of connection sharing, but some of
>>> the APIs use https, so in that case the data passes through squid
>>> without the benefits of caching or connection sharing.
>>>
>>> I would like to configure squid to connect to these servers securely and
>>> pass it unencrypted to clients. Security isn't really an issue since
>>> this step is all within the one server. I'll have to configure libcurl
>>> to allow unencrypted data.
>
>> There are several approaches you can take;
>>
>> 1) configure libcurl and/or the apps to send https:// URLs to Squid in
>> regular HTTP requests. Leaving Squid to handle all the HTTPS portion.
> In 2017, curl did not support "GET https" requests:
> https://curl.haxx.se/mail/lib-2017-12/0019.html
>
> AFAICT from the curl v7.68 man page, curl still does not support "GET
> https" requests: The https scheme in the request URI implies CONNECT
> through the proxy (including through the HTTPS proxy discussed below).
>
> Perhaps there is an API trick to force libcurl into sending "GET https"
> requests to proxies. If not, you would have to use SslBump (item 3 on
> Amos' list).

Reading further into the thread you linked I found the suggestion to use
the request-target option of curl. By setting the url to the proxy
location and the request-target to the actual url it is working exactly
the way I want. API trick for the win.


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users