Question: Force the caching of 302 responses without Expires header and with Strict-Transport-Security max-age header?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Question: Force the caching of 302 responses without Expires header and with Strict-Transport-Security max-age header?

Andrei Pozolotin

Hello.


1. this question was asked before, but not yet resolved:

http://www.squid-cache.org/mail-archive/squid-users/200701/0000.html


2. use case:

the following url goes though double redirect, both times not providing "Expires:" header,

which results in repeated TCP_MISS/302 entries in the squid logs:

2020-Jan-03 17:45:14    125 192.168.1.106 TCP_MISS/302 565 GET https://archive.archlinux.org/repos/2020/01/01/community/os/x86_64/python-wheel-0.33.6-3-any.pkg.tar.xz - HIER_DIRECT/88.198.91.70 text/html                                   

2020-Jan-03 17:45:14     82 192.168.1.106 TCP_MISS/302 461 GET https://archive.org/download/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz - HIER_DIRECT/207.241.224.2 text/html                                             

2020-Jan-03 17:45:14    215 192.168.1.106 NONE/200 0 CONNECT ia803100.us.archive.org:443 - HIER_DIRECT/207.241.232.150 -   

2020-Jan-03 17:45:14      1 192.168.1.106 TCP_HIT/200 38605 GET https://ia803100.us.archive.org/6/items/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz - HIER_NONE/- application/octet-stream                               


3. here are response details via curl:

a)

curl --head https://archive.archlinux.org/repos/2020/01/01/community/os/x86_64/python-wheel-0.33.6-3-any.pkg.tar.xz

HTTP/2 302  
server: nginx/1.16.1
date: Fri, 03 Jan 2020 17:56:14 GMT
content-type: text/html
content-length: 145
location: https://archive.org/download/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz
strict-transport-security: max-age=31536000; includeSubdomains; preload

b)

curl --head https://archive.org/download/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz

HTTP/1.1 302 Found
Server: nginx/1.14.0 (Ubuntu)
Date: Fri, 03 Jan 2020 17:56:42 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Accept-Ranges: bytes
Location: https://ia803100.us.archive.org/6/items/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz
Strict-Transport-Security: max-age=15724800

4. it seems that Strict-Transport-Security: max-age header is ignored here by squid 


5. any attempt to use any of the refresh_pattern options also has no effect:

http://www.squid-cache.org/Doc/config/refresh_pattern/


6. full squid.conf is posted here:

https://github.com/random-python/nspawn/blob/master/src/main/nspawn/app/hatcher/service/image-proxy/etc/squid/squid.conf


Question: how can one force the caching of 302 responses

without the Expires header and with Strict-Transport-Security max-age header?


Thank you.



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question: Force the caching of 302 responses without Expires header and with Strict-Transport-Security max-age header?

Alex Rousskov
On 1/3/20 11:14, Andrei Pozolotin wrote:

> 3. here are response details via curl:
>
> a)
>
> curl --head
> https://archive.archlinux.org/repos/2020/01/01/community/os/x86_64/python-wheel-0.33.6-3-any.pkg.tar.xz
>
> HTTP/2 302
> server: nginx/1.16.1
> date: Fri, 03 Jan 2020 17:56:14 GMT
> content-type: text/html
> content-length: 145
> location:
> https://archive.org/download/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz 
>
> strict-transport-security: max-age=31536000; includeSubdomains; preload
>
> b)
>
> curl --head
> https://archive.org/download/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz
>
> HTTP/1.1 302 Found
> Server: nginx/1.14.0 (Ubuntu)
> Date: Fri, 03 Jan 2020 17:56:42 GMT
> Content-Type: text/html; charset=UTF-8
> Connection: keep-alive
> Accept-Ranges: bytes
> Location:
> https://ia803100.us.archive.org/6/items/archlinux_pkg_python-wheel/python-wheel-0.33.6-3-any.pkg.tar.xz 
>
> Strict-Transport-Security: max-age=15724800
>
> 4. it seems that Strict-Transport-Security: max-age header is ignored
> here by squid


Correct. Squid does not know anything about the
Strict-Transport-Security header. The header is treated like an
extension header (i.e. it is usually forwarded without interpreting its
value).


> 5. any attempt to use any of the refresh_pattern options also has no effect:
>
> http://www.squid-cache.org/Doc/config/refresh_pattern/

Yes, the decision to avoid caching of 302 responses without Expires is
hard-coded. It is made before refresh_pattern is consulted AFAICT.


> Question: how can one force the caching of 302 responses
> without the Expires header and with Strict-Transport-Security max-age
> header?


You can modify Squid to handle Strict-Transport-Security specially or
you can write an ICAP or eCAP service that would add a "more standard"
Cache-Control:max-age header to the response (with even more work, it
would be possible to drop the added response header before it leaves Squid).


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question: Force the caching of 302 responses without Expires header and with Strict-Transport-Security max-age header?

Andrei Pozolotin
Alex:

On 2020-01-03 14:19, Alex Rousskov wrote:

>> Question: how can one force the caching of 302 responses
>> without the Expires header and with Strict-Transport-Security max-age
>> header?
>
>
> You can modify Squid to handle Strict-Transport-Security specially or
> you can write an ICAP or eCAP service that would add a "more standard"
> Cache-Control:max-age header to the response (with even more work, it
> would be possible to drop the added response header before it leaves
> Squid).

1. thank you for your suggestions

2. just to confirm I got this right:

there is no way to use any current squid configuration options
or any existing squid plugins to cache 302 responses without Expires
header,
instead must write some brand new code, correct?

Andrei
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question: Force the caching of 302 responses without Expires header and with Strict-Transport-Security max-age header?

Amos Jeffries
Administrator
On 4/01/20 11:49 pm, Andrei Pozolotin wrote:

> Alex:
>
> On 2020-01-03 14:19, Alex Rousskov wrote:
>>> Question: how can one force the caching of 302 responses
>>> without the Expires header and with Strict-Transport-Security max-age
>>> header?
>>
>>
>> You can modify Squid to handle Strict-Transport-Security specially or
>> you can write an ICAP or eCAP service that would add a "more standard"
>> Cache-Control:max-age header to the response (with even more work, it
>> would be possible to drop the added response header before it leaves
>> Squid).
>
> 1. thank you for your suggestions
>
> 2. just to confirm I got this right:
>
> there is no way to use any current squid configuration options
> or any existing squid plugins to cache 302 responses without Expires
> header,
> instead must write some brand new code, correct?

Expires header is an HTTP/1.0 protocol feature. Its absence has no meaning.

The 302 response is explicitly defined in HTTP as a *temporary* object
which can change at any time. The *presence* of Cache-Control:max-age or
Expires set a minimum time the response is guaranteed not to change.



Since your use-case is a software archive mirrors you should investigate
whether the objects stored there are truly identical. If they are, the
Store-ID feature can be used to de-duplicate the URLs the 302 are
pointing at so *they* are cached efficiently.
 <https://wiki.squid-cache.org/Features/StoreID>


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question: Force the caching of 302 responses without Expires header and with Strict-Transport-Security max-age header?

Andrei Pozolotin
Amos, hello:

On 2020-01-04 05:14, Amos Jeffries wrote:
> Expires header is an HTTP/1.0 protocol feature. Its absence has no
> meaning.
> The 302 response is explicitly defined in HTTP as a *temporary* object
> which can change at any time. The *presence* of Cache-Control:max-age
> or
> Expires set a minimum time the response is guaranteed not to change.

1. perhaps an argument could be made that these are semantically
identical:
* Cache-Control: max-age=<expire-time>
* Strict-Transport-Security: max-age=<expire-time>

2. and therefore "Strict-Transport-Security" should be handled
by squid "Cache-Control" related features such as refresh_pattern
http://www.squid-cache.org/Doc/config/refresh_pattern/

> Since your use-case is a software archive mirrors you should
> investigate
> whether the objects stored there are truly identical. If they are, the
> Store-ID feature can be used to de-duplicate the URLs the 302 are
> pointing at so *they* are cached efficiently.
>  <https://wiki.squid-cache.org/Features/StoreID>

3. thank you for the StoreID idea

4. I have already implemented it:
https://github.com/random-python/nspawn/tree/master/src/main/nspawn/app/hatcher/service/image-proxy/etc/squid

5. it does improve performance, however two preceding TCP_MISS/302 hits
for every archive url hit, do provide major contribution to the overall
response delay

Thanks again,

Andrei.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question: Force the caching of 302 responses without Expires header and with Strict-Transport-Security max-age header?

Amos Jeffries
Administrator
On 5/01/20 7:24 am, Andrei Pozolotin wrote:

> Amos, hello:
>
> On 2020-01-04 05:14, Amos Jeffries wrote:
>> Expires header is an HTTP/1.0 protocol feature. Its absence has no
>> meaning.
>> The 302 response is explicitly defined in HTTP as a *temporary* object
>> which can change at any time. The *presence* of Cache-Control:max-age or
>> Expires set a minimum time the response is guaranteed not to change.
>
> 1. perhaps an argument could be made that these are semantically identical:
> * Cache-Control: max-age=<expire-time>
> * Strict-Transport-Security: max-age=<expire-time>
>

They are not. One relates to hop-by-hop message storage. The other
relates to end-to-end connection setup.


> 2. and therefore "Strict-Transport-Security" should be handled
> by squid "Cache-Control" related features such as refresh_pattern
> http://www.squid-cache.org/Doc/config/refresh_pattern/
>

As Alex said Squid does nothing with Strict-Transport-Security headers.
They are for the client UA software, irrelevant to middleware like Squid.


>> Since your use-case is a software archive mirrors you should investigate
>> whether the objects stored there are truly identical. If they are, the
>> Store-ID feature can be used to de-duplicate the URLs the 302 are
>> pointing at so *they* are cached efficiently.
>>  <https://wiki.squid-cache.org/Features/StoreID>
>
> 3. thank you for the StoreID idea
>
> 4. I have already implemented it:
> https://github.com/random-python/nspawn/tree/master/src/main/nspawn/app/hatcher/service/image-proxy/etc/squid
>
>
> 5. it does improve performance, however two preceding TCP_MISS/302 hits
> for every archive url hit, do provide major contribution to the overall
> response delay


(Warning: I have not tested this idea yet, if it does not work it can
break the downloads completely. Treat with extreme care).

You may be able to improve that a little by adding the original 302 URL
to the Store-ID map. However you MUST then add a store_miss rule to
prevent those URLs being stored in the cache.

The idea being that one one of the real download objects is stored Squid
use it as a substitute for the 302. But the 302 payload can never be
used as a substitute for the real object.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users