caching apt package lists/Raspbian

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

caching apt package lists/Raspbian

TarotApprentice
Recently upgraded to Raspbian Buster and squid 4.6. Since then I am unable to cache the Packages.xz that apt uses. The various other Pis using this proxy all end up downloading the 30MB Packages.xz every time. Does anyone have any suggestions on how to get it to cache?

Cheers
MarkJ


squid -v
Squid Cache: Version 4.6
Service Name: squid
Raspbian linux


access.log

1563597855.786    605 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 15306 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease - HIER_DIRECT/93.93.128.193 -

1563597855.811    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 25429 GET http://archive.raspberrypi.org/debian/dists/buster/InRelease - HIER_DIRECT/93.93.128.133 -

1563597857.486    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 205801 GET http://archive.raspberrypi.org/debian/dists/buster/main/binary-armhf/Packages.gz - HIER_DIRECT/93.93.128.133 application/x-gzip

1563597936.436  80026 192.168.1.73 TCP_MISS_ABORTED/200 2641974 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz - HIER_DIRECT/93.93.128.193 application/x-xz


config file

acl localnet src 192.168.1.0/24 # internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
acl l500-020b src 192.168.1.20
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
acl ads dstdomain .ad1.pamedia.com.au
acl ads dstdomain .ad3.pamedia.com.au
acl ads dstdomain .adevents.com.au
acl ads dstdomain .adinfinity.com.au
acl ads dstdomain .ads.excitehome.net.au
acl ads dstdomain .ads.fairfax.com.au
acl ads dstdomain .ads.godaddy.com
acl ads dstdomain .ads.google.com
acl ads dstdomain .ads.ninemsn.com.au
acl ads dstdomain .ads.optusnet.com.au
acl ads dstdomain .ads.property.com.au
acl ads dstdomain .ads.youtube.com
acl ads dstdomain .adserver.news.com.au
acl ads dstdomain .au.adserver.yahoo.com
acl ads dstdomain .doubleclick.net
acl ads dstdomain .googleadservices.com
acl ads dstdomain .zoomdirect.com.au
acl malware dstdomain am10.ru
acl malware dstdomain deepspacer.com
acl malware dstdomain trafficconverter.biz
acl malware dstdomain .eu.interia.pl
acl malware dstdomain .expo9.exponential.com
acl malware dstdomain .flashtalking.com
acl malware dstdomain .funad.co.kr
acl malware dstdomain .luckytime.co.kr
acl malware dstdomain .trafficholder.com
acl malware2 dst 96.43.128.194
acl hiddenwasp dst 103.206.122.245
acl hiddenwasp dst 103.206.123.13
acl hiddenwasp2 dstdomain http://103.206.123.13
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny ads
http_access deny malware
http_access deny malware2
http_access deny hiddenwasp
http_access deny hiddenwasp2
http_access allow l500-020b manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 3128
cache_mem 448 MB
maximum_object_size 320 MB
memory_replacement_policy lru
cache_replacement_policy heap LFUDA
cache_dir aufs /var/spool/squid 18432 32 256
quick_abort_min -1 KB
client_request_buffer_max_size 128 KB
coredump_dir /var/spool/squid
access_log /var/log/squid/access.log squid
cache_log /var/log/squid/cache.log
logfile_rotate 7
netdb_filename none
refresh_pattern (\.deb|\.udeb)$ 1440    80%     10080
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
host_verify_strict on
max_filedescriptors 1200
dns_v4_first on
pinger_enable off
shutdown_lifetime 5 seconds
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: caching apt package lists/Raspbian

Amos Jeffries
Administrator
On 20/07/19 5:19 pm, TarotApprentice wrote:
> Recently upgraded to Raspbian Buster and squid 4.6. Since then I am
unable to cache the Packages.xz that apt uses. The various other Pis
using this proxy all end up downloading the 30MB Packages.xz every time.
Does anyone have any suggestions on how to get it to cache?
>
> Cheers MarkJ
>

According to both Redbot and my manual check the object is only 12MB,
not 30MB. If you are getting 30MB somebody is interfering with that
download.


It should be caching by default. The redbot tool shows the site is
providing all the required cache headers and working perfectly for
revalidation. The REFRESH_UNMODIFIED log entries show that too.

The TCP_MISS_ABORTED indicates that for that log entry there was nothing
in cache (yet) for that URL, and the client aborted the transfer with
only 2.6MB fetched.



Can you try having just one Pi do its update and seeing if the .xz
object is cached afterwards?

Alternatively try the command:
  squidclient
http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz

It the object is cacheable, but your environment tends to have the Pi's
all fetching at the same time (eg before the first finishes), then you
may find collapsed_forwarding feature of use. That helps with caching
parallel fetches of objects.

Amos


> squid -v
> Squid Cache: Version 4.6
> Service Name: squid
> Raspbian linux
>
>
> access.log
>
> 1563597855.786    605 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 15306 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease - HIER_DIRECT/93.93.128.193 -
>
> 1563597855.811    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 25429 GET http://archive.raspberrypi.org/debian/dists/buster/InRelease - HIER_DIRECT/93.93.128.133 -
>
> 1563597857.486    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 205801 GET http://archive.raspberrypi.org/debian/dists/buster/main/binary-armhf/Packages.gz - HIER_DIRECT/93.93.128.133 application/x-gzip
>
> 1563597936.436  80026 192.168.1.73 TCP_MISS_ABORTED/200 2641974 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz - HIER_DIRECT/93.93.128.193 application/x-xz
>
>
> config file
>
...
> acl hiddenwasp2 dstdomain http://103.206.123.13

The above "http://" is not a valid domain name.

> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny ads
> http_access deny malware
> http_access deny malware2
> http_access deny hiddenwasp
> http_access deny hiddenwasp2
> http_access allow l500-020b manager
> http_access deny manager


'dst' ACL is quite slow and resource intensive. You should put these
manager rules above the "malware2" denial to protect against DoS better.

...
> http_port 3128
> cache_mem 448 MB
> maximum_object_size 320 MB
> memory_replacement_policy lru
> cache_replacement_policy heap LFUDA
> cache_dir aufs /var/spool/squid 18432 32 256
> quick_abort_min -1 KB
> client_request_buffer_max_size 128 KB

...

> refresh_pattern (\.deb|\.udeb)$ 1440    80%     10080
> refresh_pattern ^ftp:           1440    20%     10080
> refresh_pattern ^gopher:        1440    0%      1440
> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
> refresh_pattern .               0       20%     4320


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: caching apt package lists/Raspbian

TarotApprentice
Doing an “apt update” on the squid machine got another TCP_MISS_ABORTED for ::1 and then subsequent IPv4 requests from other Pis get the TCP_REQUEST_UNMODIFIED.

Packages.xz was 13MB.


> On 21 Jul 2019, at 12:36 am, Amos Jeffries <[hidden email]> wrote:
>
>> On 20/07/19 5:19 pm, TarotApprentice wrote:
>> Recently upgraded to Raspbian Buster and squid 4.6. Since then I am
> unable to cache the Packages.xz that apt uses. The various other Pis
> using this proxy all end up downloading the 30MB Packages.xz every time.
> Does anyone have any suggestions on how to get it to cache?
>>
>> Cheers MarkJ
>>
>
> According to both Redbot and my manual check the object is only 12MB,
> not 30MB. If you are getting 30MB somebody is interfering with that
> download.
>
>
> It should be caching by default. The redbot tool shows the site is
> providing all the required cache headers and working perfectly for
> revalidation. The REFRESH_UNMODIFIED log entries show that too.
>
> The TCP_MISS_ABORTED indicates that for that log entry there was nothing
> in cache (yet) for that URL, and the client aborted the transfer with
> only 2.6MB fetched.
>
>
>
> Can you try having just one Pi do its update and seeing if the .xz
> object is cached afterwards?
>
> Alternatively try the command:
>  squidclient
> http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz
>
> It the object is cacheable, but your environment tends to have the Pi's
> all fetching at the same time (eg before the first finishes), then you
> may find collapsed_forwarding feature of use. That helps with caching
> parallel fetches of objects.
>
> Amos
>
>
>> squid -v
>> Squid Cache: Version 4.6
>> Service Name: squid
>> Raspbian linux
>>
>>
>> access.log
>>
>> 1563597855.786    605 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 15306 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease - HIER_DIRECT/93.93.128.193 -
>>
>> 1563597855.811    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 25429 GET http://archive.raspberrypi.org/debian/dists/buster/InRelease - HIER_DIRECT/93.93.128.133 -
>>
>> 1563597857.486    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 205801 GET http://archive.raspberrypi.org/debian/dists/buster/main/binary-armhf/Packages.gz - HIER_DIRECT/93.93.128.133 application/x-gzip
>>
>> 1563597936.436  80026 192.168.1.73 TCP_MISS_ABORTED/200 2641974 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz - HIER_DIRECT/93.93.128.193 application/x-xz
>>
>>
>> config file
>>
> ...
>> acl hiddenwasp2 dstdomain http://103.206.123.13
>
> The above "http://" is not a valid domain name.
>
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>> http_access deny ads
>> http_access deny malware
>> http_access deny malware2
>> http_access deny hiddenwasp
>> http_access deny hiddenwasp2
>> http_access allow l500-020b manager
>> http_access deny manager
>
>
> 'dst' ACL is quite slow and resource intensive. You should put these
> manager rules above the "malware2" denial to protect against DoS better.
>
> ...
>> http_port 3128
>> cache_mem 448 MB
>> maximum_object_size 320 MB
>> memory_replacement_policy lru
>> cache_replacement_policy heap LFUDA
>> cache_dir aufs /var/spool/squid 18432 32 256
>> quick_abort_min -1 KB
>> client_request_buffer_max_size 128 KB
>
> ...
>
>> refresh_pattern (\.deb|\.udeb)$ 1440    80%     10080
>> refresh_pattern ^ftp:           1440    20%     10080
>> refresh_pattern ^gopher:        1440    0%      1440
>> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
>> refresh_pattern .               0       20%     4320
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: caching apt package lists/Raspbian

Amos Jeffries
Administrator
On 21/07/19 4:20 pm, Mark James wrote:
> Doing an “apt update” on the squid machine got another TCP_MISS_ABORTED for ::1 and then subsequent IPv4 requests from other Pis get the TCP_REQUEST_UNMODIFIED.
>

That hints that there is something broken in your local network IPv6
connectivity. Perhapse ICMPv6 is not working properly?

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: caching apt package lists/Raspbian

TarotApprentice
It whatever Raspbian and the router do by default, although I do use an iptables firewall. I normally don't see any IPv6 from the other Pis, so maybe something to do with localhost and the loopback interface.

Cheers






On Sunday, 21 July 2019, 2:45:59 pm AEST, Amos Jeffries <[hidden email]> wrote:





On 21/07/19 4:20 pm, Mark James wrote:
> Doing an “apt update” on the squid machine got another TCP_MISS_ABORTED for ::1 and then subsequent IPv4 requests from other Pis get the TCP_REQUEST_UNMODIFIED.
>

That hints that there is something broken in your local network IPv6
connectivity. Perhapse ICMPv6 is not working properly?


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: caching apt package lists/Raspbian

TarotApprentice
Further to this I did find an issue with the iptables loopback and IPv6 which I corrected.

It still wasn’t caching the packages.xz from either the local machine or others in the local net. I ended up adding a refresh pattern for .gz and .xz which seems to cache them now. I am using 1440 20% 1440 which I thought was fairly conservative.

MarkJ

> On 21 Jul 2019, at 5:08 pm, TarotApprentice <[hidden email]> wrote:
>
> It whatever Raspbian and the router do by default, although I do use an iptables firewall. I normally don't see any IPv6 from the other Pis, so maybe something to do with localhost and the loopback interface.
>
> Cheers
>
>
>
>
>
>
> On Sunday, 21 July 2019, 2:45:59 pm AEST, Amos Jeffries <[hidden email]> wrote:
>
>
>
>
>
>> On 21/07/19 4:20 pm, Mark James wrote:
>> Doing an “apt update” on the squid machine got another TCP_MISS_ABORTED for ::1 and then subsequent IPv4 requests from other Pis get the TCP_REQUEST_UNMODIFIED.
>>
>
> That hints that there is something broken in your local network IPv6
> connectivity. Perhapse ICMPv6 is not working properly?
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users