Problem with caching larger files

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with caching larger files

Filip Krikava
Hi,

I try to use squid to cache rpm packages fetched from a yum
repository. With the following configuration, it works well for a
small rpms, but it would not cache larger ones like openjdk. From the
log file (below), it seems that it does not cache files larger than
4MB even though the maximum_object_size
128 MB is set. Any suggestions what might be wrong?

squid.conf:
----------

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_access allow all
http_port 3128 transparent
cache_dir ufs /var/spool/squid3 4048 16 256
maximum_object_size 128 MB
refresh_pattern Packages\.bz2$ 0       20%     4320 refresh-ims
refresh_pattern Sources\.bz2$  0       20%     4320 refresh-ims
refresh_pattern Release\.gpg$  0       20%     4320 refresh-ims
refresh_pattern Release$       0       20%     4320 refresh-ims
refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|bz2|ram|rar|bin)$  129600
100% 129600 override-expire ignore-no-cache ignore-no-store
refresh_pattern .              0       20%     4320

access log while installing openjdk:
----------

1400481405.110    408 172.17.0.3 TCP_MISS/200 852 GET
http://mirrorlist.centos.org/? - HIER_DIRECT/72.232.223.58 text/plain
1400481405.325    162 172.17.0.3 TCP_MISS/200 4202 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/repodata/repomd.xml
- HIER_DIRECT/81.194.43.155 text/xml
1400481408.357   3018 172.17.0.3 TCP_MISS/200 4614150 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/repodata/617a880b84a87969e04bfeebb1b96d5f441da42ee931d374fa6b6d57decd821e-primary.sqlite.bz2
- HIER_DIRECT/81.194.43.155 application/x-bzip2
1400481409.332    182 172.17.0.3 TCP_MISS/200 904 GET
http://mirrorlist.centos.org/? - HIER_DIRECT/72.232.223.58 text/plain
1400481409.446     41 172.17.0.3 TCP_HIT/200 3814 GET
http://centos.crazyfrogs.org/6.5/extras/x86_64/repodata/repomd.xml -
HIER_NONE/- text/xml
1400481409.451      0 172.17.0.3 TCP_HIT/200 19492 GET
http://centos.crazyfrogs.org/6.5/extras/x86_64/repodata/966094530f6744d0b6a3fc45b10cbd845577945fbd6a43448defbe1fab902c38-primary.sqlite.bz2
- HIER_NONE/- application/x-bzip2
1400481409.638    183 172.17.0.3 TCP_MISS/200 920 GET
http://mirrorlist.centos.org/? - HIER_DIRECT/72.232.223.58 text/plain
1400481409.743     53 172.17.0.3 TCP_HIT/200 3856 GET
http://mirror.ovh.net/ftp.centos.org/6.5/updates/x86_64/repodata/repomd.xml
- HIER_NONE/- application/xml
1400481409.775     30 172.17.0.3 TCP_HIT/200 3273269 GET
http://mirror.ovh.net/ftp.centos.org/6.5/updates/x86_64/repodata/395b793d10b8ac3be091e715aeeb7af5f64a699c18d14e0a19c5bc67db6425ee-primary.sqlite.bz2
- HIER_NONE/- application/x-bzip2
1400481410.703      3 172.17.0.3 TCP_HIT/200 379438 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/alsa-lib-1.0.22-3.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481410.712      2 172.17.0.3 TCP_HIT/200 249014 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/flac-1.2.1-6.1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481410.717      2 172.17.0.3 TCP_HIT/200 190514 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/fontconfig-2.8.0-3.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481410.724      5 172.17.0.3 TCP_HIT/200 368066 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/freetype-2.3.11-14.el6_3.1.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481410.728      0 172.17.0.3 TCP_HIT/200 37880 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/giflib-4.1.6-3.1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481424.776  14044 172.17.0.3 TCP_MISS/200 27050980 GET
http://mirror.ovh.net/ftp.centos.org/6.5/updates/x86_64/Packages/java-1.7.0-openjdk-1.7.0.55-2.4.7.1.el6_5.x86_64.rpm
- HIER_DIRECT/91.121.124.139 application/x-redhat-package-manager
1400481430.595   5698 172.17.0.3 TCP_MISS/200 9850394 GET
http://mirror.ovh.net/ftp.centos.org/6.5/updates/x86_64/Packages/java-1.7.0-openjdk-devel-1.7.0.55-2.4.7.1.el6_5.x86_64.rpm
- HIER_DIRECT/91.121.124.139 application/x-redhat-package-manager
1400481430.646      2 172.17.0.3 TCP_HIT/200 61088 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/jpackage-utils-1.7.5-3.12.el6.noarch.rpm
- HIER_NONE/- application/octet-stream
1400481430.650      1 172.17.0.3 TCP_HIT/200 54448 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libICE-1.0.6-1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.652      0 172.17.0.3 TCP_HIT/200 37948 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libSM-1.2.1-2.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.661      7 172.17.0.3 TCP_HIT/200 598770 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libX11-1.5.0-4.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.667      2 172.17.0.3 TCP_HIT/200 196782 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libX11-common-1.5.0-4.el6.noarch.rpm
- HIER_NONE/- application/octet-stream
1400481430.671      0 172.17.0.3 TCP_HIT/200 25300 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libXau-1.0.6-4.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.674      0 172.17.0.3 TCP_HIT/200 35888 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libXext-1.3.1-2.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.676      0 172.17.0.3 TCP_HIT/200 139605 GET
http://mirror.ovh.net/ftp.centos.org/6.5/updates/x86_64/Packages/libXfont-1.4.5-3.el6_5.x86_64.rpm
- HIER_NONE/- application/x-redhat-package-manager
1400481430.680      0 172.17.0.3 TCP_HIT/200 36028 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libXi-1.6.1-3.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.681      0 172.17.0.3 TCP_HIT/200 31400 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libXrender-0.9.7-2.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.685      1 172.17.0.3 TCP_HIT/200 30284 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libXtst-1.2.1-2.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.688      1 172.17.0.3 TCP_HIT/200 25320 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libasyncns-0.8-1.1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.691      0 172.17.0.3 TCP_HIT/200 25396 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libfontenc-1.0.5-2.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.694      1 172.17.0.3 TCP_HIT/200 178925 GET
http://mirror.ovh.net/ftp.centos.org/6.5/updates/x86_64/Packages/libjpeg-turbo-1.2.1-3.el6_5.x86_64.rpm
- HIER_NONE/- application/x-redhat-package-manager
1400481430.698      0 172.17.0.3 TCP_HIT/200 21776 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libogg-1.1.4-2.1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.703      2 172.17.0.3 TCP_HIT/200 186422 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libpng-1.2.49-1.el6_2.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.710      5 172.17.0.3 TCP_HIT/200 238658 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libsndfile-1.0.20-5.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.714      1 172.17.0.3 TCP_HIT/200 172210 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libvorbis-1.2.3-4.el6_2.1.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.717      1 172.17.0.3 TCP_HIT/200 112834 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/libxcb-1.8.1-1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.722      0 172.17.0.3 TCP_HIT/200 72289 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/pkgconfig-0.23-9.1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.730      6 172.17.0.3 TCP_HIT/200 473122 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/pulseaudio-libs-0.9.21-14.el6_3.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.734      1 172.17.0.3 TCP_HIT/200 63868 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/tcp_wrappers-libs-7.6-57.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.738      1 172.17.0.3 TCP_HIT/200 44284 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/ttmkfdir-3.0.9-32.1.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.742      0 172.17.0.3 TCP_HIT/200 151141 GET
http://mirror.ovh.net/ftp.centos.org/6.5/updates/x86_64/Packages/tzdata-java-2014b-1.el6.noarch.rpm
- HIER_NONE/- application/x-redhat-package-manager
1400481430.749      2 172.17.0.3 TCP_HIT/200 77097 GET
http://miroir.univ-paris13.fr/centos/6.5/os/x86_64/Packages/xorg-x11-font-utils-7.2-11.el6.x86_64.rpm
- HIER_NONE/- application/octet-stream
1400481430.763      6 172.17.0.3 TCP_HIT/200 532402 GET
http://miroir.univ-paris13

I also asked this on server fault for a reference:
http://serverfault.com/questions/596890/squid3-not-caching-larger-files

Thank you,
Filip
Reply | Threaded
Open this post in threaded view
|

Re: Problem with caching larger files

Eliezer Croitoru
Hey Filip,

If you can provide more information about the OS and the squid build
version using:
uname -a
squid -v

Thanks,
Eliezer

On 05/19/2014 11:06 AM, Filip Krikava wrote:
> Hi,
>
> I try to use squid to cache rpm packages fetched from a yum
> repository. With the following configuration, it works well for a
> small rpms, but it would not cache larger ones like openjdk. From the
> log file (below), it seems that it does not cache files larger than
> 4MB even though the maximum_object_size
> 128 MB is set. Any suggestions what might be wrong?
>

Reply | Threaded
Open this post in threaded view
|

Re: Problem with caching larger files

Filip Krikava
Hi Eliezer,

Thanks for quick reply!

$ squid3 -v
Squid Cache: Version 3.3.8
configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security'

$ uname -a
Linux 84194223c57e 3.14.1-tinycore64 #1 SMP Thu May 15 02:39:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

/cheers
Filip


On 19 May 2014, at 10:09, Eliezer Croitoru <[hidden email]> wrote:

> Hey Filip,
>
> If you can provide more information about the OS and the squid build version using:
> uname -a
> squid -v
>
> Thanks,
> Eliezer
>
> On 05/19/2014 11:06 AM, Filip Krikava wrote:
>> Hi,
>>
>> I try to use squid to cache rpm packages fetched from a yum
>> repository. With the following configuration, it works well for a
>> small rpms, but it would not cache larger ones like openjdk. From the
>> log file (below), it seems that it does not cache files larger than
>> 4MB even though the maximum_object_size
>> 128 MB is set. Any suggestions what might be wrong?
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Problem with caching larger files

Plamen
Hi,

maximum_object_size 128 MB

should be above the cache_dir in your squid config.



Reply | Threaded
Open this post in threaded view
|

Re: Problem with caching larger files

Filip Krikava
Thank you, now it works perfectly!

Filip

On 19 May 2014, at 13:40, Plamen <[hidden email]> wrote:

> Hi,
>
> maximum_object_size 128 MB
>
> should be above the cache_dir in your squid config.
>
>
>
>
>
>
>
> --
> View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Problem-with-caching-larger-files-tp4666004p4666006.html
> Sent from the Squid - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Problem with caching larger files

Eliezer Croitoru
In reply to this post by Filip Krikava
Happy you got it solved.
Is there a tinycore squid package? I have been looking for one for quite
some time.

Eliezer

On 05/19/2014 11:17 AM, Filip Krikava wrote:
> $ uname -a
> Linux 84194223c57e 3.14.1-tinycore64 #1 SMP Thu May 15 02:39:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> /cheers
> Filip