Squid Seems to be Rate-Limiting TCP_HIT Downloads

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid Seems to be Rate-Limiting TCP_HIT Downloads

RW-8
I'm using  Squid 2.6.13 locally on a FreeBSD 6.2 Desktop PC with a ufs
store, and I am seeing slow download speeds on TCP_HIT downloads, which
should be instantaneous on localhost. With cache misses I get normal
fluctuating speeds, but with hits I get a steady 20KB/s per connection,
which suggests there is some kind of rate-limiting going on.

I tried deleting the cache and reinitializing, to no effect. I don't have
delay pools built-in. There is nothing unusual in my squid.conf file, and I
don't see anything new in the default version. I'm seeing this with both
browsers and commandline tools.

I'm pretty sure something has changed in Squid, a few months ago large
images cached in squid were displaying instantaneously, even after deleting
the browser cache.



Reply | Threaded
Open this post in threaded view
|

Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

Adrian Chadd
On Fri, Jun 08, 2007, RW wrote:
> I'm using  Squid 2.6.13 locally on a FreeBSD 6.2 Desktop PC with a ufs
> store, and I am seeing slow download speeds on TCP_HIT downloads, which
> should be instantaneous on localhost. With cache misses I get normal
> fluctuating speeds, but with hits I get a steady 20KB/s per connection,
> which suggests there is some kind of rate-limiting going on.

Try running aufs rather than ufs. FreeBSD-6 has perfectly functional
threading support for disk IO.



Adrian

Reply | Threaded
Open this post in threaded view
|

Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

RW-8
Adrian Chadd wrote:

> On Fri, Jun 08, 2007, RW wrote:
>> I'm using  Squid 2.6.13 locally on a FreeBSD 6.2 Desktop PC with a ufs
>> store, and I am seeing slow download speeds on TCP_HIT downloads, which
>> should be instantaneous on localhost. With cache misses I get normal
>> fluctuating speeds, but with hits I get a steady 20KB/s per connection,
>> which suggests there is some kind of rate-limiting going on.
>
> Try running aufs rather than ufs. FreeBSD-6 has perfectly functional
> threading support for disk IO.

This is a single user PC and any I'm seeing this with a single download,
with large files. There shouldn't be any significant difference in the
download of a single 5MB file between aufs and ufs on a system with no
other workload. And it's always a round 20KB/s, if I start a second,
simultaneous download it also runs at exactly 20KB/s.

This either a bug or a feature, I can't see 20KB/s being a disk access
bottleneck.

Reply | Threaded
Open this post in threaded view
|

Re: Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

Adrian Chadd
On Fri, Jun 08, 2007, RW wrote:

> Adrian Chadd wrote:
>
> > On Fri, Jun 08, 2007, RW wrote:
> >> I'm using  Squid 2.6.13 locally on a FreeBSD 6.2 Desktop PC with a ufs
> >> store, and I am seeing slow download speeds on TCP_HIT downloads, which
> >> should be instantaneous on localhost. With cache misses I get normal
> >> fluctuating speeds, but with hits I get a steady 20KB/s per connection,
> >> which suggests there is some kind of rate-limiting going on.
> >
> > Try running aufs rather than ufs. FreeBSD-6 has perfectly functional
> > threading support for disk IO.
>
> This is a single user PC and any I'm seeing this with a single download,
> with large files. There shouldn't be any significant difference in the
> download of a single 5MB file between aufs and ufs on a system with no
> other workload. And it's always a round 20KB/s, if I start a second,
> simultaneous download it also runs at exactly 20KB/s.
>
> This either a bug or a feature, I can't see 20KB/s being a disk access
> bottleneck.

Then post the config sans-comments. Ie:

$ cat squid.conf | grep -v ^# | grep -v ^$

Then we can see if you're running delay pools, what else could be causing the issue.




Adrian

Reply | Threaded
Open this post in threaded view
|

Re: Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

RW-8
Adrian Chadd wrote:

> On Fri, Jun 08, 2007, RW wrote:

>> This either a bug or a feature, I can't see 20KB/s being a disk access
>> bottleneck.
>
> Then post the config sans-comments. Ie:
>
> $ cat squid.conf | grep -v ^# | grep -v ^$
>
> Then we can see if you're running delay pools, what else could be causing
> the issue.


$ cat squid.conf | grep -v ^# | grep -v ^$
http_port 3128
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
maximum_object_size 12 MB
cache_dir ufs /usr/local/squid/cache   160 16 256
cache_swap_low  90
cache_swap_high 95
logfile_rotate 4
access_log /usr/local/squid/logs/access.log squid
refresh_pattern    ^ftp:               1440    33%     10080
refresh_pattern    ^gopher:            1440    0%      1440
refresh_pattern -i \.(zip|rar|bz2|gz)$ 10      5%      999 reload-into-ims
refresh_pattern -i \.(pdf|ps)$         0       25%     999 reload-into-ims
refresh_pattern -i \.(jpe?g|gif)$      0       25%     18000
refresh_pattern -i \.(swf)$            0        5%     4320
refresh_pattern    .                   0       25%     4320
quick_abort_min 32 KB
quick_abort_max 200 KB
quick_abort_pct 33
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl mylan src 192.168.1.0/255.255.255.0
http_access allow mylan
http_access allow localhost
http_access deny all
http_reply_access allow all
icp_access allow all
maximum_single_addr_tries 4
coredump_dir /usr/local/squid


Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

Adrian Chadd
On Fri, Jun 08, 2007, RW wrote:

> Adrian Chadd wrote:
>
> > On Fri, Jun 08, 2007, RW wrote:
>
> >> This either a bug or a feature, I can't see 20KB/s being a disk access
> >> bottleneck.
> >
> > Then post the config sans-comments. Ie:
> >
> > $ cat squid.conf | grep -v ^# | grep -v ^$
> >
> > Then we can see if you're running delay pools, what else could be causing
> > the issue.

Everything looks fine, save the ufs filesystem. Please realise that UFS is a pretty
naive store implementation and doesn't even try to schedule multiple reads/writes
at the same time; so if you've got some objects being read off completely different
parts of the disk (and they're not cached) then your disk performance might be
horrible - and network IO will follow that.

Just humour me and try aufs using libthr :P




Adrian

> $ cat squid.conf | grep -v ^# | grep -v ^$
> http_port 3128
> hierarchy_stoplist cgi-bin ?
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> maximum_object_size 12 MB
> cache_dir ufs /usr/local/squid/cache   160 16 256
> cache_swap_low  90
> cache_swap_high 95
> logfile_rotate 4
> access_log /usr/local/squid/logs/access.log squid
> refresh_pattern    ^ftp:               1440    33%     10080
> refresh_pattern    ^gopher:            1440    0%      1440
> refresh_pattern -i \.(zip|rar|bz2|gz)$ 10      5%      999 reload-into-ims
> refresh_pattern -i \.(pdf|ps)$         0       25%     999 reload-into-ims
> refresh_pattern -i \.(jpe?g|gif)$      0       25%     18000
> refresh_pattern -i \.(swf)$            0        5%     4320
> refresh_pattern    .                   0       25%     4320
> quick_abort_min 32 KB
> quick_abort_max 200 KB
> quick_abort_pct 33
> acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 443
> acl Safe_ports port 80          # http
> acl Safe_ports port 21          # ftp
> acl Safe_ports port 443         # https
> acl Safe_ports port 70          # gopher
> acl Safe_ports port 210         # wais
> acl Safe_ports port 1025-65535  # unregistered ports
> acl Safe_ports port 280         # http-mgmt
> acl Safe_ports port 488         # gss-http
> acl Safe_ports port 591         # filemaker
> acl Safe_ports port 777         # multiling http
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> acl mylan src 192.168.1.0/255.255.255.0
> http_access allow mylan
> http_access allow localhost
> http_access deny all
> http_reply_access allow all
> icp_access allow all
> maximum_single_addr_tries 4
> coredump_dir /usr/local/squid
>

--
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level bandwidth-capped VPSes available in WA -
Reply | Threaded
Open this post in threaded view
|

Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

RW-8
Adrian Chadd wrote:
> Just humour me and try aufs using libthr :P

I rebuilt and reconfigured squid for aufs, and then deleted and
reinitialized the cache. As I expected, it made no difference.

It's very consistent too - I downloaded a 4MB file several times with fetch,
and several times with wget, and on each TCP_HIT it took 3 minutes 25
seconds, which wget reports as 20.01 KB/s.

As I said before, I'm pretty sure that something has changed here. When I
first setup squid, a few months ago, cached images were loading
instantaneously.  It's pretty hard to say when it changed because my large
downloads don't go through squid, and 20.01 KB/s times several connections
simply makes a cached page look like an uncached page.

 
 



Reply | Threaded
Open this post in threaded view
|

Re: Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

Adrian Chadd
On Sat, Jun 09, 2007, RW wrote:
> Adrian Chadd wrote:
> > Just humour me and try aufs using libthr :P
>
> I rebuilt and reconfigured squid for aufs, and then deleted and
> reinitialized the cache. As I expected, it made no difference.

Thats fine, now we know.

> It's very consistent too - I downloaded a 4MB file several times with fetch,
> and several times with wget, and on each TCP_HIT it took 3 minutes 25
> seconds, which wget reports as 20.01 KB/s.

Does that match your IO fetch speed reported in iostat?

> As I said before, I'm pretty sure that something has changed here. When I
> first setup squid, a few months ago, cached images were loading
> instantaneously.  It's pretty hard to say when it changed because my large
> downloads don't go through squid, and 20.01 KB/s times several connections
> simply makes a cached page look like an uncached page.

It shouldn't be anything with Squid, but its hard to tell without being able
to duplicate the issue. And I've tested squid at home with your basic config,
everything works fine here under FreeBSD, with:

[adrian@jacinta ~]$ cat /etc/libmap.conf
squid
     libpthread.so.2         libthr.so.2
     libpthread.so           libthr.so



Adrian

Reply | Threaded
Open this post in threaded view
|

Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

Henrik Nordström
In reply to this post by RW-8
fre 2007-06-08 klockan 15:36 +0100 skrev RW:
> I'm using  Squid 2.6.13 locally on a FreeBSD 6.2 Desktop PC with a ufs
> store, and I am seeing slow download speeds on TCP_HIT downloads, which
> should be instantaneous on localhost. With cache misses I get normal
> fluctuating speeds, but with hits I get a steady 20KB/s per connection,
> which suggests there is some kind of rate-limiting going on.

20KB/s cache hit speed? Very odd indeed.

Is there any change if you have a network download at the same time?

What kind of comm loop are you using, and cache_dir?

Regards
Henrik

signature.asc (316 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

Henrik Nordström
In reply to this post by Adrian Chadd
fre 2007-06-08 klockan 23:34 +0800 skrev Adrian Chadd:

> Try running aufs rather than ufs. FreeBSD-6 has perfectly functional
> threading support for disk IO.

ufs shouldn't see this kind of odd effects as we are then blocking on
read.

aufs/diskd might however, if there is some problem with the disk I/O
notification polling.

Regards
Henrik

signature.asc (316 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Squid Seems to be Rate-Limiting TCP_HIT Downloads

RW-8
In reply to this post by Henrik Nordström
Henrik Nordstrom wrote:

> 20KB/s cache hit speed? Very odd indeed.
>
> Is there any change if you have a network download at the same time?

No, it's just the same.

> What kind of comm loop are you using, and cache_dir?

I was using ufs, which was changed aufs on Adrian's suggestion, and made no
difference. I'm not sure what you mean by "comm loop", but if you mean
kqueue v's poll, I'm using kqueue, the default.