Delay_pools problem in Squid 3.5.20

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Delay_pools problem in Squid 3.5.20

Alex Tang
Dear All,

I had installed a squid 3.5.20 on Centos7. I'm also setting up a delay_pools and cache_peer function on this squid server .

but I don't know why delay_pools function not working now. before I tested is OK and I can limit user download less then 512 kb. but now, all user download is over 512kb and can't control. I tried reboot the squid or use the other delay_pools command to try to test which part has problem to cause the delay_pools problem. I've checked my config file seem OK (this is my first time to install and use squid proxy server) , would you mind tell me is it my config problem or squid bug (maybe)

here is my full config file.

====================================================
#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 111.11.0.0/12 # RFC1918 possible internal network
acl localnet src 11.123.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
#acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
#acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
#acl CONNECT method CONNECT



#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
#http_access allow !Safe_ports

# Deny CONNECT to other than secure SSL ports
#http_access allow CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
#log_fqdn on

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
#http_port 8000
#http_port 8001
icp_port 3130

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 1500 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

#
# Add any of your own refresh_pattern entries above these.
#
#refresh_pattern ^ftp: 1440 20% 10080
#refresh_pattern ^gopher: 1440 0% 1440
#refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
#refresh_pattern . 0 20% 4320

http_port 8000 name=port_8000
http_port 8001 name=port_8001

nonhierarchical_direct off

acl port_8000_acl myportname port_8000
acl port_8001_acl myportname port_8001

always_direct deny port_8000_acl
always_direct deny port_8001_acl

never_direct allow port_8000_acl
never_direct allow port_8001_acl

# 8000
cache_peer xxx.xxxx.com parent 8000 3130 weight=20 no-digest no-query name=proxy8000
cache_peer_access proxy8000 allow port_8000_acl
cache_peer_access proxy8000 deny all

# 8001
cache_peer xxx.xxxx.com parent 8001 3130 weight=20 no-digest no-query name=proxy8001
cache_peer_access proxy8001 allow port_8001_acl
cache_peer_access proxy8001 deny all

cache_mem 100 MB
cache_swap_low 90
cache_swap_high 95


acl work_day time MTWHFAS 09:00-18:30
acl BBHK src 11.123.0.0/16
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 512000/512000
delay_access 1 allow work_day
delay_access 1 allow BBHK


acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
maximum_object_size 2048 KB
ipcache_size 1024
ipcache_low 90
ipcache_high 95
fqdncache_size 1024
logformat squid      %ts.%03tu %6tr %>A %Ss/%03>Hs %<st %rm %ru %[un %Sh/%<a %mt
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log none
pid_filename /var/run/squid.pid
auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 30
auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
auth_param basic children 30
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
request_body_max_size 15 MB



acl AuthorizedUsers proxy_auth REQUIRED
acl shockwaveplayer browser Shockwave
acl Java browser Java/1.4 Java/1.5 Java/1.6
acl BBhknet src "/etc/squid/ACL/allow_net"
acl allow_ip src "/etc/squid/ACL/allow_ip"
#acl deny_ip src "/etc/squid/ACL/deny_ip"
#acl DenyUsers proxy_auth "/etc/squid/ACL/deny_users"
acl allow_pattern dstdom_regex "/etc/squid/ACL/allow_domain"

acl allow_sites url_regex "/etc/squid/ACL/allow_url"
#http_access allow allow_sites

acl deny_site url_regex "/etc/squid/ACL/deny_url"
#http_access deny deny_site

acl deny_pattern dstdom_regex "/etc/squid/ACL/deny_domain"
#http_access deny deny_pattern

acl deny_domain dstdom_regex "/etc/squid/ACL/deny_domain"
acl deny_url url_regex "/etc/squid/ACL/deny_url"

acl fwdurl dstdomain .salesforce.com .force.com
never_direct allow fwdurl

acl fwdhkpurl dstdomain "/etc/squid/ACL/fwdhkpdomain"
never_direct allow fwdhkpurl


http_access allow fwdurl
http_access allow fwdhkpurl
http_access allow shockwaveplayer
http_access allow Java
http_access allow allow_ip
http_access allow allow_pattern
http_access allow allow_sites
http_access deny deny_domain
http_access allow BBhknet
http_access allow BBHK 
http_access deny all


http_reply_access allow all
icp_access allow all
miss_access allow all
cache_mgr BBHK Network Admin Email
cache_effective_user squid
cache_effective_group squid
visible_hostname proxy6.hkg.xerox.com
max_filedesc 4096
never_direct allow all
error_directory /usr/share/squid/errors/English
coredump_dir /var/spool/squid

=======================================================

Thanks all

Alex

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Delay_pools problem in Squid 3.5.20

Amos Jeffries
Administrator
On 26/07/17 13:49, Alex Tang wrote:

> Dear All,
>
> I had installed a squid 3.5.20 on Centos7. I'm also setting up a
> delay_pools and cache_peer function on this squid server .
>
> but I don't know why delay_pools function not working now. before I
> tested is OK and I can limit user download less then 512 kb. but now,
> all user download is over 512kb and can't control. I tried reboot the
> squid or use the other delay_pools command to try to test which part has
> problem to cause the delay_pools problem. I've checked my config file
> seem OK (this is my first time to install and use squid proxy server) ,
> would you mind tell me is it my config problem or squid bug (maybe)
>

What do you mean exactly by "before I tested is OK". The same Squid
version? the same config file?



> here is my full config file.
>
> ====================================================
> #
> # Recommended minimum configuration:
> #
>
> # Example rule allowing access from your local networks.
> # Adapt to list your (internal) IP networks from where browsing
> # should be allowed
> acl localnet src 10.0.0.0/8 <http://10.0.0.0/8># RFC1918 possible
> internal network
> acl localnet src 111.11.0.0/12 <http://111.11.0.0/12># RFC1918 possible
> internal network
> acl localnet src 11.123.0.0/16 <http://11.123.0.0/16># RFC1918 possible
> internal network

Ah, 111/8 and 11/8 are not RFC 1918 network ranges. If you have been
assigned those ranges and are using them internally that is fine, but
remove the RFC1918 comment to avoid confusing anyone about them.


> acl localnet src fc00::/7       # RFC 4193 local private network range
> acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged)
> machines
>
> acl SSL_ports port 443 563
> acl Safe_ports port 80# http
> acl Safe_ports port 21# ftp
> #acl Safe_ports port 443# https
> acl Safe_ports port 70# gopher
> acl Safe_ports port 210# wais
> #acl Safe_ports port 1025-65535# unregistered ports
> acl Safe_ports port 280# http-mgmt
> acl Safe_ports port 488# gss-http
> acl Safe_ports port 591# filemaker
> acl Safe_ports port 777# multiling http
> #acl CONNECT method CONNECT
>
>
>
> #
> # Recommended minimum Access Permission configuration:
> #
> # Deny requests to certain unsafe ports
> #http_access allow !Safe_ports
>
> # Deny CONNECT to other than secure SSL ports
> #http_access allow CONNECT !SSL_ports
>

Do not do the above commenting-out of those lines. They are the basic
security protection against certain types of DoS and attacks on your
proxy. Which is also why they should always be first.


> # Only allow cachemgr access from localhost
> http_access allow localhost manager
> http_access deny manager
>
> # We strongly recommend the following be uncommented to protect innocent
> # web applications running on the proxy server who think the only
> # one who can access services on "localhost" is a local user
> #http_access deny to_localhost
>
> #
> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
> #
>
> # Example rule allowing access from your local networks.
> # Adapt localnet in the ACL section to list your (internal) IP networks
> # from where browsing should be allowed
> http_access allow localnet
> http_access allow localhost
>
> # And finally deny all other access to this proxy
> http_access deny all
>
> icp_port 3130
 >

> # Leave coredumps in the first cache dir
> coredump_dir /var/spool/squid
>
> #
> # Add any of your own refresh_pattern entries above these.
> #
> #refresh_pattern ^ftp:144020%10080
> #refresh_pattern ^gopher:14400%1440
> #refresh_pattern -i (/cgi-bin/|\?) 00%0
> #refresh_pattern .020%4320

The above refresh_pattern lines are tuned specifically for HTTP/1.1
required behaviour with dynamic content. If you are caching I highly
recommend leaving them active. If you are not using caching they are
irrelevant.

>
> http_port 8000 name=port_8000
> http_port 8001 name=port_8001
>
> nonhierarchical_direct off
>
> acl port_8000_acl myportname port_8000
> acl port_8001_acl myportname port_8001
>
> always_direct deny port_8000_acl
> always_direct deny port_8001_acl
>
> never_direct allow port_8000_acl
> never_direct allow port_8001_acl
>

The above are redundant. always_direct overrides never_direct. The
normal practice is to use only never_direct to forbid use of DNS
identified origin servers and leave the cache_peer as an available routing.


> # 8000
> cache_peer xxx.xxxx.com parent 8000 3130 weight=20
> no-digest no-query name=proxy8000
> cache_peer_access proxy8000 allow port_8000_acl
> cache_peer_access proxy8000 deny all
>
> # 8001
> cache_peer xxx.xxxx.com parent 8001 3130 weight=20
> no-digest no-query name=proxy8001
> cache_peer_access proxy8001 allow port_8001_acl
> cache_peer_access proxy8001 deny all
>
> cache_mem 100 MB
> cache_swap_low 90
> cache_swap_high 95
>
>
> acl work_day time MTWHFAS 09:00-18:30
> acl BBHK src 11.123.0.0/16
> delay_pools 1
> delay_class 1 2
> delay_parameters 1 -1/-1 512000/512000
> delay_access 1 allow work_day
> delay_access 1 allow BBHK


Three things to be careful about. I order them here based on the
likelihood that they are your problem:

  1) the work_day times are in 'local time', according to whatever the
local wall-clock of your Squid machine is set to.

   This local time detail can catch people out if the production proxy
is set to use UTC as its local time, or is hosted in another timezone
(ie cloud service).

  2) delay pools operate by limiting Squid->server connection traffic.
That means that cache HITs are not delayed. Also, HTTP/1.1 revalidations
use very much less server connection bytes compared to client connection
bytes. So the delay may have the appearance of not working, even when
working perfectly.
   For proper bandwidth shaping it is best to use your system QoS
functionality, not Squid delay pools. Especially given your criteria
does not actually use any HTTP layer details to classify the traffic for
delay.



  3) these lines form an OR condition. The BBHK are *always* delayed,
and so is every client during the work_day times.

Check carefully where (from what client IPs to what proxy IPs), how, and
when your previous working tests were done compared to the current
non-working situation.


  4) the delay pools measure traffic in Bytes (KB or kB) not bits (Kb or
kb). KB/sec limits are 8x larger than kb/sec.

I assume your "kb" word was just sloppy typing. But if you did mean bits
this could be affecting your measurement results.


>
> acl QUERY urlpath_regex cgi-bin \?
> no_cache deny QUERY

The above is a misconfiguration in Squid-3. The refresh_pattern which
you commented out are the correct way to prevent broken dynamic content
caching.

> maximum_object_size 2048 KB
> ipcache_size 1024
> ipcache_low 90
> ipcache_high 95
> fqdncache_size 1024
> logformat squid      %ts.%03tu %6tr %>A %Ss/%03>Hs %<st %rm %ru %[un
> %Sh/%<a %mt

Please do not re-define the "squid" format name. Current Squid releases
will ignore your definition, and the log will not end up with what you
want in it (the real built-in format does not have the exact output in
some cases as those %-codes perform).
  Either use the built-in format (removing those above line), or use a
different format name for your custom format.


> cache_access_log /var/log/squid/access.log

cache_access_log is long ago deprecated.

Use access_log instead, like so:
  access_log stdio:/var/log/access.log

or for better performance:
  access_log daemon:/var/log/access.log


> cache_log /var/log/squid/cache.log
> cache_store_log none
> pid_filename /var/run/squid.pid
> auth_param ntlm program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-ntlmssp
> auth_param ntlm children 30
> auth_param basic program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-basic
> auth_param basic children 30
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours
> request_body_max_size 15 MB
>

NTLM plus delay pools. Ouch.

>
> acl AuthorizedUsers proxy_auth REQUIRED
> acl shockwaveplayer browser Shockwave
> acl Java browser Java/1.4 Java/1.5 Java/1.6
> acl BBhknet src "/etc/squid/ACL/allow_net"
> acl allow_ip src "/etc/squid/ACL/allow_ip"
> #acl deny_ip src "/etc/squid/ACL/deny_ip"
> #acl DenyUsers proxy_auth "/etc/squid/ACL/deny_users"
> acl allow_pattern dstdom_regex "/etc/squid/ACL/allow_domain"
>
> acl allow_sites url_regex "/etc/squid/ACL/allow_url"
> #http_access allow allow_sites
>
> acl deny_site url_regex "/etc/squid/ACL/deny_url"
> #http_access deny deny_site
>
> acl deny_pattern dstdom_regex "/etc/squid/ACL/deny_domain"
> #http_access deny deny_pattern
>
> acl deny_domain dstdom_regex "/etc/squid/ACL/deny_domain"
> acl deny_url url_regex "/etc/squid/ACL/deny_url"
>
> acl fwdurl dstdomain .salesforce.com <http://salesforce.com> .force.com
> <http://force.com>
> never_direct allow fwdurl
>
> acl fwdhkpurl dstdomain "/etc/squid/ACL/fwdhkpdomain"
> never_direct allow fwdhkpurl
>
>
> http_access allow fwdurl
> http_access allow fwdhkpurl

Note: The fwdurl and fwdhkpurl ACL are the same type, and used together
for exactly the same things. You can simplify your config by merging
them into one ACL name like so:
  acl fwdurl dstdomain .salesforce.com .force.com
  acl fwdurl dstdomain "/etc/squid/ACL/fwdhkpdomain"

then removing the *_access lines using fwdhkpurl.


> http_access allow shockwaveplayer
> http_access allow Java
> http_access allow allow_ip
> http_access allow allow_pattern
> http_access allow allow_sites
> http_access deny deny_domain
> http_access allow BBhknet
> http_access allow BBHK
> http_access deny all
>
>
> http_reply_access allow all
> icp_access allow all
> miss_access allow all
> cache_mgr BBHK Network Admin Email > cache_effective_user squid
> cache_effective_group squid
> visible_hostname proxy6.hkg.xerox.com
> max_filedesc 4096
> never_direct allow all
> error_directory /usr/share/squid/errors/English
> coredump_dir /var/spool/squid

Most of the above are default config settings. You can simplify your
config by removing the default lines.

squid -k parse should give you a lot of hints about improvements as
well. That is always being improved, so a quick -k parse after upgrades
is recommended best practice.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...