FreeBSD Squid timeout issue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

FreeBSD Squid timeout issue

dave-5
Hello,
    This is a strange one. I've got a FreeBSD 6.2 router with squid on it
for transparent proxy. It has been working fine, until about 3 days ago when
i noticed one of my internal machines wasn't completing an http transaction,
in this case downloading of it's ports index. The connection would start out
extremely slowly and eventually timeout then giving an error about a
truncated file. I checked my firewall, hard disks, debug logs, messages
everything, couldn't find any error msgs or anything obviously wrong. This
morning an internal machine, this one a centos box, began failing with it's
yum updates, giving timeout errors on retrieving rpm files from http sites.
Again, i ran the log checks, didn't see anything. So, i next checked the
firewall, nothing in debug.log or messages, but in the /usr/local/squid/logs
directory i found two things. The first was that the failed transactions
were all returning http response code 206, timeout? The second was that i
had a 43 mb core file. I shut down squid, turned off the transparent
redirect rule in my firewall and everything started working normally. If
this core file is useful in debugging i'd like to know what to do or where
to send it, i can post it on a web page if interested. I'd appreciate any
suggestions.
    The squid version used is 2.6.13, memory and cpu usage during the times
in question are not even moving, this system is not in any way heavily
loaded.
    Any suggestions appreciated.
Thanks.
Dave.

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

dave-5
    Hello,
    Thank you for your reply. Yes i did check my cache_dir partition and it
isn't even close to being full in terms of how much space is available on
the filesystem. My cache_dir entry looks like:

cache_dir diskd /usr/local/squid/cache 600 32 512

The size of the cache directory itself is 519 mb.
Thanks.
Dave.

----- Original Message -----
From: "Michel Santos" <[hidden email]>
To: "Dave" <[hidden email]>
Sent: Sunday, June 24, 2007 1:01 PM
Subject: Re: [squid-users] FreeBSD Squid timeout issue


>
> Dave disse na ultima mensagem:
>> Hello,
>>     This is a strange one. I've got a FreeBSD 6.2 router with squid on it
>
> Hi
> did you checked your cache_dirpartition  if it is full and swap.state size
> by case?
>
> Michel
>
>
>
>> for transparent proxy. It has been working fine, until about 3 days ago
>> when
>> i noticed one of my internal machines wasn't completing an http
>> transaction,
>> in this case downloading of it's ports index. The connection would start
>> out
>> extremely slowly and eventually timeout then giving an error about a
>> truncated file. I checked my firewall, hard disks, debug logs, messages
>> everything, couldn't find any error msgs or anything obviously wrong.
>> This
>> morning an internal machine, this one a centos box, began failing with
>> it's
>> yum updates, giving timeout errors on retrieving rpm files from http
>> sites.
>> Again, i ran the log checks, didn't see anything. So, i next checked the
>> firewall, nothing in debug.log or messages, but in the
>> /usr/local/squid/logs
>> directory i found two things. The first was that the failed transactions
>> were all returning http response code 206, timeout? The second was that i
>> had a 43 mb core file. I shut down squid, turned off the transparent
>> redirect rule in my firewall and everything started working normally. If
>> this core file is useful in debugging i'd like to know what to do or
>> where
>> to send it, i can post it on a web page if interested. I'd appreciate any
>> suggestions.
>>     The squid version used is 2.6.13, memory and cpu usage during the
>> times
>> in question are not even moving, this system is not in any way heavily
>> loaded.
>>     Any suggestions appreciated.
>> Thanks.
>> Dave.
>>
>>
>>
>>
>>
>>
>>
>>
>> A mensagem foi scaneada pelo sistema de e-mail e pode ser considerada
>> segura.
>> Service fornecido pelo Datacenter Matik  https://datacenter.matik.com.br
>>
>
>
>
>
> ...
>
>
>
>
> ****************************************************
> Datacenter Matik http://datacenter.matik.com.br
> E-Mail e Data Hosting Service para Profissionais.
> ****************************************************

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Tek Bahadur Limbu
In reply to this post by dave-5
Dave wrote:

> Hello,
>    This is a strange one. I've got a FreeBSD 6.2 router with squid on it
> for transparent proxy. It has been working fine, until about 3 days ago
> when i noticed one of my internal machines wasn't completing an http
> transaction, in this case downloading of it's ports index. The
> connection would start out extremely slowly and eventually timeout then
> giving an error about a truncated file. I checked my firewall, hard
> disks, debug logs, messages everything, couldn't find any error msgs or
> anything obviously wrong. This morning an internal machine, this one a
> centos box, began failing with it's yum updates, giving timeout errors
> on retrieving rpm files from http sites. Again, i ran the log checks,
> didn't see anything. So, i next checked the firewall, nothing in
> debug.log or messages, but in the /usr/local/squid/logs directory i
> found two things. The first was that the failed transactions were all
> returning http response code 206, timeout? The second was that i had a
> 43 mb core file. I shut down squid, turned off the transparent redirect
> rule in my firewall and everything started working normally. If this
> core file is useful in debugging i'd like to know what to do or where to
> send it, i can post it on a web page if interested. I'd appreciate any
> suggestions.
>    The squid version used is 2.6.13, memory and cpu usage during the
> times in question are not even moving, this system is not in any way
> heavily loaded.

Hi Dave,

Are you sure that you did not modify anything substantial in the past 3
days? Firewall rules, sysctl tunables in your FreeBSD box?

The core file suggests that your Squid might have crashed. Are you sure
that there is nothing in your cache.log?


Which firewall do you use to do transproxy? Can you show us your
relevant firewall rules?

Also posting your squid.conf might help. How many users are accessing
your cache normally? If you manually use your proxy server in your web
browser, does it help speed up the web requests?

Thanking you...



>    Any suggestions appreciated.
> Thanks.
> Dave.
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

dave-5
Hi,
    In my cache.log i am seeing one error: "page faults with physical i/o:
1" other than that nothing on this box has changed.
    For my firewall i'm using pf, and i use an rdr pass rule to redirect any
internal lan traffic to the router's squid port, this has worked fine so
far.
    Would the core file be helpful?
    Here's my squid.conf file, thanks a lot.
Dave.

http_port 127.0.0.1:3128 transparent
 icp_port 0
hierarchy_stoplist cgi-bin ?
 cache_mem 48 MB
 cache_swap_high 100%
 cache_swap_low 80%
 maximum_object_size 4096 KB
 minimum_object_size 0 KB
 maximum_object_size_in_memory 8 KB
 ipcache_size 1024
 ipcache_low 90
 ipcache_high 95
 fqdncache_size 1024
 cache_replacement_policy lru
 memory_replacement_policy lru
 cache_dir diskd /usr/local/squid/cache 600 32 512
access_log /usr/local/squid/logs/access.log squid
 cache_log /usr/local/squid/logs/cache.log
 cache_store_log none
 emulate_httpd_log off
 log_ip_on_direct on
 mime_table /usr/local/etc/squid/mime.conf
 log_mime_hdrs off
 pid_filename /usr/local/squid/logs/squid.pid
 log_fqdn off
 check_hostnames off
 allow_underscore off
 unlinkd_program /usr/local/libexec/squid/unlinkd
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443
acl Safe_ports port 80  # http
acl Safe_ports port 21  # ftp
acl Safe_ports port 443  # https
acl Safe_ports port 70  # gopher
acl Safe_ports port 210  # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280  # http-mgmt
acl Safe_ports port 488  # gss-http
acl Safe_ports port 591  # filemaker
acl Safe_ports port 777  # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
acl our_networks src 192.168.5.0/24
acl chat dstdomain "/usr/local/etc/squid/chat.txt"
acl porn url_regex "/usr/local/etc/squid/porn"
acl spyware dstdomain "/usr/local/etc/squid/spyware.acl"
http_access allow our_networks !chat !spyware !porn
http_access deny all
http_reply_access allow all
 cache_mgr [hidden email]
 cache_effective_user squid
 httpd_suppress_version_string on
half_closed_clients off
   redirect_program /usr/local/libexec/bannerfilter/redirector.pl

----- Original Message -----
From: "Tek Bahadur Limbu" <[hidden email]>
To: "Dave" <[hidden email]>
Cc: <[hidden email]>
Sent: Monday, June 25, 2007 2:07 PM
Subject: Re: [squid-users] FreeBSD Squid timeout issue


> Dave wrote:
>> Hello,
>>    This is a strange one. I've got a FreeBSD 6.2 router with squid on it
>> for transparent proxy. It has been working fine, until about 3 days ago
>> when i noticed one of my internal machines wasn't completing an http
>> transaction, in this case downloading of it's ports index. The connection
>> would start out extremely slowly and eventually timeout then giving an
>> error about a truncated file. I checked my firewall, hard disks, debug
>> logs, messages everything, couldn't find any error msgs or anything
>> obviously wrong. This morning an internal machine, this one a centos box,
>> began failing with it's yum updates, giving timeout errors on retrieving
>> rpm files from http sites. Again, i ran the log checks, didn't see
>> anything. So, i next checked the firewall, nothing in debug.log or
>> messages, but in the /usr/local/squid/logs directory i found two things.
>> The first was that the failed transactions were all returning http
>> response code 206, timeout? The second was that i had a 43 mb core file.
>> I shut down squid, turned off the transparent redirect rule in my
>> firewall and everything started working normally. If this core file is
>> useful in debugging i'd like to know what to do or where to send it, i
>> can post it on a web page if interested. I'd appreciate any suggestions.
>>    The squid version used is 2.6.13, memory and cpu usage during the
>> times in question are not even moving, this system is not in any way
>> heavily loaded.
>
> Hi Dave,
>
> Are you sure that you did not modify anything substantial in the past 3
> days? Firewall rules, sysctl tunables in your FreeBSD box?
>
> The core file suggests that your Squid might have crashed. Are you sure
> that there is nothing in your cache.log?
>
>
> Which firewall do you use to do transproxy? Can you show us your relevant
> firewall rules?
>
> Also posting your squid.conf might help. How many users are accessing your
> cache normally? If you manually use your proxy server in your web browser,
> does it help speed up the web requests?
>
> Thanking you...
>
>
>
>>    Any suggestions appreciated.
>> Thanks.
>> Dave.
>>
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Tek Bahadur Limbu
Dave wrote:
> Hi,
>    In my cache.log i am seeing one error: "page faults with physical
> i/o: 1" other than that nothing on this box has changed.

Hi Dave,

It does not matter if there is only 1 page fault.

>    For my firewall i'm using pf, and i use an rdr pass rule to redirect
> any internal lan traffic to the router's squid port, this has worked
> fine so far.

Since it was working a few days ago, pf must be doing it's job right.

>    Would the core file be helpful?

Well the core file would be helpful for Squid developers. I am not
familiar with it:)

>    Here's my squid.conf file, thanks a lot.
> Dave.
>
> http_port 127.0.0.1:3128 transparent

You could use:

http_port 3128 transparent right? Have you tried using your proxy server
manually in your web browser and tried browsing?


> icp_port 0
> hierarchy_stoplist cgi-bin ?
> cache_mem 48 MB
> cache_swap_high 100%
> cache_swap_low 80%

Why use 100% for cache_swap_high? You can use 95% right?


> maximum_object_size 4096 KB
> minimum_object_size 0 KB
> maximum_object_size_in_memory 8 KB
> ipcache_size 1024
> ipcache_low 90
> ipcache_high 95
> fqdncache_size 1024
> cache_replacement_policy lru
> memory_replacement_policy lru
> cache_dir diskd /usr/local/squid/cache 600 32 512

600 MB could be a bit small for a cache don't you think so? But it's
perfectly your choice.

> access_log /usr/local/squid/logs/access.log squid
> cache_log /usr/local/squid/logs/cache.log
> cache_store_log none
> emulate_httpd_log off

I prefer emulate_httpd_log on. But your choice might be different.

> log_ip_on_direct on
> mime_table /usr/local/etc/squid/mime.conf
> log_mime_hdrs off
> pid_filename /usr/local/squid/logs/squid.pid
> log_fqdn off
> check_hostnames off
> allow_underscore off
> unlinkd_program /usr/local/libexec/squid/unlinkd
> acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 443
> acl Safe_ports port 80  # http
> acl Safe_ports port 21  # ftp
> acl Safe_ports port 443  # https
> acl Safe_ports port 70  # gopher
> acl Safe_ports port 210  # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280  # http-mgmt
> acl Safe_ports port 488  # gss-http
> acl Safe_ports port 591  # filemaker
> acl Safe_ports port 777  # multiling http
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny to_localhost
> acl our_networks src 192.168.5.0/24
> acl chat dstdomain "/usr/local/etc/squid/chat.txt"
> acl porn url_regex "/usr/local/etc/squid/porn"
> acl spyware dstdomain "/usr/local/etc/squid/spyware.acl"
> http_access allow our_networks !chat !spyware !porn

How many entries do you have in the above ACLs? If there are thousands
of them, it could be one of the reasons why your Squid is slow and
giving timeouts.

> http_access deny all
> http_reply_access allow all
> cache_mgr [hidden email]
> cache_effective_user squid
> httpd_suppress_version_string on
> half_closed_clients off
>   redirect_program /usr/local/libexec/bannerfilter/redirector.pl

If all fails, why do you try deleting your cache_dir and recreate it.
Might work for your case!

Thanking you...

>
> ----- Original Message ----- From: "Tek Bahadur Limbu"
> <[hidden email]>
> To: "Dave" <[hidden email]>
> Cc: <[hidden email]>
> Sent: Monday, June 25, 2007 2:07 PM
> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>
>
>> Dave wrote:
>>> Hello,
>>>    This is a strange one. I've got a FreeBSD 6.2 router with squid on
>>> it for transparent proxy. It has been working fine, until about 3
>>> days ago when i noticed one of my internal machines wasn't completing
>>> an http transaction, in this case downloading of it's ports index.
>>> The connection would start out extremely slowly and eventually
>>> timeout then giving an error about a truncated file. I checked my
>>> firewall, hard disks, debug logs, messages everything, couldn't find
>>> any error msgs or anything obviously wrong. This morning an internal
>>> machine, this one a centos box, began failing with it's yum updates,
>>> giving timeout errors on retrieving rpm files from http sites. Again,
>>> i ran the log checks, didn't see anything. So, i next checked the
>>> firewall, nothing in debug.log or messages, but in the
>>> /usr/local/squid/logs directory i found two things. The first was
>>> that the failed transactions were all returning http response code
>>> 206, timeout? The second was that i had a 43 mb core file. I shut
>>> down squid, turned off the transparent redirect rule in my firewall
>>> and everything started working normally. If this core file is useful
>>> in debugging i'd like to know what to do or where to send it, i can
>>> post it on a web page if interested. I'd appreciate any suggestions.
>>>    The squid version used is 2.6.13, memory and cpu usage during the
>>> times in question are not even moving, this system is not in any way
>>> heavily loaded.
>>
>> Hi Dave,
>>
>> Are you sure that you did not modify anything substantial in the past
>> 3 days? Firewall rules, sysctl tunables in your FreeBSD box?
>>
>> The core file suggests that your Squid might have crashed. Are you
>> sure that there is nothing in your cache.log?
>>
>>
>> Which firewall do you use to do transproxy? Can you show us your
>> relevant firewall rules?
>>
>> Also posting your squid.conf might help. How many users are accessing
>> your cache normally? If you manually use your proxy server in your web
>> browser, does it help speed up the web requests?
>>
>> Thanking you...
>>
>>
>>
>>>    Any suggestions appreciated.
>>> Thanks.
>>> Dave.
>>>
>>>
>>>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Juan C. Crespo R.
page faults with physical i/o: 1?

    Are you sure about your Hard Disk? I mean did you check there is no
Bad Sectors on The HD?

JC

Tek Bahadur Limbu escribió:

> Dave wrote:
>> Hi,
>>    In my cache.log i am seeing one error: "page faults with physical
>> i/o: 1" other than that nothing on this box has changed.
>
> Hi Dave,
>
> It does not matter if there is only 1 page fault.
>
>>    For my firewall i'm using pf, and i use an rdr pass rule to
>> redirect any internal lan traffic to the router's squid port, this
>> has worked fine so far.
>
> Since it was working a few days ago, pf must be doing it's job right.
>
>>    Would the core file be helpful?
>
> Well the core file would be helpful for Squid developers. I am not
> familiar with it:)
>
>>    Here's my squid.conf file, thanks a lot.
>> Dave.
>>
>> http_port 127.0.0.1:3128 transparent
>
> You could use:
>
> http_port 3128 transparent right? Have you tried using your proxy
> server manually in your web browser and tried browsing?
>
>
>> icp_port 0
>> hierarchy_stoplist cgi-bin ?
>> cache_mem 48 MB
>> cache_swap_high 100%
>> cache_swap_low 80%
>
> Why use 100% for cache_swap_high? You can use 95% right?
>
>
>> maximum_object_size 4096 KB
>> minimum_object_size 0 KB
>> maximum_object_size_in_memory 8 KB
>> ipcache_size 1024
>> ipcache_low 90
>> ipcache_high 95
>> fqdncache_size 1024
>> cache_replacement_policy lru
>> memory_replacement_policy lru
>> cache_dir diskd /usr/local/squid/cache 600 32 512
>
> 600 MB could be a bit small for a cache don't you think so? But it's
> perfectly your choice.
>
>> access_log /usr/local/squid/logs/access.log squid
>> cache_log /usr/local/squid/logs/cache.log
>> cache_store_log none
>> emulate_httpd_log off
>
> I prefer emulate_httpd_log on. But your choice might be different.
>
>> log_ip_on_direct on
>> mime_table /usr/local/etc/squid/mime.conf
>> log_mime_hdrs off
>> pid_filename /usr/local/squid/logs/squid.pid
>> log_fqdn off
>> check_hostnames off
>> allow_underscore off
>> unlinkd_program /usr/local/libexec/squid/unlinkd
>> acl all src 0.0.0.0/0.0.0.0
>> acl manager proto cache_object
>> acl localhost src 127.0.0.1/255.255.255.255
>> acl to_localhost dst 127.0.0.0/8
>> acl SSL_ports port 443
>> acl Safe_ports port 80  # http
>> acl Safe_ports port 21  # ftp
>> acl Safe_ports port 443  # https
>> acl Safe_ports port 70  # gopher
>> acl Safe_ports port 210  # wais
>> acl Safe_ports port 1025-65535 # unregistered ports
>> acl Safe_ports port 280  # http-mgmt
>> acl Safe_ports port 488  # gss-http
>> acl Safe_ports port 591  # filemaker
>> acl Safe_ports port 777  # multiling http
>> acl CONNECT method CONNECT
>> http_access allow manager localhost
>> http_access deny manager
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>> http_access deny to_localhost
>> acl our_networks src 192.168.5.0/24
>> acl chat dstdomain "/usr/local/etc/squid/chat.txt"
>> acl porn url_regex "/usr/local/etc/squid/porn"
>> acl spyware dstdomain "/usr/local/etc/squid/spyware.acl"
>> http_access allow our_networks !chat !spyware !porn
>
> How many entries do you have in the above ACLs? If there are thousands
> of them, it could be one of the reasons why your Squid is slow and
> giving timeouts.
>
>> http_access deny all
>> http_reply_access allow all
>> cache_mgr [hidden email]
>> cache_effective_user squid
>> httpd_suppress_version_string on
>> half_closed_clients off
>>   redirect_program /usr/local/libexec/bannerfilter/redirector.pl
>
> If all fails, why do you try deleting your cache_dir and recreate it.
> Might work for your case!
>
> Thanking you...
>
>>
>> ----- Original Message ----- From: "Tek Bahadur Limbu"
>> <[hidden email]>
>> To: "Dave" <[hidden email]>
>> Cc: <[hidden email]>
>> Sent: Monday, June 25, 2007 2:07 PM
>> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>>
>>
>>> Dave wrote:
>>>> Hello,
>>>>    This is a strange one. I've got a FreeBSD 6.2 router with squid
>>>> on it for transparent proxy. It has been working fine, until about
>>>> 3 days ago when i noticed one of my internal machines wasn't
>>>> completing an http transaction, in this case downloading of it's
>>>> ports index. The connection would start out extremely slowly and
>>>> eventually timeout then giving an error about a truncated file. I
>>>> checked my firewall, hard disks, debug logs, messages everything,
>>>> couldn't find any error msgs or anything obviously wrong. This
>>>> morning an internal machine, this one a centos box, began failing
>>>> with it's yum updates, giving timeout errors on retrieving rpm
>>>> files from http sites. Again, i ran the log checks, didn't see
>>>> anything. So, i next checked the firewall, nothing in debug.log or
>>>> messages, but in the /usr/local/squid/logs directory i found two
>>>> things. The first was that the failed transactions were all
>>>> returning http response code 206, timeout? The second was that i
>>>> had a 43 mb core file. I shut down squid, turned off the
>>>> transparent redirect rule in my firewall and everything started
>>>> working normally. If this core file is useful in debugging i'd like
>>>> to know what to do or where to send it, i can post it on a web page
>>>> if interested. I'd appreciate any suggestions.
>>>>    The squid version used is 2.6.13, memory and cpu usage during
>>>> the times in question are not even moving, this system is not in
>>>> any way heavily loaded.
>>>
>>> Hi Dave,
>>>
>>> Are you sure that you did not modify anything substantial in the
>>> past 3 days? Firewall rules, sysctl tunables in your FreeBSD box?
>>>
>>> The core file suggests that your Squid might have crashed. Are you
>>> sure that there is nothing in your cache.log?
>>>
>>>
>>> Which firewall do you use to do transproxy? Can you show us your
>>> relevant firewall rules?
>>>
>>> Also posting your squid.conf might help. How many users are
>>> accessing your cache normally? If you manually use your proxy server
>>> in your web browser, does it help speed up the web requests?
>>>
>>> Thanking you...
>>>
>>>
>>>
>>>>    Any suggestions appreciated.
>>>> Thanks.
>>>> Dave.
>>>>
>>>>
>>>>
>>
>>
>>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

dave-5
In reply to this post by Tek Bahadur Limbu
Hello,
    Thanks for all replies.
    I've got a good hard disk, i've been checking that and haven't found any
problems or seen any error msgs in my logs.
    I've adjusted my high cache size from 100% to 95% but i'm starting to
look at is squid purging oldest items from my cache? It seems like when the
cache gets full or nearly so i start having this issue?
    As for my pornography and spyware rejection files they are each a
considerable size, they are lists of sites i don't want visited, downloaded,
or to have anything to do with. If there's a way to speed this up i'm all
for it.
Thanks.
Dave.

----- Original Message -----
From: "Tek Bahadur Limbu" <[hidden email]>
To: "Dave" <[hidden email]>
Cc: <[hidden email]>
Sent: Tuesday, June 26, 2007 4:14 AM
Subject: Re: [squid-users] FreeBSD Squid timeout issue


> Dave wrote:
>> Hi,
>>    In my cache.log i am seeing one error: "page faults with physical i/o:
>> 1" other than that nothing on this box has changed.
>
> Hi Dave,
>
> It does not matter if there is only 1 page fault.
>
>>    For my firewall i'm using pf, and i use an rdr pass rule to redirect
>> any internal lan traffic to the router's squid port, this has worked fine
>> so far.
>
> Since it was working a few days ago, pf must be doing it's job right.
>
>>    Would the core file be helpful?
>
> Well the core file would be helpful for Squid developers. I am not
> familiar with it:)
>
>>    Here's my squid.conf file, thanks a lot.
>> Dave.
>>
>> http_port 127.0.0.1:3128 transparent
>
> You could use:
>
> http_port 3128 transparent right? Have you tried using your proxy server
> manually in your web browser and tried browsing?
>
>
>> icp_port 0
>> hierarchy_stoplist cgi-bin ?
>> cache_mem 48 MB
>> cache_swap_high 100%
>> cache_swap_low 80%
>
> Why use 100% for cache_swap_high? You can use 95% right?
>
>
>> maximum_object_size 4096 KB
>> minimum_object_size 0 KB
>> maximum_object_size_in_memory 8 KB
>> ipcache_size 1024
>> ipcache_low 90
>> ipcache_high 95
>> fqdncache_size 1024
>> cache_replacement_policy lru
>> memory_replacement_policy lru
>> cache_dir diskd /usr/local/squid/cache 600 32 512
>
> 600 MB could be a bit small for a cache don't you think so? But it's
> perfectly your choice.
>
>> access_log /usr/local/squid/logs/access.log squid
>> cache_log /usr/local/squid/logs/cache.log
>> cache_store_log none
>> emulate_httpd_log off
>
> I prefer emulate_httpd_log on. But your choice might be different.
>
>> log_ip_on_direct on
>> mime_table /usr/local/etc/squid/mime.conf
>> log_mime_hdrs off
>> pid_filename /usr/local/squid/logs/squid.pid
>> log_fqdn off
>> check_hostnames off
>> allow_underscore off
>> unlinkd_program /usr/local/libexec/squid/unlinkd
>> acl all src 0.0.0.0/0.0.0.0
>> acl manager proto cache_object
>> acl localhost src 127.0.0.1/255.255.255.255
>> acl to_localhost dst 127.0.0.0/8
>> acl SSL_ports port 443
>> acl Safe_ports port 80  # http
>> acl Safe_ports port 21  # ftp
>> acl Safe_ports port 443  # https
>> acl Safe_ports port 70  # gopher
>> acl Safe_ports port 210  # wais
>> acl Safe_ports port 1025-65535 # unregistered ports
>> acl Safe_ports port 280  # http-mgmt
>> acl Safe_ports port 488  # gss-http
>> acl Safe_ports port 591  # filemaker
>> acl Safe_ports port 777  # multiling http
>> acl CONNECT method CONNECT
>> http_access allow manager localhost
>> http_access deny manager
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>> http_access deny to_localhost
>> acl our_networks src 192.168.5.0/24
>> acl chat dstdomain "/usr/local/etc/squid/chat.txt"
>> acl porn url_regex "/usr/local/etc/squid/porn"
>> acl spyware dstdomain "/usr/local/etc/squid/spyware.acl"
>> http_access allow our_networks !chat !spyware !porn
>
> How many entries do you have in the above ACLs? If there are thousands of
> them, it could be one of the reasons why your Squid is slow and giving
> timeouts.
>
>> http_access deny all
>> http_reply_access allow all
>> cache_mgr [hidden email]
>> cache_effective_user squid
>> httpd_suppress_version_string on
>> half_closed_clients off
>>   redirect_program /usr/local/libexec/bannerfilter/redirector.pl
>
> If all fails, why do you try deleting your cache_dir and recreate it.
> Might work for your case!
>
> Thanking you...
>
>>
>> ----- Original Message ----- From: "Tek Bahadur Limbu"
>> <[hidden email]>
>> To: "Dave" <[hidden email]>
>> Cc: <[hidden email]>
>> Sent: Monday, June 25, 2007 2:07 PM
>> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>>
>>
>>> Dave wrote:
>>>> Hello,
>>>>    This is a strange one. I've got a FreeBSD 6.2 router with squid on
>>>> it for transparent proxy. It has been working fine, until about 3 days
>>>> ago when i noticed one of my internal machines wasn't completing an
>>>> http transaction, in this case downloading of it's ports index. The
>>>> connection would start out extremely slowly and eventually timeout then
>>>> giving an error about a truncated file. I checked my firewall, hard
>>>> disks, debug logs, messages everything, couldn't find any error msgs or
>>>> anything obviously wrong. This morning an internal machine, this one a
>>>> centos box, began failing with it's yum updates, giving timeout errors
>>>> on retrieving rpm files from http sites. Again, i ran the log checks,
>>>> didn't see anything. So, i next checked the firewall, nothing in
>>>> debug.log or messages, but in the /usr/local/squid/logs directory i
>>>> found two things. The first was that the failed transactions were all
>>>> returning http response code 206, timeout? The second was that i had a
>>>> 43 mb core file. I shut down squid, turned off the transparent redirect
>>>> rule in my firewall and everything started working normally. If this
>>>> core file is useful in debugging i'd like to know what to do or where
>>>> to send it, i can post it on a web page if interested. I'd appreciate
>>>> any suggestions.
>>>>    The squid version used is 2.6.13, memory and cpu usage during the
>>>> times in question are not even moving, this system is not in any way
>>>> heavily loaded.
>>>
>>> Hi Dave,
>>>
>>> Are you sure that you did not modify anything substantial in the past 3
>>> days? Firewall rules, sysctl tunables in your FreeBSD box?
>>>
>>> The core file suggests that your Squid might have crashed. Are you sure
>>> that there is nothing in your cache.log?
>>>
>>>
>>> Which firewall do you use to do transproxy? Can you show us your
>>> relevant firewall rules?
>>>
>>> Also posting your squid.conf might help. How many users are accessing
>>> your cache normally? If you manually use your proxy server in your web
>>> browser, does it help speed up the web requests?
>>>
>>> Thanking you...
>>>
>>>
>>>
>>>>    Any suggestions appreciated.
>>>> Thanks.
>>>> Dave.
>>>>
>>>>
>>>>
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Amos Jeffries
Administrator
> Hello,
>     Thanks for all replies.
>     I've got a good hard disk, i've been checking that and haven't found
> any
> problems or seen any error msgs in my logs.
>     I've adjusted my high cache size from 100% to 95% but i'm starting to
> look at is squid purging oldest items from my cache? It seems like when
> the
> cache gets full or nearly so i start having this issue?
>     As for my pornography and spyware rejection files they are each a
> considerable size, they are lists of sites i don't want visited,
> downloaded,
> or to have anything to do with. If there's a way to speed this up i'm all
> for it.
> Thanks.
> Dave.
>

Make sure that you are using dst or dstdomain as the ACL types on teh
lareg lists instead of regex.
 The regex is quite slow and large lists often become a drag. After
splitting the lists into 'need regex' and dstdomain eth speed increase is
still often worth the extra time spent maintaining two lists.

Make sure there is extra space on the cache disk. All the tutorials
mention making the cache 60%-80% of drive size. I can't recall what the
exact reasons were but it had something to do with OS-level handling on
the drive.

Amos


Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

dave-5
Hello,
    Thanks for your suggestions. I checked my squid.conf and the acls for
chat and spyware were of type dstdomain, porn was url_regexp, i changed that
to dstdomain and now when i do a squid -k reconfigure i am getting syntax
errors. AS for the file sizes chat has 2 lines, spyware has 1440 lines, and
of course the big one the porn rejection file has 15025 lines. The error i'm
repeatedly getting now and i didn't get it when the file was url_regexp was
that i have subdomains of parent domains and they are ignored. Does anyone
use spyware, porn, and chat rejections, and if so where did you obtain them?
    Also, i'm wondering why my cache isn't clearing out the oldest items, is
my cache replacement policy bad?
Thanks.
Dave.

----- Original Message -----
From: <[hidden email]>
To: <[hidden email]>
Sent: Tuesday, June 26, 2007 9:27 PM
Subject: Re: [squid-users] FreeBSD Squid timeout issue


>> Hello,
>>     Thanks for all replies.
>>     I've got a good hard disk, i've been checking that and haven't found
>> any
>> problems or seen any error msgs in my logs.
>>     I've adjusted my high cache size from 100% to 95% but i'm starting to
>> look at is squid purging oldest items from my cache? It seems like when
>> the
>> cache gets full or nearly so i start having this issue?
>>     As for my pornography and spyware rejection files they are each a
>> considerable size, they are lists of sites i don't want visited,
>> downloaded,
>> or to have anything to do with. If there's a way to speed this up i'm all
>> for it.
>> Thanks.
>> Dave.
>>
>
> Make sure that you are using dst or dstdomain as the ACL types on teh
> lareg lists instead of regex.
> The regex is quite slow and large lists often become a drag. After
> splitting the lists into 'need regex' and dstdomain eth speed increase is
> still often worth the extra time spent maintaining two lists.
>
> Make sure there is extra space on the cache disk. All the tutorials
> mention making the cache 60%-80% of drive size. I can't recall what the
> exact reasons were but it had something to do with OS-level handling on
> the drive.
>
> Amos
>

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Amos Jeffries
Administrator
> Hello,
>     Thanks for your suggestions. I checked my squid.conf and the acls for
> chat and spyware were of type dstdomain, porn was url_regexp, i changed
> that
> to dstdomain and now when i do a squid -k reconfigure i am getting syntax
> errors. AS for the file sizes chat has 2 lines, spyware has 1440 lines,
> and
> of course the big one the porn rejection file has 15025 lines.

Oh, aye, that way huge for regexp to handle.

> The error
> i'm
> repeatedly getting now and i didn't get it when the file was url_regexp
> was
> that i have subdomains of parent domains and they are ignored.

Hmm, sure this is an error? not a warning? It sound to me like a little
maintenance needs doing on that file.
 - Duplicates can be removed.
 - 'example.com can' be removed if you have '.example.com' elsewhere.
 - 'www.example.com' can be removed if you have '.example.com' elsewhere.
Sounds like the last of these two are what you are being warned about.

If your still having trouble you can email me the file and I'll check it
myself.

> Does anyone
> use spyware, porn, and chat rejections, and if so where did you obtain
> them?
>     Also, i'm wondering why my cache isn't clearing out the oldest items,
> is
> my cache replacement policy bad?

Quite possibly, my squid expertise doesn't extend into the replacement
policies, yet. You will have to look to one oef the others for help.


> Thanks.
> Dave.
>
> ----- Original Message -----
> From: <[hidden email]>
> To: <[hidden email]>
> Sent: Tuesday, June 26, 2007 9:27 PM
> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>
>
>>> Hello,
>>>     Thanks for all replies.
>>>     I've got a good hard disk, i've been checking that and haven't
>>> found
>>> any
>>> problems or seen any error msgs in my logs.
>>>     I've adjusted my high cache size from 100% to 95% but i'm starting
>>> to
>>> look at is squid purging oldest items from my cache? It seems like when
>>> the
>>> cache gets full or nearly so i start having this issue?
>>>     As for my pornography and spyware rejection files they are each a
>>> considerable size, they are lists of sites i don't want visited,
>>> downloaded,
>>> or to have anything to do with. If there's a way to speed this up i'm
>>> all
>>> for it.
>>> Thanks.
>>> Dave.
>>>
>>
>> Make sure that you are using dst or dstdomain as the ACL types on teh
>> lareg lists instead of regex.
>> The regex is quite slow and large lists often become a drag. After
>> splitting the lists into 'need regex' and dstdomain eth speed increase
>> is
>> still often worth the extra time spent maintaining two lists.
>>
>> Make sure there is extra space on the cache disk. All the tutorials
>> mention making the cache 60%-80% of drive size. I can't recall what the
>> exact reasons were but it had something to do with OS-level handling on
>> the drive.
>>
>> Amos
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

dave-5
Hello,
    Thanks to everyone who has offered suggestions with this issue. To
[hidden email] i tried a direct email, but your email server rejected
my msg.
    I am getting a warning with my porn rejectionslist, which only occurs
when the configuration is changed from url_regexp to dstdomain, that
subdomains are not valid. The file itself is at:
http://www.davemehler.com/porn.gz
    I thought that would be easier than trying to push an attachment through
to the list to everyone.
    I'm also wondering if my cache replacement policy is wrong, old items
don't seem to be being removed, even though the cache still has 81 mb before
its full.
    If the rest of my config would be helpful i'll post it.
Thanks.
Dave.

----- Original Message -----
From: <[hidden email]>
To: "Dave" <[hidden email]>
Cc: <[hidden email]>; <[hidden email]>
Sent: Thursday, June 28, 2007 6:50 PM
Subject: Re: [squid-users] FreeBSD Squid timeout issue


>> Hello,
>>     Thanks for your suggestions. I checked my squid.conf and the acls for
>> chat and spyware were of type dstdomain, porn was url_regexp, i changed
>> that
>> to dstdomain and now when i do a squid -k reconfigure i am getting syntax
>> errors. AS for the file sizes chat has 2 lines, spyware has 1440 lines,
>> and
>> of course the big one the porn rejection file has 15025 lines.
>
> Oh, aye, that way huge for regexp to handle.
>
>> The error
>> i'm
>> repeatedly getting now and i didn't get it when the file was url_regexp
>> was
>> that i have subdomains of parent domains and they are ignored.
>
> Hmm, sure this is an error? not a warning? It sound to me like a little
> maintenance needs doing on that file.
> - Duplicates can be removed.
> - 'example.com can' be removed if you have '.example.com' elsewhere.
> - 'www.example.com' can be removed if you have '.example.com' elsewhere.
> Sounds like the last of these two are what you are being warned about.
>
> If your still having trouble you can email me the file and I'll check it
> myself.
>
>> Does anyone
>> use spyware, porn, and chat rejections, and if so where did you obtain
>> them?
>>     Also, i'm wondering why my cache isn't clearing out the oldest items,
>> is
>> my cache replacement policy bad?
>
> Quite possibly, my squid expertise doesn't extend into the replacement
> policies, yet. You will have to look to one oef the others for help.
>
>
>> Thanks.
>> Dave.
>>
>> ----- Original Message -----
>> From: <[hidden email]>
>> To: <[hidden email]>
>> Sent: Tuesday, June 26, 2007 9:27 PM
>> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>>
>>
>>>> Hello,
>>>>     Thanks for all replies.
>>>>     I've got a good hard disk, i've been checking that and haven't
>>>> found
>>>> any
>>>> problems or seen any error msgs in my logs.
>>>>     I've adjusted my high cache size from 100% to 95% but i'm starting
>>>> to
>>>> look at is squid purging oldest items from my cache? It seems like when
>>>> the
>>>> cache gets full or nearly so i start having this issue?
>>>>     As for my pornography and spyware rejection files they are each a
>>>> considerable size, they are lists of sites i don't want visited,
>>>> downloaded,
>>>> or to have anything to do with. If there's a way to speed this up i'm
>>>> all
>>>> for it.
>>>> Thanks.
>>>> Dave.
>>>>
>>>
>>> Make sure that you are using dst or dstdomain as the ACL types on teh
>>> lareg lists instead of regex.
>>> The regex is quite slow and large lists often become a drag. After
>>> splitting the lists into 'need regex' and dstdomain eth speed increase
>>> is
>>> still often worth the extra time spent maintaining two lists.
>>>
>>> Make sure there is extra space on the cache disk. All the tutorials
>>> mention making the cache 60%-80% of drive size. I can't recall what the
>>> exact reasons were but it had something to do with OS-level handling on
>>> the drive.
>>>
>>> Amos
>>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Amos Jeffries
Administrator
Dave wrote:
> Hello,
>    Thanks to everyone who has offered suggestions with this issue. To
> [hidden email] i tried a direct email, but your email server
> rejected my msg.

Ah, maybe you have other problems to. My squid3 address is only
protected from current spam sources. You'll have to check the bounce
message to see what it was the mailing list hides your source info for
me to look up from this end and whitelist you..

>    I am getting a warning with my porn rejectionslist, which only occurs
> when the configuration is changed from url_regexp to dstdomain, that
> subdomains are not valid. The file itself is at:
> http://www.davemehler.com/porn.gz

Your have rather a mix of content to that file. TO be fast and well
handled  I would suggest breaking it into three parts in the squid
config. Like so:

   acl porn dst '.../porn.ips'
   acl porn dstdomain '.../porn.domains'
   acl porn url_regex '.../porn.regex'

I'm not sure if all versions of squid can take one acl with multiple
types. If it does not, they may need different names.

Where the:
  *.ips gets the lines like '192.168.0.0'
  *.domains gets lines like '.zugs-model-portal.com'
  *.regex gets lines with '=female+wrestling', etc.

(note the preceding '.' in dstdomain, it wil catch any sub-domain
funkiness they try.)

That way each line is handled by an appropriate ACL, and most of them
have fast types.

>    I thought that would be easier than trying to push an attachment
> through to the list to everyone.
>    I'm also wondering if my cache replacement policy is wrong, old items
> don't seem to be being removed, even though the cache still has 81 mb
> before its full.
>    If the rest of my config would be helpful i'll post it.

You posted a copy of it 26 June, if its changed it might be worth a look
at the new version. Otherwise, I just took a look back at that and diskd
is one of the filesystems I thought was unused these days. aufs if its
available is easier on the disk.

I just noticed you have an object size of 0 accepted, I wonder of the
'old' objects are those ones which have no headers to math age against
(or I might be talking garbage here, I really don't know much about the
stores).

Hmm, have you checked out all the stats/settings squidclient can give you?
('squidclient mgr:menu' for a list, try the store-related entries.
  ie 'squidclient mgr:storedir' to see the LRU policy stats)


> Thanks.
> Dave.
>
> ----- Original Message ----- From: <[hidden email]>
> To: "Dave" <[hidden email]>
> Cc: <[hidden email]>; <[hidden email]>
> Sent: Thursday, June 28, 2007 6:50 PM
> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>
>
>>> Hello,
>>>     Thanks for your suggestions. I checked my squid.conf and the acls
>>> for
>>> chat and spyware were of type dstdomain, porn was url_regexp, i changed
>>> that
>>> to dstdomain and now when i do a squid -k reconfigure i am getting
>>> syntax
>>> errors. AS for the file sizes chat has 2 lines, spyware has 1440 lines,
>>> and
>>> of course the big one the porn rejection file has 15025 lines.
>>
>> Oh, aye, that way huge for regexp to handle.
>>
>>> The error
>>> i'm
>>> repeatedly getting now and i didn't get it when the file was url_regexp
>>> was
>>> that i have subdomains of parent domains and they are ignored.
>>
>> Hmm, sure this is an error? not a warning? It sound to me like a little
>> maintenance needs doing on that file.
>> - Duplicates can be removed.
>> - 'example.com can' be removed if you have '.example.com' elsewhere.
>> - 'www.example.com' can be removed if you have '.example.com' elsewhere.
>> Sounds like the last of these two are what you are being warned about.
>>
>> If your still having trouble you can email me the file and I'll check it
>> myself.
>>
>>> Does anyone
>>> use spyware, porn, and chat rejections, and if so where did you obtain
>>> them?
>>>     Also, i'm wondering why my cache isn't clearing out the oldest
>>> items,
>>> is
>>> my cache replacement policy bad?
>>
>> Quite possibly, my squid expertise doesn't extend into the replacement
>> policies, yet. You will have to look to one oef the others for help.
>>
>>
>>> Thanks.
>>> Dave.
>>>
>>> ----- Original Message -----
>>> From: <[hidden email]>
>>> To: <[hidden email]>
>>> Sent: Tuesday, June 26, 2007 9:27 PM
>>> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>>>
>>>
>>>>> Hello,
>>>>>     Thanks for all replies.
>>>>>     I've got a good hard disk, i've been checking that and haven't
>>>>> found
>>>>> any
>>>>> problems or seen any error msgs in my logs.
>>>>>     I've adjusted my high cache size from 100% to 95% but i'm starting
>>>>> to
>>>>> look at is squid purging oldest items from my cache? It seems like
>>>>> when
>>>>> the
>>>>> cache gets full or nearly so i start having this issue?
>>>>>     As for my pornography and spyware rejection files they are each a
>>>>> considerable size, they are lists of sites i don't want visited,
>>>>> downloaded,
>>>>> or to have anything to do with. If there's a way to speed this up i'm
>>>>> all
>>>>> for it.
>>>>> Thanks.
>>>>> Dave.
>>>>>
>>>>
>>>> Make sure that you are using dst or dstdomain as the ACL types on teh
>>>> lareg lists instead of regex.
>>>> The regex is quite slow and large lists often become a drag. After
>>>> splitting the lists into 'need regex' and dstdomain eth speed increase
>>>> is
>>>> still often worth the extra time spent maintaining two lists.
>>>>
>>>> Make sure there is extra space on the cache disk. All the tutorials
>>>> mention making the cache 60%-80% of drive size. I can't recall what the
>>>> exact reasons were but it had something to do with OS-level handling on
>>>> the drive.
>>>>
>>>> Amos
>>>>
>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

dave-5
Hi,
    Thanks for your reply. The following is the ip and abbreviated msg:
(reason: 554 5.7.1 Service unavailable; Client host [65.24.5.137] blocked
using dnsbl-1.uceprotect.net;
    To my squid issue, if aufs is less intensive and more efficient i'll
definitely switch over to it. As for your suggestion about splitting in to
multiple files I believe the version i have can do this, it has multiple acl
statements for the safe_ports definition. My issue though is there's like
15000+ lines in this file, and investigating some like 500 are duplicates.
I'd rather not have to manually go through this and do the split, is there a
way i can split based on the dst, dstdomain, or url_regexp you referenced?
I'll check out squidclient output, but last time i did i didn't see anything
that stood out.
Thanks.
Dave.


----- Original Message -----
From: "Amos Jeffries" <[hidden email]>
To: "Dave" <[hidden email]>
Cc: <[hidden email]>
Sent: Tuesday, July 03, 2007 7:42 AM
Subject: Re: [squid-users] FreeBSD Squid timeout issue


> Dave wrote:
>> Hello,
>>    Thanks to everyone who has offered suggestions with this issue. To
>> [hidden email] i tried a direct email, but your email server
>> rejected my msg.
>
> Ah, maybe you have other problems to. My squid3 address is only protected
> from current spam sources. You'll have to check the bounce message to see
> what it was the mailing list hides your source info for me to look up from
> this end and whitelist you..
>
>>    I am getting a warning with my porn rejectionslist, which only occurs
>> when the configuration is changed from url_regexp to dstdomain, that
>> subdomains are not valid. The file itself is at:
>> http://www.davemehler.com/porn.gz
>
> Your have rather a mix of content to that file. TO be fast and well
> handled  I would suggest breaking it into three parts in the squid config.
> Like so:
>
>   acl porn dst '.../porn.ips'
>   acl porn dstdomain '.../porn.domains'
>   acl porn url_regex '.../porn.regex'
>
> I'm not sure if all versions of squid can take one acl with multiple
> types. If it does not, they may need different names.
>
> Where the:
>  *.ips gets the lines like '192.168.0.0'
>  *.domains gets lines like '.zugs-model-portal.com'
>  *.regex gets lines with '=female+wrestling', etc.
>
> (note the preceding '.' in dstdomain, it wil catch any sub-domain
> funkiness they try.)
>
> That way each line is handled by an appropriate ACL, and most of them have
> fast types.
>
>>    I thought that would be easier than trying to push an attachment
>> through to the list to everyone.
>>    I'm also wondering if my cache replacement policy is wrong, old items
>> don't seem to be being removed, even though the cache still has 81 mb
>> before its full.
>>    If the rest of my config would be helpful i'll post it.
>
> You posted a copy of it 26 June, if its changed it might be worth a look
> at the new version. Otherwise, I just took a look back at that and diskd
> is one of the filesystems I thought was unused these days. aufs if its
> available is easier on the disk.
>
> I just noticed you have an object size of 0 accepted, I wonder of the
> 'old' objects are those ones which have no headers to math age against (or
> I might be talking garbage here, I really don't know much about the
> stores).
>
> Hmm, have you checked out all the stats/settings squidclient can give you?
> ('squidclient mgr:menu' for a list, try the store-related entries.
>  ie 'squidclient mgr:storedir' to see the LRU policy stats)
>
>
>> Thanks.
>> Dave.
>>
>> ----- Original Message ----- From: <[hidden email]>
>> To: "Dave" <[hidden email]>
>> Cc: <[hidden email]>; <[hidden email]>
>> Sent: Thursday, June 28, 2007 6:50 PM
>> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>>
>>
>>>> Hello,
>>>>     Thanks for your suggestions. I checked my squid.conf and the acls
>>>> for
>>>> chat and spyware were of type dstdomain, porn was url_regexp, i changed
>>>> that
>>>> to dstdomain and now when i do a squid -k reconfigure i am getting
>>>> syntax
>>>> errors. AS for the file sizes chat has 2 lines, spyware has 1440 lines,
>>>> and
>>>> of course the big one the porn rejection file has 15025 lines.
>>>
>>> Oh, aye, that way huge for regexp to handle.
>>>
>>>> The error
>>>> i'm
>>>> repeatedly getting now and i didn't get it when the file was url_regexp
>>>> was
>>>> that i have subdomains of parent domains and they are ignored.
>>>
>>> Hmm, sure this is an error? not a warning? It sound to me like a little
>>> maintenance needs doing on that file.
>>> - Duplicates can be removed.
>>> - 'example.com can' be removed if you have '.example.com' elsewhere.
>>> - 'www.example.com' can be removed if you have '.example.com' elsewhere.
>>> Sounds like the last of these two are what you are being warned about.
>>>
>>> If your still having trouble you can email me the file and I'll check it
>>> myself.
>>>
>>>> Does anyone
>>>> use spyware, porn, and chat rejections, and if so where did you obtain
>>>> them?
>>>>     Also, i'm wondering why my cache isn't clearing out the oldest
>>>> items,
>>>> is
>>>> my cache replacement policy bad?
>>>
>>> Quite possibly, my squid expertise doesn't extend into the replacement
>>> policies, yet. You will have to look to one oef the others for help.
>>>
>>>
>>>> Thanks.
>>>> Dave.
>>>>
>>>> ----- Original Message -----
>>>> From: <[hidden email]>
>>>> To: <[hidden email]>
>>>> Sent: Tuesday, June 26, 2007 9:27 PM
>>>> Subject: Re: [squid-users] FreeBSD Squid timeout issue
>>>>
>>>>
>>>>>> Hello,
>>>>>>     Thanks for all replies.
>>>>>>     I've got a good hard disk, i've been checking that and haven't
>>>>>> found
>>>>>> any
>>>>>> problems or seen any error msgs in my logs.
>>>>>>     I've adjusted my high cache size from 100% to 95% but i'm
>>>>>> starting
>>>>>> to
>>>>>> look at is squid purging oldest items from my cache? It seems like
>>>>>> when
>>>>>> the
>>>>>> cache gets full or nearly so i start having this issue?
>>>>>>     As for my pornography and spyware rejection files they are each a
>>>>>> considerable size, they are lists of sites i don't want visited,
>>>>>> downloaded,
>>>>>> or to have anything to do with. If there's a way to speed this up i'm
>>>>>> all
>>>>>> for it.
>>>>>> Thanks.
>>>>>> Dave.
>>>>>>
>>>>>
>>>>> Make sure that you are using dst or dstdomain as the ACL types on teh
>>>>> lareg lists instead of regex.
>>>>> The regex is quite slow and large lists often become a drag. After
>>>>> splitting the lists into 'need regex' and dstdomain eth speed increase
>>>>> is
>>>>> still often worth the extra time spent maintaining two lists.
>>>>>
>>>>> Make sure there is extra space on the cache disk. All the tutorials
>>>>> mention making the cache 60%-80% of drive size. I can't recall what
>>>>> the
>>>>> exact reasons were but it had something to do with OS-level handling
>>>>> on
>>>>> the drive.
>>>>>
>>>>> Amos
>>>>>
>>>>
>>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Amos Jeffries
Administrator
Dave wrote:

> Hi,
>    Thanks for your reply. The following is the ip and abbreviated msg:
> (reason: 554 5.7.1 Service unavailable; Client host [65.24.5.137]
> blocked using dnsbl-1.uceprotect.net;
>    To my squid issue, if aufs is less intensive and more efficient i'll
> definitely switch over to it. As for your suggestion about splitting in
> to multiple files I believe the version i have can do this, it has
> multiple acl statements for the safe_ports definition. My issue though
> is there's like 15000+ lines in this file, and investigating some like
> 500 are duplicates. I'd rather not have to manually go through this and
> do the split, is there a way i can split based on the dst, dstdomain, or
> url_regexp you referenced?

I just used the following commands, pulled off most of the job in a few
minutes. The remainders that got left as regex was small. There are some
that are duplicates of the domain-only list, but that can be dealt with
later.


# Pull out the IPs
grep -v -E "[a-z]+" porn | sort -u >porn.ipa

# copy everything else into a temp file
grep -v -E "[a-z]+" porn | sort -u >temp.1

# pull out lines with only domain name
grep -E "^([0-9a-z\-]\.)+[a-z]+$" temp.1 | sort -u >temp.d

# pull out everthing without a domain name into another temp
grep -v -E "^([0-9a-z\-]\.)+[a-z]+$" temp.1 | sort -u >temp.2
rm temp.1

# pull out lines that are domain/ or domain<space> and drop the end
grep -E "^([0-9a-z\-]\.)+[a-z]+[\/ ]$" temp.2 | sed s/\\/// | sed s/\\
// | sort -u >>temp.d

# leave the rest as regex patterns
grep -v -E "^([0-9a-z\-]\.)+[a-z]+[\/ ]$" temp.2 | sort -u >porn.regex
rm temp.2

# sort the just-domains and make sure there are no duplicate.
cat temp.d | sort -u > porn.domains
rm temp.d

Amos
Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD Squid timeout issue

Chris Robertson-2
Amos Jeffries wrote:

> Dave wrote:
>> Hi,
>>    Thanks for your reply. The following is the ip and abbreviated msg:
>> (reason: 554 5.7.1 Service unavailable; Client host [65.24.5.137]
>> blocked using dnsbl-1.uceprotect.net;
>>    To my squid issue, if aufs is less intensive and more efficient
>> i'll definitely switch over to it. As for your suggestion about
>> splitting in to multiple files I believe the version i have can do
>> this, it has multiple acl statements for the safe_ports definition.
>> My issue though is there's like 15000+ lines in this file, and
>> investigating some like 500 are duplicates. I'd rather not have to
>> manually go through this and do the split, is there a way i can split
>> based on the dst, dstdomain, or url_regexp you referenced?
>
> I just used the following commands, pulled off most of the job in a
> few minutes. The remainders that got left as regex was small. There
> are some that are duplicates of the domain-only list, but that can be
> dealt with later.
>
>
> # Pull out the IPs
> grep -v -E "[a-z]+" porn | sort -u >porn.ipa
>
> # copy everything else into a temp file
> grep -v -E "[a-z]+" porn | sort -u >temp.1
>
> # pull out lines with only domain name
> grep -E "^([0-9a-z\-]\.)+[a-z]+$" temp.1 | sort -u >temp.d
>
> # pull out everthing without a domain name into another temp
> grep -v -E "^([0-9a-z\-]\.)+[a-z]+$" temp.1 | sort -u >temp.2
> rm temp.1
>
> # pull out lines that are domain/ or domain<space> and drop the end
> grep -E "^([0-9a-z\-]\.)+[a-z]+[\/ ]$" temp.2 | sed s/\\/// | sed s/\\
> // | sort -u >>temp.d
>
> # leave the rest as regex patterns
> grep -v -E "^([0-9a-z\-]\.)+[a-z]+[\/ ]$" temp.2 | sort -u >porn.regex
> rm temp.2
>
> # sort the just-domains and make sure there are no duplicate.
> cat temp.d | sort -u > porn.domains
> rm temp.d
>
> Amos
For what it's worth, this method will not remove overlapping domains (if
http://yahoo.com/, http://www.yahoo.com/index.html and
http://mail.yahoo.com are all included, you will have more entries than
you need).  Minor issue, perhaps, but it can lead to unpredictable
results (the dstdomain acl type will disregard overlaps to keep the tree
sort simple).

Find attached a (less than pretty) Perl script that will resolve these
issues*.  Critiques and patches welcome (hopefully it's commented enough
to make sense to someone else).  It's likely not optimized, but in my
case the input list is not changed often, so optimization is not critical.

Chris

* It is only set up to handle a list of URLs.  It stuffs IP addresses in
with the other reg exes.  It will not account for what I assume to be
commented lines (starting with a #), or strings such as
"=female+wrestling" but will treat them as part of the domain name.  
Error checking is minimal.  It works for me, but comes without
warranty.  Salt to taste.

#!/usr/bin/perl

# Parses a the file, determines if a line should really be a site or
# domain block and pushes the data into the proper files.

use strict;

# Define variables;
$| = 1;
my ($url, $host, $scope, $time, $final);
my %domains = ();
my %regex = ();
my @hosts;
my @site_array;
my @lines;
# Open a bunch of file handles.
my $urlfile = "/etc/squid/acls/ExternalLinks.txt";
open (URLFILE, "< $urlfile");
my $allowurlfile = "/etc/squid/acls/allowurls";
unlink ($allowurlfile);
open (ALLOWURLS, "> $allowurlfile");
my $allowdomfile = "/etc/squid/acls/allowdoms";
unlink ($allowdomfile);
open (ALLOWDOMS, "> $allowdomfile");

# Start reading input
print "Working...";
while ($url = <URLFILE>) {
  chomp $url;
  my $time = time();
  # grab the host & (if it exists) path
  (undef, undef, $final) = $url =~ m#^(http(s)?://)?(.*)$#i;
  # Split the string on forward slashes
  my @url_array = split "/", $final;
  # Grab the host
  $host = shift @url_array;
  # Split the host into domain components
  my @host_array = split '\.', $host;
  # Check for a leading www (get rid of it!)
  if (@host_array[0] eq "www") {
    shift @host_array;
  }
  # Put the fqdn back together.
  $host = join (".", @host_array);
  if (scalar(@url_array[0]) || isIP(@url_array)) { # Is this REALLY a site allow?
    # Yes, it's a site.
    my $time = time();
    # grab the host & (if it exists) path
    (undef, undef, $final) = $url =~ m#^(http(s)?://)?(.*)$#;
    # Escape special regex characters
    $final =~ s/(\\|\||\(|\)|\[|\{|\^|\$|\*|\+|\?)/\\$1/g;
    # Split the string on forward slashes
    my @url_array = split "/", $final;
    # Grab the host
    my $host = shift @url_array;
    # Split the host into domain components
    my @host_array = split '\.', $host;
    # Check for a leading www (get rid of it!)
    if (@host_array[0] eq "www") {
      shift @host_array;
    }
    # Put the fqdn back together.
    $host = join (".", @host_array);
    $final = join ('.', @host_array);
    $final .= "/";
    $final .= join ("/", @url_array);
    $final =~ s/\./\\\./g;
    # Now check for a duplicate site block
    if (1 != $regex{$final}->{defined}) {
      $regex{$final}->{defined} = 1;
      # Create the entry
#print "Added site $url\n";
      $scope = "Site";
      $domains{$url}->{host} = $host;
      $domains{$url}->{final} = $final;
      $domains{$url}->{scope} = $scope;
      $domains{$url}->{time} = $time;
    }
  } else {
    # It's a Domain.
    # Is it a repeat?
    if (1 != $domains{$host}->{defined}) {
      # Haven't seen this one before.  Mark it as seen.
      $domains{$host}->{defined} = 1;
      $scope = "Domain";
      # Clear out empty array elements
      $final = join ('.', @host_array);
      $final = ".$final";
      # Create the entry
#print "Added domain $url\n";
      $domains{$url}->{host} = $host;
      $domains{$url}->{final} = $final;
      $domains{$url}->{scope} = $scope;
      $domains{$url}->{time} = $time;
      push @hosts, $host;
    }
  }
}
# Done reading the file.  Let's filter the data to remove duplication.
# Sort by number of host elements, remove subdomains of defined domains
sub byNumberOfHostElements { $a <=> $b }
# Somehow, this performs the desired sort.  Perl is weird.
my @sortedHosts = map { $_->[0] }
                  sort {
                      my @a_fields = @$a[1..$#$a];
                      my @b_fields = @$b[1..$#$b];

                      scalar(@a_fields) <=> scalar(@b_fields)
                  }
                  map { [$_, split'\.'] } @hosts;
foreach $host (@sortedHosts) {
  my $dotHost = ".$host";
  foreach my $urlToTest (keys %domains) {
    my $hostToTest = $domains{$urlToTest}->{host};
    my $dotHostToTest = ".$hostToTest";
    my $deleted = 0;
    my $different = 0;
    # If a subdomain of the host is found, drop it from the list
    if (($hostToTest =~ m/$host$/)) {
#print "$dotHost - $dotHostToTest - $urlToTest\n";
      # We have a potential match.  Verify further...
      my @host1 = split'\.', $hostToTest;
      my @host2 = split'\.', $host;
      my ($test1, $test2);
      while ($test1 = pop (@host1)) {
        $test2 = pop (@host2);
        if (defined($test1) && defined($test2)) {
          if ($test1 eq $test2) {
#print "# They match so far ($test1 eq $test2), check the next element\n";
            # They match so far, check the next element
            next;
          } else {
#print "# The hosts are different ($hostToTest $host). Break out of here.\n";
            # The hosts are different. Break out of here.
            $different = 1;
            last;
          }
        } elsif (!defined($test2)) {
          # We have a match.  Drop the subdomain.
#print "$hostToTest is a subdomain of $host.  Deleting.\n";
print "."; # So there is SOME indication of work progressing...
          delete $domains{$urlToTest};
          #delete @sortedHosts[$host];
          $deleted = 1;
        }
      }
      if (!$deleted && !$different && ("Domain" ne $domains{$urlToTest}->{scope})) {
#print "$urlToTest is a subdomain of $host.  Deleting.\n";
print "."; # More progress indication
        delete $domains{$urlToTest};
      }
    }
  }
}
print "\n";
# Write the data
print ALLOWDOMS ".apexvs.com\n";
foreach $url (keys %domains) {
  $final = $domains{$url}->{final};
  $time = $domains{$url}->{time};
  if ("Site" eq $domains{$url}->{scope}) {
    $scope = "Site";
    print ALLOWURLS "$final\n";
  } else {
    $scope = "Domain";
    print ALLOWDOMS "$final\n";
  }
}
# Close it all up
close URLFILE;
close ALLOWURLS;
close ALLOWDOMS;
# Set proper ownership
ERROR! CHANGE 15 TO THE UID OF squiduser ON YOUR MACHINE chown (15, -1, $allowurlfile, $allowdomfile);
chmod (0644, $allowurlfile, $allowdomfile);
print "Done.  Don't forget to reload Squid to make changes effective.\n";
exit 0;

sub isIP {
  my @array = shift;
  for (my $i = 1; $i <= 4; $i++) {
    # Search the first 4 parts of the array for alpha and hyphen
    # Return 0 if found or if the array is shorter than 4 parts
    @array[$i] =~ /[a-zA-Z\-]/ || (!defined(@array[$i])) && return 0;
  }
  # No alpha or hyphen found, and there are at least four parts? It
  # could be an IP address.
  return 1;
}