squid cache takes a break

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

squid cache takes a break

Vieri
Hi,

Sorry for the title, but I really don't know how to describe what just happened today. It's really odd.

I previously posted a few similar issues which were all fixed if I increased certain parameters (ulimits, children-{max,startup,idle}, TTL, etc.).

This time however, after several days trouble-free I got another show-stopper. The local squid cache stopped serving for almost half an hour. After that, it all started working again magically. I had the chance to log into the server with ssh and try a few things:

- In the cache log I could see these messages:
Starting new bllookup helpers...
helperOpenServers: Starting 10/80 'squid_url_lookup.pl' processes
WARNING: Cannot run '/opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl' process.
WARNING: Cannot run '/opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl' process.
WARNING: Cannot run '/opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl' process.

It doesn't say much as to why it "cannot run" the external program.

This is how the program is defined in squid.conf:
external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=40 children-idle=10 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]

Other than that, the log is pretty quiet.

The HTTP clients do not get served at all. They keep waiting for a reply.

# ps aux | grep squid
root      3043  0.0  0.0  84856  1728 ?        Ss   Aug31   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.http.conf -n squidhttp
squid     3046  0.0  0.0 128232 31052 ?        S    Aug31   0:35 (squid-1) -YC -f /etc/squid/squid.http.conf -n squidhttp
root      3538  0.0  0.0  86912  1740 ?        Ss   Aug31   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.https.conf -n squidhttps
squid     3540  0.0  0.1 134616 35608 ?        S    Aug31   1:09 (squid-1) -YC -f /etc/squid/squid.https.conf -n squidhttps
root      5690  0.0  0.0  87444  1736 ?        Ss   Aug31   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.conf -n squid
squid     5694  2.4  6.5 3769624 2136968 ?     S    Aug31 293:24 (squid-1) -YC -f /etc/squid/squid.conf -n squid
squid     5727  0.0  0.0   4008   524 ?        S    Aug31   0:01 (unlinkd)
squid     5728  0.0  0.0  13904  1576 ?        S    Aug31   2:09 diskd 5830660 5830661 5830662
squid    11927  0.0  0.0   4156   644 ?        S    Sep07   0:36 (logfile-daemon) /var/log/squid/access.log
squid    11937  1.7  0.0  41792  6232 ?        S    Sep07  31:08 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11939  0.1  0.0  41776  6288 ?        S    Sep07   3:09 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11940  0.0  0.0  41784  6356 ?        S    Sep07   0:28 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11941  0.0  0.0  41800  6308 ?        S    Sep07   0:07 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11942  0.0  0.0  41800  6308 ?        S    Sep07   0:02 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11943  0.0  0.0  41784  6320 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11944  0.0  0.0  41784  6068 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11945  0.0  0.0  41780  6372 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11946  0.0  0.0  41800  6852 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11947  0.0  0.0  41784  6756 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11948  0.0  0.0  41792  6784 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11949  0.0  0.0  41780  6672 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11950  0.0  0.0  41780  6660 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11951  0.0  0.0  41760  6308 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11952  0.0  0.0  41772  6336 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11953  0.0  0.0  41772  6284 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11954  0.0  0.0  41776  6956 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11955  0.0  0.0  41772  6524 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11956  0.0  0.0  41772  6664 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11958  0.0  0.0  41772  6284 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11959  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11960  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11968  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11969  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11970  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11971  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11972  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11973  0.0  0.0  40444  3528 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11974  0.0  0.0  40444  3364 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11977  0.0  0.0  40444  3528 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11979  0.0  0.0  40444  3528 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11980  0.0  0.0  40444  3528 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11981  0.0  0.0  40444  3532 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11982  0.0  0.0  40444  3528 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11983  0.0  0.0  40444  3524 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11984  0.0  0.0  40444  3528 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11986  0.0  0.0  40444  3364 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11987  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11988  0.0  0.0  40444  3336 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11989  0.0  0.0  40444  3368 ?        S    Sep07   0:00 (ssl_crtd) -s /var/lib/squid/ssl_db -M 16MB
squid    11990  0.0  0.0  27204  8000 ?        S    Sep07   0:00 /usr/bin/perl -w /usr/libexec/squid/ext_wbinfo_group_acl -K
squid    11991  0.0  0.0  27204  7744 ?        S    Sep07   0:00 /usr/bin/perl -w /usr/libexec/squid/ext_wbinfo_group_acl -K
squid    11992  0.0  0.0  27140  7816 ?        S    Sep07   0:00 /usr/bin/perl -w /usr/libexec/squid/ext_wbinfo_group_acl -K
squid    11993  0.0  0.0  27140  7824 ?        S    Sep07   0:00 /usr/bin/perl -w /usr/libexec/squid/ext_wbinfo_group_acl -K
squid    11994  0.0  0.0  27140  7756 ?        S    Sep07   0:00 /usr/bin/perl -w /usr/libexec/squid/ext_wbinfo_group_acl -K
squid    11995  0.1  0.0  69048 12904 ?        S    Sep07   3:32 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    11996  0.0  0.0  69052 12764 ?        S    Sep07   0:22 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    11997  0.0  0.0  69064 12796 ?        S    Sep07   0:07 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    11998  0.0  0.0  69056 13232 ?        S    Sep07   0:04 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    11999  0.0  0.0  69036 13108 ?        S    Sep07   0:03 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12000  0.0  0.0  69028 12944 ?        S    Sep07   0:02 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12001  0.0  0.0  69044 12824 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12003  0.0  0.0  68920 12716 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12004  0.0  0.0  68792 12596 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12005  0.0  0.0  68792 12408 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12006  0.0  0.0  68792 12712 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12007  0.0  0.0  68792 12716 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12008  0.0  0.0  68792 12532 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12009  0.0  0.0  68792 12664 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12010  0.0  0.0  68792 12504 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12011  0.0  0.0  68792 12660 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12012  0.0  0.0  68792 12556 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12013  0.0  0.0  68792 13816 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12014  0.0  0.0  68792 13740 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12015  0.0  0.0  68792 13756 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12016  0.0  0.0  68792 13820 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12017  0.0  0.0  68792 13812 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12018  0.0  0.0  68792 13820 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12019  0.0  0.0  68792 13792 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12020  0.0  0.0  32932  9372 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12021  0.0  0.0  32932  9396 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12022  0.0  0.0  32932  9364 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12023  0.0  0.0  32932  9400 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12025  0.0  0.0  32932  9396 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12026  0.0  0.0  32932  9396 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12027  0.0  0.0  32932  9372 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12028  0.0  0.0  32932  9368 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12029  0.0  0.0  32932  9372 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12032  0.0  0.0  32932  9372 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12033  0.0  0.0  32932  9368 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12034  0.0  0.0  32932  9392 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12036  0.0  0.0  32932  9404 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12037  0.0  0.0  32932  9400 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12038  0.0  0.0  32932  9368 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    12039  0.0  0.0  32932  9368 ?        S    Sep07   0:00 /usr/bin/perl -w -s /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl -tbl_name=shallalist_bl adv,aggressive,alcohol,anonvpn,automobile_bikes,automobile_boats,automobile_cars,automobile_planes,chat,costtraps,dating,drugs,dynamic,finance_insurance,finance_moneylending,finance_other,finance_realestate,finance_trading,fortunetelling,forum,gamble,hacking,hobby_cooking,hobby_games-misc,hobby_games-online,hobby_gardening,hobby_pets,homestyle,imagehosting,isp,jobsearch,military,models,movies,music,podcasts,politics,porn,radiotv,recreation_humor,recreation_martialarts,recreation_restaurants,recreation_sports,recreation_travel,recreation_wellness,redirector,religion,remotecontrol,ringtones,science_astronomy,science_chemistry,sex_education,sex_lingerie,shopping,socialnet,spyware,tracker,updatesites,urlshortener,violence,warez,weapons,webphone,webradio,webtv
squid    14108  0.0  0.0  24404  1392 ?        S    02:13   0:00 (negotiate_kerberos_auth) -s HTTP/[hidden email]
root     28492  0.0  0.0  86908  4220 ?        Ss   Sep05   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.owa.conf -n squidowa
squid    28495  0.0  0.0 103616 18364 ?        S    Sep05   0:12 (squid-1) -YC -f /etc/squid/squid.owa.conf -n squidowa
root     29120  0.0  0.0  86908  4220 ?        Ss   Sep05   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.owa2.conf -n squidowa2
squid    29123  0.2  0.6 293460 206776 ?       S    Sep05  10:30 (squid-1) -YC -f /etc/squid/squid.owa2.conf -n squidowa2
squid    30291  0.0  0.0  24404  2392 ?        S    Sep07   0:00 (negotiate_kerberos_auth) -s HTTP/[hidden email]
squid    30330  0.0  6.4 3769624 2127928 ?     S    13:42   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
squid    30866  0.0  6.4 3769624 2127928 ?     S    13:44   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid


top - 13:52:38 up 9 days,  6:19,  2 users,  load average: 2.04, 1.82, 1.65
Tasks: 405 total,   1 running, 404 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
%Cpu1  :  0.0 us,  0.3 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
%Cpu2  :  0.3 us,  0.3 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
%Cpu5  :  0.0 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu6  :  0.3 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu7  :  0.3 us,  0.0 sy,  0.0 ni, 95.7 id,  0.0 wa,  0.0 hi,  4.0 si,  0.0 st
KiB Mem : 32865056 total, 12324092 free, 16396808 used,  4144156 buff/cache
KiB Swap: 37036988 total, 35197252 free,  1839736 used. 15977208 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
29123 squid     20   0  293460 206776   8480 S   0.7  0.6  10:31.16 squid
427 root      20   0   20280   2912   2168 R   0.3  0.0   0:00.05 top
7902 named     20   0  821424 111644   5720 S   0.3  0.3  95:30.99 named
16766 suricata  20   0 2127364 358940   7356 S   0.3  1.1   3:12.03 Suricata-Main
1 root      20   0    4176   1512   1464 S   0.0  0.0   0:06.34 init

I then issued:
# strace -o squid.trace /usr/sbin/squid -YC -f /etc/squid/squid.conf -n squid
I didn't get much out of it, but I can post it if someone would like to see it.

I tried to stop Squid with my openrc script and failed.
So I issued the following manually several times:
# squid -n squid -k shutdown
The first result was:
# squid -n squid -k shutdown
2017/09/08 13:56:01.125| 24,8| SBuf.cc(124) ~SBuf: SBuf59 destructed
2017/09/08 13:56:01.125| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd74290 id=blob170 capacity=40 size=16
2017/09/08 13:56:01.125| 24,8| SBuf.cc(124) ~SBuf: SBuf48 destructed
2017/09/08 13:56:01.125| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd709a0 id=blob62 capacity=40 size=30
2017/09/08 13:56:01.125| 24,8| SBuf.cc(124) ~SBuf: SBuf44 destructed
2017/09/08 13:56:01.125| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xcdc6e0 id=blob51 capacity=40 size=12
2017/09/08 13:56:01.125| 28,3| Acl.cc(384) ~ACL: freeing ACL adaptation_access
2017/09/08 13:56:01.125| 45,9| cbdata.cc(321) cbdataInternalFree: 0xdf8298
2017/09/08 13:56:01.125| 45,9| cbdata.cc(338) cbdataInternalFree: Freeing 0xdf8298
2017/09/08 13:56:01.125| 24,8| SBuf.cc(124) ~SBuf: SBuf38 destructed
2017/09/08 13:56:01.125| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd60be0 id=blob37 capacity=16388 size=8
2017/09/08 13:56:01.125| 24,8| SBuf.cc(124) ~SBuf: SBuf37 destructed
2017/09/08 13:56:01.125| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd5cb90 id=blob36 capacity=16388 size=5
[...etc...]
Subsequent calls yielded:
squid: ERROR: Could not send signal 15 to process 5694: (3) No such process
2017/09/08 13:56:09.993| 24,8| SBuf.cc(124) ~SBuf: SBuf59 destructed
2017/09/08 13:56:09.993| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd74290 id=blob170 capacity=40 size=16
2017/09/08 13:56:09.993| 24,8| SBuf.cc(124) ~SBuf: SBuf48 destructed
2017/09/08 13:56:09.993| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd709a0 id=blob62 capacity=40 size=30
2017/09/08 13:56:09.993| 24,8| SBuf.cc(124) ~SBuf: SBuf44 destructed
2017/09/08 13:56:09.993| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xcdc6e0 id=blob51 capacity=40 size=12
2017/09/08 13:56:09.993| 28,3| Acl.cc(384) ~ACL: freeing ACL adaptation_access
2017/09/08 13:56:09.993| 45,9| cbdata.cc(321) cbdataInternalFree: 0xdf8298
2017/09/08 13:56:09.993| 45,9| cbdata.cc(338) cbdataInternalFree: Freeing 0xdf8298
2017/09/08 13:56:09.993| 24,8| SBuf.cc(124) ~SBuf: SBuf38 destructed
2017/09/08 13:56:09.993| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd60be0 id=blob37 capacity=16388 size=8
2017/09/08 13:56:09.993| 24,8| SBuf.cc(124) ~SBuf: SBuf37 destructed
2017/09/08 13:56:09.994| 24,9| MemBlob.cc(83) ~MemBlob: destructed, this=0xd5cb90 id=blob36 capacity=16388 size=5
[...etc...]
# ps aux | grep squid | grep '\-n'
squid      805  0.0  6.4 3769624 2127988 ?     S    13:53   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
squid     1452  0.0  6.4 3769624 2127988 ?     S    13:55   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
root      3043  0.0  0.0  84856  1728 ?        Ss   Aug31   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.http.conf -n squidhttp
squid     3046  0.0  0.0 128232 31052 ?        S    Aug31   0:35 (squid-1) -YC -f /etc/squid/squid.http.conf -n squidhttp
root      3538  0.0  0.0  86912  1740 ?        Ss   Aug31   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.https.conf -n squidhttps
squid     3540  0.0  0.1 134940 36244 ?        S    Aug31   1:09 (squid-1) -YC -f /etc/squid/squid.https.conf -n squidhttps
root     28492  0.0  0.0  86908  4220 ?        Ss   Sep05   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.owa.conf -n squidowa
squid    28495  0.0  0.0 103616 18364 ?        S    Sep05   0:12 (squid-1) -YC -f /etc/squid/squid.owa.conf -n squidowa
root     29120  0.0  0.0  86908  4220 ?        Ss   Sep05   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.owa2.conf -n squidowa2
squid    29123  0.2  0.6 293460 206776 ?       S    Sep05  10:31 (squid-1) -YC -f /etc/squid/squid.owa2.conf -n squidowa2
squid    30330  0.0  6.4 3769624 2127928 ?     S    13:42   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
squid    30866  0.0  6.4 3769624 2127928 ?     S    13:44   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
squid    31507  0.0  6.4 3769624 2127928 ?     S    13:47   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
squid    32055  0.0  6.4 3769624 2127928 ?     S    13:49   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
squid    32659  0.0  6.4 3769624 2127928 ?     S    13:51   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid

I changed the debug_options to ALL,9 and tried to restart or "reconfigure" Squid.
I had trouble with that and decided to:
- stop all squid instances (the other reverse proxies, eg. squidhttp, squidhttps, squidowa, squidowa2, were working fine)
- manually kill all squid processes related to the failing local cache (including ssl_crtd)
- made sure there were no more squid processes and that debug_options was ALL,9
- started the local squid cache only (only 1 daemon - no reverse proxies)

Surprisingly, it still didn't work...
Clients could not browse. The squid log only had this set of messages once in a while (very quiet log):

2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(180) ipcCreate: ipcCreate: prfd FD 94
2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(181) ipcCreate: ipcCreate: pwfd FD 94
2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(182) ipcCreate: ipcCreate: crfd FD 93
2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(183) ipcCreate: ipcCreate: cwfd FD 93
2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(196) ipcCreate: ipcCreate: FD 94 sockaddr [::1]:40905
2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(212) ipcCreate: ipcCreate: FD 93 sockaddr [::1]:50647
2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(222) ipcCreate: ipcCreate: FD 93 listening...
2017/09/08 14:18:10.322 kid1| 5,3| comm.cc(868) _comm_close: comm_close: start closing FD 93
2017/09/08 14:18:10.322 kid1| 5,3| comm.cc(540) commUnsetFdTimeout: Remove timeout for FD 93
2017/09/08 14:18:10.322 kid1| 5,5| comm.cc(721) commCallCloseHandlers: commCallCloseHandlers: FD 93
2017/09/08 14:18:10.322 kid1| 5,4| AsyncCall.cc(26) AsyncCall: The AsyncCall comm_close_complete constructed, this=0xeb7c10 [call123]
2017/09/08 14:18:10.322 kid1| 5,4| AsyncCall.cc(93) ScheduleCall: comm.cc(941) will call comm_close_complete(FD 93) [call123]
2017/09/08 14:18:10.322 kid1| 5,9| comm.cc(602) comm_connect_addr: connecting socket FD 94 to [::1]:50647 (want family: 10)
2017/09/08 14:18:10.322 kid1| 21,3| tools.cc(543) leave_suid: leave_suid: PID 8303 called
2017/09/08 14:18:10.323 kid1| 21,3| tools.cc(636) no_suid: no_suid: PID 8303 giving up root priveleges forever
2017/09/08 14:18:10.323 kid1| 54,3| ipc.cc(304) ipcCreate: ipcCreate: calling accept on FD 93
[...]
2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(644) comm_connect_addr: sock=98, addrinfo( flags=4, family=10, socktype=1, protocol=6, &addr=0xeb79a0, addrlen=28 )
2017/09/08 14:24:49.682 kid1| 5,9| comm.cc(645) comm_connect_addr: connect FD 98: (-1) (110) Connection timed out
2017/09/08 14:24:49.682 kid1| 14,9| comm.cc(646) comm_connect_addr: connecting to: [::1]:60557
2017/09/08 14:24:49.682 kid1| 5,3| comm.cc(868) _comm_close: comm_close: start closing FD 98
2017/09/08 14:24:49.682 kid1| 5,3| comm.cc(540) commUnsetFdTimeout: Remove timeout for FD 98
2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(721) commCallCloseHandlers: commCallCloseHandlers: FD 98
2017/09/08 14:24:49.682 kid1| 5,4| AsyncCall.cc(26) AsyncCall: The AsyncCall comm_close_complete constructed, this=0xeb7e90 [call128]
2017/09/08 14:24:49.682 kid1| 5,4| AsyncCall.cc(93) ScheduleCall: comm.cc(941) will call comm_close_complete(FD 98) [call128]
2017/09/08 14:24:49.682 kid1| WARNING: Cannot run '/usr/libexec/squid/ext_wbinfo_group_acl' process.
2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(347) comm_openex: comm_openex: Attempt open socket for: [::1]
2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(388) comm_openex: comm_openex: Opened socket local=[::1] remote=[::] FD 99 flags=1 : family=10, type=1, protocol=0
2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(420) comm_init_opened: local=[::1] remote=[::] FD 99 flags=1 is a new socket
2017/09/08 14:24:49.682 kid1| 51,3| fd.cc(198) fd_open: fd_open() FD 99 ext_wbinfo_group_acl
2017/09/08 14:24:49.682 kid1| 50,6| comm.cc(209) commBind: commBind: bind socket FD 99 to [::1]
2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(347) comm_openex: comm_openex: Attempt open socket for: [::1]
2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(388) comm_openex: comm_openex: Opened socket local=[::1] remote=[::] FD 100 flags=1 : family=10, type=1, protocol=0
2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(420) comm_init_opened: local=[::1] remote=[::] FD 100 flags=1 is a new socket
2017/09/08 14:24:49.682 kid1| 51,3| fd.cc(198) fd_open: fd_open() FD 100 ext_wbinfo_group_acl
2017/09/08 14:24:49.682 kid1| 50,6| comm.cc(209) commBind: commBind: bind socket FD 100 to [::1]
2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(180) ipcCreate: ipcCreate: prfd FD 100
2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(181) ipcCreate: ipcCreate: pwfd FD 100
2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(182) ipcCreate: ipcCreate: crfd FD 99
2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(183) ipcCreate: ipcCreate: cwfd FD 99
2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(196) ipcCreate: ipcCreate: FD 100 sockaddr [::1]:56367
2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(212) ipcCreate: ipcCreate: FD 99 sockaddr [::1]:35643
2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(222) ipcCreate: ipcCreate: FD 99 listening...
2017/09/08 14:24:49.683 kid1| 5,3| comm.cc(868) _comm_close: comm_close: start closing FD 99
2017/09/08 14:24:49.683 kid1| 5,3| comm.cc(540) commUnsetFdTimeout: Remove timeout for FD 99
2017/09/08 14:24:49.683 kid1| 5,5| comm.cc(721) commCallCloseHandlers: commCallCloseHandlers: FD 99
2017/09/08 14:24:49.683 kid1| 5,4| AsyncCall.cc(26) AsyncCall: The AsyncCall comm_close_complete constructed, this=0xeb7f10 [call129]
2017/09/08 14:24:49.683 kid1| 5,4| AsyncCall.cc(93) ScheduleCall: comm.cc(941) will call comm_close_complete(FD 99) [call129]
2017/09/08 14:24:49.683 kid1| 5,9| comm.cc(602) comm_connect_addr: connecting socket FD 100 to [::1]:35643 (want family: 10)
2017/09/08 14:24:49.683 kid1| 21,3| tools.cc(543) leave_suid: leave_suid: PID 10064 called
2017/09/08 14:24:49.683 kid1| 21,3| tools.cc(636) no_suid: no_suid: PID 10064 giving up root priveleges forever
2017/09/08 14:24:49.683 kid1| 54,3| ipc.cc(304) ipcCreate: ipcCreate: calling accept on FD 99

I also tried restarting Winbind, c-icap server, clamd, but nothing changed. They all seemed to be working properly anyway.

Finally, at some point (after half an hour) right after killing all squid processes yet again, I restarted the squid services one last time (reverted to debug_options ALL,1) before giving up and deciding to let users bypass the proxy. Well, it all started working again...

So now I'd really like to know what I can do the next time it stops working like this.
I'm considering setting debug_options to ALL,6 while I still can, and wait to see if it fails again. When it does, I might have more information.

Any suggestions?

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Amos Jeffries
Administrator
On 09/09/17 05:39, Vieri wrote:

> Hi,
>
> Sorry for the title, but I really don't know how to describe what just happened today. It's really odd.
>
> I previously posted a few similar issues which were all fixed if I increased certain parameters (ulimits, children-{max,startup,idle}, TTL, etc.).
>
> This time however, after several days trouble-free I got another show-stopper. The local squid cache stopped serving for almost half an hour. After that, it all started working again magically. I had the chance to log into the server with ssh and try a few things:
>
> - In the cache log I could see these messages:
> Starting new bllookup helpers...
> helperOpenServers: Starting 10/80 'squid_url_lookup.pl' processes
> WARNING: Cannot run '/opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl' process.
> WARNING: Cannot run '/opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl' process.
> WARNING: Cannot run '/opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl' process.
>
> It doesn't say much as to why it "cannot run" the external program.
>

Looking at the code that message only seems to get logged if there is a
TCP/UDP connection involved and it is having packet errors (many reasons
for that).



> This is how the program is defined in squid.conf:
> external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=40 children-idle=10 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]
>

... so far matching, external ACL use a private/localhost TCP connection
between each helper and Squid.

> Other than that, the log is pretty quiet.
>
> The HTTP clients do not get served at all. They keep waiting for a reply.
>

Squid is waiting for a reply from a helper about whether the request is
allowed or not.


>
> top - 13:52:38 up 9 days,  6:19,  2 users,  load average: 2.04, 1.82, 1.65
> Tasks: 405 total,   1 running, 404 sleeping,   0 stopped,   0 zombie
> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
> %Cpu1  :  0.0 us,  0.3 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
> %Cpu2  :  0.3 us,  0.3 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
> %Cpu5  :  0.0 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
> %Cpu6  :  0.3 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
> %Cpu7  :  0.3 us,  0.0 sy,  0.0 ni, 95.7 id,  0.0 wa,  0.0 hi,  4.0 si,  0.0 st
> KiB Mem : 32865056 total, 12324092 free, 16396808 used,  4144156 buff/cache
> KiB Swap: 37036988 total, 35197252 free,  1839736 used. 15977208 avail Mem
>
> PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 29123 squid     20   0  293460 206776   8480 S   0.7  0.6  10:31.16 squid

Starting each helper requires 206766 MB of RAM to be allocated in
swap/virtual. Copying 10x that data (+10 helpers per occurance) may take
a while.

We workaround that normally by using concurrency. Helpers that support
high levels of concurrency can handle a lot more than blocking /

Note that concurrency is just a way of pipelining requests to a helper,
it does not require multi-threading though helpers using MT are
naturally even better with concurrency.


> # ps aux | grep squid | grep '\-n'
> squid      805  0.0  6.4 3769624 2127988 ?     S    13:53   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
> squid     1452  0.0  6.4 3769624 2127988 ?     S    13:55   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
> root      3043  0.0  0.0  84856  1728 ?        Ss   Aug31   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.http.conf -n squidhttp
> squid     3046  0.0  0.0 128232 31052 ?        S    Aug31   0:35 (squid-1) -YC -f /etc/squid/squid.http.conf -n squidhttp
> root      3538  0.0  0.0  86912  1740 ?        Ss   Aug31   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.https.conf -n squidhttps
> squid     3540  0.0  0.1 134940 36244 ?        S    Aug31   1:09 (squid-1) -YC -f /etc/squid/squid.https.conf -n squidhttps
> root     28492  0.0  0.0  86908  4220 ?        Ss   Sep05   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.owa.conf -n squidowa
> squid    28495  0.0  0.0 103616 18364 ?        S    Sep05   0:12 (squid-1) -YC -f /etc/squid/squid.owa.conf -n squidowa
> root     29120  0.0  0.0  86908  4220 ?        Ss   Sep05   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.owa2.conf -n squidowa2
> squid    29123  0.2  0.6 293460 206776 ?       S    Sep05  10:31 (squid-1) -YC -f /etc/squid/squid.owa2.conf -n squidowa2
> squid    30330  0.0  6.4 3769624 2127928 ?     S    13:42   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
> squid    30866  0.0  6.4 3769624 2127928 ?     S    13:44   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
> squid    31507  0.0  6.4 3769624 2127928 ?     S    13:47   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
> squid    32055  0.0  6.4 3769624 2127928 ?     S    13:49   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
> squid    32659  0.0  6.4 3769624 2127928 ?     S    13:51   0:00 (squid-1) -YC -f /etc/squid/squid.conf -n squid
>
> I changed the debug_options to ALL,9 and tried to restart or "reconfigure" Squid.
> I had trouble with that and decided to:
> - stop all squid instances (the other reverse proxies, eg. squidhttp, squidhttps, squidowa, squidowa2, were working fine)
> - manually kill all squid processes related to the failing local cache (including ssl_crtd)
> - made sure there were no more squid processes and that debug_options was ALL,9
> - started the local squid cache only (only 1 daemon - no reverse proxies)
>
> Surprisingly, it still didn't work...
> Clients could not browse. The squid log only had this set of messages once in a while (very quiet log):
>
> 2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(180) ipcCreate: ipcCreate: prfd FD 94
> 2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(181) ipcCreate: ipcCreate: pwfd FD 94
> 2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(182) ipcCreate: ipcCreate: crfd FD 93
> 2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(183) ipcCreate: ipcCreate: cwfd FD 93
> 2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(196) ipcCreate: ipcCreate: FD 94 sockaddr [::1]:40905
> 2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(212) ipcCreate: ipcCreate: FD 93 sockaddr [::1]:50647
> 2017/09/08 14:18:10.322 kid1| 54,3| ipc.cc(222) ipcCreate: ipcCreate: FD 93 listening...
> 2017/09/08 14:18:10.322 kid1| 5,3| comm.cc(868) _comm_close: comm_close: start closing FD 93
> 2017/09/08 14:18:10.322 kid1| 5,3| comm.cc(540) commUnsetFdTimeout: Remove timeout for FD 93
> 2017/09/08 14:18:10.322 kid1| 5,5| comm.cc(721) commCallCloseHandlers: commCallCloseHandlers: FD 93
> 2017/09/08 14:18:10.322 kid1| 5,4| AsyncCall.cc(26) AsyncCall: The AsyncCall comm_close_complete constructed, this=0xeb7c10 [call123]
> 2017/09/08 14:18:10.322 kid1| 5,4| AsyncCall.cc(93) ScheduleCall: comm.cc(941) will call comm_close_complete(FD 93) [call123]
> 2017/09/08 14:18:10.322 kid1| 5,9| comm.cc(602) comm_connect_addr: connecting socket FD 94 to [::1]:50647 (want family: 10)
> 2017/09/08 14:18:10.322 kid1| 21,3| tools.cc(543) leave_suid: leave_suid: PID 8303 called
> 2017/09/08 14:18:10.323 kid1| 21,3| tools.cc(636) no_suid: no_suid: PID 8303 giving up root priveleges forever
> 2017/09/08 14:18:10.323 kid1| 54,3| ipc.cc(304) ipcCreate: ipcCreate: calling accept on FD 93

Whatever this helper being started on FD 93/94 was, it is successful.
We know it is a helper because the FD are used by ipc.cc (helper I/O
channels) for something, but that is all.


> [...]
> 2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(644) comm_connect_addr: sock=98, addrinfo( flags=4, family=10, socktype=1, protocol=6, &addr=0xeb79a0, addrlen=28 )
> 2017/09/08 14:24:49.682 kid1| 5,9| comm.cc(645) comm_connect_addr: connect FD 98: (-1) (110) Connection timed out
> 2017/09/08 14:24:49.682 kid1| 14,9| comm.cc(646) comm_connect_addr: connecting to: [::1]:60557
> 2017/09/08 14:24:49.682 kid1| 5,3| comm.cc(868) _comm_close: comm_close: start closing FD 98
> 2017/09/08 14:24:49.682 kid1| 5,3| comm.cc(540) commUnsetFdTimeout: Remove timeout for FD 98
> 2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(721) commCallCloseHandlers: commCallCloseHandlers: FD 98
> 2017/09/08 14:24:49.682 kid1| 5,4| AsyncCall.cc(26) AsyncCall: The AsyncCall comm_close_complete constructed, this=0xeb7e90 [call128]
> 2017/09/08 14:24:49.682 kid1| 5,4| AsyncCall.cc(93) ScheduleCall: comm.cc(941) will call comm_close_complete(FD 98) [call128]
> 2017/09/08 14:24:49.682 kid1| WARNING: Cannot run '/usr/libexec/squid/ext_wbinfo_group_acl' process.

The comm TCP timeout is for FD 98. It may or may not be related to the
wbinfo helper. The trace shown does not include any ipc.cc or fd.cc
output indicating what that FD is used for. see the below trace.


This trace is more complete:

> 2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(347) comm_openex: comm_openex: Attempt open socket for: [::1]
> 2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(388) comm_openex: comm_openex: Opened socket local=[::1] remote=[::] FD 99 flags=1 : family=10, type=1, protocol=0
> 2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(420) comm_init_opened: local=[::1] remote=[::] FD 99 flags=1 is a new socket
> 2017/09/08 14:24:49.682 kid1| 51,3| fd.cc(198) fd_open: fd_open() FD 99 ext_wbinfo_group_acl
> 2017/09/08 14:24:49.682 kid1| 50,6| comm.cc(209) commBind: commBind: bind socket FD 99 to [::1]
> 2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(347) comm_openex: comm_openex: Attempt open socket for: [::1]
> 2017/09/08 14:24:49.682 kid1| 50,3| comm.cc(388) comm_openex: comm_openex: Opened socket local=[::1] remote=[::] FD 100 flags=1 : family=10, type=1, protocol=0
> 2017/09/08 14:24:49.682 kid1| 5,5| comm.cc(420) comm_init_opened: local=[::1] remote=[::] FD 100 flags=1 is a new socket
> 2017/09/08 14:24:49.682 kid1| 51,3| fd.cc(198) fd_open: fd_open() FD 100 ext_wbinfo_group_acl
> 2017/09/08 14:24:49.682 kid1| 50,6| comm.cc(209) commBind: commBind: bind socket FD 100 to [::1]
> 2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(180) ipcCreate: ipcCreate: prfd FD 100
> 2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(181) ipcCreate: ipcCreate: pwfd FD 100
> 2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(182) ipcCreate: ipcCreate: crfd FD 99
> 2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(183) ipcCreate: ipcCreate: cwfd FD 99
> 2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(196) ipcCreate: ipcCreate: FD 100 sockaddr [::1]:56367
> 2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(212) ipcCreate: ipcCreate: FD 99 sockaddr [::1]:35643
> 2017/09/08 14:24:49.682 kid1| 54,3| ipc.cc(222) ipcCreate: ipcCreate: FD 99 listening...
> 2017/09/08 14:24:49.683 kid1| 5,3| comm.cc(868) _comm_close: comm_close: start closing FD 99
> 2017/09/08 14:24:49.683 kid1| 5,3| comm.cc(540) commUnsetFdTimeout: Remove timeout for FD 99
> 2017/09/08 14:24:49.683 kid1| 5,5| comm.cc(721) commCallCloseHandlers: commCallCloseHandlers: FD 99
> 2017/09/08 14:24:49.683 kid1| 5,4| AsyncCall.cc(26) AsyncCall: The AsyncCall comm_close_complete constructed, this=0xeb7f10 [call129]
> 2017/09/08 14:24:49.683 kid1| 5,4| AsyncCall.cc(93) ScheduleCall: comm.cc(941) will call comm_close_complete(FD 99) [call129]
> 2017/09/08 14:24:49.683 kid1| 5,9| comm.cc(602) comm_connect_addr: connecting socket FD 100 to [::1]:35643 (want family: 10)
> 2017/09/08 14:24:49.683 kid1| 21,3| tools.cc(543) leave_suid: leave_suid: PID 10064 called
> 2017/09/08 14:24:49.683 kid1| 21,3| tools.cc(636) no_suid: no_suid: PID 10064 giving up root priveleges forever
> 2017/09/08 14:24:49.683 kid1| 54,3| ipc.cc(304) ipcCreate: ipcCreate: calling accept on FD 99

This ext_wbinfo_group_acl helper on FD 99/100 was started successfully
AFAIK.


>
> I also tried restarting Winbind, c-icap server, clamd, but nothing changed. They all seemed to be working properly anyway.
>
> Finally, at some point (after half an hour) right after killing all squid processes yet again, I restarted the squid services one last time (reverted to debug_options ALL,1) before giving up and deciding to let users bypass the proxy. Well, it all started working again...
>
> So now I'd really like to know what I can do the next time it stops working like this.

Couple of workarounds.

a) start fewer helpers at a time.

b) reduce cache_mem.

c) add concurrency support to the helpers.

All of the above are aimed at reducing the amount of background
administrative work Squid has to do for helpers, or the amount of memory
consumed. And thus reducing the duration and effects of these pauses,
even if (a) causes them to be slightly more frequent.


> I'm considering setting debug_options to ALL,6 while I still can, and wait to see if it fails again. When it does, I might have more information.
>

The log lines above indicate sections "5,9 50,6 51,3 54,9" are logging
the helper startup details. You can reduce debug_options to those for
smaller logs to see if the TCP connect timeout is affecting helper startup.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Vieri

________________________________
From: Amos Jeffries <[hidden email]>
>
> a) start fewer helpers at a time.
>
> b) reduce cache_mem.
>
> c) add concurrency support to the helpers.


So I decreased the startup, idle, cache_mem values:

# egrep 'startup=|idle=' squid.conf
external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]
sslcrtd_children 128 startup=10 idle=3

# grep cache_mem squid.conf
cache_mem 64 MB

I also set debug_options to "ALL,1 5,9 50,6 51,3 54,9".

As far as concurrency is concerned, I never programmed a helper to support this feature.
If it were to be done in Perl, do you know by any chance if it would require Perl6 "promises" with await/start function calls?

Currently, my "bllookup" helper is a simple Perl5 script which reads from standard input like so:

while( <STDIN> )
{
[...lookup URI in a MySQL database and reply accordingly to Squid...]
}

It does not handle the channel-ID field.

I haven't found many Squid concurrency-enabled helper examples out there.

By the way, I see that Squid defaults to IPv6 for helper communications. I suppose it wouldn't make any real difference if I tried "ipv4" with "external_acl_type".
If I don't get any new info next time Squid slows down to a crawl, I'll probably try ipv4 just for kicks.

What I still don't get is how long it takes for Squid to get back to work after I do a complete restart (after thoroughly killing all related processes, including helpers). I'm talking more than 5 minutes here...
If I ever get the same issue again, I understand that I can:

- stop squid & eventually kill all apparently stalled processes

- modify squid.conf, and decrease or comment out all *startup= and *idle= options

- start squid

At this point, I should expect Squid to be up and serving within a reasonable amount of time, even if I may get squid warnings later on asking me to increase those values.
Or maybe not, because the Linux kernel might be busy cleaning up the swap space anyway?

One last thing. I'm running squid 3.5.26. I'll try to upgrade to 3.5.27 asap.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Amos Jeffries
Administrator
On 11/09/17 20:49, Vieri wrote:

>
> ________________________________
> From: Amos Jeffries
>>
>> a) start fewer helpers at a time.
>>
>> b) reduce cache_mem.
>>
>> c) add concurrency support to the helpers.
>
>
> So I decreased the startup, idle, cache_mem values:
>
> # egrep 'startup=|idle=' squid.conf
> external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]
> sslcrtd_children 128 startup=10 idle=3
>
> # grep cache_mem squid.conf
> cache_mem 64 MB
>
> I also set debug_options to "ALL,1 5,9 50,6 51,3 54,9".
>
> As far as concurrency is concerned, I never programmed a helper to support this feature.
> If it were to be done in Perl, do you know by any chance if it would require Perl6 "promises" with await/start function calls?
>

Don't know the answer to that one sorry. But ...

> Currently, my "bllookup" helper is a simple Perl5 script which reads from standard input like so:
>
> while( <STDIN> )
> {
> [...lookup URI in a MySQL database and reply accordingly to Squid...]
> }
>
> It does not handle the channel-ID field.

That is all it needs to do to begin with; parse off the numeric value
from the input line and send it back as prefix on the output line. The
helper does not need threading or anything particularly special for the
minimal support.

>
> I haven't found many Squid concurrency-enabled helper examples out there.
>

Nod.

> By the way, I see that Squid defaults to IPv6 for helper communications. I suppose it wouldn't make any real difference if I tried "ipv4" with "external_acl_type".

If the helper is running at all without it, then no.

> If I don't get any new info next time Squid slows down to a crawl, I'll probably try ipv4 just for kicks.
>
> What I still don't get is how long it takes for Squid to get back to work after I do a complete restart (after thoroughly killing all related processes, including helpers). I'm talking more than 5 minutes here...
> If I ever get the same issue again, I understand that I can:
>
> - stop squid & eventually kill all apparently stalled processes
>
> - modify squid.conf, and decrease or comment out all *startup= and *idle= options
>
> - start squid
>
> At this point, I should expect Squid to be up and serving within a reasonable amount of time, even if I may get squid warnings later on asking me to increase those values.
> Or maybe not, because the Linux kernel might be busy cleaning up the swap space anyway?

Something along those lines, though the disk cache related things can
sometimes take a surprisingly long time to complete. It's hard to tell
these possibilities apart without a trace of some kind to provide clues
about what is going on during the pause.

>
> One last thing. I'm running squid 3.5.26. I'll try to upgrade to 3.5.27 asap.
>

Nod.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Vieri

________________________________
From: Amos Jeffries <[hidden email]>
>
> That is all it needs to do to begin with; parse off the numeric value
> from the input line and send it back as prefix on the output line. The

> helper does not need threading or anything particularly special for the > minimal support.


I thought it had to be asynchronous.
The docs say "Only used with helpers capable of processing more than one query at a time."

Example:
Squid sends "1 URI1" (or whatever) to the helper.
It does not wait for an immediate response.
In fact, Squid can send "2 URI2" before getting the reply to ID 1, right?
In my case, the helper is synchronous, non-MT. I don't think it will improve the time responses per-se.

In any case, my helper won't be able to process more than one query AT A TIME.

I tried it anyway. So here's the relevant code:

while( <STDIN> )
{
s/^\s*//;
s/\s*$//;
my @squidin = split;
my $squidn = scalar @squidin;
undef $url;
undef $channelid;
if ( ($squidn == 2 ) && (defined $squidin[0]) && ($squidin[0] =~ /^\d+?$/) ) {
$channelid = $squidin[0];
$url = $squidin[1] if (defined $squidin[1]);
} else {
$url = $squidin[0] if (defined $squidin[0]);
}

[...]
logtofile("Channel-ID: ".$channelid."\n") if ((defined $channelid) && ($debug >= 1));

[...do DB lookups, reply accordingly...]

if (defined $channelid) {
print( $channelid." OK\n" );
logtofile( $channelid." OK\n" ) if ($debug >= 1);
} else {
print( "OK\n" );
logtofile( "OK\n" ) if ($debug >= 1);
}
[...similar responses for ERR messages...]
}

Here's the relevant squid.conf line:
external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]

How can I check in the Squid log that concurrency is "working"?

If the helper logs to a text file as in the trimmed code above, I notice that the channel ID is always 0. I get messages such as:
Channel-ID: 0
0 OK
Channel-ID: 0
0 ERR ...

Is this expected?

Despite this, I can see that the number of helper processes does not increase over time for now, and that HTTP/S client browsing is responsive enough.
# ps aux | grep -c squid_url_lookup.pl
11

One last thing. I'm using:
cache_dir diskd /var/cache/squid 100 16 256
I may want to try to comment out this directive for improved I/O performance.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Amos Jeffries
Administrator
On 12/09/17 22:59, Vieri wrote:

>
> ________________________________
> From: Amos Jeffries <[hidden email]>
>>
>> That is all it needs to do to begin with; parse off the numeric value
>> from the input line and send it back as prefix on the output line. The
>
>> helper does not need threading or anything particularly special for the > minimal support.
>
>
> I thought it had to be asynchronous.
> The docs say "Only used with helpers capable of processing more than one query at a time."
>
> Example:
> Squid sends "1 URI1" (or whatever) to the helper.
> It does not wait for an immediate response.
> In fact, Squid can send "2 URI2" before getting the reply to ID 1, right?

Yes.


> In my case, the helper is synchronous, non-MT. I don't think it will improve the time responses per-se.
>
> In any case, my helper won't be able to process more than one query AT A TIME.
>
> I tried it anyway. So here's the relevant code:
>
> while( <STDIN> )
> {
> s/^\s*//;
> s/\s*$//;
> my @squidin = split;
> my $squidn = scalar @squidin;
> undef $url;
> undef $channelid;
> if ( ($squidn == 2 ) && (defined $squidin[0]) && ($squidin[0] =~ /^\d+?$/) ) {
> $channelid = $squidin[0];
> $url = $squidin[1] if (defined $squidin[1]);
> } else {
> $url = $squidin[0] if (defined $squidin[0]);
> }
>
> [...]
> logtofile("Channel-ID: ".$channelid."\n") if ((defined $channelid) && ($debug >= 1));
>
> [...do DB lookups, reply accordingly...]
>
> if (defined $channelid) {
> print( $channelid." OK\n" );
> logtofile( $channelid." OK\n" ) if ($debug >= 1);
> } else {
> print( "OK\n" );
> logtofile( "OK\n" ) if ($debug >= 1);
> }
> [...similar responses for ERR messages...]
> }
>
> Here's the relevant squid.conf line:
> external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]
>
> How can I check in the Squid log that concurrency is "working"?

Section 82, level 2 or 4 should log the queries.

Better info is in the cachemgr/squidclient "external_acl" report. Each
helper is listed with its total and summary stats for each helper child.

>
> If the helper logs to a text file as in the trimmed code above, I notice that the channel ID is always 0. I get messages such as:
> Channel-ID: 0
> 0 OK
> Channel-ID: 0
> 0 ERR ...
>
> Is this expected?

Maybe.

If you make the helper pause a bit and throw a large number of different
URLs at Squid you should see it grow a bit higher than 0.

>
> Despite this, I can see that the number of helper processes does not increase over time for now, and that HTTP/S client browsing is responsive enough.
> # ps aux | grep -c squid_url_lookup.pl
> 11
>

Yay.

> One last thing. I'm using:
> cache_dir diskd /var/cache/squid 100 16 256
> I may want to try to comment out this directive for improved I/O performance.
>
> Thanks,
>
> Vieri

Cheers
Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Eliezer Croitoru
I just must add that if you understand how TCP works(which the helpers use to communicate with squid) then it makes sense that it is possible that...
The sender (ie squid) sent 100 lines but the client software yet to process them since it's in the OS or other software\hardware related buffer.

For me it was hard to understand at first since it's an STDIN\STDOUT interface so it would block after every write but it's not...

There is a possibility that if the helper can process every incoming request with threading or other method of concurrency then the performance of the helper and by that squid will be better but if only using the basic buffer works fine for you then great.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
Sent: Tuesday, September 12, 2017 16:08
To: [hidden email]
Subject: Re: [squid-users] squid cache takes a break

On 12/09/17 22:59, Vieri wrote:

>
> ________________________________
> From: Amos Jeffries <[hidden email]>
>>
>> That is all it needs to do to begin with; parse off the numeric value
>> from the input line and send it back as prefix on the output line. The
>
>> helper does not need threading or anything particularly special for the > minimal support.
>
>
> I thought it had to be asynchronous.
> The docs say "Only used with helpers capable of processing more than one query at a time."
>
> Example:
> Squid sends "1 URI1" (or whatever) to the helper.
> It does not wait for an immediate response.
> In fact, Squid can send "2 URI2" before getting the reply to ID 1, right?

Yes.


> In my case, the helper is synchronous, non-MT. I don't think it will improve the time responses per-se.
>
> In any case, my helper won't be able to process more than one query AT A TIME.
>
> I tried it anyway. So here's the relevant code:
>
> while( <STDIN> )
> {
> s/^\s*//;
> s/\s*$//;
> my @squidin = split;
> my $squidn = scalar @squidin;
> undef $url;
> undef $channelid;
> if ( ($squidn == 2 ) && (defined $squidin[0]) && ($squidin[0] =~ /^\d+?$/) ) {
> $channelid = $squidin[0];
> $url = $squidin[1] if (defined $squidin[1]);
> } else {
> $url = $squidin[0] if (defined $squidin[0]);
> }
>
> [...]
> logtofile("Channel-ID: ".$channelid."\n") if ((defined $channelid) && ($debug >= 1));
>
> [...do DB lookups, reply accordingly...]
>
> if (defined $channelid) {
> print( $channelid." OK\n" );
> logtofile( $channelid." OK\n" ) if ($debug >= 1);
> } else {
> print( "OK\n" );
> logtofile( "OK\n" ) if ($debug >= 1);
> }
> [...similar responses for ERR messages...]
> }
>
> Here's the relevant squid.conf line:
> external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]
>
> How can I check in the Squid log that concurrency is "working"?

Section 82, level 2 or 4 should log the queries.

Better info is in the cachemgr/squidclient "external_acl" report. Each
helper is listed with its total and summary stats for each helper child.

>
> If the helper logs to a text file as in the trimmed code above, I notice that the channel ID is always 0. I get messages such as:
> Channel-ID: 0
> 0 OK
> Channel-ID: 0
> 0 ERR ...
>
> Is this expected?

Maybe.

If you make the helper pause a bit and throw a large number of different
URLs at Squid you should see it grow a bit higher than 0.

>
> Despite this, I can see that the number of helper processes does not increase over time for now, and that HTTP/S client browsing is responsive enough.
> # ps aux | grep -c squid_url_lookup.pl
> 11
>

Yay.

> One last thing. I'm using:
> cache_dir diskd /var/cache/squid 100 16 256
> I may want to try to comment out this directive for improved I/O performance.
>
> Thanks,
>
> Vieri

Cheers
Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Yuri Voinov
It is just enough not to reinvent the wheel. What needs op - already
exists and is called ufdbguard. And it's works perfectly with shallalist :)


13.09.2017 2:51, Eliezer Croitoru пишет:

> I just must add that if you understand how TCP works(which the helpers use to communicate with squid) then it makes sense that it is possible that...
> The sender (ie squid) sent 100 lines but the client software yet to process them since it's in the OS or other software\hardware related buffer.
>
> For me it was hard to understand at first since it's an STDIN\STDOUT interface so it would block after every write but it's not...
>
> There is a possibility that if the helper can process every incoming request with threading or other method of concurrency then the performance of the helper and by that squid will be better but if only using the basic buffer works fine for you then great.
>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: [hidden email]
>
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
> Sent: Tuesday, September 12, 2017 16:08
> To: [hidden email]
> Subject: Re: [squid-users] squid cache takes a break
>
> On 12/09/17 22:59, Vieri wrote:
>> ________________________________
>> From: Amos Jeffries <[hidden email]>
>>> That is all it needs to do to begin with; parse off the numeric value
>>> from the input line and send it back as prefix on the output line. The
>>> helper does not need threading or anything particularly special for the > minimal support.
>>
>> I thought it had to be asynchronous.
>> The docs say "Only used with helpers capable of processing more than one query at a time."
>>
>> Example:
>> Squid sends "1 URI1" (or whatever) to the helper.
>> It does not wait for an immediate response.
>> In fact, Squid can send "2 URI2" before getting the reply to ID 1, right?
> Yes.
>
>
>> In my case, the helper is synchronous, non-MT. I don't think it will improve the time responses per-se.
>>
>> In any case, my helper won't be able to process more than one query AT A TIME.
>>
>> I tried it anyway. So here's the relevant code:
>>
>> while( <STDIN> )
>> {
>> s/^\s*//;
>> s/\s*$//;
>> my @squidin = split;
>> my $squidn = scalar @squidin;
>> undef $url;
>> undef $channelid;
>> if ( ($squidn == 2 ) && (defined $squidin[0]) && ($squidin[0] =~ /^\d+?$/) ) {
>> $channelid = $squidin[0];
>> $url = $squidin[1] if (defined $squidin[1]);
>> } else {
>> $url = $squidin[0] if (defined $squidin[0]);
>> }
>>
>> [...]
>> logtofile("Channel-ID: ".$channelid."\n") if ((defined $channelid) && ($debug >= 1));
>>
>> [...do DB lookups, reply accordingly...]
>>
>> if (defined $channelid) {
>> print( $channelid." OK\n" );
>> logtofile( $channelid." OK\n" ) if ($debug >= 1);
>> } else {
>> print( "OK\n" );
>> logtofile( "OK\n" ) if ($debug >= 1);
>> }
>> [...similar responses for ERR messages...]
>> }
>>
>> Here's the relevant squid.conf line:
>> external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]
>>
>> How can I check in the Squid log that concurrency is "working"?
> Section 82, level 2 or 4 should log the queries.
>
> Better info is in the cachemgr/squidclient "external_acl" report. Each
> helper is listed with its total and summary stats for each helper child.
>
>> If the helper logs to a text file as in the trimmed code above, I notice that the channel ID is always 0. I get messages such as:
>> Channel-ID: 0
>> 0 OK
>> Channel-ID: 0
>> 0 ERR ...
>>
>> Is this expected?
> Maybe.
>
> If you make the helper pause a bit and throw a large number of different
> URLs at Squid you should see it grow a bit higher than 0.
>
>> Despite this, I can see that the number of helper processes does not increase over time for now, and that HTTP/S client browsing is responsive enough.
>> # ps aux | grep -c squid_url_lookup.pl
>> 11
>>
> Yay.
>
>> One last thing. I'm using:
>> cache_dir diskd /var/cache/squid 100 16 256
>> I may want to try to comment out this directive for improved I/O performance.
>>
>> Thanks,
>>
>> Vieri
> Cheers
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Eliezer Croitoru
Well, the ready to use products are not always what you need or want.
Even squid is not good enough for many scenarios...

If it works with shallalist it's nice but not the real deal for most cases.
Vieri might or might not clarify his scenario, but the issue here is not other then working with squid and a helper.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Yuri
Sent: Tuesday, September 12, 2017 23:54
To: [hidden email]
Subject: Re: [squid-users] squid cache takes a break

It is just enough not to reinvent the wheel. What needs op - already
exists and is called ufdbguard. And it's works perfectly with shallalist :)


13.09.2017 2:51, Eliezer Croitoru пишет:

> I just must add that if you understand how TCP works(which the helpers use to communicate with squid) then it makes sense that it is possible that...
> The sender (ie squid) sent 100 lines but the client software yet to process them since it's in the OS or other software\hardware related buffer.
>
> For me it was hard to understand at first since it's an STDIN\STDOUT interface so it would block after every write but it's not...
>
> There is a possibility that if the helper can process every incoming request with threading or other method of concurrency then the performance of the helper and by that squid will be better but if only using the basic buffer works fine for you then great.
>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: [hidden email]
>
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
> Sent: Tuesday, September 12, 2017 16:08
> To: [hidden email]
> Subject: Re: [squid-users] squid cache takes a break
>
> On 12/09/17 22:59, Vieri wrote:
>> ________________________________
>> From: Amos Jeffries <[hidden email]>
>>> That is all it needs to do to begin with; parse off the numeric value
>>> from the input line and send it back as prefix on the output line. The
>>> helper does not need threading or anything particularly special for the > minimal support.
>>
>> I thought it had to be asynchronous.
>> The docs say "Only used with helpers capable of processing more than one query at a time."
>>
>> Example:
>> Squid sends "1 URI1" (or whatever) to the helper.
>> It does not wait for an immediate response.
>> In fact, Squid can send "2 URI2" before getting the reply to ID 1, right?
> Yes.
>
>
>> In my case, the helper is synchronous, non-MT. I don't think it will improve the time responses per-se.
>>
>> In any case, my helper won't be able to process more than one query AT A TIME.
>>
>> I tried it anyway. So here's the relevant code:
>>
>> while( <STDIN> )
>> {
>> s/^\s*//;
>> s/\s*$//;
>> my @squidin = split;
>> my $squidn = scalar @squidin;
>> undef $url;
>> undef $channelid;
>> if ( ($squidn == 2 ) && (defined $squidin[0]) && ($squidin[0] =~ /^\d+?$/) ) {
>> $channelid = $squidin[0];
>> $url = $squidin[1] if (defined $squidin[1]);
>> } else {
>> $url = $squidin[0] if (defined $squidin[0]);
>> }
>>
>> [...]
>> logtofile("Channel-ID: ".$channelid."\n") if ((defined $channelid) && ($debug >= 1));
>>
>> [...do DB lookups, reply accordingly...]
>>
>> if (defined $channelid) {
>> print( $channelid." OK\n" );
>> logtofile( $channelid." OK\n" ) if ($debug >= 1);
>> } else {
>> print( "OK\n" );
>> logtofile( "OK\n" ) if ($debug >= 1);
>> }
>> [...similar responses for ERR messages...]
>> }
>>
>> Here's the relevant squid.conf line:
>> external_acl_type bllookup ttl=86400 negative_ttl=86400 children-max=80 children-startup=10 children-idle=3 concurrency=8 %URI /opt/custom/scripts/run/scripts/firewall/squid_url_lookup.pl [...]
>>
>> How can I check in the Squid log that concurrency is "working"?
> Section 82, level 2 or 4 should log the queries.
>
> Better info is in the cachemgr/squidclient "external_acl" report. Each
> helper is listed with its total and summary stats for each helper child.
>
>> If the helper logs to a text file as in the trimmed code above, I notice that the channel ID is always 0. I get messages such as:
>> Channel-ID: 0
>> 0 OK
>> Channel-ID: 0
>> 0 ERR ...
>>
>> Is this expected?
> Maybe.
>
> If you make the helper pause a bit and throw a large number of different
> URLs at Squid you should see it grow a bit higher than 0.
>
>> Despite this, I can see that the number of helper processes does not increase over time for now, and that HTTP/S client browsing is responsive enough.
>> # ps aux | grep -c squid_url_lookup.pl
>> 11
>>
> Yay.
>
>> One last thing. I'm using:
>> cache_dir diskd /var/cache/squid 100 16 256
>> I may want to try to comment out this directive for improved I/O performance.
>>
>> Thanks,
>>
>> Vieri
> Cheers
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid cache takes a break

Vieri
Thanks for the suggestion. I'm sure ufdbguard works great even though it's not maintained/updated on my distro (Gentoo).

I use ready-made helpers/redirectors like squidGuard on other systems.
However, on this system I wanted to avoid depending on extra software. I also wanted to make my own helper so I could then combine Squid ACLs and do things such as:
- block access to blacklisted URLs on a Squid setup with transparent ssl_bump (no proxy auth)

- show custom deny web page with optional auth form to bypass this restriction
- authenticate via LDAP using a custom web form, and insert the user's client IP address into a database with a timeout
- auto-redirect the request to the restricted web site so the user on a particular client host can access the site for a given time frame

- use a squid ACL to look up the user's host IP address in the DB, and decide to allow or not


In any case, I've been experiencing lots of issues with Squid during the past 2 weeks. I can finally say that I've fine-tuned my setup thanks to the great help I found on this ML. One of the things that were nagging me was the helper part. Knowing how helpers work, and how they can be optimized on heavy traffic loads is "a good thing". For starters, I did not know how to use the concurrency option and how the use of it could benefit overall performance.


Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users