Squid and c-icap's srv_url_check module

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid and c-icap's srv_url_check module

Amiq Nahas
Hi Guys,

I am trying to use the srv_url_check module to block websites.
I have configured squid with proxy authentication and followed this
wiki: https://sourceforge.net/p/c-icap/wiki/UrlCheckProfiles/
to configure c-icap and srv_url_check. Now, I am having trouble
configuring squid.conf. Below I have shared my configuration of squid.

I suspect that the last svcBlocker line in squid.conf, is the faulty
one, among other possible faults.
Please point out what is it that I am doing wrong.

/etc/squid/squid.conf
-----
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/squid_passwd
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED

acl SSL_ports port 443
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443        # https
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

http_access deny !authenticated
http_access allow localhost
http_access deny all

http_port 3128
coredump_dir /var/spool/squid
refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0
refresh_pattern (Release|Packages(.gz)*)$      0       20%
2880refresh_pattern .        0    20%    4320

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Authenticated-User
icap_client_username_encode on
icap_preview_enable on
icap_preview_size 1024

icap_service svcBlocker reqmod_precache
icap://127.0.0.1:1344/srv_url_check bypass=off
-----



Below are c-icap related files in case they are required too.


/usr/local/etc/c-icap.conf
-----
PidFile /var/run/c-icap/c-icap.pid
CommandsSocket /var/run/c-icap/c-icap.ctl
Timeout 300
MaxKeepAliveRequests 100
KeepAliveTimeout 600
StartServers 3
MaxServers 10
MinSpareThreads     10
MaxSpareThreads     20
ThreadsPerChild     10
MaxRequestsPerChild  0
Port 1344
[ciphers=ciph1:ciph2...] [tls_options=[!]Opt1|[!]Opt2|...]
ServerAdmin [hidden email]
ServerName YourServerName
TmpDir /var/tmp
MaxMemObject 131072
DebugLevel 1
Pipelining on
SupportBuggyClients off
ModulesDir /usr/local/lib/c_icap
ServicesDir /usr/local/lib/c_icap
TemplateDir /usr/local/share/c_icap/templates/
TemplateDefaultLanguage en
LoadMagicFile /usr/local/etc/c-icap.magic

RemoteProxyUsers on
RemoteProxyUserHeader X-Authenticated-User
RemoteProxyUserHeaderEncoded on
GroupSourceByGroup hash:/usr/local/etc/c-icap-groups.txt
acl all src 0.0.0.0/0.0.0.0

LogFormat myFormat "%a %la %lp %>a %<A %ts %tl %tg %tr %>hi %>ho %huo
%hu %<hi %<ho %Hs %Hso %iu %im %is %>ih %<ih %ipl %Ih %Oh %Ib %Ob %I
%O %bph %un %Sl %Sa"

ServerLog /usr/local/var/log/server.log
AccessLog /usr/local/var/log/access.log myFormat all

Service echo srv_echo.so
Include srv_url_check.conf
-----

/usr/local/etc/c-icap-groups.txt
-----
Users: user1
-----

/usr/local/etc/urls.txt
-----
www.facebook.com/
-----

/usr/local/etc/srv_url_check.conf
-----
%{url_check:action_cat}Sa] [Action: %{url_check:action}Sa]"
Service url_check_module srv_url_check.so
url_check.LookupTableDB urls url hash:/usr/local/etc/urls.txt
url_check.Profile social_media block urls
url_check.Profile default pass ALL
acl facebook group Users
url_check.ProfileAccess social_media facebook
-----

Thanks
Amiq
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amos Jeffries
Administrator
On 16/06/20 1:55 am, Amiq Nahas wrote:
> Hi Guys,
>
> I am trying to use the srv_url_check module to block websites.
> I have configured squid with proxy authentication and followed this
> wiki: https://sourceforge.net/p/c-icap/wiki/UrlCheckProfiles/
> to configure c-icap and srv_url_check. Now, I am having trouble
> configuring squid.conf. Below I have shared my configuration of squid.
>


"I am having trouble" is not sufficient details to investigate a problem.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amiq Nahas
On Wed, Jun 17, 2020 at 10:23 AM Amos Jeffries <[hidden email]> wrote:

>
> On 16/06/20 1:55 am, Amiq Nahas wrote:
> > Hi Guys,
> >
> > I am trying to use the srv_url_check module to block websites.
> > I have configured squid with proxy authentication and followed this
> > wiki: https://sourceforge.net/p/c-icap/wiki/UrlCheckProfiles/
> > to configure c-icap and srv_url_check. Now, I am having trouble
> > configuring squid.conf. Below I have shared my configuration of squid.
> >
>
>
> "I am having trouble" is not sufficient details to investigate a problem.

Sorry my bad.
So After doing all the above configuration. The browser does not block
the websites in the blocklist.
Browser does prompt for user credentials just like the squid.conf is
configured to do, but it is not blocking websites.

However, when I execute c-icap-client from command line it blocks the
blocklisted websites.
To check with c-icap-client I have used the below command:
`c-icap-client -s url_check -x "X-Authenticated-User: dXNlcjE=" -req
"https://www.facebook.com/" -v`

So if the blocking from c-icap client is working but blocking from
browser is not working then
something must be wrong with my squid.conf or the configuration part
responsible for making c-icap and squid work together, right?

So what could be it? Please let me know if any other piece of
information is required, I am not sure what else could be of use.

Thanks
Amiq
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amos Jeffries
Administrator
On 18/06/20 1:32 am, Amiq Nahas wrote:

> On Wed, Jun 17, 2020 at 10:23 AM Amos Jeffries wrote:
>>
>> On 16/06/20 1:55 am, Amiq Nahas wrote:
>>> Hi Guys,
>>>
>>> I am trying to use the srv_url_check module to block websites.
>>> I have configured squid with proxy authentication and followed this
>>> wiki: https://sourceforge.net/p/c-icap/wiki/UrlCheckProfiles/
>>> to configure c-icap and srv_url_check. Now, I am having trouble
>>> configuring squid.conf. Below I have shared my configuration of squid.
>>>
>>
>>
>> "I am having trouble" is not sufficient details to investigate a problem.
>
> Sorry my bad.
> So After doing all the above configuration. The browser does not block
> the websites in the blocklist.

If the Browser is doing blocking the request would never reach Squid.


> Browser does prompt for user credentials just like the squid.conf is
> configured to do, but it is not blocking websites.
>
> However, when I execute c-icap-client from command line it blocks the
> blocklisted websites.
> To check with c-icap-client I have used the below command:
> `c-icap-client -s url_check -x "X-Authenticated-User: dXNlcjE=" -req
> "https://www.facebook.com/" -v`
>
> So if the blocking from c-icap client is working but blocking from
> browser is not working then

Please define "blocking from c-icap"

Please define "blocking from browser"


> something must be wrong with my squid.conf or the configuration part
> responsible for making c-icap and squid work together, right?

Unknown.

>
> So what could be it? Please let me know if any other piece of
> information is required, I am not sure what else could be of use.
>

Log trace(s) from a transaction that you think is failing to start with.
Squid access.log and c-icap log. Maybe a Squid cache.log trace with
debug_options 11,2 or ALL,2


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amiq Nahas
On Wed, Jun 17, 2020 at 8:28 PM Amos Jeffries <[hidden email]> wrote:

>
> > Browser does prompt for user credentials just like the squid.conf is
> > configured to do, but it is not blocking websites.
> >
> > However, when I execute c-icap-client from command line it blocks the
> > blocklisted websites.
> > To check with c-icap-client I have used the below command:
> > `c-icap-client -s url_check -x "X-Authenticated-User: dXNlcjE=" -req
> > "https://www.facebook.com/" -v`
> >
> > So if the blocking from c-icap client is working but blocking from
> > browser is not working then
>
> Please define "blocking from c-icap"
> Please define "blocking from browser"

Squid is being used as a manual proxy at localhost:3128 in firefox,
and as a way of authenticating a user (proxy authentication), in my
case I am the sole user.
Squid is configured and is being used by the browser on the same machine.

When I said blocking from browser is not happening,
I meant that when I use the browser to access a website it is not blocked,
where as, when c-icap-client is used to make a request using the command
`c-icap-client -s url_check -x "X-Authenticated-User: dXNlcjE=" -req
"https://www.facebook.com/" -v`
I get the response saying the website is blocked.

ls> > So what could be it? Please let me know if any other piece of
> > information is required, I am not sure what else could be of use.
> >
>
> Log trace(s) from a transaction that you think is failing to start with.
> Squid access.log and c-icap log. Maybe a Squid cache.log trace with
> debug_options 11,2 or ALL,2

Squid logs access.log and cache.log do contain the logs of requested
websites requested via browser.
I enabled debug_options ALL,2 and here are the logs:
access.log: https://paste.debian.net/1152807/
cache.log: https://justpaste.it/45fca
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amos Jeffries
Administrator
On 19/06/20 8:46 pm, Amiq Nahas wrote:

> On Wed, Jun 17, 2020 at 8:28 PM Amos Jeffries wrote:
>>
>>> Browser does prompt for user credentials just like the squid.conf is
>>> configured to do, but it is not blocking websites.
>>>
>>> However, when I execute c-icap-client from command line it blocks the
>>> blocklisted websites.
>>> To check with c-icap-client I have used the below command:
>>> `c-icap-client -s url_check -x "X-Authenticated-User: dXNlcjE=" -req
>>> "https://www.facebook.com/" -v`
>>>
>>> So if the blocking from c-icap client is working but blocking from
>>> browser is not working then
>>
>> Please define "blocking from c-icap"
>> Please define "blocking from browser"
>
> Squid is being used as a manual proxy at localhost:3128 in firefox,
> and as a way of authenticating a user (proxy authentication), in my
> case I am the sole user.
> Squid is configured and is being used by the browser on the same machine.
>
> When I said blocking from browser is not happening,
> I meant that when I use the browser to access a website it is not blocked,
> where as, when c-icap-client is used to make a request using the command
> `c-icap-client -s url_check -x "X-Authenticated-User: dXNlcjE=" -req
> "https://www.facebook.com/" -v`
> I get the response saying the website is blocked.
>
> ls> > So what could be it? Please let me know if any other piece of
>>> information is required, I am not sure what else could be of use.
>>>
>>
>> Log trace(s) from a transaction that you think is failing to start with.
>> Squid access.log and c-icap log. Maybe a Squid cache.log trace with
>> debug_options 11,2 or ALL,2
>
> Squid logs access.log and cache.log do contain the logs of requested
> websites requested via browser.
> I enabled debug_options ALL,2 and here are the logs:


Looks like traffic is fine and Squid operational, but no sign of any
ICAP activity. I think try adding this to your config:

  adaptation_access svcBlocker allow all

Its supposed to be the default action, but just to be sure add it
explicitly.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amiq Nahas
> Looks like traffic is fine and Squid operational, but no sign of any
> ICAP activity. I think try adding this to your config:
>
>   adaptation_access svcBlocker allow all
>
> Its supposed to be the default action, but just to be sure add it
> explicitly.

I added the above line and browser could not open any page, this was
the notice shown: https://ibb.co/HVQYD2c
cache.log: https://justpaste.it/38eyl
access.log: https://paste.debian.net/1152817/
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amos Jeffries
Administrator
On 19/06/20 10:07 pm, Amiq Nahas wrote:

>> Looks like traffic is fine and Squid operational, but no sign of any
>> ICAP activity. I think try adding this to your config:
>>
>>   adaptation_access svcBlocker allow all
>>
>> Its supposed to be the default action, but just to be sure add it
>> explicitly.
>
> I added the above line and browser could not open any page, this was
> the notice shown: https://ibb.co/HVQYD2c
> cache.log: https://justpaste.it/38eyl
> access.log: https://paste.debian.net/1152817/
>

The problem is this:

"
2020/06/19 15:11:09.998 kid1| 93,2| Xaction.cc(272)
dieOnConnectionFailure: Adaptation::Icap::OptXact failed to connect to
icap://127.0.0.1:1344/srv_url_check

2020/06/19 15:11:09.998 kid1| essential ICAP service is down after an
options fetch failure: icap://127.0.0.1:1344/srv_url_check [down,!opt]
"

Next step is to debug why the OPTIONS request to the ICAP service causes
it to break.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amiq Nahas
On Fri, Jun 19, 2020 at 4:20 PM Amos Jeffries <[hidden email]> wrote:

>
> On 19/06/20 10:07 pm, Amiq Nahas wrote:
> >> Looks like traffic is fine and Squid operational, but no sign of any
> >> ICAP activity. I think try adding this to your config:
> >>
> >>   adaptation_access svcBlocker allow all
> >>
> >> Its supposed to be the default action, but just to be sure add it
> >> explicitly.
> >
> > I added the above line and browser could not open any page, this was
> > the notice shown: https://ibb.co/HVQYD2c
> > cache.log: https://justpaste.it/38eyl
> > access.log: https://paste.debian.net/1152817/
> >
>
> The problem is this:
>
> "
> 2020/06/19 15:11:09.998 kid1| 93,2| Xaction.cc(272)
> dieOnConnectionFailure: Adaptation::Icap::OptXact failed to connect to
> icap://127.0.0.1:1344/srv_url_check
>
> 2020/06/19 15:11:09.998 kid1| essential ICAP service is down after an
> options fetch failure: icap://127.0.0.1:1344/srv_url_check [down,!opt]
> "
>
> Next step is to debug why the OPTIONS request to the ICAP service causes
> it to break.

Problem solved. The name of service was wrong.
Service name should be icap://127.0.0.1:1344/url_check instead of
icap://127.0.0.1:1344/srv_url_check.
Thanks for helping in narrowing the problem.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid and c-icap's srv_url_check module

Amos Jeffries
Administrator
On 22/06/20 7:37 pm, Amiq Nahas wrote:
>
> Problem solved. The name of service was wrong.
> Service name should be icap://127.0.0.1:1344/url_check instead of
> icap://127.0.0.1:1344/srv_url_check.
> Thanks for helping in narrowing the problem.
>

Welcome, thats what we are here for. Sorry for not being familiar enough
with c-icap to pick up that much faster.


FWIW; I have sent a note to Christos the c-icap author about the wiki
docs needing to specify this detail clearly on the page you were working
from.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users