multiple log_file_daemon settings in squid.conf

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

multiple log_file_daemon settings in squid.conf

Rafael Akchurin

Greetings all,

 

I am trying to find the best (easiest, least interfering) solution for the following problem.

 

Our custom ICAP server writes various information about ICAP transaction (user name, policy ip, detection module, timings, words triggered detection, etc) into the record database. This happens at the time when ICAP transaction ends. Based on that information we build extensive reports of web filtering activity.

 

As known, end of the ICAP transaction does not mean the end of the original transaction in Squid – e.g. after scanning CONNECT request and allowing it to proceed, the actual data transfer from the Squid’s perspective may end much much later. The piece of information that would be very interesting for reporting module is how many bytes were pumped through that connection. This information is only available after the original Squid transaction ends.

 

So somehow in the record database we must correlate the ICAP transaction(s) with original Squid transaction.

 

Question 1:

Is there any unique transaction ID in the Squid’s inner workings that I can see in the ICAP server, by passing it as additional X-* ICAP header?

I see some references to so called “master transaction” in the docs but could not find any log format like identifier that can be used for ICAP header value.

 

Question 2:

If there is no such transaction ID, I can use ICAP header to pass the ICAP specific transaction ID back to Squid *and* I can get that ID written to Squid’s access log as "X-WebSafety-IID=%{X-WebSafety-IID}adapt::<last_h". Is it a valid approach?

 

Question 3:

If q1 or q2 is answered positively, I still need to somehow get the data from squid’s access log the ICAP record database. Currently the idea is to have the custom logfile_daemon setting that would fork original log_file_daemon to have log entries written to access_log *and* parse out the ICAP ID *and* update the corresponding ICAP record in the database with transferred bytes information. But this seems complex and fragile.

 

Is it possible to have *two* daemon log settings in the squid.conf? One (original) would write access_log is usual and another one would parse out pumped bytes and update the ICAP records database.

 

Hope I could explain it :(

 

Thanks in advance for everyone taking time to respond.

 

Best regards,

Rafael Akchurin

Diladele B.V.

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: multiple log_file_daemon settings in squid.conf

Amos Jeffries
Administrator
On 20/03/18 23:38, Rafael Akchurin wrote:

> Greetings all,
>
> I am trying to find the best (easiest, least interfering) solution for
> the following problem.
>
> Our custom ICAP server writes various information about ICAP transaction
> (user name, policy ip, detection module, timings, words triggered
> detection, etc) into the record database. This happens at the time when
> ICAP transaction ends. Based on that information we build extensive
> reports of web filtering activity.
>
>  
>
> As known, end of the ICAP transaction does not mean the end of the
> original transaction in Squid – e.g. after scanning CONNECT request and
> allowing it to proceed, the actual data transfer from the Squid’s
> perspective may end much much later. The piece of information that would
> be very interesting for reporting module is how many bytes were pumped
> through that connection. This information is only available after the
> original Squid transaction ends.
>
>  
>
> So somehow in the record database we must correlate the ICAP
> transaction(s) with original Squid transaction.
>
>  
>
> Question 1:
>
> Is there any unique transaction ID in the Squid’s inner workings that I
> can see in the ICAP server, by passing it as additional X-* ICAP header?
>
> I see some references to so called “master transaction” in the docs but
> could not find any log format like identifier that can be used for ICAP
> header value.

see <http://www.squid-cache.org/Doc/config/adaptation_meta/>

There is also the note directive. Although that is only added right at
the point of logging - so it is useful for Squid adding notes if the
ICAP service does not add any itself. All Squid helpers used for a
transaction add notes about their results to the transaction (except
security tokens for NTLM, Kerberos and Digest).
 <http://www.squid-cache.org/Doc/config/note/>


>
> Question 2:
>
> If there is no such transaction ID, I can use ICAP header to pass the
> ICAP specific transaction ID back to Squid **and** I can get that ID
> written to Squid’s access log as
> "X-WebSafety-IID=%{X-WebSafety-IID}adapt::<last_h". Is it a valid approach?
>

Yes.


>  
>
> Question 3:
>
> If q1 or q2 is answered positively, I still need to somehow get the data
> from squid’s access log the ICAP record database. Currently the idea is
> to have the custom logfile_daemon setting that would fork original
> log_file_daemon to have log entries written to access_log **and** parse
> out the ICAP ID **and** update the corresponding ICAP record in the
> database with transferred bytes information. But this seems complex and
> fragile.
>

There is only one daemon at a time supported.

That said the daemon can be instructed via access_log parameter
"daemon:PLACE" to send its data to a specific output PLACE - where PLACE
is a string passed to the helper command line and interpreted in any way
the helper author desires.
 The bundled helpers treat is respectively as the name+path of a file,
or a database DSN to drop the relevant access_log lines to.


Alternatively the "tcp:" logging module can send the log lines as a raw
TCP stream to some place that handles input with the logformat defined
syntax as a series of lines.
 I've seen people defining custom logformat with variously SQL, XML and
JSON markup for delivery to custom daemon or tcp modules.


There is also the option of an external ACL helper that gets passed the
%note details along with anything else you wish it to use. That helper
can do any updates you want prior to the logformat using the same
details. The only restriction here is that it must run in a slow-group
access check. The last one that seems reasonable is http_reply_access
which preceeds ICAP RESPMOD alterations.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: multiple log_file_daemon settings in squid.conf

Alex Rousskov
In reply to this post by Rafael Akchurin
On 03/20/2018 04:38 AM, Rafael Akchurin wrote:

> Is there any unique transaction ID in the Squid’s inner workings that I
> can see in the ICAP server, by passing it as additional X-* ICAP header?

Unfortunately, no. %sn comes close to that but due to implementation
bugs and backward compatibility reasons, it is not usable in adaptation
context. You can construct a fairly unique ID using SMP macros, time,
and TCP connection details, but it is not going to be reliable. Master
transaction ID is a missing Squid feature.

The first ICAP REQMOD service that sees the transaction can set this ID,
but ICAP services do not support returning annotations to Squid. Another
missing Squid feature. You might be able to work around this limitation
by using %adapt::<last_h discussed below.


> I can use ICAP header to pass the
> ICAP specific transaction ID back to Squid **and** I can get that ID
> written to Squid’s access log as
> "X-WebSafety-IID=%{X-WebSafety-IID}adapt::<last_h". Is it a valid approach?

I am not sure what your definition of "valid" in this context is, but
this approach probably can be used as a workaround.

Instead of matching Squid access log records and ICAP log records, you
can also tell Squid to log all of the ICAP details into Squid access
log, using the same %adapt::<last_h workaround. No unique IDs or ID
matching is necessary in that case. Whether to put Squid access log
records into a database (instead or in addition to the file) is up to
you, of course.


> Question 3:
>
> If q1 or q2 is answered positively, I still need to somehow get the data
> from squid’s access log the ICAP record database. Currently the idea is
> to have the custom logfile_daemon setting that would fork original
> log_file_daemon to have log entries written to access_log **and** parse
> out the ICAP ID **and** update the corresponding ICAP record in the
> database with transferred bytes information. But this seems complex and
> fragile.

Squid supports multiple access loggers so, yes, you can also add a
non-"daemon" logger that will do the above. Please see Amos response for
details.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users