Squid full request logging

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid full request logging

Niels Hofmans
Hi,

In addition to my ICAP server now authorising requests for my squid instance, is it possible to do full request/response logging?
I do not see the appropriate log_format directive in the docs.
I was hoping not having to do this in my ICAP service since this slows down approval of the HTTP request. (Empty preview v.s. a request capped at 1MB that needs to be sent over every time)
Any suggestions on what the best way forward is here?
Thank you.

Regards,
Niels Hofmans

SITE   https://ironpeak.be
BTW   BE0694785660
BANK BE76068909740795


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid full request logging

Alex Rousskov
On 3/4/21 2:52 PM, Niels Hofmans wrote:

> is it possible to do full request/response logging?

Squid can log HTTP headers with %>h and %<h logformat codes.

Squid cannot log HTTP message bodies.


> I do not see the appropriate log_format directive in the docs.
> I was hoping not having to do this in my ICAP service since this slows
> down approval of the HTTP request. (Empty preview v.s. a request capped
> at 1MB that needs to be sent over every time)

FWIW, an ICAP or eCAP service can start responding to the request
_before_ the service receives the entire HTTP message body. To get
things going, all the service needs is HTTP headers (and even that is,
technically, optional in some cases).

Using an adaptation service is still an overhead, of course, but, very
few legitimate Squid use cases involve logging message bodies, so there
is no built-in mechanism optimized for that specific rare purpose
(yet?). The fastest option available today is probably a dedicated eCAP
service that refuses to adapt the message bit continues to receive (and
log) the message body.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid full request logging

Niels Hofmans
Hi Alex,

Thanks for the feedback. Although I am not proficient in C for writing an ecap service, is there some binding available online for Go?
This was the reason I originally opted for an ICAP service since I can abstract Go behind the HTTP ICAP layer.
Now I understand this has its limitations, but AFAIK a preview cap at 100kb would be sufficient per request.
But this will slow down my current setup greatly, as I’m currently sending -only- the headers.

Would you think that a) using Go for the ecap adapter or b) using two ICAP services.
One would validate the headers and return OK or NOT (bypass=0), while the other only pushes the 1kb request/response to a queue.
Ideally those two would be contacted simultaneously while only the first one is blocking.
..just thinking aloud tough.

Regards,
Niels Hofmans

SITE   https://ironpeak.be
BTW   BE0694785660
BANK BE76068909740795

On 4 Mar 2021, at 22:23, Alex Rousskov <[hidden email]> wrote:

On 3/4/21 2:52 PM, Niels Hofmans wrote:

is it possible to do full request/response logging?

Squid can log HTTP headers with %>h and %<h logformat codes.

Squid cannot log HTTP message bodies.


I do not see the appropriate log_format directive in the docs.
I was hoping not having to do this in my ICAP service since this slows
down approval of the HTTP request. (Empty preview v.s. a request capped
at 1MB that needs to be sent over every time)

FWIW, an ICAP or eCAP service can start responding to the request
_before_ the service receives the entire HTTP message body. To get
things going, all the service needs is HTTP headers (and even that is,
technically, optional in some cases).

Using an adaptation service is still an overhead, of course, but, very
few legitimate Squid use cases involve logging message bodies, so there
is no built-in mechanism optimized for that specific rare purpose
(yet?). The fastest option available today is probably a dedicated eCAP
service that refuses to adapt the message bit continues to receive (and
log) the message body.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid full request logging

Alex Rousskov
On 3/4/21 4:33 PM, Niels Hofmans wrote:

> Although I am not proficient in C for writing
> an ecap service, is there some binding available online for Go?

eCAP is C++ and, unfortunately, the first version of its API evidently
requires elevated programming skills. I am not aware of Go bindings, and
I would not recommend eCAP for those who are not fluent in C++ (or
cannot hire somebody fluent in C++).


> This was the reason I originally opted for an ICAP service since I can
> abstract Go behind the HTTP ICAP layer.

I do not know what "abstract Go behind the HTTP ICAP layer" means, but I
hope that you are not writing an ICAP service from scratch. If you do,
be prepared to battle many problems, both short- and long-term. ICAP
looks simple but it is difficult to get right.


> Now I understand this has its limitations, but AFAIK a preview cap at
> 100kb would be sufficient per request.

I am not sure Squid supports 100KB previews -- the internal buffers may
not be big enough for that. 64KB might be the limit, but do not quote me
on that.

BTW, please note that, AFAICT, Squid ignores icap_preview_size in
squid.conf (a bug) -- your ICAP service has to request the desired
maximum preview size via an ICAP OPTIONS response.


> Would you think that ... using two ICAP services.
> One would validate the headers and return OK or NOT (bypass=0), while
> the other only pushes the 1kb request/response to a queue.
> Ideally those two would be contacted simultaneously while only the first
> one is blocking.
> ..just thinking aloud tough.

Sorry, I cannot evaluate this design because I do not know what you want
to optimize and what your logging requirements/limitations are.


Good luck,

Alex.


> On 4 Mar 2021, at 22:23, Alex Rousskov wrote:
>
> On 3/4/21 2:52 PM, Niels Hofmans wrote:
>
>> is it possible to do full request/response logging?
>
> Squid can log HTTP headers with %>h and %<h logformat codes.
>
> Squid cannot log HTTP message bodies.
>
>
>> I do not see the appropriate log_format directive in the docs.
>> I was hoping not having to do this in my ICAP service since this slows
>> down approval of the HTTP request. (Empty preview v.s. a request capped
>> at 1MB that needs to be sent over every time)
>
> FWIW, an ICAP or eCAP service can start responding to the request
> _before_ the service receives the entire HTTP message body. To get
> things going, all the service needs is HTTP headers (and even that is,
> technically, optional in some cases).
>
> Using an adaptation service is still an overhead, of course, but, very
> few legitimate Squid use cases involve logging message bodies, so there
> is no built-in mechanism optimized for that specific rare purpose
> (yet?). The fastest option available today is probably a dedicated eCAP
> service that refuses to adapt the message bit continues to receive (and
> log) the message body.
>
>
> HTH,
>
> Alex.
> _______________________________________________
> squid-users mailing list
> [hidden email] <mailto:[hidden email]>
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid full request logging

Eliezer Croitoru-3
In reply to this post by Niels Hofmans
Hey Niels,

Take a peek at:

I am using it in production and it was written because of squid limitations.
squid is great but take a peek and see how it works for you.
I have 2 servers in ha cluster which works great.

An example I wrote to filter youtube traffic is at:

let me know if it helps you or gives you any direction.

בתאריך יום ה׳, 4 במרץ 2021, 23:33, מאת Niels Hofmans ‏<[hidden email]>:
Hi Alex,

Thanks for the feedback. Although I am not proficient in C for writing an ecap service, is there some binding available online for Go?
This was the reason I originally opted for an ICAP service since I can abstract Go behind the HTTP ICAP layer.
Now I understand this has its limitations, but AFAIK a preview cap at 100kb would be sufficient per request.
But this will slow down my current setup greatly, as I’m currently sending -only- the headers.

Would you think that a) using Go for the ecap adapter or b) using two ICAP services.
One would validate the headers and return OK or NOT (bypass=0), while the other only pushes the 1kb request/response to a queue.
Ideally those two would be contacted simultaneously while only the first one is blocking.
..just thinking aloud tough.

Regards,
Niels Hofmans

SITE   https://ironpeak.be
BTW   BE0694785660
BANK BE76068909740795

On 4 Mar 2021, at 22:23, Alex Rousskov <[hidden email]> wrote:

On 3/4/21 2:52 PM, Niels Hofmans wrote:

is it possible to do full request/response logging?

Squid can log HTTP headers with %>h and %<h logformat codes.

Squid cannot log HTTP message bodies.


I do not see the appropriate log_format directive in the docs.
I was hoping not having to do this in my ICAP service since this slows
down approval of the HTTP request. (Empty preview v.s. a request capped
at 1MB that needs to be sent over every time)

FWIW, an ICAP or eCAP service can start responding to the request
_before_ the service receives the entire HTTP message body. To get
things going, all the service needs is HTTP headers (and even that is,
technically, optional in some cases).

Using an adaptation service is still an overhead, of course, but, very
few legitimate Squid use cases involve logging message bodies, so there
is no built-in mechanism optimized for that specific rare purpose
(yet?). The fastest option available today is probably a dedicated eCAP
service that refuses to adapt the message bit continues to receive (and
log) the message body.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users