Squid full request logging
Locked
 
|
Hi,
In addition to my ICAP server now authorising requests for my squid instance, is it possible to do full request/response logging? I do not see the appropriate log_format directive in the docs. I was hoping not having to do this in my ICAP service since this slows down approval of the HTTP request. (Empty preview v.s. a request capped at 1MB that needs to be sent over every time) Any suggestions on what the best way forward is here? Thank you. _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
On 3/4/21 2:52 PM, Niels Hofmans wrote:
> is it possible to do full request/response logging? Squid can log HTTP headers with %>h and %<h logformat codes. Squid cannot log HTTP message bodies. > I do not see the appropriate log_format directive in the docs. > I was hoping not having to do this in my ICAP service since this slows > down approval of the HTTP request. (Empty preview v.s. a request capped > at 1MB that needs to be sent over every time) FWIW, an ICAP or eCAP service can start responding to the request _before_ the service receives the entire HTTP message body. To get things going, all the service needs is HTTP headers (and even that is, technically, optional in some cases). Using an adaptation service is still an overhead, of course, but, very few legitimate Squid use cases involve logging message bodies, so there is no built-in mechanism optimized for that specific rare purpose (yet?). The fastest option available today is probably a dedicated eCAP service that refuses to adapt the message bit continues to receive (and log) the message body. HTH, Alex. _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
Hi Alex,
Thanks for the feedback. Although I am not proficient in C for writing an ecap service, is there some binding available online for Go? This was the reason I originally opted for an ICAP service since I can abstract Go behind the HTTP ICAP layer. Now I understand this has its limitations, but AFAIK a preview cap at 100kb would be sufficient per request. But this will slow down my current setup greatly, as I’m currently sending -only- the headers. Would you think that a) using Go for the ecap adapter or b) using two ICAP services. One would validate the headers and return OK or NOT (bypass=0), while the other only pushes the 1kb request/response to a queue. Ideally those two would be contacted simultaneously while only the first one is blocking. ..just thinking aloud tough. Regards, On 4 Mar 2021, at 22:23, Alex Rousskov <[hidden email]> wrote: On 3/4/21 2:52 PM, Niels Hofmans wrote: is it possible to do full request/response logging? Squid can log HTTP headers with %>h and %<h logformat codes. Squid cannot log HTTP message bodies. I do not see the appropriate log_format directive in the docs. FWIW, an ICAP or eCAP service can start responding to the request _before_ the service receives the entire HTTP message body. To get things going, all the service needs is HTTP headers (and even that is, technically, optional in some cases). Using an adaptation service is still an overhead, of course, but, very few legitimate Squid use cases involve logging message bodies, so there is no built-in mechanism optimized for that specific rare purpose (yet?). The fastest option available today is probably a dedicated eCAP service that refuses to adapt the message bit continues to receive (and log) the message body. HTH, Alex. _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
|
|
On 3/4/21 4:33 PM, Niels Hofmans wrote:
> Although I am not proficient in C for writing > an ecap service, is there some binding available online for Go? eCAP is C++ and, unfortunately, the first version of its API evidently requires elevated programming skills. I am not aware of Go bindings, and I would not recommend eCAP for those who are not fluent in C++ (or cannot hire somebody fluent in C++). > This was the reason I originally opted for an ICAP service since I can > abstract Go behind the HTTP ICAP layer. I do not know what "abstract Go behind the HTTP ICAP layer" means, but I hope that you are not writing an ICAP service from scratch. If you do, be prepared to battle many problems, both short- and long-term. ICAP looks simple but it is difficult to get right. > Now I understand this has its limitations, but AFAIK a preview cap at > 100kb would be sufficient per request. I am not sure Squid supports 100KB previews -- the internal buffers may not be big enough for that. 64KB might be the limit, but do not quote me on that. BTW, please note that, AFAICT, Squid ignores icap_preview_size in squid.conf (a bug) -- your ICAP service has to request the desired maximum preview size via an ICAP OPTIONS response. > Would you think that ... using two ICAP services. > One would validate the headers and return OK or NOT (bypass=0), while > the other only pushes the 1kb request/response to a queue. > Ideally those two would be contacted simultaneously while only the first > one is blocking. > ..just thinking aloud tough. Sorry, I cannot evaluate this design because I do not know what you want to optimize and what your logging requirements/limitations are. Good luck, Alex. > On 4 Mar 2021, at 22:23, Alex Rousskov wrote: > > On 3/4/21 2:52 PM, Niels Hofmans wrote: > >> is it possible to do full request/response logging? > > Squid can log HTTP headers with %>h and %<h logformat codes. > > Squid cannot log HTTP message bodies. > > >> I do not see the appropriate log_format directive in the docs. >> I was hoping not having to do this in my ICAP service since this slows >> down approval of the HTTP request. (Empty preview v.s. a request capped >> at 1MB that needs to be sent over every time) > > FWIW, an ICAP or eCAP service can start responding to the request > _before_ the service receives the entire HTTP message body. To get > things going, all the service needs is HTTP headers (and even that is, > technically, optional in some cases). > > Using an adaptation service is still an overhead, of course, but, very > few legitimate Squid use cases involve logging message bodies, so there > is no built-in mechanism optimized for that specific rare purpose > (yet?). The fastest option available today is probably a dedicated eCAP > service that refuses to adapt the message bit continues to receive (and > log) the message body. > > > HTH, > > Alex. > _______________________________________________ > squid-users mailing list > [hidden email] <mailto:[hidden email]> > http://lists.squid-cache.org/listinfo/squid-users > _______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
Re: Squid full request logging
|
|
In reply to this post by Niels Hofmans
Hey Niels, Take a peek at: I am using it in production and it was written because of squid limitations. squid is great but take a peek and see how it works for you. I have 2 servers in ha cluster which works great. An example I wrote to filter youtube traffic is at: let me know if it helps you or gives you any direction. בתאריך יום ה׳, 4 במרץ 2021, 23:33, מאת Niels Hofmans <[hidden email]>:
_______________________________________________ squid-users mailing list [hidden email] http://lists.squid-cache.org/listinfo/squid-users |
«
Return to Squid - Users
|
1 view|%1 views
| Free forum by Nabble | Edit this page |