Display eCAP meta-information on Squid error-page

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Display eCAP meta-information on Squid error-page

chgerber
Hi Squid users

I have a question concerning eCAP implementation in Squid 3.5.

My goal is to display data which is provided by an eCAP adapter on a Squid error-page. My primary goal is to achieve this with the eCAP transaction meta-information which is provided by the eCAP adapter with Adapter::Xaction::visitEachOption().

I know that with "adapt::<last_h" (1) this meta-information (options) can be logged to access.log. Is it somehow possible that Squid access these key-value pairs and hands it over to the Error page? Similar to "deny_info" tags (2) which ensures the availability of the URL (%U) etc. in the html code of the error-page.

The approach that the eCAP adapter would create the error-page and send it back to squid as response body instead of calling blockVirgin() (3) is known by me but this would only be the second choice if the approach in question is not possible.

References:
(1) http://www.squid-cache.org/Doc/config/logformat/
(2) http://www.squid-cache.org/Doc/config/deny_info/
(3) https://answers.launchpad.net/ecap/+faq/2516

Thanks in advance for your help.
Best

Christof

--
Christof Gerber

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Fwd: Display eCAP meta-information on Squid error-page

chgerber
Hi Squid users

I have a question concerning eCAP implementation in Squid 3.5.

My goal is to display data which is provided by an eCAP adapter on a
Squid error-page. My primary goal is to achieve this with the eCAP
transaction meta-information which is provided by the eCAP adapter
with Adapter::Xaction::visitEachOption().

I know that with "adapt::<last_h" (1) this meta-information (options)
can be logged to access.log. Is it somehow possible that Squid access
these key-value pairs and hands it over to the Error page? Similar to
"deny_info" tags (2) which ensures the availability of the URL (%U)
etc. in the html code of the error-page.

The approach that the eCAP adapter would create the error-page and
send it back to squid as response body instead of calling
blockVirgin() (3) is known by me but this would only be the second
choice if the approach in question is not possible.

References:
(1) http://www.squid-cache.org/Doc/config/logformat/
(2) http://www.squid-cache.org/Doc/config/deny_info/
(3) https://answers.launchpad.net/ecap/+faq/2516

Thanks in advance for your help.
Best

Christof

--
Christof Gerber
Email: [hidden email]
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

Alex Rousskov
In reply to this post by chgerber
On 09/27/2017 11:52 PM, Christof Gerber wrote:

> I have a question concerning eCAP implementation in Squid 3.5.

Actually, you do not :-). You have a question about using so called
annotations in Squid error pages. eCAP is just one of several annotation
sources in Squid. The other sources are ICAP services, helpers, and the
"note" directive in squid.conf.


> I know that with "adapt::<last_h" (1) this meta-information (options)
> can be logged to access.log.

FYI: It is best to use %note logformat %code for logging annotations.
The %adapt::<last_h code is meant for adaptation services debugging (and
to work around the current ICAP code lack of support for annotations).


> Is it somehow possible that Squid access
> these key-value pairs and hands it over to the Error page?

Unfortunately, Squid error page generation code is unaware of
annotations -- there is currently no code to add them to the generated
error page. Supporting %note and probably all other access.log
formatting codes (via some common namespace like %logformat::note) would
be generally useful IMO.

http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

chgerber
"It is best to use %note logformat %code for logging annotations.
The %adapt::<last_h code is meant for adaptation services debugging (and
to work around the current ICAP code lack of support for annotations)."

How exactly can I use %note to log the same information to access.log? For
example assume I use "%{my-ecap-header}adapt::<last_h" how can I log the
same using %note as you suggested?

Related question:

Can I apply ACL's to annotations served by eCAP adapters. Say when
%{my-ecap-header}adapt::<last_h or the same solution with %note respectively
(see first part of post) returns "bad" I want squid to deny the access and
grant access when it returns "good"? I know about the eCAP specific
virginBlock() function.



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

Amos Jeffries
Administrator
On 05/06/18 21:27, chgerber wrote:
> "It is best to use %note logformat %code for logging annotations.
> The %adapt::<last_h code is meant for adaptation services debugging (and
> to work around the current ICAP code lack of support for annotations)."
>
> How exactly can I use %note to log the same information to access.log? For
> example assume I use "%{my-ecap-header}adapt::<last_h" how can I log the
> same using %note as you suggested?

%note{key-name}

or %<h{header-name}

or %>h{header-name}

Depending on how your adaptor produces it. As an annotation (note) or as
an HTTP message header to be delivered to the client or server.


>
> Related question:
>
> Can I apply ACL's to annotations served by eCAP adapters. Say when
> %{my-ecap-header}adapt::<last_h or the same solution with %note respectively
> (see first part of post) returns "bad" I want squid to deny the access and
> grant access when it returns "good"? I know about the eCAP specific
> virginBlock() function.
>

That sounds like a rather inefficient use of an adaptor.

The adaptor API purpose is to alter HTTP messages as they travel through
the proxy, not to be a substitute for access control logic already
available in the proxy. So what your adaptor SHOULD be doing is simply
producing the 403 Forbidden message itself.

By using a header as described you are forcing Squid to:
  receive adapted message from eCAP
  re-parse that altered message,
  erase that altered message,
  generate a new denial (403) message, and
  deliver to client.

Instead of:
 receive adapted (403) message from eCAP
  re-parse that altered message,
 deliver to client.

As you can see its a whole extra round of message processing and memory
allocation. Doubling the CPU cycles spent, and the traffic latency costs
incurred by using the proxy.


There is an "external ACL" interface provided for complex authorization
logics to be offloaded to a helper process with more capabilities than
the proxy. That should be used instead of eCAP/ICAP adaptors.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

chgerber
Does that mean it is possible to apply ACL to headers/notes after ecap
processing or not?

I agree with your efficiency considerations but can you tell me how you
would solve the following requirement with squid and without eCAP/ICAP:

Parse the body of all requests with a non-empty body and block all requests
containing a certain string "foo".

This is not content adaptation as you mentioned as main use case of ecap but
access control based on content analysis. I see no way other than using
ecap/icap. The reason why we would like to avoid to create the error message
in the ecap adapter itself is that we would like to do the access control
within squid only and using the ecap adapter as content analysis tool with
feedback on which basis squid could decide (ACL).




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

Alex Rousskov
In reply to this post by chgerber
On 06/05/2018 03:27 AM, chgerber wrote:
> "It is best to use %note logformat %code for logging annotations.
> The %adapt::<last_h code is meant for adaptation services debugging (and
> to work around the current ICAP code lack of support for annotations)."

> How exactly can I use %note to log the same information to access.log? For
> example assume I use "%{my-ecap-header}adapt::<last_h" how can I log the
> same using %note as you suggested?


  logformat myLog ... adapter-decision=%{my-ecap-header}note ...
  access_log ... myLog ...


Newer Squids may also support a more natural %note{my-ecap-header}
syntax. Use that if you can.


> Can I apply ACL's to annotations served by eCAP adapters.

Yes, of course. The note ACL does not know where the annotation came
from. Just make sure that you are using the directives that are checked
after Squid receives transaction annotations from your eCAP adapter.

For example, using http_access will not work in most cases because
(SslBump exceptions aside) that directive is only checked before Squid
talks to eCAP adapter(s). However, there is adapted_http_access that is
checked after request adaptations.


> Say when
> %{my-ecap-header}adapt::<last_h or the same solution with %note respectively
> (see first part of post) returns "bad" I want squid to deny the access and
> grant access when it returns "good"?

This sketch is a possible starting point:

  acl badRequest note my-ecap-header bad
  adapted_http_access deny badRequest
  adapted_http_access allow all

The exact correct configuration would depend on the specifics of your
use case. For example, the above allows unrated requests, but you may
want to block (some of) them.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

Alex Rousskov
In reply to this post by chgerber
On 06/05/2018 05:29 AM, chgerber wrote:

> can you tell me how you
> would solve the following requirement with squid and without eCAP/ICAP:

> Parse the body of all requests with a non-empty body and block all requests
> containing a certain string "foo".

You cannot satisfy the above requirement without eCAP or ICAP. Amos
probably assumed that your adapter does not care about the message body,
but that assumption is incorrect in your environment.


> This is not content adaptation as you mentioned as main use case of ecap

Lots of eCAP adapters and ICAP services do not modify message content.
In this context, the unfortunate standard "content adaptation" term
should be interpreted broadly to include content analysis. Content
analysis is one of the primary use cases for eCAP adapters and ICAP
services.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

chgerber
In reply to this post by Alex Rousskov
> logformat myLog ... adapter-decision=%{my-ecap-header}note ...
>  access_log ... myLog ..

I tried this but it didn't work as it does with
"%{my-ecap-header}adapt::<last_h". I am not sure if "my-ecap-header" is a
really a header as I called it as I hand it over to squid as
libecap::NamedValueVisitor when visitEachOption() is called. I guess it is
an ecap option or ecap meta information rather than a header, right? I guess
when you talk about ecap headers you mean http headers set by the eCAP
adapter.

Should the logging with %note and applying ACL also work with ecap meta
information and am I doing something wrong or is this not supported with
ecap meta information? In the latter case I guess I would have to change it
to a solution where the eCAP adapter sets a proper http header which can
then be logged as described above.

Source of confusion:
"adapt::<last_h The header of the last ICAP response or meta-information
from the last eCAP transaction related to the HTTP transaction."
http://www.squid-cache.org/Versions/v3/3.5/cfgman/logformat.html




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

Alex Rousskov
In reply to this post by Amos Jeffries
On 06/05/2018 04:51 AM, Amos Jeffries wrote:

> The adaptor API purpose is to alter HTTP messages as they travel through
> the proxy, not to be a substitute for access control logic already
> available in the proxy.

This statement is incorrect for legitimate use cases where the required
access control logic is not supported by Squid internally. A typical
alternation-free example is content analysis (which Squid ACLs cannot
perform).


> So what your adaptor SHOULD be doing is simply
> producing the 403 Forbidden message itself.

Sometimes, that is the best solution indeed, but it may also be a bad
solution in some cases because it can be slower and because it
duplicates (or discards) a lot of advanced functionality already
implemented in Squid. The rule of thumb here is "If Squid can generate
the right blocking message, use Squid (instead of the adapter) to
generate the right blocking message".


> By using a header as described you are forcing Squid to:
>   receive adapted message from eCAP
>   re-parse that altered message,
>   erase that altered message,
>   generate a new denial (403) message, and
>   deliver to client.

Adapter implementors are not "forced" to use the above sequence of
steps: The first three steps do not have to happen (they are optional).
An optimized adapter implementation that lets Squid generate the
blocking message is limited to the last two steps from your list:

   generate a new denial (403) message, and
   deliver to client.


> There is an "external ACL" interface provided for complex authorization
> logics to be offloaded to a helper process with more capabilities than
> the proxy. That should be used instead of eCAP/ICAP adaptors.

An adapter may be a better solution than the external ACL in some cases.
The actual decision logic here is roughly as follows:

* If built-in ACLs alone are sufficient, then
  use just the built-in ACLs. They are usually simpler and faster.

* If an external ACL is sufficient and performance is not an issue, then
  use an external ACL. It is a lot simpler to implement than an adapter.

* If the decision logic involves message body analysis, then
  use an eCAP adapter or an ICAP service. Others do not get content.

* Otherwise, carefully evaluate external ACLs vs eCAP adapter choice
  given your use case specifics. eCAP can be faster than an external
  ACL or vice versa.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

Alex Rousskov
In reply to this post by chgerber
On 06/05/2018 09:48 AM, chgerber wrote:
>> logformat myLog ... adapter-decision=%{my-ecap-header}note ...
>>  access_log ... myLog ..
>
> I tried this but it didn't work as it does with
> "%{my-ecap-header}adapt::<last_h".

If it does not work, then most likely there is a bug in your adapter, in
your Squid configuration, or in Squid.


> I am not sure if "my-ecap-header" is a
> really a header as I called it as I hand it over to squid as
> libecap::NamedValueVisitor when visitEachOption() is called. I guess it is
> an ecap option or ecap meta information rather than a header, right?

It is a meta-header or annotation, similar to the ICAP response header
(not to be confused with the HTTP header embedded in an ICAP response).


> I guess when you talk about ecap headers you mean http headers set by
> the eCAP adapter.
If I used the "ecap headers" terminology in this context, then I did not
mean the HTTP header. I meant the meta-header or annotation set by your
eCAP adapter via the libecap::Options API presented by your
libecap::adapter::Xaction implementation.


> Should the logging with %note and applying ACL also work with ecap meta
> information and am I doing something wrong or is this not supported with
> ecap meta information?

AFAIK, it is supported in modern Squids. It used to work when I last
tested it. I believe it is supported in v3.5 as well, but I have not
tested it recently.


> Source of confusion:
> "adapt::<last_h The header of the last ICAP response or meta-information
> from the last eCAP transaction related to the HTTP transaction."
> http://www.squid-cache.org/Versions/v3/3.5/cfgman/logformat.html

That definition sounds correct to me. Once you figure it out, please
consider improving the documentation to be less confusing.


Thank you,

Alex.


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

chgerber
>> If it does not work, then most likely there is a bug in your adapter, in
>> your Squid configuration, or in Squid.

By now I see no other way than this being a misunderstanding between us or a
bug in Squid 3.5.

##########
Squid 3.5 XactionRep.cc line 492:

// Store received meta headers for adapt::<last_h logformat code use.
Adaptation::History::Pointer ah = request->adaptLogHistory();
    if (ah != NULL) {
        HttpHeader meta(hoReply);
        OptionsExtractor extractor(meta);
        theMaster->visitEachOption(extractor);
        ah->recordMeta(&meta);
    }
###########

This is where Squid demands the ecap meta headers. This succeeds as I follow
fromt the fact that I can log them with the adapt::<last_h logformat. The
comment also indicates the usage of these ecap meta headers with
adapt::<last_h logformat. But with %{my-ecap-header}note it does not work.
Even when I add %note which should log all notes according to the logformat
configuration docs the result is the same ("-"). I tested this in both
Respmode and Reqmode.  Why are you so sure that %note should work with ecap
meta headers?

Is it due to the changelog of Squid-3.4:
"New format code %note to log a transaction annotation linked to the
transaction by ICAP, eCAP, a helper, or the note squid.conf directive."

To elaborate again why this is important to me: I want to apply ACL's on
information provided by the eCAP adapter. As ACL's can be applied to notes
and because you mentioned that eCAP can provide ecap meta headers as notes I
found this as the way to go. As a starting point I first wanted to verify
that ecap meta headers really can be provided to Squid in the form of notes
so I tried to log them with %{my-ecap-header}note and %note which failed.

squid.conf logformats tried:
logformat test %{my-ecap-header}note                     // resulting
output: "-"
logformat test %note                                                //
resulting output: "-"
logformat test %{my-ecap-header}adapt::<last_h     //  resulting output:
"value-of-my-ecap-header"
         



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Display eCAP meta-information on Squid error-page

Alex Rousskov
On 06/22/2018 03:14 AM, chgerber wrote:
>>> If it does not work, then most likely there is a bug in your adapter, in
>>> your Squid configuration, or in Squid.

> By now I see no other way than this being a misunderstanding between us or a
> bug in Squid 3.5.

> Why are you so sure that %note should work with ecap meta headers?

I have not carefully investigated or tested this, but I believe it
should work (bugs notwithstanding) because I recall projects that
added/used that feature and because documentation/code seem to support
my recollection.

If you think you found a Squid bug, I recommend reporting it. Posting
minimal adapter code that illustrates the bug would increase your
chances of somebody volunteering to triage your bug report.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users