Need help about ICAP scan timeout/max file size for big files

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Need help about ICAP scan timeout/max file size for big files

Schroeffu

Hi all,

i am trying to solve the problem, that SQUID is caching all the big files (for example 1GB) before sending them to the client, but the connected ICAP virus scanner is configured with max_file_size 2MB and scan_timeout 5 seconds. So all bigger files, or longer scanning times, should result in "clean" state from the icap virus scanner.

I am running antivirus FSIGK (F-Secure Internet GateKeeper) as an ICAP daemon connected to Squid with this configuration:

#ICAP
icap_enable on
acl domains_dont_icapscan url_regex -i "/etc/squid/ka/domains_dont_icapscan.acl"
acl audio rep_mime_type -i ^(audio\/x-mpegurl|audio\/mpeg|audio\/ogg|audio\/aac|audio/mp3)$

icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/request
adaptation_access service_req allow !domains_dont_icapscan
icap_service service_resp respmod_precache bypass=1 icap://127.0.0.1:1344/response
adaptation_access service_resp allow !domains_dont_icapscan !audio

Detecting viruses is working, but downloading large files is a huge problem. Squid is downloading them completely first into the servers memory and caching them, before sending them to the client. Its not stop scanning & caching after 2MB/5Seconds. When downloading big files (f.e. 1gb) the browser just does nothing but waiting a long time, because squid is downloading and caching 1gb before forward to client.

I tried change respmod_precache to respmod_postcache but it seems not to be implemented yet, with respmod_postcache fsigk icap log is empty , no virus detection works anymore.
I have a test-virus-file with 100MB (https://schroeffu.ch/100mbrandomvirus_begin.txt eicar+randomcontent) and the virus is detected by fsigk with settings max_scan_size=104400136 / scan_timeout=9000 , change them to max_scan_size=2147483 (2mb) and scan_timeout=5 (5Seconds) the virus is correctly not detected anymore, but, squid still does cache the 100mb before sending to the client.

How can I configure the ICAP Service to truly let bigger files/longer scan times through the icap service marked as "clean"?

Thanks for any help!
Schroeffu

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help about ICAP scan timeout/max file size for big files

Amos Jeffries
Administrator
On 4/01/19 11:38 pm, [hidden email] wrote:

>
> Hi all,
>
> i am trying to solve the problem, that SQUID is caching all the big
> files (for example 1GB) before sending them to the client, but the
> connected ICAP virus scanner is configured with max_file_size 2MB and
> scan_timeout 5 seconds. So all bigger files, or longer scanning times,
> should result in "clean" state from the icap virus scanner.
>
> I am running antivirus FSIGK (F-Secure Internet GateKeeper) as an ICAP
> daemon connected to Squid with this configuration:
>
> #ICAP
> icap_enable on
> acl domains_dont_icapscan url_regex -i
> "/etc/squid/ka/domains_dont_icapscan.acl"
> acl audio rep_mime_type -i
> ^(audio\/x-mpegurl|audio\/mpeg|audio\/ogg|audio\/aac|audio/mp3)$
>
> icap_service service_req reqmod_precache bypass=1
> icap://127.0.0.1:1344/request
> adaptation_access service_req allow !domains_dont_icapscan

> icap_service service_resp respmod_precache bypass=1
> icap://127.0.0.1:1344/response

> adaptation_access service_resp allow !domains_dont_icapscan !audio

The above line says that everything which is not *both* an audio file
and on your dont-scan list does get scanned.

In other words, you can only whitelist audio responses.


>
> Detecting viruses is working, but downloading large files is a huge
> problem. Squid is downloading them completely first into the servers
> memory and caching them,> before sending them to the client. Its not stop
> scanning & caching after 2MB/5Seconds.

Squid is not scanning. The whole point of ICAP is that something *else*
is doing the content manipulation/scanning.

Squid is just a relaying the content octets blindly between the various
agents using it.


> When downloading big files (f.e.
> 1gb) the browser just does nothing but waiting a long time, because
> squid is downloading and caching 1gb before forward to client.

The amount of memory used will depend on other config settings which you
have not shown. Please provide all your config so we can analyze the
problem in full context of what is going on around these ICAP services.


>
> I tried change respmod_precache to respmod_postcache but it seems not to
> be implemented yet, with respmod_postcache fsigk icap log is empty , no
> virus detection works anymore.

Correct. Post-cache ICAP hooks are not supported/implemented by Squid.

If scanning it once (pre-cache) is slow then scanning it per-fetch / N
times (aka post-cache) would be at least N times slower.



> I have a test-virus-file with 100MB
> (https://schroeffu.ch/100mbrandomvirus_begin.txt eicar+randomcontent)
> and the virus is detected by fsigk with settings max_scan_size=104400136
> / scan_timeout=9000 , change them to max_scan_size=2147483 (2mb) and
> scan_timeout=5 (5Seconds) the virus is correctly not detected anymore,

These sound like config setting for the scanning operation. None of that
has any relevance to Squid.

The file is a TXT file not an audio file, so as far as Squid can tell it
is always to be delivered to the scanner.


> but, squid still does cache the 100mb before sending to the client.
>
> How can I configure the ICAP Service to truly let bigger files/longer
> scan times through the icap service marked as "clean"?


What you describe sounds like problems identified with ClamAV early on.
It turned out clam was storing the object to disk and waiting for it to
complete before scanning and providing any output to Squid.

Please check whether the scanner you are using does that type of behaviour.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help about ICAP scan timeout/max file size for big files

Alex Rousskov
In reply to this post by Schroeffu
On 1/4/19 3:38 AM, [hidden email] wrote:
> How can I configure the ICAP Service to truly let bigger files/longer
> scan times through the icap service marked as "clean"?

Which of the following questions are you asking?

1. How to configure Squid to never send huge files to your ICAP service?

2. How to configure your ICAP service to speed up huge-file decisions?

3. How to configure Squid to send huge files to your ICAP service
   without storing them in Squid memory or in Squid disk cache?

For all questions, do the huge files that you are dealing with have an
HTTP Content-Length response header?

And, if it is question #2, does your ICAP service support ICAP Preview
mode? Have you enabled ICAP previews in your Squid configuration?

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help about ICAP scan timeout/max file size for big files

Schroeffu
Hi Alex (& hi Amos)

it depends on the ICAP Service. The one I am trying to use is F-Secure FSICAPD which is not working as expected.

So i compared with ClamAV C-ICAP: With ClamAV C-ICAP there is defined "MaxStreamSize 25M" as default, so after 25MB scanned by ICAP I can see with tcpdump on port 1344 "ICAP/1.0 200 OK" from ICAP to Squid which triggers the browser to start the download. Thats what i want also for F-Secure ICAP.

#ClamAV MaxStreamSize reached ICAP response:
ICAP/1.0 200 OK
Server: C-ICAP/0.4.4
Connection: keep-alive
ISTag: CI0001-1-squidclamav-10
Encapsulated: res-hdr=0, res-body=331

Unfortunately, the F-Secure ICAP is not sending this "ICAP/1.0 200 OK" after X MB or X Seconds. I am in touch with them if this is a bug, i dont know yet, they're checking that. So, if their ICAP really is not sending "ICAP/1.0 200 OK" after X Seconds/MB, can I configure SQUID with a workaround?

So, to your questions:

> 1. How to configure Squid to never send huge files to your ICAP service?

Yes, as a workaround, but how? Header of big files are usually not included.

> 2. How to configure your ICAP service to speed up huge-file decisions?

The header seems not include the file size. Here is an example of 100MB Virus File (EICAR Signature at the beginning) Header:

RESPMOD icap://127.0.0.1:1344/response ICAP/1.0
Host: 127.0.0.1:1344
Date: Fri, 04 Jan 2019 15:56:48 GMT
Encapsulated: req-hdr=0, res-hdr=434, res-body=676

GET https://schroeffu.ch/100mbrandomvirus_begin.txt HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Cookie: _pk_id.n/a.1636=5b8e9d8d8516ea65.1546604985.1.1546604985.1546604985.
Upgrade-Insecure-Requests: 1
Host: schroeffu.ch

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Jan 2019 15:56:48 GMT
Content-Type: text/plain
Last-Modified: Fri, 04 Jan 2019 15:31:19 GMT
Vary: Accept-Encoding
ETag: W/"5c2f7c47-61a8088"
X-Powered-By: PleskLin
Content-Encoding: gzip

The 200 OK reaches Squid after 100% of 100MB has been scanned by F-secure ICAP after 114 Seconds (!),  means, the browser is 114 Seconds doing nothing but watiting:

ICAP/1.0 200 OK
Server: F-Secure ICAP Server
ISTag: "FSAV-2019-01-02_04"
Connection: keep-alive
Expires: Fri, 04 Jan 2019 16:58:42 GMT
X-FSecure-Scan-Result: clean
X-FSecure-ORSP-FRS-Duration: 5.005693
X-FSecure-Transaction-Duration: 114.205939
X-FSecure-Versions: F-Secure Corporation Hydra/5.22 build 28/2018-12-28_01 F-Secure Corporation Aquarius/1.0 build 8/2019-01-02_04 fsavd/1.0/0148 fsicapd/1.1.277-263d28a
Encapsulated: res-hdr=0, res-body=242

> 3. How to configure Squid to send huge files to your ICAP service without storing them in Squid memory or in Squid disk cache?

No, this point we can forget.

I think best would be to configure squid, if ICAP is not able to scan the complete request in 10 seconds, skip (or mark as clean) and let browser download it. 10 seconds icap scan timeout seems to be the default in ESET Linux Gateway ICAP too. Can I configure that in Squid?
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help about ICAP scan timeout/max file size for big files

Alex Rousskov
On 1/8/19 9:46 AM, [hidden email] wrote:

> With ClamAV C-ICAP there is defined "MaxStreamSize 25M" as default,
> so after 25MB scanned by ICAP I can see with tcpdump on port 1344
> "ICAP/1.0 200 OK" from ICAP to Squid which triggers the browser to
> start the download. Thats what i want also for F-Secure ICAP.

The best solution would be for F-Secure to add support for (or enable in
your setup) "data trickling" or "patience pages". Any workarounds inside
Squid would be either nasty (e.g., timeouts, abandoned transactions,
etc.) or expensive (require Squid or eCAP/ICAP wrapper development).


> if their ICAP really is not sending "ICAP/1.0 200 OK" after X
> Seconds/MB, can I configure SQUID with a workaround?

You can try to specify a timeout via icap_io_timeout. Bugs
notwithstanding, Squid would terminate a connection to the ICAP service
that does not respond in X seconds. You may need to adjust
icap_service_failure_limit and/or icap_service_revival_delay to avoid
marking the affected ICAP service as "down" [too often]. Again, this is
not a proper solution and it may have negative side effects such as
memory leaks and unresponsive ICAP service. It may be worth trying while
you wait for F-Secure.

Unfortunately, the icap_io_timeout may not work if Squid is constantly
writing to the ICAP service (to deliver more virgin body bytes). Squid
should be treating each such write as an I/O, resetting the timeout.


You can also hack Squid to treat these cases specially. For example, you
could add adaptation_response_timeout or a similar directive that would
work like icap_io_timeout but ignore write activity. If you go down that
route, I suggest posting an RFC with new option description to squid-dev
as the first step.


You can even write an ICAP service (or eCAP adapter) that will add data
trickling or patience pages support to any ICAP service, but that is a
lot of development work!


> The header seems not include the file size. Here is an example of
> 100MB Virus File

Please note that you should test/analyze "real" transactions, not
requests for test files. If real transactions of interest usually lack
the Content-Length header, then timeout-based knobs are your best bet
(see above): There are no ACLs that can match accumulated response size
and, more importantly, there is no directive that would repeatedly
evaluate such ACLs as Squid accumulates the response body while waiting
for the ICAP response.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users