Squid proxy server to log HTTPS traffic

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid proxy server to log HTTPS traffic

Helen Rai
Hi,

I am using  Mikrotik Routerboard Model CRS125-24G-1S-2HnD which Links the internal network with WAN. One of the port is connected with WiFi Router which is broadcasting WiFi signal. Some of the devices work on WiFi network. I have installed squid3 (Version 3.5.27) in Raspberry Pi which is also connected with Mikrotik. Squid is installed with --enable-SSL.

Now, what I want to do is Redirect or log all the HTTP and HTTPS requests are done from devices which are in WiFi network to squid proxy server.

I have tried doing it for months now, I am able to log all HTTP traffic but am unable to do it for HTTPS.

Please help me with this and if you need any detail from me, please contact me.

----
Helen Rai
Platform operations, Nepallink
<a href="tel:44260822" style="color:rgb(84,84,84);text-decoration:none;font-size:12px" target="_blank">44260822 || <a href="tel:9841262275" style="color:rgb(84,84,84);text-decoration:none;font-size:12px" target="_blank">9841262275 || [hidden email] 
http://www.nepallink.net || Link Road, Khusibu 
  

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid proxy server to log HTTPS traffic

Amos Jeffries
Administrator
On 30/07/18 22:58, Helen Rai wrote:
> Hi,
>
> I am using  Mikrotik Routerboard Model CRS125-24G-1S-2HnD which Links
> the internal network with WAN. One of the port is connected with WiFi
> Router which is broadcasting WiFi signal. Some of the devices work on
> WiFi network. I have installed squid3 (Version 3.5.27) in Raspberry Pi
> which is also connected with Mikrotik. Squid is installed with --enable-SSL.
>

The --enable-ssl option was removed in Squid-3.5.
<http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.28-RELEASENOTES.html#ss4.3>.
Use --with-openssl instead, and ensure the Debian libssl1.0-dev package
is installed when building.

To see any URLs inside the HTTPS you will need --enable-ssl-crtd.


> Now, what I want to do is Redirect or log all the HTTP and HTTPS
> requests are done from devices which are in WiFi network to squid proxy
> server.
>
> I have tried doing it for months now, I am able to log all HTTP traffic
> but am unable to do it for HTTPS.
>
> Please help me with this and if you need any detail from me, please
> contact me.
>

If using the correct build option(s) does not resolve your problem we
will need to know what the Mikrotik routign settings, Rasberry Pi
routing and NAT settings, and squid.conf *_port lines are to provide any
assistance.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid proxy server to log HTTPS traffic

Helen Rai
Hi,

The MikroTik router I am using has 24 ports.
There is 4 department and for each department 4 ports of Mikrotik is given.
That is
Department A has 4 computers and each of them is connected with ethernet cable from port 4,5,6 and 7 of Mikrotik Router.
Department B, C, and D have the same number of the computer as Department A and they are connected to 8,9,10,11 and 12,13,14,15 and 16,17,18,19 port of Mikrotik respectively with the ethernet cable.
Each department is in different network (different private IP range).
Each department is later combined into each bridge interface.
Port 20 of Mikrotik is connected to Unifi-AP, which is broadcasting WiFi signals. Mobile devices and Laptop which are not connected via ethernet cable uses WiFi signals.
One of the port from Department A is connected to Raspberry Pi where squid is configured.
Now what I want to do is log all the web traffic that is requested from those devices which are using WiFi.
I have attached the configuration of squid and Mikrotik which I had done for HTTP. (Please note that when I had done this for HTTP, the squid wasn't configured with --enable-SSL or --enable-crtd)

I had referred some online sites to do for HTTPS but none of them worked. Please help if anyone knows how to log all HTTPS request.

I have attached document regarding the rules applied in Mikrotik and squid3 configuration for HTTP.



----
Helen Rai
Platform operations, Nepallink
<a href="tel:44260822" style="color:rgb(84,84,84);text-decoration:none;font-size:12px" target="_blank">44260822 || <a href="tel:9841262275" style="color:rgb(84,84,84);text-decoration:none;font-size:12px" target="_blank">9841262275 || [hidden email] 
http://www.nepallink.net || Link Road, Khusibu 
  

On Mon, Jul 30, 2018 at 7:34 PM, Amos Jeffries <[hidden email]> wrote:
On 30/07/18 22:58, Helen Rai wrote:
> Hi,
>
> I am using  Mikrotik Routerboard Model CRS125-24G-1S-2HnD which Links
> the internal network with WAN. One of the port is connected with WiFi
> Router which is broadcasting WiFi signal. Some of the devices work on
> WiFi network. I have installed squid3 (Version 3.5.27) in Raspberry Pi
> which is also connected with Mikrotik. Squid is installed with --enable-SSL.
>

The --enable-ssl option was removed in Squid-3.5.
<http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.28-RELEASENOTES.html#ss4.3>.
Use --with-openssl instead, and ensure the Debian libssl1.0-dev package
is installed when building.

To see any URLs inside the HTTPS you will need --enable-ssl-crtd.


> Now, what I want to do is Redirect or log all the HTTP and HTTPS
> requests are done from devices which are in WiFi network to squid proxy
> server.
>
> I have tried doing it for months now, I am able to log all HTTP traffic
> but am unable to do it for HTTPS.
>
> Please help me with this and if you need any detail from me, please
> contact me.
>

If using the correct build option(s) does not resolve your problem we
will need to know what the Mikrotik routign settings, Rasberry Pi
routing and NAT settings, and squid.conf *_port lines are to provide any
assistance.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Mikrotik_rules (438 bytes) Download Attachment
squid_conf (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Squid proxy server to log HTTPS traffic

Amos Jeffries
Administrator
On 02/08/18 19:05, Helen Rai wrote:
> Now what I want to do is log all the web traffic that is requested from
> those devices which are using WiFi.
> I have attached the configuration of squid and Mikrotik which I had done
> for HTTP. (Please note that when I had done this for HTTP, the squid
> wasn't configured with --enable-SSL or --enable-crtd)
>

Yes, those options are required to handle the TLS which is part of
HTTPS. HTTP is not encrypted. Or rather: --with-openssl is required

Since you are rebuilding it with these extra options I recommend using
the Squid-4 code from the Debian Sid repository if you can. It contains
much better handling for HTTPS traffic which will be useful later to
avoid non-HTTPS traffic arriving on port 443.


> I had referred some online sites to do for HTTPS but none of them
> worked. Please help if anyone knows how to log all HTTPS request.
>
> I have attached document regarding the rules applied in Mikrotik and
> squid3 configuration for HTTP.
>

Okay. So first thing to do is fix some mistakes in your squid.conf

* "transparent" has long ago been replaced by "intercept"

 http_port 3127 intercept

* remove incomplete / broken config line

 #http_access deny

* remove the first two of these repeated lines:

 http_access allow localhost
 ...
 http_access allow localhost
 http_access allow localhost
 ...

* stop using "localhost" as the ACL name to represent Department-A.
Localhost is a reserved term in networking with a very specific meaning
(different to what you are using it for) which is also baked into Squid
default configuration settings.

Instead use an ACL which names the thing being matched. For example:

 acl Department-A src 10.0.7.0/24


* move your custom setting down below the default security http_access
rules. Those are the deny lines referring to SSL_ports and Safe_ports.

Your http_access lines should now look nice and simple, like this:


 http_access deny !Safe_ports
 http_access deny CONNECT !SSL_ports

 acl test dstdomain "/etc/squid3/acl/test"
 http_access deny test

 acl Department-A src 10.0.7.0/24
 http_access allow Department-A

 http_access deny all


*** Your HTTP traffic should still be fully operational and doing what
you want after these changes. If not something is broken we need to fix
before continuing.


Second, add the pieces necessary for handling port 443 traffic to
squid.conf.

You *will* need to generate a self-signed CA certificate with the
signing-cert (CA) and and HTTP server properties to use SSL-Bump features.
  But that will only need to be installed on clients machine *if* you go
as far as decrypting the HTTPS traffic or producing errors for clients.
This is where Squid-4 handles TLS far more cleanly than Squid-3.5.


 # A port to receive port 443 traffic from Mikrotik
 https_port 3129 intercept ssl-bump cert=/etc/squid3/your-CA.pem

 ... note the 's' in that directive name. A typo there breaks HTTPS in
confusing ways.


 # SSL-Bump rules to peek at details for logging, then relay the traffic
 acl step1 at_step SslBump1
 ssl_bump peek step1
 ssl_bump splice all

 # Squid-4 can pass any non-HTTPS through or omit the below line to
deliver error pages signed by the your-CA.pem certificate.

  on_unsupported_protocol tunnel


Thirdly, you need to add an entry to the Raspberry Pi NAT rules. Same as
for the port 80 NAT rule that should also exist, but for port 443.

 ** at this point you should be able to test the Raspberry PI is working
for HTTPS by setting a test machine to have the R-Pi IP address as its
gateway and requesting any https:// URL in a Browser, or wget, curl,
squidclient etc.


Lastly, when Squid is happily running with all that add to your Mikrotik
config a second line identical to the first one in your attachment -
except that it ends with "dst-port=443".

 As soon as that line is active in your router the HTTPS traffic should
start arriving and being logged at the Squid.



Please be aware of the following details:

* Do expect to see at least half of HTTPS traffic being CONNECT with
raw-IPs and 0 bytes transferred. That is from the first step of
SSL-Bump'ing. They should be followed by another CONNECT with the TLS
SNI domain name and correct size of data transferred (one-way, to client).


* The SNI is only accurate for honest clients. It can be forged and the
real HTTPS messages be using another domain name hosted on the same
server/CDN network.
 However, to be properly sure that the traffic inside the TLS is going
to the domain the SNI claimed you will have to fully decrypt the traffic
(bump action) and that means installing your CA cert on all clients
machines. Plus configuring dynamic cert generation on the https_port line.
 Start with getting the above non-decrypt setup working before going
near bumping.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid proxy server to log HTTPS traffic

Helen Rai
HI,

Thanks, will try and update.

----
Helen Rai
Platform operations, Nepallink
<a href="tel:44260822" style="color:rgb(84,84,84);text-decoration:none;font-size:12px" target="_blank">44260822 || <a href="tel:9841262275" style="color:rgb(84,84,84);text-decoration:none;font-size:12px" target="_blank">9841262275 || [hidden email] 
http://www.nepallink.net || Link Road, Khusibu 
  

On Thu, Aug 2, 2018 at 4:29 PM, Amos Jeffries <[hidden email]> wrote:
On 02/08/18 19:05, Helen Rai wrote:
> Now what I want to do is log all the web traffic that is requested from
> those devices which are using WiFi.
> I have attached the configuration of squid and Mikrotik which I had done
> for HTTP. (Please note that when I had done this for HTTP, the squid
> wasn't configured with --enable-SSL or --enable-crtd)
>

Yes, those options are required to handle the TLS which is part of
HTTPS. HTTP is not encrypted. Or rather: --with-openssl is required

Since you are rebuilding it with these extra options I recommend using
the Squid-4 code from the Debian Sid repository if you can. It contains
much better handling for HTTPS traffic which will be useful later to
avoid non-HTTPS traffic arriving on port 443.


> I had referred some online sites to do for HTTPS but none of them
> worked. Please help if anyone knows how to log all HTTPS request.
>
> I have attached document regarding the rules applied in Mikrotik and
> squid3 configuration for HTTP.
>

Okay. So first thing to do is fix some mistakes in your squid.conf

* "transparent" has long ago been replaced by "intercept"

 http_port 3127 intercept

* remove incomplete / broken config line

 #http_access deny

* remove the first two of these repeated lines:

 http_access allow localhost
 ...
 http_access allow localhost
 http_access allow localhost
 ...

* stop using "localhost" as the ACL name to represent Department-A.
Localhost is a reserved term in networking with a very specific meaning
(different to what you are using it for) which is also baked into Squid
default configuration settings.

Instead use an ACL which names the thing being matched. For example:

 acl Department-A src 10.0.7.0/24


* move your custom setting down below the default security http_access
rules. Those are the deny lines referring to SSL_ports and Safe_ports.

Your http_access lines should now look nice and simple, like this:


 http_access deny !Safe_ports
 http_access deny CONNECT !SSL_ports

 acl test dstdomain "/etc/squid3/acl/test"
 http_access deny test

 acl Department-A src 10.0.7.0/24
 http_access allow Department-A

 http_access deny all


*** Your HTTP traffic should still be fully operational and doing what
you want after these changes. If not something is broken we need to fix
before continuing.


Second, add the pieces necessary for handling port 443 traffic to
squid.conf.

You *will* need to generate a self-signed CA certificate with the
signing-cert (CA) and and HTTP server properties to use SSL-Bump features.
  But that will only need to be installed on clients machine *if* you go
as far as decrypting the HTTPS traffic or producing errors for clients.
This is where Squid-4 handles TLS far more cleanly than Squid-3.5.


 # A port to receive port 443 traffic from Mikrotik
 https_port 3129 intercept ssl-bump cert=/etc/squid3/your-CA.pem

 ... note the 's' in that directive name. A typo there breaks HTTPS in
confusing ways.


 # SSL-Bump rules to peek at details for logging, then relay the traffic
 acl step1 at_step SslBump1
 ssl_bump peek step1
 ssl_bump splice all

 # Squid-4 can pass any non-HTTPS through or omit the below line to
deliver error pages signed by the your-CA.pem certificate.

  on_unsupported_protocol tunnel


Thirdly, you need to add an entry to the Raspberry Pi NAT rules. Same as
for the port 80 NAT rule that should also exist, but for port 443.

 ** at this point you should be able to test the Raspberry PI is working
for HTTPS by setting a test machine to have the R-Pi IP address as its
gateway and requesting any https:// URL in a Browser, or wget, curl,
squidclient etc.


Lastly, when Squid is happily running with all that add to your Mikrotik
config a second line identical to the first one in your attachment -
except that it ends with "dst-port=443".

 As soon as that line is active in your router the HTTPS traffic should
start arriving and being logged at the Squid.



Please be aware of the following details:

* Do expect to see at least half of HTTPS traffic being CONNECT with
raw-IPs and 0 bytes transferred. That is from the first step of
SSL-Bump'ing. They should be followed by another CONNECT with the TLS
SNI domain name and correct size of data transferred (one-way, to client).


* The SNI is only accurate for honest clients. It can be forged and the
real HTTPS messages be using another domain name hosted on the same
server/CDN network.
 However, to be properly sure that the traffic inside the TLS is going
to the domain the SNI claimed you will have to fully decrypt the traffic
(bump action) and that means installing your CA cert on all clients
machines. Plus configuring dynamic cert generation on the https_port line.
 Start with getting the above non-decrypt setup working before going
near bumping.


Amos


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users