(no subject)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

(no subject)

Vieri Di Paola
Hi,

I'm trying to connect from a LAN client with IP addr. 10.215.144.48 to
a web server through Squid 3 + Tproxy.

As you can see from the logs here below, there seems to be a timeout:

https://pastebin.com/2Jka4es1

The Squid machine has no issues if I browse the web from command line,
eg. 'links http://www.linuxheadquarters.com' works fine.

What should I be looking for?

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Amos Jeffries
Administrator
On 12/10/19 2:35 am, Vieri Di Paola wrote:
> Hi,
>
> I'm trying to connect from a LAN client with IP addr. 10.215.144.48 to
> a web server through Squid 3 + Tproxy.
>
> As you can see from the logs here below, there seems to be a timeout:
>
> https://pastebin.com/2Jka4es1

That log contains only a few lines of actual relevance to the problem:

2019/10/11 15:13:48.003 kid1| 5,5| comm.cc(1574) checkTimeouts:
checkTimeouts: FD 14 Expired
2019/10/11 15:13:48.003 kid1| 5,5| comm.cc(1577) checkTimeouts:
checkTimeouts: FD 14: Call timeout handler
2019/10/11 15:13:48.003 kid1| 5,4| AsyncCall.cc(93) ScheduleCall:
comm.cc(1580) will call Comm::ConnOpener::timeout(local=10.215.144.48
remote=172.217.17.5:443 flags=25, data=0x149ac58) [call7547]
2019/10/11 15:13:48.023 kid1| 5,4| AsyncCallQueue.cc(55) fireNext:
entering Comm::ConnOpener::timeout(local=10.215.144.48
remote=172.217.17.5:443 flags=25, data=0x149ac58)
2019/10/11 15:13:48.023 kid1| 5,4| AsyncCall.cc(38) make: make call
Comm::ConnOpener::timeout [call7547]
2019/10/11 15:13:48.023 kid1| 45,9| cbdata.cc(492) cbdataReferenceValid:
0x149ac58
2019/10/11 15:13:48.023 kid1| 45,9| cbdata.cc(492) cbdataReferenceValid:
0x149ac58
2019/10/11 15:13:48.023 kid1| 45,9| cbdata.cc(492) cbdataReferenceValid:
0x149ac58
2019/10/11 15:13:48.023 kid1| 45,9| cbdata.cc(492) cbdataReferenceValid:
0x149ac58
2019/10/11 15:13:48.023 kid1| 5,4| AsyncJob.cc(123) callStart:
Comm::ConnOpener status in: [ job929]
2019/10/11 15:13:48.023 kid1| 45,9| cbdata.cc(492) cbdataReferenceValid:
0x149ac58
2019/10/11 15:13:48.023 kid1| 5,5| ConnOpener.cc(442) timeout:
local=10.215.144.48 remote=172.217.17.5:443 flags=25: * - ERR took too
long to receive response.


Note that this last entry is about a connection to port 443, whereas the
rest of the log is all about traffic to port 80.


>
> The Squid machine has no issues if I browse the web from command line,
> eg. 'links http://www.linuxheadquarters.com' works fine.
>
> What should I be looking for?

TCP/IP level packet routing. Squid is trying to open a TCP connection to
that "remote=" server. TCP SYN is sent, and then ... ... ... nothing.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Vieri Di Paola
On Fri, Oct 11, 2019 at 3:50 PM Amos Jeffries <[hidden email]> wrote:

>
> Note that this last entry is about a connection to port 443, whereas the
> rest of the log is all about traffic to port 80.
> >
> > The Squid machine has no issues if I browse the web from command line,
> > eg. 'links http://www.linuxheadquarters.com' works fine.
> >
> > What should I be looking for?
>
> TCP/IP level packet routing. Squid is trying to open a TCP connection to
> that "remote=" server. TCP SYN is sent, and then ... ... ... nothing.

I noticed the ":80 to :443" flaw in the log, and I don't know why this
shows up if it's not a redirection.
So I did another test to another destination, and I tried to connect
to host with IP addr. 104.113.250.104 on port 80.
Now the log is consistent, but I'm still getting the same connection
timeout even though I can connect without any issues with an HTTP
client from the Squid machine itself. If it were a packet routing
issue, wouldn't the connection time out also with this HTTP client on
the server itself?

Do you see anything fishy in the squid log I've pasted below?

https://pastebin.com/yJZYw28A

Thanks again,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Amos Jeffries
Administrator
On 19/10/19 1:21 am, Vieri Di Paola wrote:

> On Fri, Oct 11, 2019 at 3:50 PM Amos Jeffries wrote:
>>
>> Note that this last entry is about a connection to port 443, whereas the
>> rest of the log is all about traffic to port 80.
>>>
>>> The Squid machine has no issues if I browse the web from command line,
>>> eg. 'links http://www.linuxheadquarters.com' works fine.
>>>
>>> What should I be looking for?
>>
>> TCP/IP level packet routing. Squid is trying to open a TCP connection to
>> that "remote=" server. TCP SYN is sent, and then ... ... ... nothing.
>
> I noticed the ":80 to :443" flaw in the log, and I don't know why this
> shows up if it's not a redirection.

If you are able to share your config maybe we could help spot something,
both for that and for the timeout issue.


> So I did another test to another destination, and I tried to connect
> to host with IP addr. 104.113.250.104 on port 80.
> Now the log is consistent, but I'm still getting the same connection
> timeout even though I can connect without any issues with an HTTP
> client from the Squid machine itself. If it were a packet routing
> issue, wouldn't the connection time out also with this HTTP client on
> the server itself?

You said Squid used TPROXY. The spoofing of packets causes a different
set of routing tables and rules to be applied than normal server
outgoing traffic.

>
> Do you see anything fishy in the squid log I've pasted below?
>

Looks like Squid is doing everything right and the issues is somewhere
between the TCP SYN send and SYN ACK returning.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Vieri Di Paola
Hi,

On Fri, Oct 18, 2019 at 10:13 PM Amos Jeffries <[hidden email]> wrote:
>
> If you are able to share your config maybe we could help spot something,
> both for that and for the timeout issue.

I prepared and tested a trimmed-down squid conf:

# cat squid.conf
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 901         # SWAT
acl CONNECT method CONNECT

http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports

http_access allow localhost manager
http_access deny manager

acl explicit myportname 3128
acl intercepted myportname 3129
acl interceptedssl myportname 3130

http_port 3128
http_port 3129 tproxy
https_port 3130 tproxy ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=16MB cert=/etc/ssl/squid/proxyserver.pem
sslflags=NO_DEFAULT_CA
sslproxy_flags DONT_VERIFY_PEER

sslcrtd_program /usr/libexec/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 16MB
sslcrtd_children 40 startup=20 idle=10

cache_dir diskd /var/cache/squid 32 16 256

acl localnet src 10.0.0.0/8
acl localnet src 192.168.0.0/16

acl good_useragents req_header User-Agent Firefox/
acl good_useragents req_header User-Agent Edge/
acl good_useragents req_header User-Agent Microsoft-CryptoAPI/

http_access deny intercepted !localnet
http_access deny interceptedssl !localnet

http_access allow CONNECT interceptedssl SSL_ports
http_access deny !good_useragents

http_access allow localnet

debug_options rotate=1 ALL,9

reply_header_access Alternate-Protocol deny all
ssl_bump stare all
ssl_bump bump all

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024
icap_service antivirus respmod_precache bypass=0 icap://127.0.0.1:1344/clamav
adaptation_access antivirus allow all

email_err_data on
client_lifetime 480 minutes

httpd_suppress_version_string on
dns_v4_first on
via off
forwarded_for transparent

cache_mem 32 MB

max_filedescriptors 65536
icap_service_failure_limit -1
icap_persistent_connections off

http_access allow localhost

http_access deny all

coredump_dir /var/cache/squid

refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

> You said Squid used TPROXY. The spoofing of packets causes a different
> set of routing tables and rules to be applied than normal server
> outgoing traffic.

I use Shorewall on this system. This program configures iptables and routing.
I dumped all the network information while trying to access port 80 on
host with IP addr. 104.113.250.104 form local host with IP addr.
10.215.144.48:
https://drive.google.com/file/d/13Pr2OCgCInY6E72krCci9BiHrB1lrMce/view?usp=sharing

> Looks like Squid is doing everything right and the issues is somewhere
> between the TCP SYN send and SYN ACK returning.

I suspect there must be something wrong with my routing or marking
(please see dump).

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Amos Jeffries
Administrator
On 22/10/19 11:22 pm, Vieri Di Paola wrote:
>
> I use Shorewall on this system. This program configures iptables and routing.
> I dumped all the network information while trying to access port 80 on
> host with IP addr. 104.113.250.104 form local host with IP addr.
> 10.215.144.48:


I do not see any DIVERT rule at all in your firewall config dump. That
is at least part of the problem.

Have you run through the notes and troubleshooting checks on the TPROXY
feature page?
<https://wiki.squid-cache.org/Features/Tproxy4>


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Vieri Di Paola
On Tue, Oct 22, 2019 at 1:48 PM Amos Jeffries <[hidden email]> wrote:
>
> On 22/10/19 11:22 pm, Vieri Di Paola wrote:
> >
> > I use Shorewall on this system. This program configures iptables and routing.
> > I dumped all the network information while trying to access port 80 on
> > host with IP addr. 104.113.250.104 form local host with IP addr.
> > 10.215.144.48:
> I do not see any DIVERT rule at all in your firewall config dump. That
> is at least part of the problem.

I don't know why.. I must have taken the wrong dump. Here's a new one
I just tested:

https://drive.google.com/file/d/1iqIU8SrvmOfSHs7wv2tjLLx1DXWNrP8h/view?usp=sharing

> Have you run through the notes and troubleshooting checks on the TPROXY
> feature page?
> <https://wiki.squid-cache.org/Features/Tproxy4>

Yes, but I'm obviously overlooking something.
I'll work on it.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Vieri Di Paola
In reply to this post by Amos Jeffries
On Tue, Oct 22, 2019 at 1:48 PM Amos Jeffries <[hidden email]> wrote:
>
> I do not see any DIVERT rule at all in your firewall config dump. That
> is at least part of the problem.

I opened the previous dump and saw the divert rules here below:

Chain PREROUTING (policy ACCEPT 573K packets, 462M bytes)
 pkts bytes target     prot opt in     out     source
destination
 573K  462M CONNMARK   all  --  *      *       0.0.0.0/0
0.0.0.0/0            CONNMARK restore mask 0xff
 1213  181K routemark  all  --  ppp1   *       0.0.0.0/0
0.0.0.0/0            mark match 0x0/0xff
 3195  308K routemark  all  --  ppp2   *       0.0.0.0/0
0.0.0.0/0            mark match 0x0/0xff
 1320 79360 routemark  all  --  ppp3   *       0.0.0.0/0
0.0.0.0/0            mark match 0x0/0xff
 311K  277M tcpre      all  --  *      *       0.0.0.0/0
0.0.0.0/0            mark match 0x0/0xff
    0     0 divert     tcp  --  ppp1   *       0.0.0.0/0
10.215.144.48       [goto]  tcp spt:80 flags:!0x17/0x02 socket
--transparent
    0     0 divert     tcp  --  ppp2   *       0.0.0.0/0
10.215.144.48       [goto]  tcp spt:80 flags:!0x17/0x02 socket
--transparent
    0     0 divert     tcp  --  ppp3   *       0.0.0.0/0
10.215.144.48       [goto]  tcp spt:80 flags:!0x17/0x02 socket
--transparent
   76  7484 TPROXY     tcp  --  enp10s0 *       10.215.144.48
0.0.0.0/0            tcp dpt:80 TPROXY redirect 0.0.0.0:3129 mark
0x200/0x200
    0     0 divert     tcp  --  ppp1   *       0.0.0.0/0
10.215.144.48       [goto]  tcp spt:443 flags:!0x17/0x02 socket
--transparent
    0     0 divert     tcp  --  ppp2   *       0.0.0.0/0
10.215.144.48       [goto]  tcp spt:443 flags:!0x17/0x02 socket
--transparent
    0     0 divert     tcp  --  ppp3   *       0.0.0.0/0
10.215.144.48       [goto]  tcp spt:443 flags:!0x17/0x02 socket
--transparent
   10  1060 TPROXY     tcp  --  enp10s0 *       10.215.144.48
0.0.0.0/0            tcp dpt:443 TPROXY redirect 0.0.0.0:3130 mark
0x200/0x200

Aren't these the DIVERT rules you are referring to?
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Amos Jeffries
Administrator
On 23/10/19 1:23 am, Vieri Di Paola wrote:

> On Tue, Oct 22, 2019 at 1:48 PM Amos Jeffries wrote:
>>
>> I do not see any DIVERT rule at all in your firewall config dump. That
>> is at least part of the problem.
>
> I opened the previous dump and saw the divert rules here below:
>
> Chain PREROUTING (policy ACCEPT 573K packets, 462M bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>  573K  462M CONNMARK   all  --  *      *       0.0.0.0/0
> 0.0.0.0/0            CONNMARK restore mask 0xff
>  1213  181K routemark  all  --  ppp1   *       0.0.0.0/0
> 0.0.0.0/0            mark match 0x0/0xff
>  3195  308K routemark  all  --  ppp2   *       0.0.0.0/0
> 0.0.0.0/0            mark match 0x0/0xff
>  1320 79360 routemark  all  --  ppp3   *       0.0.0.0/0
> 0.0.0.0/0            mark match 0x0/0xff
>  311K  277M tcpre      all  --  *      *       0.0.0.0/0
> 0.0.0.0/0            mark match 0x0/0xff
>     0     0 divert     tcp  --  ppp1   *       0.0.0.0/0
> 10.215.144.48       [goto]  tcp spt:80 flags:!0x17/0x02 socket
> --transparent
>     0     0 divert     tcp  --  ppp2   *       0.0.0.0/0
> 10.215.144.48       [goto]  tcp spt:80 flags:!0x17/0x02 socket
> --transparent
>     0     0 divert     tcp  --  ppp3   *       0.0.0.0/0
> 10.215.144.48       [goto]  tcp spt:80 flags:!0x17/0x02 socket
> --transparent
>    76  7484 TPROXY     tcp  --  enp10s0 *       10.215.144.48
> 0.0.0.0/0            tcp dpt:80 TPROXY redirect 0.0.0.0:3129 mark
> 0x200/0x200
>     0     0 divert     tcp  --  ppp1   *       0.0.0.0/0
> 10.215.144.48       [goto]  tcp spt:443 flags:!0x17/0x02 socket
> --transparent
>     0     0 divert     tcp  --  ppp2   *       0.0.0.0/0
> 10.215.144.48       [goto]  tcp spt:443 flags:!0x17/0x02 socket
> --transparent
>     0     0 divert     tcp  --  ppp3   *       0.0.0.0/0
> 10.215.144.48       [goto]  tcp spt:443 flags:!0x17/0x02 socket
> --transparent
>    10  1060 TPROXY     tcp  --  enp10s0 *       10.215.144.48
> 0.0.0.0/0            tcp dpt:443 TPROXY redirect 0.0.0.0:3130 mark
> 0x200/0x200
>
> Aren't these the DIVERT rules you are referring to?
>


Oh, case sensitivity. I was grep'ing for the upper case chain name.

So you have a 'divert' chain.

First problem with these rules is they depend on an IP address. IP is
the one detail guaranteed not to match properly when TPROXY spoofing is
going on.

Second problem is that they also depend on a source port number of 80 or
443. The traffic needing to be marked comes from both directions, so
this will break half the traffic flow.


Third is that you are using the --transparent option. If I am
understanding it correctly, that will cause the connections out of Squid
(which are marked as transparent) to skip divert action and hit the
TPROXY intercept all over again - infinite loop.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: (no subject)

Vieri Di Paola
On Wed, Oct 23, 2019 at 1:06 PM Amos Jeffries <[hidden email]> wrote:
>
> First problem with these rules is they depend on an IP address. IP is
> the one detail guaranteed not to match properly when TPROXY spoofing is
> going on.

Thank you for giving me clues.
Actually, my whole setup was OK except for one detail.
Where I specify only "10.215.144.48" for TProxy, I needed to also add
the public IP addresses of my 3 ppp links to Internet, ie. the "inet"
values that are shown with:
# ip a s ppp1
# ip a s ppp2
# ip a s ppp3

I don't know how to avoid that. However, it's not a big deal because
they are static addresses.

Thanks again,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users