how to go from connect/tunnel in squid4 ->GET

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

how to go from connect/tunnel in squid4 ->GET

L A Walsh
I had a version of this working in squid3.x, but it didn't work
for some sites and didn't work well with a newer Opera, but did
ok with an older FF-clone.

I bumped to squid4 a few months ago, but stil haven't gotten to the point
where I can see and cache individual requests and following config examples
@ https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit,
I'm feeling rather clueless as to what I'm missing.

If someone could throw a few hints/clueballs my way I'd really appreciate
knowing what I'm doing wrong.

My port line looks like (it's all 1 line).
http_port ishtar.sc.tlinx.org:8118 ssl-bump
generate-host-certificates=on dynamic_cert_mem_cache_size=64MB
tls-cert=/etc/squid/ssl_cert/myCA.pem
options=SINGLE_DH_USE,SINGLE_ECDH_USE
tls-dh=secp521r1,/etc/squid/ssl_cert/dhparam-4096.pem

myCA.pem contains both private+public sigs.  I generated a separate
dhparam file, but don't know if I was supposed to include the curve
type in the generation command or if it only uses that later.

I pre-generated the cert dir and it seems to be running, but I don't
see any certs appearing in the dir


Looking at squid w/ps, I see:
root     56805     1  0 04:28 ?        00:00:00 /usr/sbin/squid
squid    56807 56805 42 04:28 ?        00:00:03 (squid-1) --kid squid-1
squid    56809 56807  0 04:28 ?        00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid    56810 56807  0 04:28 ?        00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid    56811 56807  0 04:28 ?        00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid    56812 56807  0 04:28 ?        00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid    56813 56807  0 04:28 ?        00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid    56814 56807  0 04:28 ?        00:00:00 (logfile-daemon)
/var/log/squid/access.log
squid    56815 56807  0 04:28 ?        00:00:00 (pinger)

Any ideas where I might be missing things?  I can decomment and
send the active lines from the config file if that would help.

Thanks for any pointers...




_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: how to go from connect/tunnel in squid4 ->GET

Alex Rousskov
On 11/29/18 5:33 AM, L A Walsh wrote:

> I bumped to squid4 a few months ago, but stil haven't gotten to the point
> where I can see and cache individual requests and following config examples
> @ https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit,
> I'm feeling rather clueless as to what I'm missing.

What record(s) does your access.log contain for a single test
transaction (preferably using curl or wget rather than a browser)? Any
messages in cache.log for that test transaction? Any ERRORs or WARNINGs
in cache.log at Squid startup?

Alex.



> My port line looks like (it's all 1 line).
> http_port ishtar.sc.tlinx.org:8118 ssl-bump
> generate-host-certificates=on dynamic_cert_mem_cache_size=64MB
> tls-cert=/etc/squid/ssl_cert/myCA.pem
> options=SINGLE_DH_USE,SINGLE_ECDH_USE
> tls-dh=secp521r1,/etc/squid/ssl_cert/dhparam-4096.pem
>
> myCA.pem contains both private+public sigs.  I generated a separate
> dhparam file, but don't know if I was supposed to include the curve
> type in the generation command or if it only uses that later.
>
> I pre-generated the cert dir and it seems to be running, but I don't
> see any certs appearing in the dir
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: how to go from connect/tunnel in squid4 ->GET

L A Walsh
BTW, I posted this a 2nd time because I didn't see the 1st post
ever post (or maybe I didn't see the 2nd post post?...) but it
sorta looks like you responded to the 1st post, and my 2nd post
came in immediate after...strange...
Thank you very much, for your reply, answers are below...
Linda


On 11/29/2018 7:53 AM, Alex Rousskov wrote:

> On 11/29/18 5:33 AM, L A Walsh wrote:
>
>  
>> I bumped to squid4 a few months ago, but stil haven't gotten to the point
>> where I can see and cache individual requests and following config examples
>> @ https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit,
>> I'm feeling rather clueless as to what I'm missing.
>>    
>
> What record(s) does your access.log contain for a single test
> transaction (preferably using curl or wget rather than a browser)? Any
> messages in cache.log for that test transaction? Any ERRORs or WARNINGs
> in cache.log at Squid startup?
>  
----
 From the latest startup:
2018/11/29 09:26:17| Created PID file (/run/squid.pid)
2018/11/29 09:26:17 kid1| Set Current Directory to /var/cache/squid
2018/11/29 09:26:17 kid1| Starting Squid Cache version 4.0.25 for
x86_64-pc-linux-gnu...
2018/11/29 09:26:17 kid1| Service Name: squid
2018/11/29 09:26:17 kid1| Process ID 2344
2018/11/29 09:26:17 kid1| Process Roles: worker
2018/11/29 09:26:17 kid1| With 16384 file descriptors available
2018/11/29 09:26:17 kid1| Initializing IP Cache...
2018/11/29 09:26:17 kid1| DNS Socket created at 0.0.0.0, FD 5
2018/11/29 09:26:17 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding nameserver 192.168.3.1 from
/etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding domain sc.tlinx.org from /etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding domain tlinx.org from /etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding ndots 1 from /etc/resolv.conf
2018/11/29 09:26:17 kid1| helperOpenServers: Starting 5/32
'security_file_certgen' processes
2018/11/29 09:26:17 kid1| Logfile: opening log
daemon:/var/log/squid/access.log
2018/11/29 09:26:17 kid1| Logfile Daemon: opening log
/var/log/squid/access.log
2018/11/29 09:26:17 kid1| Store logging disabled
2018/11/29 09:26:17 kid1| Swap maxSize 100663296 + 262144 KB, estimated
394240 objects
2018/11/29 09:26:17 kid1| Target number of buckets: 12320
2018/11/29 09:26:17 kid1| Using 16384 Store buckets
2018/11/29 09:26:17 kid1| Max Mem  size: 262144 KB
2018/11/29 09:26:17 kid1| Max Swap size: 100663296 KB
2018/11/29 09:26:18 kid1| Rebuilding storage in /var/cache/squid (dirty log)
2018/11/29 09:26:18 kid1| Using Least Load store dir selection
2018/11/29 09:26:18 kid1| Set Current Directory to /var/cache/squid
2018/11/29 09:26:18 kid1| Finished loading MIME types and icons.
2018/11/29 09:26:18 kid1| WARNING: No ssl_bump configured. Disabling
ssl-bump on http_port 192.168.3.1:8118
2018/11/29 09:26:18 kid1| HTCP Disabled.
2018/11/29 09:26:18 kid1| Pinger socket opened on FD 27
2018/11/29 09:26:18 kid1| Squid plugin modules loaded: 0
2018/11/29 09:26:18 kid1| Adaptation support is off.
2018/11/29 09:26:18 kid1| Accepting HTTP Socket connections at
local=192.168.3.1:8118 remote=[::] FD 23 flags=9
2018/11/29 09:26:18 kid1| Accepting HTTP Socket connections at
local=192.168.3.1:8080 remote=[::] FD 24 flags=9
2018/11/29 09:26:18 kid1| Accepting HTTP Socket connections at
local=127.0.0.1:8080 remote=[::] FD 25 flags=9
2018/11/29 09:26:18 kid1| Store rebuilding is 0.60% complete
2018/11/29 09:26:18| pinger: Initialising ICMP pinger ...
2018/11/29 09:26:18| pinger: ICMP socket opened.
2018/11/29 09:26:21 kid1| Done reading /var/cache/squid swaplog (663690
entries)
2018/11/29 09:26:21 kid1| Finished rebuilding storage from disk.
2018/11/29 09:26:21 kid1|    663558 Entries scanned
2018/11/29 09:26:21 kid1|         0 Invalid entries.
2018/11/29 09:26:21 kid1|         0 With invalid flags.
2018/11/29 09:26:21 kid1|    663504 Objects loaded.
2018/11/29 09:26:21 kid1|         0 Objects expired.
2018/11/29 09:26:21 kid1|        95 Objects cancelled.
2018/11/29 09:26:21 kid1|         0 Duplicate URLs purged.
2018/11/29 09:26:21 kid1|        54 Swapfile clashes avoided.
2018/11/29 09:26:21 kid1|   Took 3.76 seconds (176329.00 objects/sec).
2018/11/29 09:26:21 kid1| Beginning Validation Procedure
2018/11/29 09:26:21 kid1|   262144 Entries Validated so far.
2018/11/29 09:26:22 kid1|   524288 Entries Validated so far.
2018/11/29 09:26:22 kid1|   Completed Validation Procedure
2018/11/29 09:26:22 kid1|   Validated 663462 Entries
2018/11/29 09:26:22 kid1|   store_swap_size = 90578908.00 KB
2018/11/29 09:26:22 kid1| storeLateRelease: released 95 objects
2018/11/29 10:10:32 kid1| ipcacheParse No Address records in response to
'ipv6.msftncsi.com'
2018/11/29 10:11:43 kid1| Logfile: opening log
stdio:/var/cache/squid/cache/squid/netdb.state
2018/11/29 10:11:43 kid1| netdbSaveState
stdio:/var/cache/squid/cache/squid/netdb.state: (0) No error.

---
When I tried to do a wget on "www.slashdot.org", I in my short-hand
monitor of the access log, I see:

[1129_101306.00]  129ms; 266  (0/2.0K) MISS/301     <Ishtar [HEAD
http://www.slashdot.org/ - 216.105.38.15 text/html]
  +0.10    48ms; 39   (419/813) TUNNEL/200   <Ishtar [CONNECT
www.slashdot.org:443 - 216.105.38.15 -]

---
and the form directly from the access log shows:
1543515186.809    129 192.168.3.1 TCP_MISS/301 266 HEAD
http://www.slashdot.org/ - HIER_DIRECT/216.105.38.15 text/html
[User-Agent: "Mozilla/4.0 (compatible; MSIE 6.1; Windows NT
5.1)"\r\nAccept: */*\r\nConnection: Keep-Alive\r\nProxy-Connection:
Keep-Alive\r\nHost: www.slashdot.org\r\n] [HTTP/1.1 301 Moved
Permanently\r\nServer: nginx/1.13.12\r\nDate: Thu, 29 Nov 2018 18:13:06
GMT\r\nContent-Type: text/html\r\nContent-Length: 186\r\nConnection:
keep-alive\r\nLocation: https://www.slashdot.org/\r\n\r]
1543515186.902     48 192.168.3.1 TCP_TUNNEL/200 39 CONNECT
www.slashdot.org:443 - HIER_DIRECT/216.105.38.15 - [User-Agent:
"Mozilla/4.0 (compatible; MSIE 6.1; Windows NT 5.1)"\r\nHost:
www.slashdot.org:443\r\n] []


---
the wget showed:

>  wget "http://www.slashdot.org"
--2018-11-29 10:13:06--  http://www.slashdot.org/
Resolving ishtar.sc.tlinx.org (ishtar.sc.tlinx.org)... 192.168.3.1
Connecting to ishtar.sc.tlinx.org
(ishtar.sc.tlinx.org)|192.168.3.1|:8118... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://www.slashdot.org/ [following]
--2018-11-29 10:13:06--  https://www.slashdot.org/
Connecting to ishtar.sc.tlinx.org
(ishtar.sc.tlinx.org)|192.168.3.1|:8118... connected.
Unable to establish SSL connection.
Converted 0 files in 0 seconds.

and curl shows:

>  curl --http1.0 "http://www.slashdot.org" -D headers.txt -o out_.htm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  
Current
                                 Dload  Upload   Total   Spent    Left  
Speed
100   186  100   186    0     0   3358      0 --:--:-- --:--:--
--:--:--  3381
Ishtar:/tmp> cat headers.txt
HTTP/1.1 301 Moved Permanently
Server: nginx/1.13.12
Date: Thu, 29 Nov 2018 18:27:31 GMT
Content-Type: text/html
Content-Length: 186
Connection: close
Location: https://www.slashdot.org/
>  cat out_.htm
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.13.12</center>
</body>
</html>








_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: how to go from connect/tunnel in squid4 ->GET

Alex Rousskov
On 11/29/18 11:38 AM, L A Walsh wrote:
  
>>> I bumped to squid4 a few months ago, but stil haven't gotten to the
>>> point where I can see and cache individual requests

> 2018/11/29 09:26:18 kid1| WARNING: No ssl_bump configured. Disabling
> ssl-bump on http_port 192.168.3.1:8118

You have not configured any ssl_bump rules. Thus, you are effectively
not using any SslBump features. All HTTPS traffic is simply tunneled
through without decryption/analysis.

Your final ssl_bump rule set may become completely different, but you
can start lab-testing with something simple like

    ssl_bump stare all
    ssl_bump bump all

For more rule examples and associated discussion, see
https://wiki.squid-cache.org/Features/SslPeekAndSplice


>  curl --http1.0 "http://www.slashdot.org" -D headers.txt -o out_.htm

Please note that you should test SslBump features using https://...
URLs, not http://... URLs.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: how to go from connect/tunnel in squid4 ->GET

L A Walsh
On 11/29/2018 12:41 PM, Alex Rousskov wrote:
> You have not configured any ssl_bump rules. Thus, you are effectively
> not using any SslBump features. All HTTPS traffic is simply tunneled
> through without decryption/analysis.
---
        Ok....I didn't do any of that in squid 3.x when I had something
working.  I had set my proxy up to have all protos use 1 port,
like 8080 or such.  I placed a rootCA in all of the clients
that I wanted to use the proxy.  And then...it worked for 99%
of the sites.  Some things didn't work right, and maybe these
highlight areas of misconfiguration -- most notably, Opera and
Google sites often failed to connect.  FF-derivative Palemoon
did work with google as did explorer.  I think opera was more
up-to-date with best-practices for encryption usage.

        For sites that I needed that didn't work or for sites
I wanted to remain encrypted (bank, forexample), I'd have use
a straight through connect+tunnel.

        Where were the ssl_bump options set in 3.x.  I thought
the 'ssl-bump' keyword in the http_port options enabled the bumping.

        Did it work that way in 3.x and now just doesn't work
that way in 4.x?

        I'm wanting to know why the old setup worked (mostly)
while the 4.x version seems to be missing "basic bumping"
that you highlighted.



> Please note that you should test SslBump features using https://...
> URLs, not http://... URLs.
---
        Only started with http addresses that I new redirected
to https.


What is the 'ssl-bump' option for in the http_port statement?
It seems like it it a little confusing.

Thanks much!
-linda
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: how to go from connect/tunnel in squid4 ->GET

Alex Rousskov
On 11/30/18 10:39 AM, L A Walsh wrote:
> On 11/29/2018 12:41 PM, Alex Rousskov wrote:
>> You have not configured any ssl_bump rules. Thus, you are effectively
>> not using any SslBump features. All HTTPS traffic is simply tunneled
>> through without decryption/analysis.

> Where were the ssl_bump options set in 3.x.

Not sure I understand the question: The location of ssl_bump directives
has not changed. They are and have always been squid.conf directives. In
modern Squids, they exact location within squid.conf does not matter
(but their order does).


> I thought
> the 'ssl-bump' keyword in the http_port options enabled the bumping.

It enables SslBump processing, which may or may not include bumping
connections (depending on the matching ssl_bump rule and other factors).

All modern Squid versions need ssl_bump rules. It is _possible_ that
(but I do not remember whether) omitting those rules worked by accident
in some older Squid versions. You should use explicit ssl_bump rules in
any modern Squid version.


> Did it work that way in 3.x and now just doesn't work
> that way in 4.x?

I do not know or do not remember. And 3.x is a large range; things may
have changed from v3.1 to v3.5... However, again, explicit ssl_bump
rules should be used in any version that supports ssl_bump directive.


>     I'm wanting to know why the old setup worked (mostly)
> while the 4.x version seems to be missing "basic bumping"
> that you highlighted.

I understand that you want to know that. I cannot spend more free cycles
on this (secondary) question/investigation. FWIW, whether your old setup
"worked" or not, it was wrong.


> What is the 'ssl-bump' option for in the http_port statement?

To tell Squid that the corresponding http_port should pay the cost (and
take the risks) of SslBump processing (validating relevant port
configuration options, creating associated SSL structures at start time,
checking ssl_bump rules at runtime, etc.).

In many Squid deployments, only certain ports do SslBump. Consider
traffic on the other ports: What should happen to it when it matches a,
say, "ssl_bump bump" rule? The only correct answer is ... not to ask
that question in the first place! An ssl-bump flag on a _port line
allows us to avoid that question (and all the other risks/expenses
associated with SslBump).


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users