[squid-users] how to go from connect/tunnel in squid4 ->GET

Discussion:

L A Walsh

2018-11-29 12:33:58 UTC

I had a version of this working in squid3.x, but it didn't work
for some sites and didn't work well with a newer Opera, but did
ok with an older FF-clone.

I bumped to squid4 a few months ago, but stil haven't gotten to the point
where I can see and cache individual requests and following config examples
@ https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit,
I'm feeling rather clueless as to what I'm missing.

If someone could throw a few hints/clueballs my way I'd really appreciate
knowing what I'm doing wrong.

My port line looks like (it's all 1 line).
http_port ishtar.sc.tlinx.org:8118 ssl-bump
generate-host-certificates=on dynamic_cert_mem_cache_size=64MB
tls-cert=/etc/squid/ssl_cert/myCA.pem
options=SINGLE_DH_USE,SINGLE_ECDH_USE
tls-dh=secp521r1,/etc/squid/ssl_cert/dhparam-4096.pem

myCA.pem contains both private+public sigs. I generated a separate
dhparam file, but don't know if I was supposed to include the curve
type in the generation command or if it only uses that later.

I pre-generated the cert dir and it seems to be running, but I don't
see any certs appearing in the dir

Looking at squid w/ps, I see:
root 56805 1 0 04:28 ? 00:00:00 /usr/sbin/squid
squid 56807 56805 42 04:28 ? 00:00:03 (squid-1) --kid squid-1
squid 56809 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56810 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56811 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56812 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56813 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56814 56807 0 04:28 ? 00:00:00 (logfile-daemon)
/var/log/squid/access.log
squid 56815 56807 0 04:28 ? 00:00:00 (pinger)

Any ideas where I might be missing things? I can decomment and
send the active lines from the config file if that would help.

Thanks for any pointers...

Alex Rousskov

2018-11-29 15:53:11 UTC

Permalink

Post by L A Walsh
I bumped to squid4 a few months ago, but stil haven't gotten to the point
where I can see and cache individual requests and following config examples
@ https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit,
I'm feeling rather clueless as to what I'm missing.

What record(s) does your access.log contain for a single test
transaction (preferably using curl or wget rather than a browser)? Any
messages in cache.log for that test transaction? Any ERRORs or WARNINGs
in cache.log at Squid startup?

Alex.

Post by L A Walsh
My port line looks like (it's all 1 line).
http_port ishtar.sc.tlinx.org:8118 ssl-bump
generate-host-certificates=on dynamic_cert_mem_cache_size=64MB
tls-cert=/etc/squid/ssl_cert/myCA.pem
options=SINGLE_DH_USE,SINGLE_ECDH_USE
tls-dh=secp521r1,/etc/squid/ssl_cert/dhparam-4096.pem
myCA.pem contains both private+public sigs. I generated a separate
dhparam file, but don't know if I was supposed to include the curve
type in the generation command or if it only uses that later.
I pre-generated the cert dir and it seems to be running, but I don't
see any certs appearing in the dir

L A Walsh

2018-11-29 18:38:55 UTC

Permalink

BTW, I posted this a 2nd time because I didn't see the 1st post
ever post (or maybe I didn't see the 2nd post post?...) but it
sorta looks like you responded to the 1st post, and my 2nd post
came in immediate after...strange...
Thank you very much, for your reply, answers are below...
Linda

Post by Alex Rousskov

----
From the latest startup:
2018/11/29 09:26:17| Created PID file (/run/squid.pid)
2018/11/29 09:26:17 kid1| Set Current Directory to /var/cache/squid
2018/11/29 09:26:17 kid1| Starting Squid Cache version 4.0.25 for
x86_64-pc-linux-gnu...
2018/11/29 09:26:17 kid1| Service Name: squid
2018/11/29 09:26:17 kid1| Process ID 2344
2018/11/29 09:26:17 kid1| Process Roles: worker
2018/11/29 09:26:17 kid1| With 16384 file descriptors available
2018/11/29 09:26:17 kid1| Initializing IP Cache...
2018/11/29 09:26:17 kid1| DNS Socket created at 0.0.0.0, FD 5
2018/11/29 09:26:17 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding nameserver 192.168.3.1 from
/etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding domain sc.tlinx.org from /etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding domain tlinx.org from /etc/resolv.conf
2018/11/29 09:26:17 kid1| Adding ndots 1 from /etc/resolv.conf
2018/11/29 09:26:17 kid1| helperOpenServers: Starting 5/32
'security_file_certgen' processes
2018/11/29 09:26:17 kid1| Logfile: opening log
daemon:/var/log/squid/access.log
2018/11/29 09:26:17 kid1| Logfile Daemon: opening log
/var/log/squid/access.log
2018/11/29 09:26:17 kid1| Store logging disabled
2018/11/29 09:26:17 kid1| Swap maxSize 100663296 + 262144 KB, estimated
394240 objects
2018/11/29 09:26:17 kid1| Target number of buckets: 12320
2018/11/29 09:26:17 kid1| Using 16384 Store buckets
2018/11/29 09:26:17 kid1| Max Mem size: 262144 KB
2018/11/29 09:26:17 kid1| Max Swap size: 100663296 KB
2018/11/29 09:26:18 kid1| Rebuilding storage in /var/cache/squid (dirty log)
2018/11/29 09:26:18 kid1| Using Least Load store dir selection
2018/11/29 09:26:18 kid1| Set Current Directory to /var/cache/squid
2018/11/29 09:26:18 kid1| Finished loading MIME types and icons.
2018/11/29 09:26:18 kid1| WARNING: No ssl_bump configured. Disabling
ssl-bump on http_port 192.168.3.1:8118
2018/11/29 09:26:18 kid1| HTCP Disabled.
2018/11/29 09:26:18 kid1| Pinger socket opened on FD 27
2018/11/29 09:26:18 kid1| Squid plugin modules loaded: 0
2018/11/29 09:26:18 kid1| Adaptation support is off.
2018/11/29 09:26:18 kid1| Accepting HTTP Socket connections at
local=192.168.3.1:8118 remote=[::] FD 23 flags=9
2018/11/29 09:26:18 kid1| Accepting HTTP Socket connections at
local=192.168.3.1:8080 remote=[::] FD 24 flags=9
2018/11/29 09:26:18 kid1| Accepting HTTP Socket connections at
local=127.0.0.1:8080 remote=[::] FD 25 flags=9
2018/11/29 09:26:18 kid1| Store rebuilding is 0.60% complete
2018/11/29 09:26:18| pinger: Initialising ICMP pinger ...
2018/11/29 09:26:18| pinger: ICMP socket opened.
2018/11/29 09:26:21 kid1| Done reading /var/cache/squid swaplog (663690
entries)
2018/11/29 09:26:21 kid1| Finished rebuilding storage from disk.
2018/11/29 09:26:21 kid1| 663558 Entries scanned
2018/11/29 09:26:21 kid1| 0 Invalid entries.
2018/11/29 09:26:21 kid1| 0 With invalid flags.
2018/11/29 09:26:21 kid1| 663504 Objects loaded.
2018/11/29 09:26:21 kid1| 0 Objects expired.
2018/11/29 09:26:21 kid1| 95 Objects cancelled.
2018/11/29 09:26:21 kid1| 0 Duplicate URLs purged.
2018/11/29 09:26:21 kid1| 54 Swapfile clashes avoided.
2018/11/29 09:26:21 kid1| Took 3.76 seconds (176329.00 objects/sec).
2018/11/29 09:26:21 kid1| Beginning Validation Procedure
2018/11/29 09:26:21 kid1| 262144 Entries Validated so far.
2018/11/29 09:26:22 kid1| 524288 Entries Validated so far.
2018/11/29 09:26:22 kid1| Completed Validation Procedure
2018/11/29 09:26:22 kid1| Validated 663462 Entries
2018/11/29 09:26:22 kid1| store_swap_size = 90578908.00 KB
2018/11/29 09:26:22 kid1| storeLateRelease: released 95 objects
2018/11/29 10:10:32 kid1| ipcacheParse No Address records in response to
'ipv6.msftncsi.com'
2018/11/29 10:11:43 kid1| Logfile: opening log
stdio:/var/cache/squid/cache/squid/netdb.state
2018/11/29 10:11:43 kid1| netdbSaveState
stdio:/var/cache/squid/cache/squid/netdb.state: (0) No error.

---
When I tried to do a wget on "www.slashdot.org", I in my short-hand
monitor of the access log, I see:

[1129_101306.00] 129ms; 266 (0/2.0K) MISS/301 <Ishtar [HEAD
http://www.slashdot.org/ - 216.105.38.15 text/html]
+0.10 48ms; 39 (419/813) TUNNEL/200 <Ishtar [CONNECT
www.slashdot.org:443 - 216.105.38.15 -]

---
and the form directly from the access log shows:
1543515186.809 129 192.168.3.1 TCP_MISS/301 266 HEAD
http://www.slashdot.org/ - HIER_DIRECT/216.105.38.15 text/html
[User-Agent: "Mozilla/4.0 (compatible; MSIE 6.1; Windows NT
5.1)"\r\nAccept: */*\r\nConnection: Keep-Alive\r\nProxy-Connection:
Keep-Alive\r\nHost: www.slashdot.org\r\n] [HTTP/1.1 301 Moved
Permanently\r\nServer: nginx/1.13.12\r\nDate: Thu, 29 Nov 2018 18:13:06
GMT\r\nContent-Type: text/html\r\nContent-Length: 186\r\nConnection:
keep-alive\r\nLocation: https://www.slashdot.org/\r\n\r]
1543515186.902 48 192.168.3.1 TCP_TUNNEL/200 39 CONNECT
www.slashdot.org:443 - HIER_DIRECT/216.105.38.15 - [User-Agent:
"Mozilla/4.0 (compatible; MSIE 6.1; Windows NT 5.1)"\r\nHost:
www.slashdot.org:443\r\n] []

---

Post by Alex Rousskov
wget "http://www.slashdot.org"

--2018-11-29 10:13:06-- http://www.slashdot.org/
Resolving ishtar.sc.tlinx.org (ishtar.sc.tlinx.org)... 192.168.3.1
Connecting to ishtar.sc.tlinx.org
(ishtar.sc.tlinx.org)|192.168.3.1|:8118... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://www.slashdot.org/ [following]
--2018-11-29 10:13:06-- https://www.slashdot.org/
Connecting to ishtar.sc.tlinx.org
(ishtar.sc.tlinx.org)|192.168.3.1|:8118... connected.
Unable to establish SSL connection.
Converted 0 files in 0 seconds.

Post by Alex Rousskov
curl --http1.0 "http://www.slashdot.org" -D headers.txt -o out_.htm

% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 186 100 186 0 0 3358 0 --:--:-- --:--:--
--:--:-- 3381
Ishtar:/tmp> cat headers.txt
HTTP/1.1 301 Moved Permanently
Server: nginx/1.13.12
Date: Thu, 29 Nov 2018 18:27:31 GMT
Content-Type: text/html
Content-Length: 186
Connection: close
Location: https://www.slashdot.org/

Post by Alex Rousskov
cat out_.htm

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.13.12</center>
</body>
</html>

Alex Rousskov

2018-11-29 20:41:06 UTC

Permalink

On 11/29/18 11:38 AM, L A Walsh wrote:

Post by L A Walsh

Post by L A Walsh
I bumped to squid4 a few months ago, but stil haven't gotten to the
point where I can see and cache individual requests

2018/11/29 09:26:18 kid1| WARNING: No ssl_bump configured. Disabling
ssl-bump on http_port 192.168.3.1:8118

You have not configured any ssl_bump rules. Thus, you are effectively
not using any SslBump features. All HTTPS traffic is simply tunneled
through without decryption/analysis.

Your final ssl_bump rule set may become completely different, but you
can start lab-testing with something simple like

ssl_bump stare all
ssl_bump bump all

For more rule examples and associated discussion, see
https://wiki.squid-cache.org/Features/SslPeekAndSplice

Post by L A Walsh
curl --http1.0 "http://www.slashdot.org" -D headers.txt -o out_.htm

Please note that you should test SslBump features using https://...
URLs, not http://... URLs.

Alex.

L A Walsh

2018-11-30 17:39:21 UTC

Permalink

Post by Alex Rousskov
You have not configured any ssl_bump rules. Thus, you are effectively
not using any SslBump features. All HTTPS traffic is simply tunneled
through without decryption/analysis.

---
Ok....I didn't do any of that in squid 3.x when I had something
working. I had set my proxy up to have all protos use 1 port,
like 8080 or such. I placed a rootCA in all of the clients
that I wanted to use the proxy. And then...it worked for 99%
of the sites. Some things didn't work right, and maybe these
highlight areas of misconfiguration -- most notably, Opera and
Google sites often failed to connect. FF-derivative Palemoon
did work with google as did explorer. I think opera was more
up-to-date with best-practices for encryption usage.

For sites that I needed that didn't work or for sites
I wanted to remain encrypted (bank, forexample), I'd have use
a straight through connect+tunnel.

Where were the ssl_bump options set in 3.x. I thought
the 'ssl-bump' keyword in the http_port options enabled the bumping.

Did it work that way in 3.x and now just doesn't work
that way in 4.x?

I'm wanting to know why the old setup worked (mostly)
while the 4.x version seems to be missing "basic bumping"
that you highlighted.

Post by Alex Rousskov
Please note that you should test SslBump features using https://...
URLs, not http://... URLs.

---
Only started with http addresses that I new redirected
to https.

What is the 'ssl-bump' option for in the http_port statement?
It seems like it it a little confusing.

Thanks much!
-linda

Alex Rousskov

2018-11-30 18:48:45 UTC

Permalink

Post by L A Walsh

Post by Alex Rousskov
You have not configured any ssl_bump rules. Thus, you are effectively
not using any SslBump features. All HTTPS traffic is simply tunneled
through without decryption/analysis.

Where were the ssl_bump options set in 3.x.

Not sure I understand the question: The location of ssl_bump directives
has not changed. They are and have always been squid.conf directives. In
modern Squids, they exact location within squid.conf does not matter
(but their order does).

Post by L A Walsh
I thought
the 'ssl-bump' keyword in the http_port options enabled the bumping.

It enables SslBump processing, which may or may not include bumping
connections (depending on the matching ssl_bump rule and other factors).

All modern Squid versions need ssl_bump rules. It is _possible_ that
(but I do not remember whether) omitting those rules worked by accident
in some older Squid versions. You should use explicit ssl_bump rules in
any modern Squid version.

Post by L A Walsh
Did it work that way in 3.x and now just doesn't work
that way in 4.x?

I do not know or do not remember. And 3.x is a large range; things may
have changed from v3.1 to v3.5... However, again, explicit ssl_bump
rules should be used in any version that supports ssl_bump directive.

Post by L A Walsh
I'm wanting to know why the old setup worked (mostly)
while the 4.x version seems to be missing "basic bumping"
that you highlighted.

I understand that you want to know that. I cannot spend more free cycles
on this (secondary) question/investigation. FWIW, whether your old setup
"worked" or not, it was wrong.

Post by L A Walsh
What is the 'ssl-bump' option for in the http_port statement?

To tell Squid that the corresponding http_port should pay the cost (and
take the risks) of SslBump processing (validating relevant port
configuration options, creating associated SSL structures at start time,
checking ssl_bump rules at runtime, etc.).

In many Squid deployments, only certain ports do SslBump. Consider
traffic on the other ports: What should happen to it when it matches a,
say, "ssl_bump bump" rule? The only correct answer is ... not to ask
that question in the first place! An ssl-bump flag on a _port line
allows us to avoid that question (and all the other risks/expenses
associated with SslBump).

HTH,

Alex.

L A Walsh

2018-11-29 17:43:43 UTC

Permalink

e***@ngtech.co.il

2018-12-04 19:57:26 UTC

Permalink

Hey,

I'm not sure I understand the scenario and the issue.
From the wiki page you quoted:
- https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit

I understand you are trying to intercept ssl connections but it's not clear if any traffic is being intercepted or not.
If possible provide the:
- OS and distribution
- "squid -v" output
- some of the access.log that might provide more details on if the traffic is passing or not thru the proxy
- if linux then iptables rules
- if possible the whole squid.conf (remove or obscure any private details)

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: ***@ngtech.co.il

-----Original Message-----
From: squid-users <squid-users-***@lists.squid-cache.org> On Behalf Of L A Walsh
Sent: Thursday, November 29, 2018 19:44
To: squid-***@squid-cache.org
Subject: [squid-users] how to go from connect/tunnel in squid4 ->GET

I had a version of this working in squid3.x, but it didn't work
for some sites and didn't work well with a newer Opera, but did
ok with an older FF-clone.

I bumped to squid4 a few months ago, but stil haven't gotten to the point
where I can see and cache individual requests and following config examples
@ https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit,
I'm feeling rather clueless as to what I'm missing.

If someone could throw a few hints/clueballs my way I'd really appreciate
knowing what I'm doing wrong.

My port line looks like (it's all 1 line).
http_port ishtar.sc.tlinx.org:8118 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=64MB tls-cert=/etc/squid/ssl_cert/myCA.pem options=SINGLE_DH_USE,SINGLE_ECDH_USE tls-dh=secp521r1,/etc/squid/ssl_cert/dhparam-4096.pem

myCA.pem contains both private+public sigs. I generated a separate
dhparam file, but don't know if I was supposed to include the curve
type in the generation command or if it only uses that later.

I pre-generated the cert dir and it seems to be running, but I don't
see any certs appearing in the dir

Looking at squid w/ps, I see:
root 56805 1 0 04:28 ? 00:00:00 /usr/sbin/squid
squid 56807 56805 42 04:28 ? 00:00:03 (squid-1) --kid squid-1
squid 56809 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56810 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56811 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56812 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56813 56807 0 04:28 ? 00:00:00 (security_file_certgen)
-s /var/cache/squid/lib/ssl_db -M 64MB
squid 56814 56807 0 04:28 ? 00:00:00 (logfile-daemon)
/var/log/squid/access.log
squid 56815 56807 0 04:28 ? 00:00:00 (pinger)

Any ideas where I might be missing things? I can decomment and
send the active lines from the config file if that would help.

Thanks for any pointers...

_______________________________________________
squid-users mailing list
squid-***@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users