[squid-users] Squid Kerberos helper leaking memory

Discussion:

[squid-users] Squid Kerberos helper leaking memory - OpenBSD 6.3

Silamael

2018-09-04 13:24:21 UTC

Hello,

I'm currently investigating a memory leak in with the Kerberos negotiate
authentication helper in Squid 3.5.27 under OpenBSD 6.3. It's a own port
with added Kerberos support since OpenBSD's port does not support
Kerberos at all.

As library Heimdal 7.5.0 is used. So far I had no luck in finding the
memory leak itself.

Would it be safe for Squid, to patch the helper code so that it does a
clean exit after every X processed requests?

Or will this bring new problems on Squid's side?

Thanks for any help!

-- Matthias

Amos Jeffries

2018-09-04 13:51:54 UTC

Permalink

Post by Silamael
Hello,
I'm currently investigating a memory leak in with the Kerberos negotiate
authentication helper in Squid 3.5.27 under OpenBSD 6.3. It's a own port
with added Kerberos support since OpenBSD's port does not support
Kerberos at all.
As library Heimdal 7.5.0 is used. So far I had no luck in finding the
memory leak itself.

Have you tried valgrind and either GCC or clang static analysis features
on your helper and/or library?

Post by Silamael
Would it be safe for Squid, to patch the helper code so that it does a
clean exit after every X processed requests?
Or will this bring new problems on Squid's side?

Should be okay so long as the helpers do reply to at least some queries,
and do not exit all at once.

Squid-3.5 will log errors about helpers exiting unexpectedly, but should
only die if the helpers did so on their startup or many are dying within
a shifting 30sec window of time.

Squid-4 can use the auth_param on-persistent-overload=ERR option to
prevent even the death cases above.

Amos

Silamael

2018-09-04 15:22:10 UTC

Permalink

Post by Amos Jeffries

Have you tried valgrind and either GCC or clang static analysis features
on your helper and/or library?

valgrind doesn't seem to work properly on OpenBSD. I get a bunch of
nonsense output and then a segmentation fault...
What are the GCC/clang statistic features? I'm no C/C++ pro ;)

Post by Amos Jeffries

Post by Silamael
Would it be safe for Squid, to patch the helper code so that it does a
clean exit after every X processed requests?
Or will this bring new problems on Squid's side?

Should be okay so long as the helpers do reply to at least some queries,
and do not exit all at once.
Squid-3.5 will log errors about helpers exiting unexpectedly, but should
only die if the helpers did so on their startup or many are dying within
a shifting 30sec window of time.

At moment a helper will call exit(0) after 10000 requests. Don't know,
how Squid distributes the requests over all helper processes and if we
have too many helpers exiting within 30 seconds...
But good to know that there aren't any general objections.

Post by Amos Jeffries
Squid-4 can use the auth_param on-persistent-overload=ERR option to
prevent even the death cases above.

Good to know.

-- Matthias

Alex Rousskov

2018-09-04 15:59:38 UTC

Permalink

Post by Silamael
At moment a helper will call exit(0) after 10000 requests.
good to know that there aren't any general objections.

Here is one: Squid is currently not designed to gracefully handle a
helper-initiated exit/death. Helpers that decide to exit may kill
in-progress transactions, and/or may slow down or even kill Squid,
depending, in part, on your Squid version and/or configuration.

AFAICT, there are a few better options for going forward, including:

1. Fixing helper memory leak (just stating the obvious for completeness
sake).

2. Wrapping leaking/exiting helper process into a
non-leaking/non-exiting helper that is going to kill/restart the wrapped
helper after N requests (transparently to Squid).

3. Hacking Squid to kill/restart a helper process after N requests.

4. Enhancing Squid and helper protocol to handle helper-initiated exits.

HTH,

Alex.

Markus Moeller

2018-09-04 17:42:23 UTC

Permalink

Can you run the helper standalone with valgrind ?

e.g.

./negotiate_kerberos_auth_test squid.example.com 3 | awk
'{sub(/Token:/,"YR"); print $0}END{print "QQ"}' |
valgrind --log-file=./negotiate_kerberos_auth.val --leak-check=full --show-reachable=yes
-v ./negotiate_kerberos_auth -d -t none -k $dir/squid.keytab -s
GSS_C_NO_NAME

Markus

Post by Silamael
At moment a helper will call exit(0) after 10000 requests.
good to know that there aren't any general objections.