[squid-users] What happens when duplicate external_acl

Post by Amish
----------
Effectively squid.conf now has two external_acl_type lines with same
name. (ipuser)
First one has %ul and other one does not.
From my tests - first one gets the priority and second one is ignored by
squid.
1) Can I assume this to be always true?

Now that you have found the lack of error message on startup one will be
added. It has not been a serious problem, so we are unlikely to make it
more than an ERROR message and explicitly ignore the second (time will
tell tough).

Post by Amish
2) Can there be a case where second gets called instead of first?

Yes. They are looked up by name when processing the config ACLs on
startup/reconfigure. The one found may change on any restart,
reconfigure, or after changes to any lines referencing their shared name.

Post by Amish
3) Can I expect this assumption to remain the same in future too?

No. It is undefined behaviour of the config parser, any change to the
ACL parts of that code might change it unexpectedly.

Amos

Amish

2018-12-01 11:15:03 UTC

Thank you for your quick response.

So if I pass %ul to external_acl_type, but dont use any auth_param,
squid dies with an error.

"Can't use proxy auth because no authentication schemes are fully
configured"

Is it possible for squid to not to die but instead warn and then just
pass "-" (dash) for %ul?

Passing "-" is what squid normally does when macro value is unknown.

Or can we have additional macro %uL (capital L)

%ul will die with error (existing behavior)
%uL will pass username if available else pass "-" (dash)

This way my external_acl_type will work with OR without proxy auth

All I need to change my external_acl_type in main.conf is to add %uL to it.

If this proposal is acceptable, I will try to create a PR.

Regards,

Amish

Amos Jeffries

2018-12-01 11:54:13 UTC

Post by Amish
----------
Effectively squid.conf now has two external_acl_type lines with same
name. (ipuser)
First one has %ul and other one does not.
From my tests - first one gets the priority and second one is
ignored by
squid.
1) Can I assume this to be always true?

The %ul code will generate an auth challenge exchange if no username is
available. So the auth system must be setup with parameters to use in
that challenge.

Use %un for when username is optional.

Amos

Amish

2018-12-01 14:17:28 UTC

Post by Amish
Thank you for your quick response.
So if I pass %ul to external_acl_type, but dont use any auth_param,
squid dies with an error.
"Can't use proxy auth because no authentication schemes are fully
configured"
Is it possible for squid to not to die but instead warn and then just
pass "-" (dash) for %ul?

The %ul code will generate an auth challenge exchange if no username is
available. So the auth system must be setup with parameters to use in
that challenge.
Use %un for when username is optional.

With %un I have a problem.

I have referenced to external acl twice in my squid.conf.

Simplified setup:

external_acl_type ipuser queue-size=40 ttl=120 children-max=1
children-startup=1 concurrency=20 %>a %un /usr/lib/squid/ip_to_user
acl proxyuser external ipuser
http_access allow proxyuser restrictedports
http_access allow proxyuser restrictedsites

where some ports and some sites are allowed only for some users.

so when I try %un (with no auth param set), external acl helper gets
request two times.

First time with "-" and then again with username that external acl
helper itself replied with.

Squid sends: 1 127.0.0.1 - -
Helper reply: 1 OK user=local
Squid sends: 2 127.0.0.1 local -

(Dash at end is due to automatic addition of %DATA macro by squid)

1 was triggered by first http_access line and
2 was triggered by second http_access because %un is either %ul or %ue
(which is now known due to 1)

In my case, it becomes completely unnecessary and an additional processing.

That is why I was thinking of additional macro %uL (capital L)

Regards,

Amish.

Alex Rousskov

2018-12-02 04:03:29 UTC

Post by Amish
Effectively squid.conf now has two external_acl_type lines with same
name. (ipuser)
First one has %ul and other one does not.
From my tests - first one gets the priority and second one is ignored by
squid.
1) Can I assume this to be always true?

Now that you have found the lack of error message on startup one will be
added.

I agree that adding an ERROR message is the right first step.

Post by Amos Jeffries
It has not been a serious problem,

We cannot know that. It is quite likely that this has been a serious
problem for somebody that we did not hear from (or do not remember
hearing from). It may even be a serious problem right now for somebody
who does not know it yet!

To reduce long-term headaches, I think we should be strict and deprecate
(and then prohibit) ignoring duplicated external_acl_type declarations.

I do not see any good reasons for ignoring this configuration error
forever. FWIW, the use case discussed in this thread is not a good
reason IMO because Squid configuration in question can and should be
easily generated (probably from a stable template) to correctly
accommodate the needs of the current authentication method.

Cheers,

Alex.

Amish

2018-12-02 04:31:28 UTC

Post by Alex Rousskov
To reduce long-term headaches, I think we should be strict and deprecate
(and then prohibit) ignoring duplicated external_acl_type declarations.
I do not see any good reasons for ignoring this configuration error
forever. FWIW, the use case discussed in this thread is not a good
reason IMO because Squid configuration in question can and should be
easily generated (probably from a stable template) to correctly
accommodate the needs of the current authentication method.

Thank you for your clarification.

Now I am looking for alternate ways I can resolve my issue.

Does squid allow defining a variable and using it as argument or macro?

For example:

File: pre.conf
----------------
define proxyauth 0
#auth_param basic program /usr/lib/squid/basic_db_auth <arguments>

File: main.conf:
------------------
include pre.conf
external_acl_type ipuser queue-size=40 ttl=120 children-max=1
children-startup=1 concurrency=20 %>a %variable{proxyauth}
/usr/lib/squid/ip_to_user
# OR alternate form
# external_acl_type ipuser queue-size=40 ttl=120 children-max=1
children-startup=1 concurrency=20 %>a /usr/lib/squid/ip_to_user
--proxyauth %variable{proxyauth}

So if I want to use proxyauth I can uncomment auth_param line and change
proxyauth to 1. ip_to_user will be smart enough to then act accordingly.

The reason I cant change main.conf directly is because its a
standardized packaged file and gets overwritten every time package is
updated.

This "define" feature can also have several other use in future.

Thank you,

Amish.

Amos Jeffries

2018-12-02 09:50:05 UTC

Thank you for your clarification.
Now I am looking for alternate ways I can resolve my issue.

What is wrong with %un that makes it unusable?

It will contain username when Squid has been told a username and '-'
when none is known.

Amos

Amish

2018-12-02 11:41:52 UTC

Thank you for your clarification.
Now I am looking for alternate ways I can resolve my issue.

What is wrong with %un that makes it unusable?
It will contain username when Squid has been told a username and '-'
when none is known.

I believe you missed my reply. Here is the archive link to it:

http://lists.squid-cache.org/pipermail/squid-users/2018-December/019759.html

Amish.

Amos Jeffries

2018-12-02 13:15:42 UTC

Thank you for your clarification.
Now I am looking for alternate ways I can resolve my issue.

What is wrong with %un that makes it unusable?
It will contain username when Squid has been told a username and '-'
when none is known.

http://lists.squid-cache.org/pipermail/squid-users/2018-December/019759.html

Ah, yes it has not arrived here for some reason.

There are actually _up to four_ helper checks being done when %ul is
used. Performance optimizations in Squid were/are preventing them being
very visible for Basic auth type and external ACL. But the helper state
is still being checked and if any of the cache TTLs end the check may
fall through to do a full helper query.
* Each test of the proxyuser ACL involves a check of the external
helper cache.
- If there was no cached result with that exact pattern a fully query
is sent.
* Each test of the cache for an external helper using %ul (aka. %LOGIN)
requires a check of the auth_param helper cache (if any).
- If there was no cached result with that exact pattern OR if the auth
scheme does not cache results, a fully query is sent to the auth_param
helper.

With your config and %ul:

- (1) the auth_param helper is asked to login the client and provide a
username

then:
- (2A) the external ACL helper is asked if "user=X" username is okay
OR:
- (2B) the external ACL helper is asked if "-" username is okay

then:
- (3) the auth_param helper is asked to login the client and provide a
username

then:
- (4A) the external ACL helper is asked if "user=X" username is okay
OR:
- (4B) the external ACL helper is asked if "-" username is okay

With your config and %un:

- (1) the external ACL helper is asked if "-" username is okay,

then:
- (2A) the external ACL helper is asked if "user=X" username is okay
OR:
- (2B) the external ACL helper is asked if "-" username is okay

For optimal performance (under either setup) you need to restructure
these lines:
http_access allow proxyuser restrictedports
http_access allow proxyuser restrictedsites

such that the helper is not being used multiple times:

http_access deny !proxyuser
http_access allow restrictedports
http_access allow restrictedsites

Or,
acl restrictedPlaces anyof restrictedports restrictedsites
http_access allow proxyuser restrictedPlaces

Amos

Alex Rousskov

2018-12-02 17:44:06 UTC

Post by Amish
Now I am looking for alternate ways I can resolve my issue.

There are probably many ways to do this. There are ready-to-use
templating tools that may be a better solution (more on that further
below). However, I can offer a simple template-free approach that should
work:

# main.conf
...
include authentication.conf
...

and then do either

cp -p foo.conf authentication.conf

or

cp -p bar.conf authentication.conf

depending on which authentication mechanism you want to use with a
particular Squid instance.

Post by Amish
Does squid allow defining a variable

No, Squid does not. However, there must be ready-to-use templating tools
that can substitute a "variable" in squid.conf.template to generate
squid.conf without variables.

Post by Amish
and using it as argument or macro?

Squid has native support for simple preprocessor conditionals. Combined
with template variables (as discussed above), that support gives you
another way to accomplish the same outcome. The following sketch uses
${Amish::var} as a custom squid.conf.template variable:

if ${Amish::ForceFoo} = 1
... configuration using approach foo ...
else
... configuration using approach bar ...
endif

Post by Amish
The reason I cant change main.conf directly is because its a
standardized packaged file and gets overwritten every time package is
updated.

That is not a valid argument IMO. You do not need to modify main.conf to
generate a configuration file that is actually used. My earlier
authentication.conf sketch is one example. And even something as simple
as the following sed-based script may work in many cases:

sed s/AmishForceFoo/true/ main.conf > squid.conf

Post by Amish
This "define" feature can also have several other use in future.

True. I had considered adding it in the past. The primary reasons it has
not been added yet are:

1. squid.conf terrible syntax makes it difficult to introduce new
syntax-related changes/features safely. For variables to be useful, they
should be usable in many contexts while not being expanded in some other
contexts. As we learned the hard way when general quoted strings were
introduced, getting this right is tricky and sometimes nearly impossible.

2. Templates and such offer a ready-to-use variable support that can
avoid the general problems of #1 in specific deployment environments.
See examples above.

3. Nobody has guided the required changes from an RFC to implementation.

HTH,

Alex.

Amish

2018-12-03 03:59:27 UTC

Post by Amish
Now I am looking for alternate ways I can resolve my issue.

There are probably many ways to do this. There are ready-to-use
templating tools that may be a better solution (more on that further
below). However, I can offer a simple template-free approach that should
# main.conf
...
include authentication.conf
...
and then do either
cp -p foo.conf authentication.conf
or
cp -p bar.conf authentication.conf
depending on which authentication mechanism you want to use with a
particular Squid instance.

This is what I decided on eventually.

Slightly different way but more or less on the same lines.

# main.conf
   ...
   include authentication.conf
   ...

# authentication.conf
   #auth_param ...
   #external_acl_type ... %ul
   external_acl_type ...

And comment / uncomment whichever method I want to use.

Post by Amish
Does squid allow defining a variable

No, Squid does not. However, there must be ready-to-use templating tools
that can substitute a "variable" in squid.conf.template to generate
squid.conf without variables.

Post by Amish
and using it as argument or macro?

Squid has native support for simple preprocessor conditionals. Combined
with template variables (as discussed above), that support gives you
another way to accomplish the same outcome. The following sketch uses
if ${Amish::ForceFoo} = 1
... configuration using approach foo ...
else
... configuration using approach bar ...
endif

Post by Amish
The reason I cant change main.conf directly is because its a
standardized packaged file and gets overwritten every time package is
updated.

Actual squid.conf is:
include pre.conf
include main.conf

So above would not not work. And I do not want to modify main.conf

Post by Amish
This "define" feature can also have several other use in future.

True. I had considered adding it in the past. The primary reasons it has
1. squid.conf terrible syntax makes it difficult to introduce new
syntax-related changes/features safely. For variables to be useful, they
should be usable in many contexts while not being expanded in some other
contexts. As we learned the hard way when general quoted strings were
introduced, getting this right is tricky and sometimes nearly impossible.

May be atleast 'if' statement can have define variable support. Because
it already has similar support for ${process_number} etc.

So template variable substitution above is not required.

pre.conf
-----------
define proxyauth 1

main.conf
-------------
if ${proxyauth} = 1
   # external acl with %ul
else
   # external acl without %ul
endif

This will also allow squid packagers to allow administrators to activate
/ deactive a feature just by simple define variable.

Sample squid.conf

# on top of squid.conf
define transparent 1

# somewhere inside
if ${transparent} = 1
   http_port ... intercept ...
else
   http_port ...
endif

This way newbie administrator does not need to know fine details about
http_port.

Post by Alex Rousskov
2. Templates and such offer a ready-to-use variable support that can
avoid the general problems of #1 in specific deployment environments.
See examples above.
3. Nobody has guided the required changes from an RFC to implementation.

Thank you for spending your time for a detailed reply.

Regards

Amish

Alex Rousskov

2018-12-03 04:21:15 UTC

Post by Alex Rousskov
You do not need to modify main.conf to
generate a configuration file that is actually used. My earlier
authentication.conf sketch is one example. And even something as simple
sed s/AmishForceFoo/true/ main.conf > squid.conf

include pre.conf
include main.conf

... but it does not have to be. In the sed-based approach, there is no
pre.conf at all, and squid.conf is generated from main.conf.

Post by Amish
And I do not want to modify main.conf

The variable/sed-based approach requires a single main.conf modification
(once). Just like the authentication.conf approach requires a single
main.conf modification (that you have already performed).

Post by Amish
May be atleast 'if' statement can have define variable support.

The "I will just add this little thing in this little corner" is how we
ended up with terrible syntax in the first place...

Post by Amish
Because it already has similar support for ${process_number} etc.

Yes, and preprocessor macros are supported throughout the configuration
file, not just in if statements. However, they are preprocessor macros,
not configuration variables. We can add support for custom preprocessor
macros (in addition to the existing built-in ones), but, again, doing so
correctly requires some serious work. Meanwhile, template substitutions
are available as a local ready-to-use alternative.

Alex.

Amish

2018-12-03 04:56:00 UTC

Post by Amish
May be atleast 'if' statement can have define variable support.

The "I will just add this little thing in this little corner" is how we
ended up with terrible syntax in the first place...

Current syntax isn't outright bad. Its decent enough.

Only reason its bit terrible is because backward compatibility is given
a priority and attempt is made to NOT to break old squid.conf.

So adding a new or better config and better defaults is bit tough.

May be squid.conf should have versioning processor at top.

$ cat squid.conf
cfgversion 4.0
# new syntax follows

If no version is mentioned existing syntax is assumed.

This way we do not have to worry much about backward compatibility.

Ofcourse this is a different subject altogether so I will end here on this.

Post by Amish
Because it already has similar support for ${process_number} etc.

Thanks a lot again for your inputs.

Regards

Amish.

Alex Rousskov

2018-12-03 16:14:18 UTC