Discussion:
[squid-users] SMP mode, workers, cache_dir with conditional
juju
2018-10-10 14:01:09 UTC
Permalink
Hello,Sorry if this question has already been posted ( I can't check as the "Search Function" is currently not working )I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GBMy squid is currently working but only on 1 core.I which to use the 2 cores. I understood that I have to use workers in squid.conf with separate cache directory. So I modified default  squid.conf  with :--- [***@JujuServerTest2 ~]# egrep -v "^#|^$" /etc/squid/squid.confacl localnet src 10.0.0.0/8     # RFC1918 possible internal networkacl localnet src 172.16.0.0/12  # RFC1918 possible internal networkacl localnet src 192.168.0.0/16 # RFC1918 possible internal networkacl localnet src fc00::/7       # RFC 4193 local private network rangeacl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machinesacl SSL_ports port 443acl Safe_ports port 80          # httpacl Safe_ports port 21          # ftpacl Safe_ports port 443         # httpsacl Safe_ports port 70          # gopheracl Safe_ports port 210         # waisacl Safe_ports port 1025-65535  # unregistered portsacl Safe_ports port 280         # http-mgmtacl Safe_ports port 488         # gss-httpacl Safe_ports port 591         # filemakeracl Safe_ports port 777         # multiling httpacl CONNECT method CONNECThttp_access deny !Safe_portshttp_access deny CONNECT !SSL_portshttp_access allow localhost managerhttp_access deny managerhttp_access allow localnethttp_access allow localhosthttp_access deny allhttp_port 3128workers 2if ${process_number} = 1cache_dir rock /var/spool/squid1 1024endifif ${process_number} = 2cache_dir rock /var/spool/squid2 1024endifcoredump_dir /var/spool/squidrefresh_pattern ^ftp:           1440    20%     10080refresh_pattern ^gopher:        1440    0%      1440refresh_pattern -i (/cgi-bin/|\?) 0     0%      0refresh_pattern .               0       20%     4320[***@JujuServerTest2 ~]#
---but/var/log/squid/squid.out :
[...]2018/10/10 13:52:15 kid2| Set Current Directory to /var/spool/squid2018/10/10 13:52:15 kid2| Creating missing swap directories2018/10/10 13:52:15 kid1| Set Current Directory to /var/spool/squid2018/10/10 13:52:15 kid1| Creating missing swap directories2018/10/10 13:52:15 kid3| Set Current Directory to /var/spool/squid2018/10/10 13:52:15 kid3| Creating missing swap directories2018/10/10 13:52:15 kid3| No cache_dir stores are configured.2018/10/10 13:52:15 kid3| Set Current Directory to /var/spool/squid2018/10/10 13:52:15 kid2| Set Current Directory to /var/spool/squid2018/10/10 13:52:15 kid1| Set Current Directory to /var/spool/squid2018/10/10 13:52:15 kid2| Creating missing swap directories2018/10/10 13:52:15 kid1| Creating missing swap directories2018/10/10 13:52:15 kid3| Creating missing swap directories2018/10/10 13:52:15 kid3| No cache_dir stores are configured    <---------ps faux |grep squidroot      4172  0.0  0.1 110512  2012 pts/1    S+   14:00   0:00                          \_ grep --color=auto squidroot      4133  0.0  0.6 346844  6848 ?        Ss   13:52   0:00 squid -f /etc/squid/squid.confsquid     4136  0.0  2.1 359828 21292 ?        S    13:52   0:00  \_ (squid-coord-3) -f /etc/squid/squid.confsquid     4140  0.0  0.1  28004  1892 ?        S    13:52   0:00      \_ (logfile-daemon) /var/log/squid/access.log


Looks like, my configuration is not good because "No cache_dir stores are configured"Coud you tell me where is my mistake ? 
Thank you very much.
Amos Jeffries
2018-10-10 15:10:26 UTC
Permalink
Post by juju
Hello,
Sorry if this question has already been posted ( I can't check as the
"Search Function" is currently not working )
I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
My squid is currently working but only on 1 core.
I which to use the 2 cores. I understood that I have to use workers in
squid.conf with separate cache directory.
This is not quite correct.

Per-worker configuration is only required for features which are not yet
SMP-aware.

The rock cache type *is* SMP-aware, so one directory can be shared by
workers.

The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
separate directories per-worker currently.



Since you are using only rock type caches you do not have to do anything
special for SMP to work.
Post by juju
---
...
Post by juju
workers 2
if ${process_number} = 1
cache_dir rock /var/spool/squid1 1024
endif
if ${process_number} = 2
cache_dir rock /var/spool/squid2 1024
endif
...
Post by juju
---
but
[...]
2018/10/10 13:52:15 kid3| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid2| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid1| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid2| Creating missing swap directories
2018/10/10 13:52:15 kid1| Creating missing swap directories
2018/10/10 13:52:15 kid3| Creating missing swap directories
2018/10/10 13:52:15 kid3| No cache_dir stores are configured    <------
Notice the kid (process) numbers in those log lines.

You only defined cache directories for process #1 and #2. This line is
being reported by process #3 - which is the process coordinator process.
So it is correct for your config.

Also, this is informative message - not an error.


Amos
Alex Rousskov
2018-10-10 17:38:36 UTC
Permalink
Post by Amos Jeffries
Post by juju
Hello,
Sorry if this question has already been posted ( I can't check as the
"Search Function" is currently not working )
I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
My squid is currently working but only on 1 core.
I which to use the 2 cores. I understood that I have to use workers in
squid.conf with separate cache directory.
This is not quite correct.
Per-worker configuration is only required for features which are not yet
SMP-aware.
Just a couple of additional clarifications, not contradicting what Amos
Post by Amos Jeffries
The rock cache type *is* SMP-aware, so one directory can be shared by
workers.
"can be" and "must be". We do not support worker-specific cache_dirs.

In case of rock cache_dirs, I do not know for sure what would happen if
cache_dirs are configured on a per-worker basis, but I suspect that
there will be no disker processes to serve them. At best, you will get
no disk caching at all.
Post by Amos Jeffries
The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
separate directories per-worker currently.
... but worker-specific cache_dirs are not supported, will violate HTTP
rules, etc.
Post by Amos Jeffries
Since you are using only rock type caches you do not have to do anything
special for SMP to work.
... with your simple configuration that does not use SMP-unaware features.

In fact, with rock cache_dirs, you do not even have to specify multiple
workers to get SMP Squid -- by default, Squid will run in SMP mode if
you have one worker (default) and at least one rock cache_dir. However,
multiple workers may be necessary to better utilize multicore CPUs.

Beware of hyperthreading: Do not expect two workers, each showing 80+%
CPU utilization to be able to share a single physical CPU core
efficiently. I do not know whether your two cores are hyperthreaded or real.

N.B. To search the mailing list archives, consider using your favorite
search engine. Many would accept a site:lists.squid-cache.org restriction.

Alex.
juju
2018-10-11 08:15:34 UTC
Permalink
Hello !
Thank you very much Alex and AmosI have 2 real cores so it will be good.

on http://www.squid-cache.org/Doc/config/cache_dir/  (available in: 4   3.5   3.4   3.3   3.2   2.7   3.1   3.0   2.6)   :"In SMP configurations, cache_dir must not precede the workers option and should use configuration macros or conditionals to give each worker interested in disk caching a dedicated cache directory"

Perhaps it would be good to specify for which version and which cache_dir type this note is for.
Best regards 
Post by Amos Jeffries
Post by juju
Hello,
Sorry if this question has already been posted ( I can't check as the
"Search Function" is currently not working )
I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
My squid is currently working but only on 1 core.
I which to use the 2 cores. I understood that I have to use workers in
squid.conf with separate cache directory.
This is not quite correct.
Per-worker configuration is only required for features which are not yet
SMP-aware.
Just a couple of additional clarifications, not contradicting what Amos
Post by Amos Jeffries
The rock cache type *is* SMP-aware, so one directory can be shared by
workers.
"can be" and "must be". We do not support worker-specific cache_dirs.

In case of rock cache_dirs, I do not know for sure what would happen if
cache_dirs are configured on a per-worker basis, but I suspect that
there will be no disker processes to serve them. At best, you will get
no disk caching at all.
Post by Amos Jeffries
The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
separate directories per-worker currently.
... but worker-specific cache_dirs are not supported, will violate HTTP
rules, etc.
Post by Amos Jeffries
Since you are using only rock type caches you do not have to do anything
special for SMP to work.
... with your simple configuration that does not use SMP-unaware features.

In fact, with rock cache_dirs, you do not even have to specify multiple
workers to get SMP Squid -- by default, Squid will run in SMP mode if
you have one worker (default) and at least one rock cache_dir. However,
multiple workers may be necessary to better utilize multicore CPUs.

Beware of hyperthreading: Do not expect two workers, each showing 80+%
CPU utilization to be able to share a single physical CPU core
efficiently. I do not know whether your two cores are hyperthreaded or real.

N.B. To search the mailing list archives, consider using your favorite
search engine. Many would accept a site:lists.squid-cache.org restriction.

Alex.
Alex Rousskov
2018-10-11 15:56:55 UTC
Permalink
Post by juju
"In SMP configurations, cache_dir must not precede the workers option
and should use configuration macros or conditionals to give each worker
interested in disk caching a dedicated cache directory"
Perhaps it would be good to specify for which version and which
cache_dir type this note is for.
Yes, that stale note should be rephrased to discourage unsupported
configurations or even replaced with Squid warnings about unsupported
use of ufs-based cache_dirs in SMP configurations.

I am not sure, but I suspect the "must not precede" part is also stale
or, at the very least, should be unnecessary -- if the right order is
still required, Squid should detect wrong order and quit with an error
message.


Quality patches welcomed.

Alex.
Post by juju
Post by Amos Jeffries
Post by juju
Hello,
Sorry if this question has already been posted ( I can't check as the
"Search Function" is currently not working )
I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
My squid is currently working but only on 1 core.
I which to use the 2 cores. I understood that I have to use workers in
squid.conf with separate cache directory.
This is not quite correct.
Per-worker configuration is only required for features which are not yet
SMP-aware.
Just a couple of additional clarifications, not contradicting what Amos
Post by Amos Jeffries
The rock cache type *is* SMP-aware, so one directory can be shared by
workers.
"can be" and "must be". We do not support worker-specific cache_dirs.
In case of rock cache_dirs, I do not know for sure what would happen if
cache_dirs are configured on a per-worker basis, but I suspect that
there will be no disker processes to serve them. At best, you will get
no disk caching at all.
Post by Amos Jeffries
The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
separate directories per-worker currently.
... but worker-specific cache_dirs are not supported, will violate HTTP
rules, etc.
Post by Amos Jeffries
Since you are using only rock type caches you do not have to do anything
special for SMP to work.
... with your simple configuration that does not use SMP-unaware features.
In fact, with rock cache_dirs, you do not even have to specify multiple
workers to get SMP Squid -- by default, Squid will run in SMP mode if
you have one worker (default) and at least one rock cache_dir. However,
multiple workers may be necessary to better utilize multicore CPUs.
Beware of hyperthreading: Do not expect two workers, each showing 80+%
CPU utilization to be able to share a single physical CPU core
efficiently. I do not know whether your two cores are hyperthreaded or real.
N.B. To search the mailing list archives, consider using your favorite
search engine. Many would accept a site:lists.squid-cache.org restriction.
Alex.
_______________________________________________
squid-users mailing list
http://lists.squid-cache.org/listinfo/squid-users
Loading...