Sun 25 February 2018
Today I am going to explain how you can run multiple docker daemons on the same host. This can be useful if you want several users to use docker and want each of them to be isolated from one another (ie don't see images, containers, … of other users). The solution relies on user namespaces and sudo.
First, we will update the sudoers file by launching
visudo to allow the users to use the docker command and force them to use the proper daemon. To do that, append the following lines at the end of the file:
USER ALL=(root) NOPASSWD: /usr/bin/docker -H unix\:///var/run/docker-USER.sock *, ! /usr/bin/docker *--priviledged*, ! /usr/bin/docker *host*
You must replace
USER (present two times) by the username of the user you want to allow (you must duplicate the line for each user you want to allow). This will allow USER to launch the docker command as root, without a password, while restricting the user to use its docker instance by talking to the proper socket. We also prevent the user to run a command that contains the priviledged option and host keyword. This is meant to avoid them disabling namespaces for security reasons. If we don't, they could run containers as true root on the machine. This would make our isolation less useful. If you don't care about that, you can simply use:
USER ALL=(root) NOPASSWD: /usr/bin/docker -H unix\:///var/run/docker-USER.sock *
/etc/subuid and /etc/subgid so you have two lines per users which will look like (depending on your system, you may already have the second line):
The first line is used to map the user of id 1 in the container (ie root) to the id of a normal user outside. The second line, defines which uid will be used to map other users: the first number gives the first id and the second the number of ids. The ranges must not collide between users. Since this is still probably obscure, let me give an example:
I have a user named
jenselme. It has uid 1000 on the system. I have a user named
julien. It has uid 1001 on the system. To enable mapping of root in the container to jenselme on the host for jenselme's containers, I add
jenselme:1000:1 to /etc/subuid and /etc/subgid To enable mapping of other users in jenselme's containers, I add:
jenselme:100000:65536. So users will be dispatched to host uids between 100000 and 165535. For instance, user with id 33 in a container will have id 100032 on the host (id mapping starts at 100000, hence 100032 and not 100033). To enable mapping of root in the container to julien on the host for julien's containers, I add
julien:1001:1 to /etc/subuid and /etc/subgid To enable mapping of other users in julien's containers, I add:
julien:165536:65536. So users will be dispatched to host uids between 165536 and 231071. Which are the 65536 ids avaiable after jenselme's ids range.
So my files will contain:
To learn more about this, you can read
my article dedicated to this subject
To do that, you first need to create a
file so you can select for which user you want to run docker. This file must be located under docker@.service /etc/systemd/system/ and have this content:
Description = Docker Application Container Engine
Documentation = http://docs.docker.com
After = network.target docker-containerd.service
Wants = docker-storage-setup.service
Requires = docker-containerd.service rhel-push-plugin.socket registries.service
Type = notify
EnvironmentFile = /run/containers/registries.conf
EnvironmentFile = -/etc/sysconfig/docker
EnvironmentFile = -/etc/sysconfig/docker-storage
EnvironmentFile = -/etc/sysconfig/docker-network
Environment = GOTRACEBACK=crash
ExecStart = /usr/bin/dockerd-current \
--add-runtime oci=/usr/libexec/docker/docker-runc-current \
--containerd /run/containerd.sock \
--exec-opt native.cgroupdriver=systemd \
--userns-remap %i \
--host unix:///var/run/docker-%i.sock \
--pidfile /var/run/docker-%i.pid \
ExecReload = /bin/kill -s HUP $MAINPID
TasksMax = 8192
LimitNOFILE = 1048576
LimitNPROC = 1048576
LimitCORE = infinity
TimeoutStartSec = 0
Restart = on-abnormal
WantedBy = multi-user.target
I added three command line options to the standard service. The
%i will be replace by what comes after the @ when we will run docker (eg USER with systemctl start docker@USER.service). Let me detail the new options:
: enables the username space for the user. This way, each users won't be able to see/use the images and containers of each other: when usermapping is enabled, docker keeps everything (images, containers, networks, volume …) separate for each mapping. --userns-remap %i
: change the name of the socket to listen to. If we don't do this, all our instances would try to listen to --host unix:///var/run/docker-%i.sock which wouldn't work. unix:///var/run/docker.sock
: change the name of the PID file of the process. If we don't do this, all our instances would try to use --pidfile /var/run/docker-%i.pid /var/run/docker.pid which wouldn't work.
If you want to use a different docker configuration file for each instance, you can add
to the list of option. This will make docker use --config-file /etc/docker/daemon-%i.json as a config file when running /etc/docker/daemon-USER.json docker@USER. Note: the file must exist and be valid JSON, otherwise, docker won't start.
Note: I have commented the part that differs from the standard docker.service on fedora. If you use a different distribution, the content of this file may differ but you should be able to adapt it from mine. If you have a question, please leave a comment.
Don't forget to make systemd take the new file into account with
systemctl . daemon-reload
At this point, everything is functional, you can start your daemons with
systemctl start docker@USER.service. To use a daemon, you need to specify to docker which socket to use with the option like this: -H sudo docker . To ease the process, you can create an alias like this in your -H unix:///var/run/docker-USER.sock .bashrc file (adapt to your shell):
alias docker = "sudo docker -H unix:///var/run/docker- $(whoami ) .sock"
After opening a new shell or doing
source , you can use the docker command directly, eg: ~/.bashrc
docker run --rm busybox ls
I suggest you play around with it on multiple accounts to see how all of this works before read the end of the article.
You can use docker-compose like this
sudo if you add the line below to you sudoers: docker-compose -H unix:///var/run/docker-$(whoami).sock
USER ALL=(root) NOPASSWD: /usr/bin/docker-compose -H unix:///var/run/docker-USER.sock *
However, since docker compose does HTTP request to your docker socket, you cannot rely on sudo to prevent the user from launching priviledged or unamespaced containers. You will need an
authorization plugin for that like HBM or Twistlock AuthZ Broker. Note: I have not tested these plugins.
Everything should be working and you should be able to run as many docker daemons as you need. I hope I was clear on this quite technical subject. If I wasn't or if you just want to leave a remark, please leave a comment below!