Jailing programs started from RC scripts

While porting Cardano Node I've got an idea to make the daemon run under a thin jail for added security. By “thin” I mean a minimal chroot environment (in contrary to full-system jails created by Bastille or iocage) that gets set up right before the daemon starts and destroyed after its shutdown. This post summarizes the knowledge I mined during the process of writing a RC service file for Cardano Node.

jail(8)

The jail command is used to create, modify and destroy jails. Creating a jail usually means running some program specified by command or exec.start parameters. When the jailed process exits (and all children processes spawned by it) the jail itself gets destroyed. The path passed in either of these parameters is resolved against jail's root, not the host filesystem.

Another important parameter is called path. It sets the directory that will be used as a root for the jail.

Now we're ready to run a simple program under a jail:

mkdir /tmp/myjail
cp /rescue/echo /tmp/myjail/
jail -c path=/tmp/myjail command=/echo Hello from jail

Hello from jail

Library dependencies

Why did I use echo program from the /rescue/ directory for the running example? Well, let’s try the usual /bin/echo:

cp /bin/echo /tmp/myjail/
jail -c path=/tmp/myjail command=/echo Hello from jail

ELF interpreter /libexec/ld-elf.so.1 not found, error 2
jail: /echo Hello from jail: exited on signal 6

Most programs in the wild are dynamically linked which means they need all libraries they are linked to and the runtime linker to start. So, to make the dynamic echo work we’d need to put all this stuff into the chroot too:

mkdir /tmp/myjail/libexec
cp /libexec/ld-elf.so.1 /tmp/myjail/libexec/
mkdir /tmp/myjail/lib
cp /lib/libc.so.7 /tmp/myjail/lib/
jail -c path=/tmp/myjail command=/echo Hello from jail

Hello from jail

The ldd utility can be used to find out which libraries are required by a given executable:

ldd /usr/local/bin/cardano-node

/usr/local/bin/cardano-node:
        libssl.so.111 => /usr/lib/libssl.so.111 (0x200000)
        libcrypto.so.111 => /lib/libcrypto.so.111 (0x200000)
        libz.so.6 => /lib/libz.so.6 (0x200000)
        libutil.so.9 => /lib/libutil.so.9 (0x200000)
        libgmp.so.10 => /usr/local/lib/libgmp.so.10 (0x200000)
        libm.so.5 => /lib/libm.so.5 (0x200000)
        librt.so.1 => /usr/lib/librt.so.1 (0x200000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x200000)
        libffi.so.8 => /usr/local/lib/libffi.so.8 (0x200000)
        libthr.so.3 => /lib/libthr.so.3 (0x200000)
        libc.so.7 => /lib/libc.so.7 (0x200000)

Instead of copying all of them by hand the following shell incantation can be used:

ldd ${command} | cut -s -d " " -f 3 | grep -E '^(/lib|/usr)' | sort -u | xargs -I % cp % ${jail_root}/lib/

devfs

A lot of programs make access to device pseudo-files under the /dev/ directory. Missing /dev/random may cause obscure runtime errors, which are hard to diagnose. To provide an access to the /dev/ stuff for the jailed program the devfs filesystem has to be mounted inside the jail:

mkdir /tmp/myjail/dev
mount -o ruleset=4 -t devfs devfs /tmp/myjail/dev

The ruleset=4 parameter corresponds to the [devfsrules_jail=4] section of the /etc/defaults/devfs.rules file. This ruleset defines a minimal set of device pseudo-files that should be available for a common jail.

Don't forget to unmount this path after the jail is destroyed:

umount /tmp/myjail/dev

Accessing parts of the host filesystem

Jailed program can't access any paths on the host filesystem on their own. To allow them to do so the loopback filesystem is used:

mkdir /tmp/not_in_jail
mkdir /tmp/myjail/in_jail
mount_nullfs /tmp/not_in_jail /tmp/myjail/in_jail

Now the jail can access the contents of /tmp/not_in_jail directory as /in_jail. Roughly speaking, mount_nullfs acts as ln -s but with ability to span chroot boundaries. Files can be mounted with mount_nullfs just like directories.

The cardano_node RC script uses this trick to hide the fact that the daemon runs under the jail. To do that the config and log directories provided by the user are null-mounted into the jail to make them reachable by the daemon.

Running under a different user

Setting exec.jail_user jail parameter will cause the starting command to run under specified user. If exec.system_jail_user is also present, the user information is taken from the host /etc/passwd. Thanks to this parameter, no in-jail user setup should be performed each time the jail starts.

Jail networking

To allow the jail to listen for and accept incoming connections it is sufficient to pass ip4=inherit, ip6=inherit and host=inherit jail parameters during its creation.

Wrapping up

There are a lot more parameters worth mentioning, but man 8 jail will do it better. For a complete example of the jailed service script see ${PORTSDIR}/net-p2p/cardano-node/files/cardano_node.in.