Daemontools and runit

Tired of PID files, needing root access, and writing init scripts just
to have your UNIX apps start when your server boots? Want a simpler,
better alternative that will also restart them if they crash? If so,
this is an introduction to process supervision with runit/daemontools.


	Background

Classic init scripts, e.g. /etc/init.d/apache, are widely used for
starting processes at system boot time, when they are executed by init.
Sadly, init scripts are cumbersome and error-prone to write, they must
typically be edited and run as root, and the processes they launch do
not get restarted automatically if they crash.

In an alternative scheme called "process supervision", each important
process is looked after by a tiny supervising process, which deals with
starting and stopping the important process on request, and re-starting
it when it exits unexpectedly. Those supervising processes can in turn
be supervised by other supervising processes.

Dan Bernstein wrote the process supervision toolkit, "daemontools",
which is a set of small, reliable programs that cooperate in the
UNIX tradition to manage process supervision trees.

Runit is a more conveniently licensed and more actively maintained
reimplementation of daemontools, written by Gerrit Pape.

Here I’ll use runit, however, the ideas are the same for other
daemontools-like projects (there are several).


	Service directories and scripts

In runit parlance a "service" is simply a directory containing a script
named "run".

There are just two key programs in runit. Firstly, runsv supervises the
process for an individual service. Service directories themselves sit
inside a containing directory, and the runsvdir program supervises that
directory, running one child runsv process for the service in each
subdirectory. A typical choice is to start an instance of runsvdir
which supervises services in subdirectories of /var/service/.

If /var/service/log/ exists, runsv will supervise two services,
and will connect stdout of main service to the stdin of log service.
This is primarily used for logging.

You can debug an individual service by running its SERVICE_DIR/run script.
In this case, its stdout and stderr go to your terminal.

You can also run "runsv SERVICE_DIR", which runs both the service
and its logger service (SERVICE_DIR/log/run) if logger service exists.
If logger service exists, the output will go to it instead of the terminal.

"runsvdir /var/service" merely runs "runsv SERVICE_DIR" for every subdirectory
in /var/service.


	Examples

This directory contains some examples of services:

    var_service/getty_<tty>

Runs a getty on <tty>. (run script looks at $PWD and extracts suffix
after "_" as tty name). Create copies (or symlinks) of this directory
with different names to run many gettys on many ttys.

    var_service/gpm

Runs gpm, the cut and paste utility and mouse server for text consoles.

    var_service/inetd

Runs inetd. This is an example of a service with log. Log service
writes timestamped, rotated log data to /var/log/service/inetd/*
using "svlogd -tt". p_log and w_log scripts demonstrage how you can
"page log" and "watch log".

Other services which have logs handle them in the same way.

    var_service/nmeter

Runs nmeter '%t %c ....' with output to /dev/tty9. This gives you
a 1-second sampling of server load and health on a dedicated text console.


	Networking examples

In many cases, network configuration makes it necessary to run several daemons:
dhcp, zeroconf, ppp, openvpn and such. They need to be controlled,
and in many cases you also want to babysit them.

They present a case where different services need to control (start, stop,
restart) each other.

    var_service/dhcp_if

controls a udhcpc instance which provides DHCP-assigned IP
address on interface named "if". Copy/rename this directory as needed to run
udhcpc on other interfaces (var_service/dhcp_if/run script uses _foo suffix
of the parent directory as interface name).

When IP address is obtained or lost, var_service/dhcp_if/dhcp_handler is run.
It saves new config data to /var/run/service/fw/dhcp_if.ipconf and (re)starts
/var/service/fw service. This example can be used as a template for other
dynamic network link services (ppp/vpn/zcip).

This is an example of service with has a "finish" script. If downed ("sv d"),
"finish" is executed. For this service, it removes DHCP address from
the interface. This is useful when ifplugd detects that the the link is dead
(cable is no longer attached anywhere) and downs us - keeping DHCP configured
addresses on the interface would make kernel still try to use it.

    var_service/zcip_if

Zeroconf IP service: assigns a 169.254.x.y/16 address to interface "if".
This allows to talk to other devices on a network without DHCP server
(if they also assign 169.254 addresses to themselves).

    var_service/ifplugd_if

Watches link status of interface "if". Downs and ups /var/service/dhcp_if
service accordingly. In effect, it allows you to unplug/plug-to-different-network
and have your IP properly re-negotiated at once.

    var_service/dhcp_if_pinger

Uses var_service/dhcp_if's data to determine router IP. Pings it.
If ping fails, restarts /var/service/dhcp_if service.
Basically, an example of watchdog service for networks which are not reliable
and need babysitting.

    var_service/supplicant_if

Wireless supplicant (wifi association and encryption daemon) service for
interface "if".

    var_service/fw

"Firewall" script, although it is tasked with much more than setting up firewall.
It is responsible for all aspects of network configuration.

This is an example of *one-shot* service.

It reconfigures network based on current known state of ALL interfaces.
Uses conf/*.ipconf (static config) and /var/run/service/fw/*.ipconf
(dynamic config from dhcp/ppp/vpn/etc) to determine what to do.

One-shot-ness of this service means that it shuts itself off after single run.
IOW: it is not a constantly running daemon sort of thing.
It starts, it configures the network, it shuts down, all done
(unlike infamous NetworkManagers which sit in RAM forever).

However, any dhcp/ppp/vpn or similar service can restart it anytime
when it senses the change in network configuration.
This even works while fw service runs: if dhcp signals fw to (re)start
while fw runs, fw will not stop after its execution, but will re-execute once,
picking up dhcp's new configuration.
This is achieved very simply by having
	# Make ourself one-shot
	sv o .
at the very beginning of fw/run script, not at the end.

Therefore, any "sv u fw" command by any other script "undoes" o(ne-shot)
command if fw still runs, thus runsv will rerun it; or start it
in a normal way if fw is not running.

This mechanism is the reason why fw is a service, not just a script.

System administrators are expected to edit fw/run script, since
network configuration needs are likely to be very complex and different
for non-trivial installations.

    var_service/ftpd
    var_service/httpd
    var_service/tftpd
    var_service/ntpd

Examples of typical network daemons.


	Process tree

Here is an example of the process tree from a live system with these services
(and a few others). An interesting detail are ftpd and vpnc services, where
you can see only logger process. These services are "downed" at the moment:
their daemons are not launched.

PID TIME COMMAND
553 0:04 runsvdir -P /var/service
561 0:00   runsv sshd
576 0:00     svlogd -tt /var/log/service/sshd
589 0:00     /usr/sbin/sshd -D -e -p22 -u0 -h /var/service/sshd/ssh_host_rsa_key
562 0:00   runsv dhcp_eth0
568 0:00     svlogd -tt /var/log/service/dhcp_eth0
850 0:00     udhcpc -vv --foreground --interface=eth0
                --pidfile=/var/service/dhcp_eth0/udhcpc.pid
                --script=/var/service/dhcp_eth0/dhcp_handler
                -x hostname bbox
563 0:00   runsv ntpd
573 0:01     svlogd -tt /var/log/service/ntpd
845 0:00     busybox ntpd -dddnNl -S ./ntp.script -p 10.x.x.x -p 10.x.x.x
564 0:00   runsv ifplugd_wlan0
598 0:00     svlogd -tt /var/log/service/ifplugd_wlan0
614 0:05     ifplugd -apqns -t3 -u0 -d0 -i wlan0
                -r /var/service/ifplugd_wlan0/ifplugd_handler
565 0:08   runsv dhcp_wlan0_pinger
911 0:00     sleep 67
566 0:00   runsv unscd
583 0:03     svlogd -tt /var/log/service/unscd
599 0:02     nscd -dddd
567 0:00   runsv dhcp_wlan0
591 0:00     svlogd -tt /var/log/service/dhcp_wlan0
802 0:00     udhcpc -vv -C -o -V  --foreground --interface=wlan0
                --pidfile=/var/service/dhcp_wlan0/udhcpc.pid
                --script=/var/service/dhcp_wlan0/dhcp_handler
569 0:00   runsv fw
570 0:00   runsv ifplugd_eth0
597 0:00     svlogd -tt /var/log/service/ifplugd_eth0
612 0:05     ifplugd -apqns -t3 -u8 -d8 -i eth0
                -r /var/service/ifplugd_eth0/ifplugd_handler
571 0:00   runsv zcip_eth0
590 0:00     svlogd -tt /var/log/service/zcip_eth0
607 0:01     zcip -fvv eth0 /var/service/zcip_eth0/zcip_handler
572 0:00   runsv ftpd
604 0:00     svlogd -tt /var/log/service/ftpd
574 0:00   runsv vpnc
603 0:00     svlogd -tt /var/log/service/vpnc
575 0:00   runsv httpd
602 0:00     svlogd -tt /var/log/service/httpd
622 0:00     busybox httpd -p80 -vvv -f -h /home/httpd_root
577 0:00   runsv supplicant_wlan0
627 0:00     svlogd -tt /var/log/service/supplicant_wlan0
638 0:03     wpa_supplicant -i wlan0
                -c /var/service/supplicant_wlan0/wpa_supplicant.conf -d

	A distro which already uses runit

I installed Void Linux, in order to see what do they have.
Xfce desktop looks fairly okay, network is up.
ps tells me they did put X, dbus, NM and udev into runsvdir-supervised tree:

    1 ?        00:00:01 runit
  623 ?        00:00:00   runsvdir
  629 ?        00:00:00     runsv
  650 tty1     00:00:00       agetty
  630 ?        00:00:00     runsv
  644 ?        00:00:09       NetworkManager
 1737 ?        00:00:00         dhclient
  631 ?        00:00:00     runsv
  639 tty4     00:00:00       agetty
  632 ?        00:00:00     runsv
  640 ?        00:00:00       sshd
 1804 ?        00:00:00         sshd
 1809 pts/3    00:00:00           sh
 1818 pts/3    00:00:00             ps
  633 ?        00:00:00     runsv
  637 tty5     00:00:00       agetty
  634 ?        00:00:00     runsv
  796 ?        00:00:00       dhclient
  635 ?        00:00:00     runsv
  649 ?        00:00:00       uuidd
  636 ?        00:00:00     runsv
  647 ?        00:00:00       acpid
  638 ?        00:00:00     runsv
  652 ?        00:00:00       console-kit-dae
  641 ?        00:00:00     runsv
  651 tty6     00:00:00       agetty
  642 ?        00:00:00     runsv
  660 tty2     00:00:00       agetty
  643 ?        00:00:00     runsv
  657 ?        00:00:02       dbus-daemon
  645 ?        00:00:00     runsv
  658 ?        00:00:00       cgmanager
  648 ?        00:00:00     runsv
  656 tty3     00:00:00       agetty
  653 ?        00:00:00     runsv
  655 ?        00:00:00       lxdm-binary
  698 tty7     00:00:14         Xorg
  729 ?        00:00:00         lxdm-session
  956 ?        00:00:00           sh
  982 ?        00:00:00             xfce4-session
 1006 ?        00:00:04               nm-applet
  654 ?        00:00:00     runsv
  659 ?        00:00:00       udevd

Here is a link to Void Linux's wiki:

    https://wiki.voidlinux.eu/Runit

Void Linux packages install their services as subdirectories of /etc/rc,
such as /etc/sv/sshd, with a script file, "run", and a link
"supervise" -> /run/runit/supervise.sshd

For sshd, "run" contains:

    #!/bin/sh
    ssh-keygen -A >/dev/null 2>&1 # generate host keys if they don't exist
    [ -r conf ] && . ./conf
    exec /usr/bin/sshd -D $OPTS

That's it from the POV of the packager.

This is pretty minimalistic, and yet, it is already distro-specific:
the link to /run/runit/* is conceptually wrong, it requires packagers
to know that /etc/rc should not be mutable and thus they need to use
a different location in filesystem for supervise/ directory.

I think a good thing would be to require just one file: the "run" script.
The rest should be handled by distro tooling, not by packager.

A similar issue is arising with logging. It would be ideal if packagers
would not need to know how a particular distro manages logs.
Whatever their daemons print to stdout/stderr, should be automagically logged
in a way distro prefers.

* * * * * * * *

	Proposed "standard" on how distros should use runit

The original idea of services-as-directories belongs to D.J.Bernstein (djb),
and his project to implement it is daemontools: https://cr.yp.to/daemontools.html

There are several reimplementations of daemontools:
- runit: by Gerrit Pape, http://smarden.org/runit/
  (busybox has it included)
- s6: by Laurent Bercot, http://skarnet.org/software/s6/

It is not required that a specific clone should be used. Let evolution work.


	Terminology

daemon: any long running background program. Common examples are sshd, getty,
ntpd, dhcp client...

service: daemon controlled by a service monitor.

service directory: a directory with an executable file (script) named "run"
which (usually) execs some daemon, possibly after some preparatory steps.
It should start it not as a child or daemonized process, but by exec'ing it
(inheriting the same PID and the place in the process tree).

service monitor: a tool which watches a set of service directories.
In daemontools package, it is called "svscan". In runit, it is called
"runsvdir". In s6, it is called "s6-svscan".
Service monitor starts a supervisor for each service directory.
If it dies, it restarts it. If service directory disappears,
service monitor will not be restarted if it dies.
runit's service monitor (runsvdir) sends SIGTERM to supervisors
whose directories disappeared.

supervisor: a tool which monitors one service directory.
It runs "run" script as its child. It restarts it if it dies.
It can be instructed to start/stop/signal its child.
In daemontools package, it is called "supervise". In runit, it is called
"runsv". In s6, it is called "s6-supervise".

Conceptually, a daemontools clone can be designed such that it does not *have*
the supervisor component: service monitor can directly monitor all its daemons
(for example, this may be a good idea for memory-constrained systems).
However all three existing projects (daemontools/runit/s6) do have a per-service
supervisor process.

log service: a service which is exclusively tasked with logging
the output of another service. It is implemented as log/ subdirectory
in a service directory. It has the same structure as "normal"
service dirs: it has a "run" script which starts a logging tool.

If log service exists, stdout of its "main" service is piped
to log service. Stops/restarts of either of them do not sever the pipe
between them.

If log service exists, daemontools and s6 run a pair of supervisors
(one for the daemon, one for the logger); runit runs only one supervisor
per service, which is handling both of them (presumably this is done
to use fewer processes and thus, fewer resources).


	User API

"Users" of service monitoring are authors of software which has daemons.
They need to package their daemons to be installed as services at package
install time. And they need to do this for many distros.
The less distros diverge, the easier users' lives are.

System-wide service dirs reside in a distro-specific location.
The recommended location is /var/service. (However, since it is not
a mandatory location, avoid depending on it in your run scripts.
Void Linux wanted to have it somewhere in /run/*, and they solved this
by making /var/service a symlink).

The install location for service dirs is /etc/rc:
when e.g. ntpd daemon is installed, it creates the /etc/rc/ntpd
directory with (minimally) one executable file (script) named "run"
which starts ntpd daemon. It can have other files there.

At boot, distro should copy /etc/rc/* to a suitable writable
directory (common choice are /var/service, /run/service etc).
It should create log/ directories in each subdirectory
and create "run" files in them with suitable (for this particular distro)
logging tool invocation, unless this directory chose to channel
all logging from all daemons through service monitor process
and log all of them into one file/database/whatever,
in which case log/ directories should not be created.

It is allowable for a distro to directly use /etc/rc/ as the only
location of its service directories. (For example,
/var/service may be a symlink to /etc/rc).
However, it poses some problems:

(1) Supervision tools will need to write to subdirectories:
the control of running daemons is implemented via some files and fifos
in automatically created supervise/ subdirectory in each /etc/rc/DIR.

(2) Creation of a new service can race with the rescanning of /etc/rc/
by service monitor: service monitor may see a directory with only some files
present. If it attempts to start the service in this state, all sorts
of bad things may happen. This may be worked around by various
heuristics in service monitor which give new service a few seconds
of "grace time" to be fully populated; but this is not yet
implemented in any of three packages.
This also may be worked around by creating a .dotdir (a directory
whose name starts with a dot), populating it, and then renaming;
but packaging tools usually do not have an option to do this
automatically - additional install scripting in packages will be needed.

Daemons' output file descriptors are handled somewhat awkwardly
by various daemontools implementations. For example, for runit tools,
daemons' stdout goes to wherever runsvdir's stdout was directed;
stderr goes to runsvdir, which in turn "rotates" it on its command line
(which is visible in ps output).

Hopefully this get changed/standardized; while it is not, the "run" file
should start with a

    exec 2>&1

command, making stderr equivalent to stdout.
An especially primitive service which does not want its output to be logged
with standard tools can do

    exec >LOGFILE 2>&1

or even

    exec >/dev/null 2>&1

To prevent creation of distro-specific log/ directory, a service directory
in /etc/rc can contain an empty "log" file.


	Controlling daemons

The "svc" tool is available for admins and scripts to control services.
In particular, often one service needs to control another:
e.g. ifplugd can detect that the network cable was just plugged in,
and it needs to (re)start DHCP service for this network device.

The name of this tool is not standard either, which is an obvious problem.
I propose to fix this by implementing a tool with fixed name and API by all
daemontools clones. Lets use original daemontools name and API. Thus:

The following form must work:

	svc -udopchaitkx DIR

Options map to up/down/once/STOP/CONT/HUP/ALRM/INT/TERM/KILL/exit
commands to the daemon being controlled.

The form with one option letter must work. If multiple-option form
is supported, there is no guarantee in which order they take effect:
svc -it DIR can deliver TERM and INT in any order.

If more than one DIR can be specified (which is not a requirement),
there is no guarantee in which order commands are sent to them.

If DIR has no slash and is not "." or "..", it is assumed to be
relative to the system-wide service directory.

[Currently, "svc" exists only in daemontools and in busybox.
This proposal asks developers of other daemontools implementations
to add "svc" command to their projects]

The "svok DIR" tool exits 0 if service supervisor is running
(with service itself either running or stopped), and nonzero if not.

Other tools with different names and APIs may exist; however
for portability scripts should use the above tools.

Creation of a new service on a running system should be done atomically.
To this end, first create and populate a new /etc/rc/DIR.

Then "activate" it by running ??????? - this copies (or symlinks,
depending on the distro) its files to the "live" service directory,
wherever it is located on this distro.

Removal of the service should be done as follows:
svc -d DIR [DIR/log], then remove the service directory:
this makes service monitor SIGTERM per-directory supervisors
(if they exist in the implementation).


	Implementation details

Top-level service monitor program name is not standardized
[svscan, runsvdir, s6-svscan ...] - it does not need to be,
as far as daemon packagers are concerned.

It may run one per-directory supervisor, or two supervisors
(one for DIR/ and one for DIR/log/); for memory-constrained systems
an implementation is possible which itself controls all services, without
intermediate supervisors.
[runsvdir runs one "runsv DIR" per DIR, runsv handles DIR/log/ if that exists]
[svscan runs a pair of "supervise DIR" and "supervise DIR/log"]

Directories are remembered by device+inode numbers, not names. Renaming a directory
does not affect the running service (unless it is renamed to a .dotdir).

Removal (or .dotdiring) of a directory sends SIGTERM to any running services.

Standard output of non-logged services goes to standard output of service monitor.
Standard output of logger services goes to standard output of service monitor.
Standard error of them always goes to standard error of service monitor.

If you want to log standard error of your logged service along with its stdout, use
"exec 2>&1" in the beginning of your "run" script.

Whether stdout/stderr of service monitor is discarded (>/dev/null)
or logged in some way is system-dependent.


	Containers

[What do containers need?]