Abstract
The purpose of this tutorial is to get you up and running with runit as quickly as possible. According to the official documentaion, runit is "a UNIX init scheme with service supervision". Basically this means that runit provides:
- Automatic starting of services when the system starts
- Automatic monitoring and restarting of services if they terminate
This tutorial will describe the basic usage of the utilities included in the runit package by providing a template to bootstrap the management of new services as well as providing a simple example service. The purpose of process supervision is to automate the task of keeping your infrastrucutre up and running. If you are serious about creating a robust environment, you should use this tutorial as a starting point to get you over the initial hump associated with learning a new tool and then continue by checking out the official documentation and performing further experimentation.
Official Documentation: http://smarden.org/runit/
What You Will Learn
- How to create a runit template used to configure new montitored services
- How to setup your first monitored service
- How to manage monitored services manually
0. Prerequisites
Make sure that you have runit installed on your system. Most linux distributions should have a runit package available in their repositories. For example, if you are running a Debian based release you can install runit by executing the following command:
# apt-get install runit
Note: Snippets starting with the '#' character represent command you should enter in your shell excluding the '#'.
To verify that you have successfully installed runit and it is up and running, look for a process running the runsvdir command by executing the following:
# ps -ef | grep runsvdir
The output of the command should be similar to this:
root 2783 1 0 15:34 ? 00:00:00 runsvdir -P /etc/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
If you see a runsvdir process then continue on. If not ensure that runit was installed properly by checking the documentation for you distro. The runsvdir utility is the first of several that we will encounter in this tutorial. The design of runit takes a very "Unixy" approach by breaking down functionality into several small utilities each responsible for a single task. This approach allows the simple components to be composed in various ways to suit you needs. The core runit utilities are runsvdir, runsv, chpst, svlogd, and sv. Throughout this tutorial we will combine these utilities to configure and run a managed service.
1. Creating a Template
The first thing we need to do is take a closer look at what the runsvdir -P /etc/service log:....... command is doing. Bascially, this command is watching the /etc/service directory for files used to configure a monitored service. A monitored service is configured by adding a subdirecotry to /etc/service with a run script in it. When runsvdir finds a new service configuration, it starts a new runsv process to manage the service. As far as runsvdir is concerned, there really is not much more to it that that. Remember that runit's philosophy is to provide simple utilities with a single responsibility. Check out the man page for further details:
# man runsvdir
Make sure that the /etc/service directory exists and if it does not, create it with the following command:
# mkdir /etc/service
Rather than adding our configuration files directly to the /etc/service directory, let's create a staging directory that we can use to develop and test our configurations. Once we are happy with our service configuration, we can deploy it by creating a symbolic link to it from the /etc/service directory. You can use any directory that you like for staging, but I like to use /etc/runit. Go ahead and create your staging directory now.
# mkdir /etc/runit
Before creating our first real service configuration, let's create a template. This template will illustrate the basics of using runit and can be used to quickly create new configurations in the future. Create a directory for our template by running this command:
# mkdir /etc/runit/template
Next, create a script named run in the template directory with the following contents:
#!/bin/sh -e
exec 2>&1
exec chpst -u USER COMMAND
Make the run script executable by running:
# chmod +x /etc/runit/template/run
This script is quite simple. First it redirects all output from stderr to stdout. Next, it executes some COMMAND as the given USER with the chpst utility. The chpst utility allows us to configure how the command will be executed. Because the run script will be executed as the root user, we can use chpst to change the context to a normal user with limited permissions. chpst has many more options for configuring the context in which to execute a command. Check out the man page for more info.
# man chpst
When the runsvdir process detects a new directory in /etc/service it will start a new runsv process that will be responible for executing and monitoring the run script. Take a look at the runsv man page for more details.
# man runsv
You may have noticed that the man page for runsv mentions that it can optionally start an appendant log service. Because logging is such a critical part of monitoring an environment, let's create a template script for logging the output of our monitored process.
To setup logging for our service, create a new log directory by running the following command:
# mkdir /etc/runit/template/log
In the new directory create a script named run with the following contets:
#!/bin/sh
exec chpst -u USER svlogd -tt LOGDIR
Make the run script executable by running the following command:
# chmod +x /etc/runit/template/log/run
This script uses chpst to start a svlogd daemon as the given USER wirting logs to the LOGDIR directory. For details on logging configuration, check out the svlogd man page.
# man svlogd
When runsvdir notices a monitor configuration in a new directory under /etc/service, it looks for a sub directory named log. If it finds one it starts a new runsv process that will execute and monitor the run script in the log directory.
2. An Example Service
Let's take our template and create a simple example service configuration. First, create a new user that will be used by chpst in the run script. You can name the user anything you want.
# adduser foo
Next, let's create a directory to host the service and change its ownership to our new user. You can choose any directory that you like. I like to setup applications in the opt directory.
# mkdir /opt/example
# chown foo:foo /opt/example
Now that we have a home for our service, let's create a simple script that will stand in for a real service. Change to the new foo user:
# su foo
Create a new file in /opt/example called foo-service.sh and add the following contents:
#!/bin/bash
echo "Started service..."
for i in {1..30}
do
echo "Doing stuff..."
sleep 1
done
echo "Oh no I crashed..." >&2
exit 1
Make this script executable by executing this command:
# chmod +x /opt/example/foo-service.sh
This script will simulate a real application that crashes periodically by logging some information for 30 seconds, logging an error to stderr, and then exiting with a non zero status. Try it out by executing:
# /opt/example/foo-service.sh
Next, create a directory that will contain the logs for the service.
# mkdir /opt/example/logs
That's everthing we need to do as the foo user, so switch back to root by executing:
# exit
Now we are ready to use the template we defined to configure runit to monitor our example service. Create the configuration in our staging directory by copying the template to a new directory named example.
# cp -R /etc/runit/template /etc/runit/example
Update the /etc/runit/example/run script to execute the foo-service.sh script as the foo user.
#!/bin/sh -e
exec 2>&1
exec chpst -u foo /opt/example/foo-service.sh
Update the /etc/runit/example/log/run log script log to the /etc/example/logs directory as the foo user.
#!/bin/sh
exec chpst -u foo svlogd -tt /opt/example/logs
Before we deploy the service to /etc/service, let's test it to make sure everything is setup properly.
# /etc/runit/example/run
If everything is ok, you should see the script log output for 30 seconds, log an error and then terminate.
Finally, we are ready to deploy the service. Create a symbolic link in /etc/service to our staging location.
# ln -s /etc/runit/example /etc/service/example
The runsvdir process should have noticed our new monitor configuration and started two runsv processes to execute the service and run the log daemon. To verify that everything is working, check the status of our service.
# sv status example
You should see output similar to what is shown below to indicate the service and logger are running.
run: example: (pid 3483) 3s; run: log: (pid 3324) 154s
You can watch the output of the service by tailing the log.
# tail -f /opt/example/logs/current
You should see the service logging output, logging an error, and then starting over as the runsv process notices the process terminate and restarts it.
3. Managing Services
Finally, let's take a look at the sv utility. sv is used to manually manage your services.
To check the status of the example service run:
# sv status example
To stop the example service run:
# sv stop example
You will notice if you watch the logs that the service is no longer writing any output. Once you stop a service with sv, it will not be restarted automatically until you explicitly restart it.
To start the service again run:
# sv start example
To stop and then start a service run:
# sv restart example
Make sure to check out the sv man page to see additional options.
# man sv
Start Your Monitors!!!
With what you have learned in this tutorial, you should now be able to configure services to be monitored with runit. Try using the template to monitor a web application, some web services, or your CI server.
Contact Me
If you have any questions or would like further clarification, please feel free to shoot me an email. I would be glad to help.
Kevin