Configure supervisord (logging, pidfile, ...)¶
We handle all logging through Supervisord, so you will probably at least want to configure where we log to.
Supervisor variables are now iniatially defined in the variables section in buildout-base.cfg and can be overridden by adding a variables section in your own buildout config file. Logging directory and rotation parameters should be redefined accordingly:
[variables]
# The full path to the supervisord log directory.
# Defaults to /path/to/devilrybuild/var/log/
# Note: This setting is added by our buildout-base.cfg, and not by the
# supervisor buildout recipe.
#logdir =
# Where logs are placed on the filesystem
logdir = ${buildout:directory}/var/log
# Max number of bytes in a single log
logfile-maxbytes = 50MB
# Number of times a log rotates - note that each program running under
# supervisor has 2 logs (stdout and stderr), and each log will consume
# ``logfile-maxbytes * logfile-backups`` of disk space.
logfile-backups = 30
For more information on how supervisord deals with rotation see: http://supervisord.org/logging.html#activity-log-rotation
Details. What is being logged and where?¶
The different modules that are configured is running as a subprocess within `supervisord and will be captured by supervisord. To mantain full control of the logging environment it is important to configure each submodule in order to avoid supervisord AUTO log mode in which every log is stored on temporary terms. Every configuration parameter will be included in the final supervisord.conf file in is [program:x] block. We never edit the configure directly but tunes the parameters from within the buildout files. So in buildout-base.cfg each process are configured with the extracted variables from the [variables] section you defined above. Check out the file buildout-base.cfg and look for ${variables:x} where x is the name of the variable you have redefined.
As an example we look into gunicorn configuration:
[gunicorn]
workers = 8
executable = ${variables:django_managepy}
address = 127.0.0.1
port = 8002
args = run_gunicorn -b ${gunicorn:address}:${gunicorn:port} -w ${gunicorn:workers} --timeout 180
cwd = ${buildout:directory}
logfile-maxbytes = ${variables:logfile-maxbytes}
logfile-backups = ${variables:logfile-backups}
logoptions = stdout_logfile=${variables:logdir}/gunicorn.stdout.log stdout_logfile_maxbytes=${gunicorn:logfile-maxbytes} stdout_logfile_backups=${gunicorn:logfile-backups} stderr_logfile=${variables:logdir}/gunicorn.stderr.log stderr_logfile_maxbytes=${gunicorn:logfile-maxbytes} stderr_logfile_backups=${gunicorn:logfile-backups}
- executable = ${variables:django_managepy}
- This is often not redefined. Gunicorn is started by using the run_gunicorn command to the django managment script to ensure that gunicorn is running with correct django setup.
- logfile-maxbytes = ${variables:logfile-maxbytes}
- Defined as the maximum number of bytes logged
- logfile-backups = ${variables:logfile-backups}`
- Defined as the number of rotations for each logged reaching logfile-maxbytes stored data
- logoptions = stdout_logfile=${variables:logdir}/gunicorn.stdout.log
- This is the options to the gunicorn application and directly affects the gunicorn logging behaviour within supervisord. every application setup for logging will have the same logfile naming convention where the logdir is defined and then the name will consist of <appname>.<stream>.log Both error and standard out streams are configured.
Gunicorn¶
Serves the the Devilry site to the outgoing server. Gunicorn is a WSGI server to be used behind a HTTP proxy server for incoming request-managment.
What is logged ?¶
Celery Beat¶
Celery Beat is the scheduler that starts the given tasks at defined intervals
What is logged ?¶
Run_Solr¶
This is a buildout recipe for creating a run_solr.sh file. A script that ease the Apache Solr configuration for use with HayStack. It is the search engine of Devilry.
What is logged ?¶
Rebuild the Supervisord config (output in parts/supervisor/supervisord.conf):
$ bin/buildout
And restart supervisord.
See the Buildout recipe and the Supervisord docs for more details.
Init script¶
The following init script works well. You need to adjust the DAEMON-variable (download):
#! /bin/sh
### BEGIN INIT INFO
# Provides: supervisord
# Required-Start: $remote_fs
# Required-Stop: $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Example initscript
# Description: This file should be used to construct scripts to be
# placed in /etc/init.d.
### END INIT INFO
# Author: Dan MacKinlay <danielm@phm.gov.au>
# Based on instructions by Bertrand Mathieu
# http://zebert.blogspot.com/2009/05/installing-django-solr-varnish-and.html
# See: https://gist.github.com/176149
# Do NOT "set -e"
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Description of the service"
NAME=supervisord
DAEMON=/usr/local/bin/supervisord
DAEMON_ARGS=""
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
#
# Function that starts the daemon/service
#
do_start()
{
# Return
# 0 if daemon has been started
# 1 if daemon was already running
# 2 if daemon could not be started
start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
|| return 1
start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
$DAEMON_ARGS \
|| return 2
# Add code here, if necessary, that waits for the process to be ready
# to handle requests from services started subsequently which depend
# on this one. As a last resort, sleep for some time.
}
#
# Function that stops the daemon/service
#
do_stop()
{
# Return
# 0 if daemon has been stopped
# 1 if daemon was already stopped
# 2 if daemon could not be stopped
# other if a failure occurred
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
RETVAL="$?"
[ "$RETVAL" = 2 ] && return 2
# Wait for children to finish too if this is a daemon that forks
# and if the daemon is only ever run from this initscript.
# If the above conditions are not satisfied then add some other code
# that waits for the process to drop all resources that could be
# needed by services started subsequently. A last resort is to
# sleep for some time.
start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
[ "$?" = 2 ] && return 2
# Many daemons don't delete their pidfiles when they exit.
rm -f $PIDFILE
return "$RETVAL"
}
#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
#
# If the daemon can reload its configuration without
# restarting (for example, when it is sent a SIGHUP),
# then implement that here.
#
start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
return 0
}
case "$1" in
start)
[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
do_start
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
#reload|force-reload)
#
# If do_reload() is not implemented then leave this commented out
# and leave 'force-reload' as an alias for 'restart'.
#
#log_daemon_msg "Reloading $DESC" "$NAME"
#do_reload
#log_end_msg $?
#;;
restart|force-reload)
#
# If the "reload" option is implemented then remove the
# 'force-reload' alias
#
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*)
# Failed to stop
log_end_msg 1
;;
esac
;;
*)
#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
exit 3
;;
esac
:
Harden supervisord¶
The default configuration if for a dedicated server. Supervisorctl uses a password with the local Supervisord server, which needs to be a better password in a shared environment. This should not be a problem since it is madness to host Devilry on a shared host in any case, but if you need to harden Supervisord, refer to the docs linked above.