Have you ever in a situation where you have multiple backend servers and your cronjob is scheduled in one server and it didn’t run when that server went down OR you have a Docker Swam based cluster where you have several worker nodes and you do not want the cronjob to execute more than once from all these worker nodes?
I ran into the latter when I originally implemented Docker Swarm across our IT projects. We treat all our worker nodes as disposable, as we have auto-scale system in the AWS and as soon as it finds one of the worker nodes offline, it will terminate the server and stand up a new. Our custom script will detect this and add this new worker node to the docker swarm cluster automatically. Hence, any cronjobs that were removed when the previous server went down, will be restored as well.
So I had to come up with a script that can make sure that the cronjobs don’t start from all the worker nodes. Since we had a common NFS drive connected across all these worker nodes, I’ve decided to make use of that partition. There are other possible methods such as using a redis, memcache etc. But this would appear to be reliable from my experience and zero additional cost.
#!/bin/bash
# Cronlock - A simple to use locking mechanism for cronjobs in a distributed environment
# Author - Vivek Vaskuttan
# Email - vivekv@vivekv.com
# Website - www.vivekv.com
# Usage
#------
# * * * * * /usr/bin/cronlock logKey ping google.com
# * * * * * cd /var/www/; /usr/bin/cronlock /usr/bin/php logKey test.php
#
# where,
# logKey is a filename where the output of the cron will be saved
# ping google.com is the command that needs to be executed
# Configuration
# LOCK_DIR and LOG_DIR must be a shared volume between the servers for this to work
# LOG_DIR/last_successful_start file is created with the timestamp when the cron starts executing
# LOG_DIR/last_successful_exit file is created when cron completes executing the command
LOCK_DIR=/mnt/volume/logs/lock
LOG_DIR=/mnt/volume/logs/cron
MD5=`echo -n "$@" | md5sum | awk '{print $1}'`
COMMAND="${@:2}"
mkdir -p ${LOG_DIR}
mkdir -p ${LOCK_DIR}
( /usr/bin/flock -n -x 200 && date > ${LOG_DIR}/last_successful_start_$1.log && ${COMMAND} && date > ${LOG_DIR}/last_successful_exit_$1.log ) 2>&1 200> ${LOCK_DIR}/${MD5} | while read line ; do echo `date` "- $line" ; done >> ${LOG_DIR}/$1.log
Download source code and examples from Github https://github.com/getvivekv/cronlock