Kill and restart a process everyday

sundaymouse

Hello.

I have a python script that needs to be kept running consistently, sometimes will crash and not deployable as a system service. In order to keep it running for as much time as possible, I have deployed a cron that restarts it unconditionally at 2:01 am every day, no matter if it's crashed or not.

The python script is placed at /usr/bin/(scriptprocessname) and is executable.

Currently my crontab (on ubuntu) is set as followed:

1 2 * * * /etc/restart.sh

/etc/restart.sh is executable, and is as follows:

#!/bin/sh
killall (scriptprocessname)
nohup (scriptprocessname) -(arguments) 2>1 &

with "(scriptprocessname)" as the name of process and "-(arguments)" as the arguments.

However, there is a problem. As according to the log of cron, if the python script has already crashed before the cronjob being executed, /etc/restart.sh will return "killall: no such process" and stuck there, without continuing on to start the python script again.

I have very little experience in bash scripting. I wish someone could provide an effective way to kill&restart a process no matter it is currently running or not. Thanks.

rm_

I don't see why you would unconditionally kill and restart it, and just once a day at that, if the aim is to ensure that it just runs always. And what if it crashes at 03am, the next 23 hours you are left without your script running?

Something along the lines of:

*/5 * * * * ps -Af | grep your-python-script.py | grep -v grep || /path/to/your-python-script.py

will check every 5 minutes and start it if it's not running.

MarkTurner

Use Supervisord http://supervisord.org/installing.html

You can setup the script to automatically restart if it crashes and you could always add in a mail notification upon restart.

sundaymouse

@rm_ @MarkTurner
Sorry I forgot to mention another purpose.

This script reads from a configuration file, and sometimes the config will be updated on a day sometime just before 2 am, so I do wish to get it restarted at 2 am daily. The script itself just forwards some commands to a target server, and does not carry data in the RAM, so I also with to restart it at 2 am anyway.

I might just cron a kill at 2:01 am everyday, and leave another cronjob as introduced by @rm_ to pick it up a little later.

Thank you.

MarkTurner

So setup up supervisord to monitor the config file.

petris

Use monit with a pid file. It will restart your script within 1 minute of dying.

sc754

!/bin/bash

service=myservicename

if (( $(ps -ef | grep -v grep | grep $service | wc -l) > 0 ))

then

echo "$service is running...."

else

/etc/init.d/$service start

fi

Run something like that every minute from cron, if it's running it'll do nothing but return $service is running (or remove that if you want and just put exit instead) and if it's not running it'll reboot it.

rm_

sc754 said: /etc/init.d/$service start

Worth noting that with most of the "proper" services you should be able to just blindly execute "start", and it won't launch dozens of copies of them, it'll check if the service is already running and refuse to start it again if it is. And this will be done very quickly and with not a lot of extra load, launching those two greps is almost slower than that:

# time /etc/init.d/tor start
[ ok ] Starting tor daemon...done (already running).

real    0m0.046s
user    0m0.020s
sys 0m0.004s

But in this thread we're talking about just a raw script that needs to be running, not a service wrapped into a real init.d script.