Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Restarting a script after some time without output
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Restarting a script after some time without output

edfoxedfox Member

Hello,
I have a piece of software that's on life-support waiting to be replaced by another one I'm finishing writing. I'm experiencing random hangups, where the software will just stop and not do anything, until i ^C it and restart.

Is there a software, or bash script, or whatever, to kill and restart process after x minutes have elapsed with no output?

Comments

  • ITLabsITLabs Member
    edited August 2019

    If the script crashes perhaps you can use ps as a quick hack:

    scriptcheck.sh
    OKORNOT=$(ps -A | grep -w process_name)
    
    if ! [ -n "$OKORNOT" ] ; then
        /path/to/my/script/restartproc.sh &
        exit
    fi
    

    Put scriptcheck.sh on a cron. Inside restartproc.sh write the logic to (re)start your script.

  • @ITLabs said:
    If the script crashes perhaps you can use ps as a quick hack:

    Hanging != crashing
    The script is in execution, but just idling. Waiting forever

  • @edfox said:

    @ITLabs said:
    If the script crashes perhaps you can use ps as a quick hack:

    Hanging != crashing
    The script is in execution, but just idling. Waiting forever

    Yep... I was just thinking about that. What type of output should the script produce, text, DB etc.?

  • rm_rm_ IPv6 Advocate, Veteran
    edited August 2019

    edfox said: Is there a software, or bash script, or whatever, to kill and restart process after x minutes have elapsed with no output?

    You could redirect the output of your program to a file, and then write a separate script that would check the size of that file, say every 5 minutes (from crontab), to kill and restart the program if the size didn't change since last time.

    Thanked by 2ITLabs uptime
  • I'm wondering if supervisord might be useful here - if suitable for the purpose and not overkill

    http://supervisord.org/events.html#event-listeners-and-event-notifications

  • @ITLabs said:

    Yep... I was just thinking about that. What type of output should the script produce, text, DB etc.?

    Right now it just produces useless log for troubleshooting. No need to store anything, just to check that is keeps producing it.

  • uptimeuptime Member
    edited August 2019

    this (linux bash) code snippet may be useful - modified from a stackoverflow answer

    killtime=300 # 5 minutes
    deadman="/path/to/logfile"
    if [ "$(( $(date +"%s") - $(stat -c "%Y" "$deadman") ))" -gt $killtime ]; then
       echo "WAKE UP LAZYBONES"
       # restart your script here
    fi
    
    Thanked by 1ITLabs
  • In addition to the solution proposed by @uptime you can also check Monit.

    https://mmonit.com/monit/documentation/monit.html#Process

  • Just output something to a tmp file. with second script monitor that file and if it is older than X than consider your main script hasn't spit any output. Something like that:

    dog.sh

    executing some sh > /tmp/my_tmp.tmp

    watchdog.sh

    !/bin/bash

    FILE="$1"
    CMD="$2"
    LAST=ls -l "$FILE"
    while true; do
    sleep 1
    NEW=ls -l "$FILE"
    if [ "$NEW" != "$LAST" ]; then
    "$CMD" "$FILE"
    LAST="$NEW"
    fi
    done

    Now CRON that mofo:

    ./watchdog.sh /tmp/my_tmp.tmp "dog.sh restart"

    This is abstract, but you get it.

Sign In or Register to comment.