Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
OpenStatus Server Monitoring System Release
New on LowEndTalk? Please Register and read our Community Rules.

OpenStatus Server Monitoring System Release

NickMNickM Member
edited October 2011 in General

For a few weeks now, I've been building upon DimeCadmium's scrd and nikkiii's status page to build a more comprehensive server monitor system that is easy for everyone to deploy to their LEBs. I figured that the LET community would like it, so I'm announcing the release here :)

OpenStatus adds process monitoring, an interface for adding servers, an improved alerting system, and a push-based updating system, as opposed to the polling-based system of scrd. OpenStatus Server’s web interface allows system administrators to view all of the information at a glance. It also sends alerts via email when a server reports high memory or disk usage, high load, a monitored service is down, or a server is completely unreachable. OpenStatus Client is the portion of OpenStatus that goes on the servers you’d like to monitor. It does the collection of all of the requested information, and sends it to OpenStatus Server.

I've set up a Debian repository for OpenStatus, to make installation a breeze. Click here for install instructions. A demo is available here. GitHub repo available here.

I've tested it extensively, but, as with all software releases, there are bound to be bugs that I just haven't encountered, so please, if you find a bug (or just plain don't like it), let me know (use GitHub's issue tracker, or post here). The great thing about using a Debian repo is that it makes it quick and easy for you to get bugfix updates as soon as they're available.

Enjoy!

«1

Comments

  • Very nice, I'll review the code and deploy this on test.excloo.com ^^

  • This looks very cool. I guess I might be the test dummy and try it out tonight.

  • fanfan Member

    It's really much faster than manually deploy everything! Thanks for this script and is there any plan to add a simple ping monitor like http://buyvmstatus.com/?

  • Wow, that's simply amazing stuff! And also made it available as Debian repo =) Very nice work , thank you, will be trying it. Keep it up :)

    ☻☻ VPS ☺ as of now:- 384-256-128-512x2 ☺☺

  • Debian 5:

    The following packages have unmet dependencies:
      openstatus-client: Depends: python2.6 but it is not installable
                         Depends: python-setproctitle but it is not installable
    E: Broken packages
    

    It looks like python2.6 is not available for Debian 5?

  • it is, but you have to install it from source

    DomainAgent - a smart tool for keeping details of domains you own. We're on twitter too!
  • I keep getting "Unable to connect to the database - please try again later."

    What am I doing wrong now?

  • trelawneytrelawney Member
    edited October 2011

    What @vedran said. On Squeeze and it will not connect to db (despite correct ownership/location etc).

    LaaS (love as a service) : e.g. https://aga.pw/Lowendtalk (greek only)

  • Both, server and client, packages work fine on Debian Squeeze for me. NickM, it would be good to add support for Debian Lenny too ;)

  • @trelawney and @vedran: Check your nginx error log, it should give you more information about the error.

    @fan: Good idea. I should definitely be able to add a check for ping response time.

    @iKocka: I will be setting up a Debian Lenny box later tonight to test on. Hopefully, it will work fine with python 2.5, which is Lenny's default python, since 2.6 is not packaged for Lenny.

  • ... @ vedran.. i think i just had to restart /init.d/php-cgi , it worked after that :)

    LaaS (love as a service) : e.g. https://aga.pw/Lowendtalk (greek only)

  • Yes, it works now. apt-get install sqlite php5-sqlite and restarting php-cgi did the trick, but I'm not sure which part :D

  • vedran said: Yes, it works now. apt-get install sqlite php5-sqlite and restarting php-cgi did the trick, but I'm not sure which part :D

    Likely both. The php5-sqlite module is required, and when installing php modules while running php-cgi, a restart of php-cgi is required. I'll add a note about that to the install instructions.

  • NickM, how do I delete or edit a server? I accidentally reloaded the page and now I have 2 servers with the same name

  • vedran said: NickM, how do I delete or edit a server? I accidentally reloaded the page and now I have 2 servers with the same name

    Delete server from openstatus.db which is located in /etc/openstatus/.

  • @vedren and @iKocka I've made a version that works with python 2.5 on Debian Lenny. I've only tested it a little bit, but everything seems to work.

    Replace deb http://deb.nickmoeck.com/debian stable main with deb http://deb.nickmoeck.com/debian oldstable main then run apt-get clean and apt-get update, then install. Let me know if you notice any further issues.

  • Thanks for the update @NickM

    It installs on Lenny now. Once installed, the client runs as

    /usr/bin/python /usr/bin/openstatus-client

    While on Squeeze it runs as just openstatus-client.

    If you try /etc/init.d/openstatus-client restart (Lenny), it won't detect the client and it will just start another instance.

    Also, clients can't connect to server (both Lenny and Squeeze), logs keep showing "Could not connect to server: Connection refused", I've set the server IP and port correctly in the config files and iptables are not blocking the traffic. What else can be wrong?

  • NickM said: @vedren and @iKocka I've made a version that works with python 2.5 on Debian Lenny. I've only tested it a little bit, but everything seems to work.

    It doesn't work for me :(

  • vedran said: If you try /etc/init.d/openstatus-client restart (Lenny), it won't detect the client and it will just start another instance.

    Ah yes, that's because setproctitle isn't available for python 2.5 on Lenny, which I didn't take into consideration when adjusting the init script. Will be fixed in an update tomorrow.

    vedran said: Also, clients can't connect to server (both Lenny and Squeeze), logs keep showing "Could not connect to server: Connection refused", I've set the server IP and port correctly in the config files and iptables are not blocking the traffic. What else can be wrong?

    I'm not sure what the problem could be here - I've tested it connecting to servers running both the Lenny and Squeeze versions of -server. My tests were done on a clean install of Lenny... perhaps something is conflicting. Can you give me a bit more information about your setup so I can try to replicate the problem? Also, if you don't mind, try setting the server IP to 96.8.120.80, port 12999, and see if the problem persists (it's the server I'm using for the demo, so I know that one works). If it still gives you "Connection refused", the issue is in the client part.

    iKocka said: It doesn't work for me :(

    Can you be more specific? What doesn't work?

    As soon as I can resolve these issues, I'll push an updated package.

  • iKockaiKocka Member
    edited October 2011

    NickM said: Can you be more specific? What doesn't work?

    Take a look at your script running on my server (s03 is a fresh install of Debian 5): http://status.jagodic.im/index.php

    openstatus-client.conf:

    [server] # What server should we send updates to?
    ip = 209.141.62.184
    port = 12999
    
    [misc] # Miscellaneous options
    interval = 60 # How often to update, in seconds
    
    [load]
    warning = 1
    critical = 2
    
    [memory] # Warn / critical alert when memory used is above...
    warning = 128
    critical = 192
    
    [disk] # Warn / critical alert when disk usage is above %...
    warning = 80
    critical = 95
    
    [processes] # What processes do you want to monitor?
      [[OpenSSH]]
        proc = sshd
        level = critical # What level should warnings be if this process is down? (warning, critical)

    Process seems to be running too:

      851 ?        00:00:00 openstatus-clie
  • Output in /var/log/openstatus-client.log ?

  • iKockaiKocka Member
    edited October 2011

    Here it is:

    Could not connect to server: Connection refusedCould not connect to server: Connection refused
  • vedranvedran Member
    edited October 2011

    @NickM

    I can connect to your server, so it looks like the problem is in server.
    netstat shows it's listening on 12999, but I can't connect to it, tried with telnet and I get the same error (connection refused).
    I'll try installing in on another VPS and see what happens.

    Found the problem:
    I changed the listening IP in the server config to actual IP, but it looks like it only works if you leave it at 0.0.0.0

    Now it connects, but I get "JSON error from [IP]" in the server log.

    Edit2: actually got that error only once, it's not repeating. But it's also not updating anything :/

  • NickMNickM Member
    edited October 2011

    In /usr/bin/openstatus-client, after the line:

    print 'JSON error from %s\n' % (addr[0])

    can you add:

    print data

    (If you're not familiar with python, make sure that you indent it to the same indentation level as the first print statement, using 4 tabs).

    Then, run openstatus-client in debug mode: openstatus-client -d

    That should output the JSON that is being sent to the server. Send me that output. Thanks.

  • @NickM: Is it possible to monitor openstatus-server too?

  • @iKocka
    Surely you can install the client on the same box where the server is running?

  • @iKocka: yes, you can monitor the server that openstatus-server is running on. You may have to use 127.0.0.1 as the IP. Actually, I had one server that required 127.0.0.1 and the other required the actual IP, which I find kind of weird.

  • @NickM

    This is the output I get on the server (sorry, I misread what you said, but I guess I'd get the same thing?)

    {u'uptime': {u'load1': 0.0, u'status': 0, u'uptime': u'4:46', u'load15': 0.0, u'load5': 0.0}, u'hostname': u'server1', u'ram': {u'status': 0, u'total': 89, u'bufcac': 65, u'free': 11, u'used': 78}, u'who': {u'root': 1}, u'ips': [{u'ip': u'[INSERT_IP_HERE]', u'host': u'server1'}], u'disk': {u'single': [{u'fs': u'/dev/xvda1', u'mount': u'/', u'avail': 2205124, u'used': 733928, u'total': 3096336, u'type': u'ext3'}, {u'fs': u'tmpfs', u'mount': u'/lib/init/rw', u'avail': 45968, u'used': 0, u'total': 45968, u'type': u'tmpfs'}, {u'fs': u'udev', u'mount': u'/dev', u'avail': 35388, u'used': 24, u'total': 35412, u'type': u'tmpfs'}, {u'fs': u'tmpfs', u'mount': u'/dev/shm', u'avail': 45964, u'used': 4, u'total': 45968, u'type': u'tmpfs'}], u'total': {u'avail': 2332444, u'status': 0, u'total': 3223684, u'used': 733956}}, u'procs': {}}
    

    I'll do the same for the client if you need it.

    As I said, that JSON error is not repeating so I'm not sure if this is helpful.

  • Heh. I missed your second edit. No need to further investigate the JSON error.

    I think you may be running into the same issue as iKocka. Check the output of /bin/hostname on the client and make sure it's the same as the hostname that you entered on the server.

  • Everything is working now, but I have no idea what was the problem. I removed the servers and added them again, perhaps I messed up names or IP's.

  • Glad to hear that it's working now.

  • @NickM What ever came of your discussion with Fran about using the buyvm+ free backup to host your openstatus page?

  • He said that it was fine to host status pages there. I don't think it'll be too good of an idea though because you'd need to copy the database file over to there, and you'd need some way to update it regularly. You'd essentially need a PHP version of openstatus-server, running on buyvm+. Probably better to just have your openstatus-server with a different provider in a different datacenter (or at the very least, on a different node).

    I might build in some kind of clustering support for openstatus-server though, so you can have multiple servers updated by just one openstatus-client. Currently, you could run two instances of openstatus-client on each box, each with different configs, updating to different servers if you wanted to have a fallback in case the primary openstatus-server fails, but that's not very efficient.

  • iKockaiKocka Member
    edited October 2011

    colspan should be 7 because there are only 7 collumns :)

  • Indeed it should. Not a huge deal since it still renders properly, but I've fixed it in the branch for the next version.

  • SimplyfastSimplyfast Member
    edited October 2011

    I get "Unable to connect to the database - please try again later." error.
    Both database file and folder are 666 and owned by www-data.
    Sqlite and python installed.
    Please help.

    My configs:

    server {
        server_name ***;
      access_log /var/log/nginx/openstatus.access.log;
            error_log /var/log/nginx/openstatus.error.log;
    
            root /usr/share/openstatus-server/public_html;
        include /etc/nginx/fastcgi_php;
        location / {
            index index.php;
            if (!-e $request_filename) {
                rewrite ^(.*)$  /index.php last;
            }
        }
    }
    
    [server]
    listen_ip = 127.0.0.1# Self explanatory?
    port = 12999
    email_from = [email protected] # Email address for the From: header of sent emails.
    
    [warning] # Options for Warn-level alerts
    email = [email protected] # Email address to send alert to. Using gmail's label feature to separate them
    
    [critical] # Options for Critical-level alerts
    email = [email protected]
    
  • diffradiffra Member
    edited October 2011

    Simplyfast: I had the same thing. Restart php by issuing /etc/init.d/php-cgi restart and you'll be good. SQLite gets installed when you install the openstatus app from apt, but it doesn't restart php to enable.

    Thanked by 1Simplyfast
  • Oh, thats easy, I've restarted NGINX instead.
    TNX :)

  • Still stuck: http://46.105.166.104/index.php.
    No RAM no DISK no UPTIME etc.

  • Did you edit openstatus-client.conf? Does your hostname entered match with the output of /bin/hostname?

    Thanked by 2Simplyfast mrm2005
  • Done with the hostname.
    Thanks

  • How do you install client on CentOS?

  • @LivingSoul: Clone the git repo, available here. The only files you really need are openstatus-client and openstatus-client.conf (you'll probably want to make your own init script, or set up some other way for it to start at boot, I'm not quite sure how it's done on CentOS). Make sure you have python and the necessary modules installed. Put the conf file at /etc/openstatus/openstatus-client.conf and the openstatus-client file somewhere in your PATH.

    Thanked by 2LivingSouL mrm2005
  • LivingSouLLivingSouL Member
    edited October 2011

    @NickM: I get this error running openstatus-client on CentOS

    # /usr/bin/openstatus-client
      File "/usr/bin/openstatus-client", line 12
        options, remainder = getopt.getopt(sys.argv[1:], 'dc:', ['debug',])
              ^
    IndentationError: expected an indented block
    

    The error above has been fixed. Now, I get this error:

    # /usr/bin/openstatus-client
    Traceback (most recent call last):
      File "/usr/bin/openstatus-client", line 2, in 
        import socket, select, signal, os, sys, atexit, re, getopt, time, setproctitle
    ImportError: No module named setproctitle
    
    
  • LivingSouLLivingSouL Member
    edited October 2011

    Whew.. having a hard time trying to run the client on CentOS

    # /usr/bin/openstatus-client
      File "/usr/bin/openstatus-client", line 232
        except IOError as (errno, strerror):
                        ^
    SyntaxError: invalid syntax
    
    

    but I've successfully ran scrd on the same box. Any ideas anyone?

  • What version of python does CentOS have? Seems like it might be 2.4, since that's the same error that you get trying to run the 2.6 version on 2.4. Use python 2.6.

  • i keep getting this: Could not connect to server: Connection refused
    no idea what i am doing wrong

  • Looks awesome! Will try it out when I can!

    Check my blog for more cool *nix tips & tricks!

  • It works really great, but sometimes the server "freezes", it stops updating status for unknown reason. When I restart openstatus-server it resumes normal operation.

    It happened twice already, anything I can do to help you find out the reason if it happens again?

  • vedran said: It works really great, but sometimes the server "freezes", it stops updating status for unknown reason. When I restart openstatus-server it resumes normal operation.

    It happened twice already, anything I can do to help you find out the reason if it happens again?

    It happened once to me too.

Sign In or Register to comment.