Need Help: my server was down at 5am for 3 days because of csf lfd SYSLOG check

jayc

My vps keeps down for the past 3 consecutive days and I had to reboot to bring the server back. After talking to my vps provider, they told me that there's nothing wrong on their end. So I did a little investigation on my end.

I have looked at my /var/log/messages, and found out that

Mar 30 05:00:09 servername lfd[25945]: SYSLOG check [IeQTp8WgxW22eMM0fVycJZlaj71N]
...
Mar 31 05:00:08 servername lfd[15753]: SYSLOG check [rixmECu0aprLJxalJwh]

after logging the lfd SYSLOG check on May 30, 31 and Apr 1, there won't be any other log (and server died unless it got rebooted)

So it must have something to do with lfd.

The thing is, lfd (csf firewall) was installed on my server for quite some time (almost a year) and it was fine until May 30.

I don't want to live without lfd, I can try uninstall then reinstall, but other than that is there anything I can do to find out what part of lfd caused the problem and fix it?

please advise.

Fritz

Probably your IPtables is broken now. Try to test the IPtables using CSF perl test to see all required modules is loaded.

jayc

perl /usr/local/csf/bin/csftest.pl

Testing ip_tables/iptable_filter...OK

Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...OK
Testing ipt_state/xt_state...OK
Testing ipt_limit/xt_limit...OK
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK

RESULT: csf should function on this server

seems like iptable is fine. any idea?

George_Fusioned

Your VPS was probably down due to something different. What you see in your messages log is SYSLOG_CHECK from lfd. Basically lfd sends a coded message every X seconds, and then checks for the coded message to see if syslog is running correctly.

# Check whether syslog is running. Many of the lfd checks require syslog to be
# running correctly. This test will send a coded message to syslog every
# SYSLOG_CHECK seconds. lfd will check SYSLOG_LOG log lines for the coded
# message. If it fails to do so within SYSLOG_CHECK seconds an alert using
# syslogalert.txt is sent
# 
# A value of betwen 300 and 3600 seconds is suggested. Set to 0 to disable 
SYSLOG_CHECK = "3600"

The answer to what caused your server to go down (if there is one) is in the last lines before it crashed. Since you say it was down for 3 days, your messages log will have entries up to a certain point in time, and then a 3 day gap before it starts with the next lines. This is where you should check.

While there's probably not much you can do now, on important thing you need to do is get yourself some monitoring - almost all monitoring services offer a free account for one server/probe. Checkout pingdom, statuscake, uptimemonitor etc.