Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


My servers got hacked - most likely Status2k - how?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

My servers got hacked - most likely Status2k - how?

myhkenmyhken Member
edited January 2014 in Help

Last week I got an e-mail from Nick @Ramnode, telling me that one of my servers was turned off because of really high CPU load from a mining script.
It was really strange, since I have never used any script at all, and since the server only was a part of my DNS fail over service, so it was not in use (since my main sites was up and running)

While reviewing Nics's email I discovered that the CPU load on another backup server went up in the roof. And started checking what cause the high CPU load. There I found a process using up my CPU (/tmp/kpoll -B -q --url=stratum+tcp://ltc.give-me-coins.com:3333 -u unixminer.am -p fiLSs)

The script (with the name kpoll) was uploaded from a Status2k site/user a couple of weeks before. Some how somebody had got access to upload the script in to the /tmp folder, and managed to start the script.
I'm using uniqe domains for each server and user that host Status2k. (like site1.mydomain.com). I do not have anything else on that domain beside Status2k.

The strange thing was that the script was uploaded on all my servers using Status2k. On servers not using it, there was no kpoll script.

I'm lost here, since I don't know how they could get access to upload a script to the /tmp folder and then manage to start the script.

Since the attack, I have destroyed all servers, rebuild them with new OS, new Virtualmin installation, uploading my sites from backups, creating new passwords on all servers and domain/virtualmin users. No server has two alike password.
I have start using AllowUsers (in /etc/ssh/sshd_config), I'm allready using a different SSH port then 22. I'm using CSF firewall (as I used before), only allowing a few ports (like 80,443,21).

I'm using CentOS 6.5 32bit with all updates and Virtualmin 4.04.gpl GPL.

I have now only one site with Status2k, and only using a multiserver script on all my other servers (most of them). It's three files, put on there own domain/user on each server, all with different passwords.

If anybody with some knowledge of security/php/websites can check the following three files, to see if there is anything in them that can let anybody get access to my server I will be very thankfully: (all files is .php files, I have only changed them to .txt now)
1. config.txt
2. index.txt
3. multiserv.txt

Anybody that can give me any good ideas how this happened? Anybody else using Status2k, anybody with more securing tips of my servers or my sites?

Edit:
I have to say that I used maldetect 1.4.2 on all servers before I destroyed them, and did not find any malware except some test files in the clamav directory (that i'm not using)

«1

Comments

  • Files not found.

  • @Jono20201 said:
    Files not found.

    Try now, forgot to remove the .php part.

  • Did find one more security feature for my Status2k site: Password protecting the complete site. So if there is any security issue with Status2k, nobody will get access to my Status2k page anymore.

    Still, it's not a good plan if there is any way somebody can get access with the three files in my first post.

  • That's just one of the cases why you don't publically show extended status of your servers.

    Thanked by 1Mark_R
  • @myhken said:
    Did find one more security feature for my Status2k site: Password protecting the complete site. So if there is any security issue with Status2k, nobody will get access to my Status2k page anymore.

    Still, it's not a good plan if there is any way somebody can get access with the three files in my first post.

    Restrict access to specific IP's, Limit Apache privileges, etc...

    Thanked by 1myhken
  • Maybe a stupid question, but did the hackers had to have SSH access to start the script? I understand that somehow they can find a way to upload a file to the /tmp folder. But how did the script start?
    I did see that it was the user of site1.mydomain.com that was the owner of the script.

    I find it strange that they have got SSH access since I used a pretty strong password, like: 47dwsa-sd4689 (of course not my password, but you get the idea)

  • @myhken I'm not an expert, but the

    if (isset($_GET["action"])) {
    $action = $_GET["action"];

    comes unsanitized

  • The other thing, ssh doesn't use any key pairs authentication.

  • @myhken they can launch a sh script with PHP using exec, I've seen this a lot especially with Joomla sites with a few popular plugins, exploit is found attacker uploads a PHP script that has inbuilt terminal, MySQL browser, file browser you name it, they either copy the whole home directory (to try sell templates?) or upload a more vicious file..

    If status2k is just for yourself do why the others mentioned and lock it down.

    Thanked by 1myhken
  • @vRozenSch00n said:
    myhken I'm not an expert, but the

    if (isset($_GET["action"])) {
    > $action = $_GET["action"];

    comes unsanitized

    It's only used to show phpinfo so no harm, if it was just inside a MySQL query you could potentially have the DB stolen.

    Thanked by 2vRozenSch00n myhken
  • @ATHK said:

    If status2k is just for yourself do why the others mentioned and lock it down.

    Don't know why I have not password protected my main (or all) Status2k site(s) before today...stupid choice.
    Since last week I moved my Status2k site to a VPS thats only used for Status2k. No other domains, no other users, nothing beside Status2k.
    I have not installed Status2k on any other server, just the three files over in their own domains, with no MySQL, no SSH access, unique passwords, etc.
    I really think I'm so secure that I could be. (of course, consider restricting the SSH access to a specific IP, but I'm using a VPN service with dynamic IP so not very fun the day my IP changes.)

  • drserverdrserver Member, Host Rep

    Scripts that you have attached are safe. 3 exec calls, i cant see anything that can be called with any post, get or request call

    Thanked by 1myhken
  • jarjar Patron Provider, Top Host, Veteran

    @mcmyhost said:
    Restrict access to specific IP's, Limit Apache privileges, etc...

    Pretty much this. You can mitigate so many web facing security holes by this alone, provided Apache doesn't have any known holes at the time. Throw a "deny from all" in .htaccess and I don't really care what's in the directory ;)

    Thanked by 1myhken
  • Could you please post apache log file ?
    Thanks.

  • bdtechbdtech Member
    edited January 2014

    Sounds like your running the web server process with too much privileges. Also check your file/folder permissions.

    Thanked by 1myhken
  • @alexvolk said:
    Could you please post apache log file ?
    Thanks.

    Thats amazing...since I'm only one using the Status2k site, and the log got bigger and bigger (around 500 MB per month(since I alway has a Status2k window open)) I decided to delete all logs prior to December for that domain. I have the access_log for December up to the moment the server was turned off.
    But I did see in Virtualmin that the script was first launched 30/Nov (don't know why it did not started use CPU power before 27/Dec)

    I can post the access_log file from that day (27/Dec) but have no records of 30/Nov.

    any other log that the access_log you want? I have destroyd the server, so I don't have access to other logs then the access_log and error_log

  • @bdtech said:
    Sounds like your running the web server process with too much privileges. Also check your file/folder permissions.

    It's all standard. Files has 644, folders 755. All web server process is standard virtualmin settings, I have not changed a thing.
    Virtualmin 4.04 on CentOS 6.5 32bit. How can I reduce webserver privileges?

  • Honestly Status2K sucks, having to run a web server on each of your slaves is crappy, although I believe Status2K supports SSH. I'd recommend looking into Observium or enabling SSH auth for Status2K and then you don't have to worry about the webserver.

    Thanked by 1myhken
  • Honestly Status2K sucks, having to run a web server on each of your slaves is crappy

    Don't like that it has to have a active website to run on. But it's a great script to monitor all my servers (or not of all them, but my servers with sites on them).
    Maybe there is some other program thats do the same, but I remember that it was the best option and recommendation on WHT back in 2009/2010 when I started use it.

  • @myhken said:
    Maybe there is some other program thats do the same, but I remember that it was the best option and recommendation on WHT back in 2009/2010 when I started use it.

    Back in the day it was the best around and I guess if you're already running a webserver for something else it's not as bad. I've used it in the past and the UI is kinda meh and I felt like the multiserver monitoring feature was a joke.

  • I like to see the CPU load, and uptime stats on my servers at all time. A fast way to see if I need to do something. Used to use top command in putty/SSH, but it's much more easy with a web interface with all servers listed.

  • Global apache exploit from last October/nov? 90% ppl not aware of it

  • @myhken said:
    I like to see the CPU load, and uptime stats on my servers at all time. A fast way to see if I need to do something. Used to use top command in putty/SSH, but it's much more easy with a web interface with all servers listed.

    Zabbix and Nagios is what I use for server monitoring. Zabbix has a client and you can use NRPE to monitor the health of the server from Nagios.

    Restrict which IP has access to the web interface and you would be good to go.

  • netomxnetomx Moderator, Veteran

    A friend got something similar, it seems they used pecl to upload a ssh client or something. Then, he Downloaded a script to mine

  • JanevskiJanevski Member
    edited January 2014

    @myhken You have been compromised, check Your home controlling computer, on the server side reinstall only the bare bones needed for work, no fancy status scripts, password and IP acl secure everything administrative (SSH to be more precise), also consider that the code of Your web apps or whatever You had now could have potential backdoors, use safer products, such as compile the latest version of nginx, php-fpm etc. and see if the problem occurs again.

    TL;DR

    Start from scratch, strict restrict access, safely rebuild service essentials, see if the problem occurs again, if it does then You're missing something, back to start.

  • joepie91joepie91 Member, Patron Provider

    Have a look at this. Looks like another case of a server compromised with the same miner.

  • I have started from scratch, I moved my main sites to iwstack (4 servers), I got two new backup servers. All was installed from scratch, no password was used on the new servers that was used on the old one.

    For the rest of my servers (that was affected, and that I did not cancel) I rebuild the servers from SolusVM, installed CentOS, Virtualmin etc. Did not use one of the old passwords on any of the servers, and they are completely different from any of the new servers I got.

    My goal are still to complete get rid of any of the servers that was hacked. They do not have any thing important on them anymore,

  • joepie91joepie91 Member, Patron Provider

    myhken said: no password was used on the new servers that was used on the old one.

    Don't use password auth at all - use keypair auth.

  • @joepie91 said:
    Have a look at this. Looks like another case of a server compromised with the same miner.

    Yes, its the same script.

  • Don't forget to use mod_sec for apache. csf automatically blocks ips from mod_sec log entries. See below of what csf automatically blocks.

    Time:     Sun Jan  5 08:07:42 2014 -0800
    IP:       217.69.133.236 (RU/Russian Federation/fetcher4-3.p.mail.ru)
    Failures: 5 (mod_security)
    Interval: 3600 seconds
    Blocked:  Temporary Block
    
    Log entries:
    
    [Sun Jan 05 08:00:00 2014] [error] [client 217.69.133.236] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\\\b(?:(?:n(?:et(?:\\\\b\\\\W+?\\\\blocalgroup|\\\\.exe)|(?:map|c)\\\\.exe)|t(?:racer(?:oute|t)|elnet\\\\.exe|clsh8?|ftp)|(?:w(?:guest|sh)|rcmd|ftp)\\\\.exe|echo\\\\b\\\\W*?\\\\by+)\\\\b|c(?:md(?:(?:32)?\\\\.exe\\\\b|\\\\b\\\\W*?\\\\/c)|d(?:\\\\b\\\\W*?[\\\\\\\\/]|\\\\W*?\\\\.\\\\.)|hmod.{0,40}? ..." at REQUEST_HEADERS:User-Agent. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "146"] [id "1234123446"] [msg "System Command Injection"] [data "; mail"] [severity "CRITICAL"] [tag "WEB_ATTACK/COMMAND_INJECTION"] [hostname "www.domain-name-removed.com"] [uri "/file.htm"] [unique_id "UsmBgMC4WcAAADFTP04AAAAJ"]
    
    Thanked by 1myhken
Sign In or Register to comment.