Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Admins: how do you analyze a compromised VPS or node?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Admins: how do you analyze a compromised VPS or node?

XeiXei Member
edited March 2019 in General

Given recent posts regarding compromised VPS's, I found the topic to be interesting. I've never actually met anyone in the real world that really had a background in this field (virtualization related forensics). I wondered how do administrators narrow down the attack vector? How do you narrow it down to a process? Is there a list of go to logs on the VPS that are critical to analysis? I remember reading rootkits hide their presence and attackers can delete logs, in those cases is all evidence involving the attacker virtually erased? If there's no foot prints on the VPS, is that the end of an investigation? How do you know if the node itself wasn't compromised? How do you determine if a neighbor was compromised and the VPS was compromised as a result of lateral movement? How do you determine if the Solus VM or whatever control panel was hacked? Or your mail server that sends out the root password? Are there any other common points that can lead to infiltration? Given the amount of VPS's most providers service is there even adequate time to do all this research or is it overkill. How often do you all see compromised VPS's, meaning is it weekly or monthly, etc. I'm bored but also intrigued by all this. I think it may be neat to receive a compromised image and try to piece together what happened for learning purposes. I think Linux and virtualization related forensics would look good on a resume.

Thanked by 2eol uptime

Comments

  • eoleol Member

    Yes.

    Thanked by 2dahartigan Janevski
  • @eol said:
    Yes.

    Good band.

    Thanked by 3eol uptime Janevski
  • uptimeuptime Member
    edited March 2019

    Perhaps.

    EDIT2:

    https://security.stackexchange.com/questions/39231/how-do-i-deal-with-a-compromised-server

    EDIT3:

    and here's big ol' pdf on things to do before you call the FBI in to help:

    https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r2.pdf

    Okay, probably not all that relevant to low-end VPS scenarios, but ...

    Could be some interesting reading. If you're interested in that kind of stuff.

    EDIT3.1:

    Just skimmed that last one. It's fairly bureaucratic and useless for the purpose of this discussion. Sorry. Leaving the link for posterity, with some humility for my haste to post it.

    EDIT4:

    In case it's not obvious, I am not an admin for anything other than my own systems, so not able to add much in the way of "practical" advice that an actual low-end host might be willing to share.

    That said, I suspect there may be a strong sentiment of "ain't nobody got time for all that" and (especially) "prevention is the best cure" ...

    Though certainly some hosts seem to be more cognizant of basic security practices than others.

    This thread hopefully could be a good place to learn a bit more about which hosts go above and beyond the (possibly quite lame) norm on the low-end.

    Thanked by 2eol Janevski
  • servarica_haniservarica_hani Member, Patron Provider

    Hi Xei,

    Let me explain to you few small actions that can make you able to detect any attack or hack attempt

    1- use remote loggers : dont log to same server that is a big NO because once the server is compromised all the logs are compromised and cannot be trusted while if you log to external server even if one server is hacked no body can delete the logs ont he external server .
    Actually some banks even log to a printer ( yes real printer) they log important message to real printer that print them so even if the whole network is compromised they will still have hard copy of papers they can refer to .

    2- have historic data (preferably graphs) hosted somewhere other than the server itself . so you can see clearly when something unusual occure. like you will see sudden spike in cpu usage or spike in network or innodes etc

    3- have backups : what ever you do is not enough to replace need for backups . have them as you WILL need them.

    4- keep your systems up to date : so only zero day attacks can work on your system

    5- master the idea of separation of concerns , dont use same network for user data and management or storage. keep critical servers and vms in separate network and on separate physical hosts than the rest of the VMs and servers.

    6- use good IDS system : some of them can protect you from even some zero day attacks .

    Now as an admin to figure out there is a breach is by the effects of this breach
    as i explained once you have historic data that are collected outside the server you can see when there is a spike (Actually in good monitoring system you should get an alarm) and usually for zero day attacks the spike pattern will repeat in more than one server.

    remember that most attackers who hack are hacking for resources so in most of the cases just monitoring the resources you will detect the attack.

    Now there are hacks that only want to steal small set of data (not abusing the resources) usually those are much harder to detect as they are targeted and they dont leave clear mark on any monitoring system for those attacks you need good IDS and good log analysis to detect anything unusual .

    let me give you real example .
    We have number of our users that use windows . unfortunately some of the users turn auto windows update off to save on disk space .

    when the wannacry attack started any unpatched windows machine was a a target.
    for us out of sudden in all servers that has windows VM on them we started to see the CPU go very high.
    Also we noticed that the vms started to report unusual stuff about there IPs etc.

    It didnt take us long to figure out there is some attack that only affect windows servers and checking their versions it was clear it was wannacry .

    now since those VMs was on cluster that has daily backup the damage for end users was very little since all their data was restored to the point before they where hit

    so you can see from this example the importance of point 2 and point 3

    I hope the answer was helpful and useful

  • I have never investigated such incidents, either. Here is my thought on this.

    How do you know a VPS is compromised? I would guess you are notified by external sources that the VPS was doing something weird, like sending spam (TCP port 25), SSH password breaking (TCP port 22), etc.

    Now, issue "netstat -n". If there are lots of connections to TCP ports 25 or 22 of external hosts, the weird thing might be still happening. Then map network connections -> processes -> user accounts -> pile of files to be examined.

    Thanked by 5eol Chuck Xei poisson imok
  • poissonpoisson Member
    edited March 2019

    Not an expert but netstat can show you all the TCP/UDP connections to external along with the processes that are associated with the connections (issue "sudo netstat -nputw").

    If a process you are not familiar with is connecting to foreign addresses, that's a red flag. As an example, on my seedbox running default Debian 9 OS template, only sshd [user] and the torrent program should be actively running and connecting to the outside world unless I do other things. Any other process connecting to the outside world is a red flag and needs to be investigated.

    EDIT: if you see unknown ssh attempts established using the above command, it is probably someone trying to brute force. Just pull up the list of users who have logged in via ssh using the "last" and you can see if there's any unexpected logins.

    Thanked by 3eol uptime Xei
  • uptimeuptime Member
    edited March 2019

    Also lsof can be useful for associating processes with open sockets (and many other file-type interfaces on unix-like systems)

    http://www.linux-magazine.com/Issues/2007/77/Isof (paywall? Sorry! Alternate link: "Track down and expose intruders with the versatile admin tool lsof.")

    And here's another (more general) article on lsof - no paywall for this one as far as I can tell:
    http://www.linux-magazine.com/Issues/2014/168/lsof

    Thanked by 4angstrom eol Xei Janevski
  • Commands dont work against hacked kernel .... they can bypass that commands

    Thanked by 2uptime poisson
  • jsgjsg Member, Resident Benchmarker

    A professional does nothing ON a supposedly hacked VM. He attaches the supposedly compromised disk image to a trusted forensic VM on a trusted node.

    As a concerned user one tries ones best to do good prevention, logging (external), HIDS, etc.

  • FHRFHR Member, Host Rep
    edited March 2019

    If you are interested in forensics:
    Note that I wrote this regarding Linux, but Windows forensics is generally similar with a few (albeit major) differences.

    The first step is gathering and securing the data:

    • Do NOT shut the server down. First thing you want is to obtain a memory dump, which can be later analyzed. If the server is a VM, you want to do a snapshot first - it's a bit easier, since using the memory dumping tool will leave extra traces, and you then need to separate your own work from that of the attacker.

    • After you obtained the memory dump, you can shut down the server and make a 1:1 copy of every disk which contains potentially useful information. DD for physical drives, simple "cp" for VMs.

    You can then analyze the gathered data:

    • Mount the cloned drives to a known good VM (which you will discard after you're done) with your favourite tools installed. For a lot of less sophisticated attacks, simple find with "time modified" params works great. Look for any suspicious files, suspicious permissions, suspicious modified times, recently added users; compare md5 sums of anything that looks weird to you. Look in system logs, application logs, cronjobs, system mail…

    • If you didn't find what you were looking for, it's time for more heavy duty tools. I would proceed by doing memory analysis. "Volatility" is a truly awesome tool that does wonders. Running basic utils like "strings" and "grep" on the memory dump also works and can sometimes reveal useful stuff without much effort.

    • If you suspect attacker created and deleted some files, you can attempt file carving. Note that this is a pretty time consuming operation, involves a lot of manual work and doesn't always yield results - success depends mainly on storage type (SSD vs HDD), how much data was written after the file was deleted etc.

    Note that most people will not do forensics (and for obvious reasons). It's time consuming and unless you're trying to track down a sophisticated attacker that potentially compromised a whole bunch of confidential data, it's not worth it.

    Also note that as a provider, I will do no such thing to client VMs. The client is responsible for data and security of their VMs. What I wrote applies to enterprise environments.

  • FHRFHR Member, Host Rep
    edited March 2019

    Xei said: I think it may be neat to receive a compromised image and try to piece together what happened for learning purposes.

    You can do that pretty easily, you can find training drive and memory dumps on various sites. Or possibly join CTF competitions.

    Thanked by 1Xei
  • uptimeuptime Member
    edited March 2019

    CTF == "Capture the Flag" hacking (cracking) exercises for "security education" purposes only.

    Good stuff @FHR - thanks for adding that helpful info and perspective

    EDIT2:

    Want. Moar. Knawledge.

    Thanked by 1Xei
  • uptime said: Want. Moar. Knawledge.

    I've been watching "LiveOverflow"'s channel when I have free time. He has a lot of videos on CTFs and they are quite entertaining to watch, even if you are not actually trying to complete that CTF.

    Ex.:

    Thanked by 2uptime Xei
  • Had a customer who complained about a slow VPS. He told me he couldn’t do anything anymore.

    Logged in, found there was a process running for crypto mining. I figured which user was running it, so I went to the hidden folder and found the hidden infected files.

    The thing to do next is to find the source of the exploit. Mostly it is an illegally downloaded (nulled) WordPress plugin. I delete it, delete the infections and kill the process.

    What I NEVER do is rebooting the server. The risk of data loss is gigantic, then.

  • eoleol Member

    Data loss vs. backups.
    Round 1.
    Fight!

    In my early days I would just wipe the machine.
    Then start from scratch + backup.

    Nowadays it doesn't happen anymore (hopefully) (yet).

  • uptimeuptime Member
    edited March 2019

    Data loss vs. backups

    https://www.forensicmag.com/article/2010/04/you-pull-plug

    EDIT2:

    The article linked above is actually not very interesting (in my opinion) however other links on that site may be more so. Or not, haven't checked.

    I like this one - more for embedded devices than vps, but some useful details on mounting and examining an image: Forensic Analysis of Plug Computers (pdf)

    (For a few years now, I've been noodling around with JTAG (and similar) interfaces to read and write firmware on things you might not expect to be doing anything "interesting" - for the lulz, mostly...)

    I guess if your vps is running on a raspberry pi (or an esp32, lol) then this ability to tap into firmware could be useful? (j/k ... or maybe I'm not. Who knows?)

    EDIT3:

    And then there is always https://zeptobars.ru if you really want to "see inside" a chip.

    Again, nothing to do with vps forensics (one would hope) but fun stuff maybe in the same ballpark if you might be curious to learn more about the electronic world around us. :)

    I just like looking at the pretty pictures of decapped chips. (Bandwidth warning, some super hi-rez images on the zeptobars site.)

    EDIT4:

    We need to go deeper.

    Thanked by 1eol
  • @DennisdeWit said:
    The thing to do next is to find the source of the exploit. Mostly it is an illegally downloaded (nulled) WordPress plugin. I delete it, delete the infections and kill the process.

    That's the thing a lot of people don't understand. Don't open random links, don't download and install random programs (Windows in particular), don't install random plug-in for your web apps, don't install random apk files and don't use the same password for all your logins. Most people are not ready to live in the digital age because they don't know or care to practice basics of digital security.

    Speaking of WordPress, I think I rather stick to old school manual HTML and CSS. Maybe the most is a respectable open-source flat file CMS. More bloat, more unnecessary risks.

    Thanked by 1uptime
  • @Osatien said:
    Commands dont work against hacked kernel .... they can bypass that commands

    Agreed. However, my view is that the probability of a hacked kernel is really low for most users. Usually it is more of kiddie scripts trying to gain elevated privileges to spam email, do hidden crypto mining etc. For the average user like me, protecting root will prevent most of these problems, and of course best to be on full virtualisation so that a compromised box on the node is unlikely to spread to yours in the event of a hacked kernel.

    Thanked by 2uptime Xei
  • What I have learned over the years, is that security is all about layers of security. There is no such thing as secure host or completely secure system/site. Security starts with backup. :smile:

    Now answering your question:

    • Forensics are carried out by focusing on files that are not part of default system/application
      • You can fetch files modified in the last x(7) days using find command
    find / -mtime 7 -type f 
    
    • File timestamp can be tampered with and hence cannot rely on it. If you are running redhat based distro linke CentOS, you can use rpm verify

      • You can obviously use tools like clamav, unhide, rkhunter
      • Check your site backup software to see file modification history of your site. This is a good starting point and very effective in detecting attacks early. It give you time to fix before going into blacklisting phase

    Most of the guys would go with a clean install, which is safer since you don't have any knowledge of what has happened.

    There are several proactive ways, but I will stick with the question.

    Never assume that you are secure and constantly audit security.

    Thanked by 2uptime Xei
  • deankdeank Member, Troll

    I can assure you that OP will read none of helpful posts.

  • Xei said: I think Linux and virtualization related forensics would look good on a resume.

    https://assets.kpmg/content/dam/kpmg/ch/pdf/the-five-most-common-cyber-security-mistakes-en.pdf

    Well, if it only for "resume", why don't try to grab some certs like CLFP & GIAC?
    Or maybe OP implicitly trying to resell certs?
    Or OP just bored with other LET topics?

  • I've never had to deal with a compromised node, but I've had to deal with my fair share of compromised shared hosting accounts and customer VPS when I worked at HostGator. There was only one time that I personally faced a compromise at the root level, which came from the big VestaCP vulnerability a while back.

    1. Shared hosting account

    This is easy. Too easy, to be honest. If it's a password compromise the only thing that matters is the login history. More frequently, it's a web application compromise. This means either an altered file or an uploaded file. Most frequently, it's an uploaded file being executed to perform malicious actions. You know the uploaded file from the massive number of "POST" requests made to it from OVH IPs in the web server access logs. Then you run "stat filename" and grab the time it was either altered or created. Take that timestamp and cross reference your web server access logs, find a "POST" that occurred at that time, and that is the file that was taken advantage of to upload the file. Rinse and repeat until you've traced it back to the origin, which is the file containing vulnerable code. Replace all code with fresh copies, make sure that file is either inaccessible or patched, leave no lingering files around that weren't part of the application's base code.

    1. VPS

    A VPS compromise can be referenced while being many things. It can be #1 above, just inside a VPS/container, and frankly it often is just that. In that case, same steps. If it's at the root level, you have some decision making to do. In my case with the VestaCP vulnerability I knew that it was automated, standardized, and identified. This meant that it was a bit unlikely that I had a compromise outside of what was known and clear. The best advice is that you wipe the system, you can't really "know" with 100% certainty that something deeper than the obvious happened. In my case I quarantined what was known, removed the scripts that re-created the malicious files, and scripted out a check to make sure I reviewed any system files that had been modified or created since the time of the initial compromise (to the best of my knowledge). I made a choice, it isn't one you should arrive at lightly.

    If a compromise is not at the root (or sudo user) level, the whole "wipe the whole system" thing is generally overkill. Shared hosting companies are not wiping their servers every time grandma fails to update Wordpress, and nor should they. That's insanity of the highest order, and would put them out of business one after another until one stood up and said "no." Yet, you'll still run into someone who suggests this, so take it with a grain of salt.

    If the compromise is at the root level, it makes much more sense to put a new system in production. Review to the best of your ability, but if logs are removed then you may not have much recourse. Perhaps the next system streams it's logs elsewhere (log management service, for example) so that you can locate the point of entry without giving them the ability to delete the logs.

    Besides my opinions here, I really love the accepted answer to this question, I feel like this comes from someone who has been there:

    https://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server

    Thanked by 3uptime Xei poisson
  • eoleol Member

    There will be no logs/suspicious logs if done right.

    Thanked by 1DigitalOceanJD
  • edited March 2019

    @eol said:
    There will be no logs/suspicious logs if done right.

    Which, quite frankly, it rarely is. I've only seen a few instances in which attackers removed evidence of their intrusion. Eventually it becomes known that it's compromised, as they don't do it without purpose. Outgoing spam, attacks, etc. Most of the time that starts right away, and the logs are still there.

    Thanked by 3Xei poisson Letzien
  • Nice thread with good advices. Thanks to everybody for sharing thoughts.
    My recipe is to use remote logger and Hetrixtools or similar to see changes in cpu%, bandwidth usage etc. No need to make deep analysis. Just see if some data can be saved. If not, then use backups and start a new server.

  • @DigitalOceanJD said:

    @eol said:
    There will be no logs/suspicious logs if done right.

    Which, quite frankly, it rarely is. I've only seen a few instances in which attackers removed evidence of their intrusion.

    When it comes to the new order of plentiful VPS and lower end, it seems that this is more of the rule than the exception. Half of the time they get caught before they do more than toss a few hundred/thousand packets around.

    Thanked by 1DigitalOceanJD
Sign In or Register to comment.