Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


How to figure out a website/account is hacked?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How to figure out a website/account is hacked?

bikegremlinbikegremlin Member
edited September 2019 in Help

As the title says, if there's one thing worse than getting hacked - it's getting hacked and not knowing about it.

The question is primarily related to shared/reseller hosting environment with a WordPress website installed.

I've read a lot about securing WordPress and the hosting account (with various quality of info on various sites). Still, little info on how to check if a website (on a cPanel account) has been hacked.

I understand it's a broad topic, so any info, including links for further reading, or terms for googling are all wellcome.

Comments

  • First you need to define what made you suspicious? Is your account being used to send spam emails? Is someone changing/adding content in WordPress? Has someone injected some code in your PHP files and now it is being used to mine shitcoin? Is your website now taking too much CPU?

  • uptimeuptime Member
    edited September 2019

    bikegremlin said: question is primarily related to shared/reseller hosting environment

    bikegremlin said: it's a broad topic

    ordinarily you might be looking for some sort of "Intrusion Detection System" (IDS)

    but the constraint of a shared hosting system is that you are typically going to be operating in userspace without any elevated system privileges

    so what can you do?

    thinking stuff like:

    • as much logging as you can manage - preferably to a remote logging setup
    • monitoring network for suspicious activity - not clear how this would work in shared hosting environment, even with a dedicated IP address. But something to think about.
    • auditing your filesystem - might compare to a "clean" reference installation elsewhere
    • auditing for known vulnerabilities such as SQL injection, yada yada yada

    It seems likely that you would be limited in your ability to monitor system processes, but whatever you can do along those lines could be helpful as well

    But keep in mind that depending on the sophistication of a system compromise, any telltale files and processes might not be easily detected. Logs can be altered, etc. So if it's possible to catch an intrusion in its early stages, then that may be somewhat easier to detect - before it's too late. I understand that's not exactly what you're talking about here - just something to keep in mind for future reference.

    First really is just to step back and clarify your threat model:

    • what are you trying to protect?
    • what is its value - to you, and to an attacker?
    • who is your most likely adversary? what are their capabilities?
    • what is the "budget" for your defense? what are the likely costs of an effective attack?

    I'm guessing we're not exactly expecting a stuxnet-type scenario here.

    It may be that for cheap shared hosting your best defense will be simply to keep good backups of your content separated as much as possible from any potentially compromised "program/system" level stuff.

    Ultimately you'll know your stuff has been pwned when it starts serving malware / sending spam / amplifying a DDoS / controlling a botnet / mining cryptocurrency ... or I don't know, what else could possibly go wrong - but presumably your ever-vigilant and diligent service provider will then promptly notify you of some specific problem that has caused them to suspend your account ... and then you'll know.

    I guess early warning signs of some of the above problems might show up as unexplained CPU use and network traffic. If a shared hosting setup provides a way to examine those parameters with some granularity then that would be a step in the right direction.

    If your concerns are more about undetected exfiltration of confidential data ... well, I don't know what to say about that other than it's probably not a great idea to put any serious secrets on a shared hosting setup in the first place, ammirite?

  • @alilet said:
    First you need to define what made you suspicious? Is your account being used to send spam emails? Is someone changing/adding content in WordPress? Has someone injected some code in your PHP files and now it is being used to mine shitcoin? Is your website now taking too much CPU?

    No problems for me so far. No reason for suspicion. Though a colleague's account got suspended by their provider for malware (posted on WHT about that) and that got me thinking.

    One of the things I'm using is WordFence plugin that reports any file changes.
    But I'm far from an expert, so any info / suggestions is more than welcome.

    CPU usage you noted is one thing to pay attention to?

  • @bikegremlin

    1. If you are using Google Webmaster, and Test the page, they will show a security warning that your page is infected with malware. They will also show this after a few days when the bot crawls (and if you don't use live test). Then you can go to the report section and find infected URLs. They may not display all, but a sample.

    2. Run your page test with any online malware scanner like Sucuri. It will warn you if the page is infected. And there are many services that help you scan entire WP pages.

    3. On-server side immunify 360 or alike services can catch such infected URLs. They scan and report you.

    4. WP plugins free/pro that you need to install to find infected URLs.

    5. Manually checking the WP core files/Plugin files by opening in FTP software, you can check it they are infected. Normally hacker would enter the script/code after <?php at top most section of page. But it can be anywhere!

    6. Keeping minimum plugins on WP. Also using ONLY reputed plugins directly from WP site can ensure you get patched if there's a infection.

  • @bikegremlin said:

    @alilet said:
    First you need to define what made you suspicious? Is your account being used to send spam emails? Is someone changing/adding content in WordPress? Has someone injected some code in your PHP files and now it is being used to mine shitcoin? Is your website now taking too much CPU?

    No problems for me so far. No reason for suspicion. Though a colleague's account got suspended by their provider for malware (posted on WHT about that) and that got me thinking.

    One of the things I'm using is WordFence plugin that reports any file changes.
    But I'm far from an expert, so any info / suggestions is more than welcome.

    CPU usage you noted is one thing to pay attention to?

    Yes CPU usage is one area. But it could be result of a badly coded plugin and doesn't necessarily mean rouge plugin. WordFence is a good idea. But overall it depends on attack vector for e.g. your host may be the most secure in the world but if your own PC is infected then it may result in getting your site hacked.

    Thanked by 2uptime bikegremlin
  • Sofia_KSofia_K Member
    edited September 2019

    Addition: Check if you close the "comments" section or manually approve each comment in WordPress. My BuyVM WP site got infected as someone plotted phishing URLs in comment section and Google caught it within 5 days after bot crawling it (again) and I was duly reported in GWT account. I removed all auto-comments and made my WP site ONLY to accept comments manually, also I closed comments for old blog posts.

    Re-submitting those URLs to Google came out clean and no warning in GWT.

  • bikegremlinbikegremlin Member
    edited September 2019

    @uptime said:

    bikegremlin said: question is primarily related to shared/reseller hosting environment

    bikegremlin said: it's a broad topic


    First really is just to step back and clarify your threat model:

    • what are you trying to protect?
    • what is its value - to you, and to an attacker?
    • who is your most likely adversary? what are their capabilities?
    • what is the "budget" for your defense? what are the likely costs of an effective attack?

    I'm guessing we're not exactly expecting a stuxnet-type scenario here.

    If your concerns are more about undetected exfiltration of confidential data ... well, I don't know what to say about that other than it's probably not a great idea to put any serious secrets on a shared hosting setup in the first place, ammirite?

    Thank you, @alilet and @Sofia_K for very educational posts and taking the time to patiently explain in a way I can understand - hope it will help others as well.

    To clarify (the quoted text):

    The thing that got me thinking - I posted on WHT.
    TL/DR: colleague had no idea he was hacked until his account was (temporarily) suspended by the hosting provider

    That account was a mess and the fact it was hacked is no surprise, the problematic part was we couldn't figure out the source. Without figuring that out, one can restore old backups and the hack is most likely to happen again - as it happened the first time. Which is another question / problem.

    First thing to consider though (apart from securing - @Sofia_K gave some very good tips on that) is, at least in my opinion, figuring out a hack as soon as possible. If for no other reason, then to know which backups are "safe".

    Also, I plan to start a (small) webshop - where hack protection would be of greater concern (can't remember the thread in which @jsg explained how using Cloudflare for anything but DNS gets them to see all the info decrypted, as it passes their servers on its way to the visitor).
    There as well - I figure that the least I should be able to do is notify any customers of a breach ASAP.

    Basically it is a bad scenario: low level of knowledge and experience + low budget.
    While I am a quick learner and quite persistent, I'd still rather do nothing than do things badly.
    A lot to learn, figuring out where to start from. :)

    @alilet
    When it comes to WP security, this is as far as I've figured so far (rather long and boring, intended to be corrected and added info):
    https://io.bikegremlin.com/8963/wordpress-security/

    Thanked by 1uptime
  • WebGuruWebGuru Member
    edited September 2019

    Take a look at "files modified" date-time in WP core files/folders and WP-Content etc directories using FTP or any FIle Manager it will you can idea if any of your WP file was modified by someone else. If you modified the files or upgraded them you already know the time-date when you did it so ignore those files and focus on the one with suspicious modification date-time that are not in your knowledge.

    You can also use some popular WP security plugins to scan for any files with injected code.

  • aliletalilet Member
    edited September 2019

    I would say get your friend's PC checked because he is using shared hosting and in that case security is host's responsibility and normally it is good and there is very little chance of being hacked (unless the host is incompetent). So check your own PC for a start.

    Thanked by 1bikegremlin
  • emreemre Member, LIR

    lsof command is your friend.

    lsof -p [apachepids]

    and look for suspicious stuff

    most likely they will appear on first rows of the command output.

    Thanked by 1uptime
  • uptimeuptime Member
    edited September 2019

    @emre indeed that's a good one - I imagine lsof stands for "list open files" - which includes sockets (ie network connections)

    but I have to wonder - would this command be available to an enduser in a shared hosting environment?

    @bikegremlin for more general "web application security" background info (not specific to WordPress) it may be useful to poke around https://www.owasp.org - this will give you some direction to go in if you want to learn about more basic concepts such as SQL injection or XSS vulnerabilities - probably a lot more too (it's been a looooong time since I last looked at that site - but it's still a good place to start I think)

    Thanked by 1bikegremlin
  • raindog308raindog308 Administrator, Veteran
    edited September 2019

    uptime said: ordinarily you might be looking for some sort of "Intrusion Detection System" (IDS)

    but the constraint of a shared hosting system is that you are typically going to be operating in userspace without any elevated system privileges

    You can roll your own in userspace. Write a script that records a hash (SHA, etc.) of all files. Any file whose hash doesn't match on later checks, or is not on the list, or is missing is darn suspicious.

    This doesn't help you on the database side. The big boys (Oracle, SQL Server, etc.) have database-side auditing tools but I don't think MySQL does, and regardless it's usually an avalanche of data to try to parse.

    Years and years and years ago, there was a script called tripwire that did these things, but I think they went all corporate. Still, not hard to roll your own.

    Of course, you need to keep the file with the hashes off site, and copy it back when you want to check.

  • emreemre Member, LIR

    uptime said: @emre indeed that's a good one - I imagine lsof stands for "list open files" - which includes sockets (ie network connections)

    I just skimmed the post and answered.

    this is usually what happens in the servers I manage.

    Some website hacked and uploaded some kind of malicious file which can be used as a mail proxy or something to send malicious traffic.

    nothing shows on ps list, but there is traffic going from the server.

    you can't be able to track this traffic using any kind of network tools, only way to identify what is generating this malicious traffic is to look for open files.

    yes, for to use this command you must be the system admin aka root.

    Thanked by 2bikegremlin uptime
  • As the question was amazingly answered already, I can only add a tip for the future - regularly change your passwords and keep them random generated. And for WordPress site, you can hide some directories, so no one should enter them too and won't see the list of files. This should be a good prevention from hacking

    Thanked by 1bikegremlin
  • You can use sites such as https://wpsec.com/ or http://isithacked.com/ to see if your site is hacked.

  • bikegremlinbikegremlin Member
    edited September 2019

    Thanks again to all the participants for taking the time and providing advice. Seriously considering making a transcript of this thread to keep on my website (and offline backups) for if/when LET goes down again. :)

    Some return info:

    When it comes to WordPress, WordFence plugin allows for regular scanning of all the files and noting any file changes that do not match the original repositries for the used themes and plugins.
    Whenever I add, or edit a file, it gets shown.
    Though no "false alarms" for uploaded pictures and other files used with website pages (not sure if that's a good thing).

    I've heard more than one hosting provider discourage the use of WordFence since it "loads the server, while our protection should be enough, you don't need any WordPress security plugin with our hosting". They don't forbid the use, just recommend not using it.

    When I tried using Grabber to scan for vulnerabilities, it did get blocked by the provider's WAF, while Owasp ZAP (passive scan) only got blocked by WordFence.

    As for the lsof suggestion:
    With reseller (shared) hosting environment, my options are Terminal (not root access) and SSH - the latter I suppose is to be used only if really needed and one really knows what they're doing - which is still not the case when I'm concerned. :neutral:

    Another thing to worry about are WordPress plugins and theme updates. Some get websites to crash (didn't have a problem in over 5 years, but it happens to people). Some come with security patches. What I do is update a test website (run on the same hosting environment as the "live" site, with the same theme and plugins). Then, if it's all good, wait a few days for any extra bug reports, then update the live site(s). Unless it's a security patch. In that case, I update immediately, then just not add any content for a week, until I confirm it's all good and I won't be needing to revert to a backup. Any better ideas are welcome. Especially since this is too much "manual" work for my taste.

    Thanked by 1uptime
  • uptimeuptime Member
    edited September 2019

    just thinking ... a good offsite backup protocol might also provide more opportunity for the type of file monitoring that would otherwise rely on WordFence. The way I see it, having a regularly tested capability to restore from backup is another level of real security. And it seems that checksumming files and scanning for malware could fit nicely into this process if done on a isolated vps where you have more control, flexibility, and available resources compared to your production shared hosting environment(s).

    Also, good on you for recognizing your own limitations in terms of "knowing what you're doing" and/or taking on a next-level workload or excessively complicated setup to manage

    I've heard it said that security is a process (or even a "posture") rather than a permanent state of being.

    just keep paying attention, comparing notes, and avoiding complacency and snakeoil alike - and at least you'll be in a much better position than 99% of the low-hanging fruit online.

    there is a story about running away from a bear (not the "you don't come here for the hunting" one, lol) ... punchline being sometimes you don't have to run faster than the bear, just faster than the other guy running next to you. Okay so this analogy maybe only goes so far with regard to online security but ... it's a bear joke so, whatever, deal with it. :smiley:

    Thanked by 1bikegremlin
  • bikegremlinbikegremlin Member
    edited September 2019

    @uptime said:
    just thinking ... a good offsite backup protocol might also provide more opportunity for the type of file monitoring that would otherwise rely on WordFence. Tested ability to restore from backup is another level of real security, in my opinion. Seems like checksumming files and scaning for malware could fit nicely into this process if done on a isolated vps where you have more control, flexibility, and available resources compared to your production shared hosting environment(s).

    and, good on you for recognizing your own limitations in terms of "knowing what you're doing" and/or taking on a next-level workload or excessively complicated setup to manage

    I've heard it said that security is a process (or even a "posture") rather than a permanent state of being.

    just keep paying attention, comparing notes, and avoiding complacency and snakeoil alike - and at least you'll be in a much better position than 99% of the low-hanging fruit online.

    there is a story about running away from a bear (not the "you don't come here for the hunting" one, lol) ... punchline being sometimes you don't have to run faster than the bear, just faster than the other guy running. Okay so this analogy maybe only goes so far with regard to online security but ... it's a bear joke so, whatever, deal with it. :smiley:

    Testing offsite backups is a must - great to note that for anyone else coming across this.
    Only after having tested and confirmed that can I sleep well. It is one of the main reasons I'm still paying for the cPanel - I'm very familiar and experienced now with backups and migrations to cPanel accounts.

    Managing a VPS properly would take a lot of time for learning, practising and even then I'm note sure how to check if I'm doing it right. So for me, it would most probably end up being more expensive.

    For security: I use that analogy for locking bicycles - as long as the bike doesn't look more expensive than the others and it's locked better than the others - it's very much safe.

    Still, when it comes to websites, it's my understanding that bots and malware don't get lazy, don't compare targets, they just go and hack whatever they can. So that is a bit different, isn't it?
    If a person with enough knowledge really put their minds to crashing/hacking a site - well, we've seen it with LET recently - that's next to impossible and probably very expensive to prevent.

    My thinking so far is: sooner or later, the probability of getting hacked is 100%. I'd just like to know ASAP when that happens and, preferably, know how it happened in order to prevent that particular scenario from re-occuring after the backup restore.

    In those terms, "hardening" security is just for having that happen as rarely as possible, saving a lot of time and hassle. I'm aware nothing is 100% secure.

    Thanked by 1uptime
  • I'm guessing that if you are able to devote a few hours a day to learn basic vps administration and security you'll be in pretty good shape to handle a simple setup within a month. Key thing is to keep it simple. May be worth a try - keep everything behind a restrictive firewall until you decide it's ready for real-world tests. "What could possibly go wrong?" ... It will cost a a little to rent a lowend VPS, and some time and effort. In return you (at the very least) get something interesting to write about.

    Agreed that bots are relentless. But also generally braindead, most relevant for chronically unpatched and otherwise neglected installations .... I am still under the impression that deployment of "zero-day" exploits are usually reserved for higher-value and/or specifically targeted scenarios. But certainly always some chance that your stuff will be in the wrong place at the wrong time, or serving the "wrong" client, or whatever ... So better to take nothing for granted - but also no reason to get overly stressed.

    Sounds like these suggestions from @Sofia_K point in a useful ("rapid detection") direction for you. Might be worthwhile to investigate a bit more along these lines as well ...

    @Sofia_K said:
    1. If you are using Google Webmaster, and Test the page, they will show a security warning that your page is infected with malware. [...]
    2. Run your page test with any online malware scanner like Sucuri. It will warn you if the page is infected. And there are many services that help you scan entire WP pages.
    3. On-server side immunify 360 or alike services can catch such infected URLs. They scan and report you.
    4. WP plugins free/pro that you need to install to find infected URLs.
    5. Manually checking the WP core files/Plugin files by opening in FTP software, you can check it they are infected. Normally hacker would enter the script/code after <?php at top most section of page. But it can be anywhere!

  • bikegremlinbikegremlin Member
    edited September 2019

    @uptime:

    1. Do this regularly - with email notifications of any problems enabled.
    2. From time to time - relying on WordFence scans for regular automated checking.
    3. Down to provider - using those who do it.
    4. WordFence?
    5. I very rarely do this. Any (free?) anti virus software to recommend?
      It is my understanding that with a Linux PC anti-virus software does more harm then good, so, if doing such scans - install it on a Windows PC, or get a Linux software for that?

    EDIT:

    P.S.
    VPS has become a sort of a "running joke" with my partner. :)
    The way things have been going, sooner or later I'm bound to start with a VPS.
    Then, the way I am, thorough, perfectionist - probably over a dedicated core VPS to a Dedicated server.
    But, to be certain all is working well, collocation just doesn't cut it - one does need to be sure the server room, power supply and all else is in good order. So an own small hosting company with servers and everything.
    Of course, relying on just one physical location is not enough redundancy.
    With dual power supply, and broadband link for each of course.
    ... But one also must have a reliable link supplier...

    :disappointed:

    That's the summary of a joking conversation, with each of us adding the next sentence. :)

    The reference being (him being a programmer and not worrying much about the other stuff): "have we went with a VPS yet?" :)

  • uptimeuptime Member
    edited September 2019

    So ... when it comes to wordpress - and the wordpress ecosystem - I am utterly ignorant. So can offer any specific suggestions only from that dubious vantage point - if we're feeling charitable let's call it "beginner's mind"

    That said, if I had to do something to implement a practical bottom line, it might go something like this - roughly sketched:

    1) obtain a vps with decent amount of cpu and storage. Budget ballpark $10 per month (mainly due to cpu requirement) - shop around and might find something closer to $5. Possibly a cheap (atom) dedi could be cost-effective if ever needing more ram and storage. But can start small and then get more capable system if/when necessary. (Can go into more details about many possible options from recommended affordable providers later ...)

    2) setup a firewall allowing access only from a few IP addresses

    3) setup regular backups from the shared hosting to the restricted vps - details to be determined, but you'll probably be wanting proper database dumps if you're working with wordpress

    3) setup a "scratch" directory on the vps in which to restore a backup

    4) collect checksums from all relevant files (can use linux utility such assha256sum etc - depending on required speed for checksum there may be other preferable options). Eventually have system compare to previously collected checksums to narrow the focus just to files that have changed. (Might look at how the rsync utility works to do this)

    5) scan for viruses and malware - I am dimly aware of https://avast.com for linux - that may be a place to start, and if it's good enough then great - otherwise something to compare to when investigating alternatives. (There is probably a lot more specific wordpress / php scannery out there as well - something to research ala WordFence)

    6) automate the external checks such as suggested by @Sofia_K (google webmaster etc) This may require setting up a webserver and opening up firewall to http/https traffic.

    7) generate/consolidate reports from checksums and scans - left as an exercise to the motivated reader ... (Seriously, lots of different approaches to achieve this goal - maybe just start with "the simplest thing that could possibly work" whatever that may be - and take it from there. Definitely quite a bit more to discuss along these lines though.)

    8) email is another can of worms but if it's just to send to your own inbox then may be a bit easier to get it all worked out. I've found yunohost to be a useful starting point for setting up an email server (and other things) - might be something to look into. If you need more widespread deliverability also might consider a service such as mxroute. Personally I would just get into habit of logging in to check on the vps (and shared hosting) on a regular basis, with or without an email to remind me. ("If you can check an inbox, you can check a server") ... though I suppose alerts via SMS etc might also be useful for next-level stuff - once everything else is dialed in.

    Now the real trick may be to get your partner on board to program all this and make a service / product / heroic open source project for great justice.

    Or maybe someone here has already put together a system along these lines and is willing to share with the rest of the class?

  • @uptime

    For automation of backups, this has been working reliably for me:
    https://io.bikegremlin.com/10008/jetbackup-automation/
    TL/DR: jetbackup automated backup of all the cPanel accounts within a reseller account - with both the database export and all the files copied. It allows creation of exceptions, to limit the amount of data copied (list which directory paths and databases to backup and/or which ones to exclude etc.), plus it works with Backblaze, which ends up costing zero $ as long as daily limit of 10 GB of uploads and/or 1 GB of downloads isn't exceeded.
    Not all the providers are happy to offer this level of JetBackup "freedom" (HostMantis does for one).

    With an extra off-line external disk copy to keep "at home".

    For emails, I've been using MXroute (lifetime license so it is practically "free"), storing emails with Gmail (never had an email lost with them, used them since the start and it's easy to backup emails from Gmail). Implementation:
    https://io.bikegremlin.com/10364/website-gmail/
    TL/DR: SMTP server is used for relays, while everything is picked up by Gmail. Thunderbird is set to use Gmail, while Gmail is set to use MXroute for sending-receiving. Has worked trouble free for some reason. :)

    If there comes a time I need more than 300 outbound emails per hour - SendGrid offers a dedicated IP with higher limits for 80$ + per month

    The other things you mentioned deserve their own topics I'd say - it would be easier to read and contribute that way IMO. They all sound interesting and worth considering.

    As for the habits - I prefer having "automatable" things be automated - I'm one of the most absent-minded persons I know, plus have a relatively busy schedule. So for me it's either another set-and scheduled daily/weekly chore (which is far from what I need), or an automated system that sends warnings when needed. Nothing in between will work in practice. And I'm not sure which option is wiser, especially when on a tight budget.

    Thanked by 1uptime
  • Few more questions:

    1)
    I'm using Cloudflare for DNS and for "the other option": the one that is offered as "free caching/CDN/Protection/IP hiding - just that your data gets decrypted and visible on our local servers, before getting re-encrypted and sent to the customers - you can trust us".

    It is my understanding that asymmetric encryption itself (such as TLS/SSL) also relies on having one entity that "everyone trusts".

    However, since Cloudflare deals with all the website data passing through it, including all the passwords, this seems a bit less secure. I am seriously considering disabling this at least for the webshop website - if for no other reason, then to protect the customer data, as best as I personally can.

    The downside is, that disabling it would make web-shop hosting IP address visible and not use any "extra protection" offered by Cloudflare.

    Any thoughts on that - weighing pros and cons? Bearing in mind that 1$ cost "hurts me" as much as about 5$ "hurt" anyone with an average US/EU pay, when considering alternative paid options.

    2)
    Also about Cloudflare. I'm using MXroute for emails. Which means that there's no DNS record on Cloudflare that doesn't use their IP masking and shows my website IP.
    If I wanted to use cpanel.mywebsite.com, I'd have to make that DNS record not go through Cloudflare (in order for it to work), hence make the IP visible. Does that introduce another "vector of attack", or should I not worry about that?

    2a)
    Same question for webmail.mywebsite.com, ftp.mywebsite.com and other stuff automatically added by cPanel in their DNS setup - even when not using Cloudflare, does leaving these records open additional attack vectors, or is it easily discovered as long as one knows the website's hosting IP address (guessing it's the latter, but not sure)?

    My thinking was to delete everything not used. I can always use IP_address/cpanel, /whm etc. Thought that would start with http, not https. Alternative would be to set the dns records on the local computer, not having them exist publicly, so just the computer "knows" where to go.

Sign In or Register to comment.