Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


WTF?! How was I pwned?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

WTF?! How was I pwned?

pwnedpwned Member

A monitoring service showed that my KVM VPS was rebooted, and when I logged in /var/log was empty and there were several strange processes running: /etc/my.conf, /var/ssh.conf, /usr/bin/.sshd, /usr/bin/pythno, /usr/bin/dpkgd/ps ax. (I did save a few of these binaries before I nuked the box, if there's a way to safely examine them.)

I happened to have a screen with /var/log/auth.log tailed. The last entry is ominous:

Jun 21 03:59:48 XXXX login[394]: pam_unix(login:auth): check pass; user unknown
Jun 21 03:59:48 XXXX login[394]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=
Jun 21 03:59:51 XXXX login[394]: FAILED LOGIN (1) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure

Wait, /dev/tty1? Isn't that the console connection? How would anyone connect there?

The down alert came in seconds later at 04:00:37 UTC. It went back up at 04:04:38 UTC.

More importantly, what did I do wrong to allow this to happen? I've been admin'ing VPSs for years, and I've never had something like this happen.

Background: Installed debian buster a few days ago (used the provider's Plesk VNC console to select the netinst kernel and initrd through grub), no root account, no services other than ssh at install. On first boot the usual: configure sshd to a non-standard port, disable sshd password auth, sshd AllowUsers to my account only, installed ntp, nginx, certbot, and munin. Configured iptables to allow tcp/80, tcp/443, udp/123, and tcp/ssh port through. Pretty sure I even disabled the VNC console through the Plesk control panel when I was done. It's an idler--there really wasn't anything running on it.

My account with the host (had!) a unique 64 char password and 2FA (TOTP) is active.

I'm stuck on that /dev/tty1 entry...all the usual login attempts come in through an IP to sshd, but not that one. What am I missing here?

Thoughts?

Thanks in advance!

Thanked by 1mxvin
«1

Comments

  • LittleCreekLittleCreek Member, Patron Provider

    Is there a console app in the control panel for the vps?

    Thanked by 1pwned
  • pwnedpwned Member

    I see a VNC/Desktop button at the management page, and under the control panel there's a VNC button. I don't think there's non-VNC console app or ssh tunnel style console option.

  • DPDP Administrator, The Domain Guy
    edited June 2020

    Doesn’t look like an actual login attempt though.

    You should inspect those confs and binaries in a sandbox - might give you clues to what’s going on.

    Edit: Infact, those auth logs entries, could that have been you yourself logging in but for some reason it’s logged as ‘unknown’?

    Thanked by 1pwned
  • pwnedpwned Member

    No, my logins all go through sshd, show my username, and log an IP address. There are no other tty1 entries in auth.log that I can see. I have another box with the same host, and no tty1 entries there, either. I was doing something else at 03:59 UTC and didn't notice the situation until about 6:30 UTC.

    Obviously I'm concentrating on that entry because of the timing, and maybe my ego doesn't want to admit I screwed up and got cracked.

  • LeviLevi Member

    @pwned said:
    No, my logins all go through sshd, show my username, and log an IP address. There are no other tty1 entries in auth.log that I can see. I have another box with the same host, and no tty1 entries there, either. I was doing something else at 03:59 UTC and didn't notice the situation until about 6:30 UTC.

    Obviously I'm concentrating on that entry because of the timing, and maybe my ego doesn't want to admit I screwed up and got cracked.

    Don't be delusional. You have not been pwned yet. Any other intrusion evidence you have?

  • pwnedpwned Member

    The running processes and empty log directory are pretty big clues, aren't they?

  • jarjar Patron Provider, Top Host, Veteran

    @pwned said:
    The running processes and empty log directory are pretty big clues, aren't they?

    For sure. Though you're probably focusing too hard on the auth.log entry. Actual authentication seems terribly unlikely unless there's a default account/pass in their OS images that you weren't aware of. What I'd have been interested in knowing, and you might take note of it next time, is what user account the processes were owned by / running under. If your individual services don't run as root, that should tell you which user account was compromised and through what service. Maybe munin has a vulnerability, for example.

    I'm assuming you hadn't deployed a dynamic web app yet. You just said Nginx not Nginx + PHP + Wordpress or something. That's the culprit in 999 out of 1000 cases but you're no doubt well aware of that.

    Thanked by 1pwned
  • pwnedpwned Member

    I used the standard debian netinst files from debian.org, so I hope there's not a default account! (As I mentioned, I selected the no root account option in the installer.)

    The only user account on the box is mine.

    No webapps, PHP, wordpress, etc. The only change to the nginx config was to create an alias to the munin directory, /var/cache/munin/www. Munin was listening on 127.0.0.1:4949, not 0.0.0.0, and iptables wasn't open for tcp/4949.

    Sorry, I did run a ps ax, not ps uax:

      PID TTY      STAT   TIME COMMAND
        1 ?        Ss     0:03 /sbin/init
        2 ?        S      0:00 [kthreadd]
        3 ?        I<     0:00 [rcu_gp]
        4 ?        I<     0:00 [rcu_par_gp]
        6 ?        I<     0:00 [kworker/0:0H-kblockd]
        8 ?        I<     0:00 [mm_percpu_wq]
        9 ?        S      0:02 [ksoftirqd/0]
       10 ?        I      0:09 [rcu_sched]
       11 ?        I      0:00 [rcu_bh]
       12 ?        S      0:00 [migration/0]
       14 ?        S      0:00 [cpuhp/0]
       15 ?        S      0:00 [kdevtmpfs]
       16 ?        I<     0:00 [netns]
       17 ?        S      0:00 [kauditd]
       18 ?        S      0:00 [khungtaskd]
       19 ?        S      0:00 [oom_reaper]
       20 ?        I<     0:00 [writeback]
       21 ?        S      0:00 [kcompactd0]
       22 ?        SN     0:00 [ksmd]
       23 ?        I<     0:00 [crypto]
       24 ?        I<     0:00 [kintegrityd]
       25 ?        I<     0:00 [kblockd]
       26 ?        I<     0:00 [edac-poller]
       27 ?        I<     0:00 [devfreq_wq]
       28 ?        S      0:00 [watchdogd]
       29 ?        S      0:00 [kswapd0]
       47 ?        I<     0:00 [kthrotld]
       48 ?        I<     0:00 [ipv6_addrconf]
       49 ?        I      0:00 [kworker/u2:1-events_unbound]
       58 ?        I<     0:00 [kstrp]
      112 ?        I<     0:00 [kworker/0:1H-kblockd]
      142 ?        I      0:00 [kworker/u2:2-events_unbound]
      169 ?        I<     0:00 [kworker/u3:0]
      171 ?        S      0:00 [jbd2/vda2-8]
      172 ?        I<     0:00 [ext4-rsv-conver]
      204 ?        Ss     0:00 /lib/systemd/systemd-journald
      222 ?        Ss     0:00 /lib/systemd/systemd-udevd
      256 ?        I<     0:00 [ata_sff]
      259 ?        S      0:00 [scsi_eh_0]
      261 ?        I<     0:00 [scsi_tmf_0]
      262 ?        S      0:00 [scsi_eh_1]
      264 ?        I<     0:00 [scsi_tmf_1]
      269 ?        I<     0:00 [ttm_swap]
      378 ?        Ssl    0:00 /usr/sbin/rsyslogd -n -iNONE
      386 ?        Ss     0:00 /usr/sbin/cron -f
      388 ?        Ss     0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
      408 ?        Ss     0:00 /usr/sbin/vnstatd -n
      411 ?        Ssl    0:01 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:112
      417 ?        Ss     0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
      418 ?        S      0:00 nginx: worker process
      650 ?        Ss     0:00 /lib/systemd/systemd --user
      651 ?        S      0:00 (sd-pam)
     2150 ?        Ss     0:00 /usr/bin/perl -wT /usr/sbin/munin-node
     2741 ?        Ss     0:00 /usr/sbin/sshd -D
     3011 ?        Ssl    0:03 /etc/my.conf
     3025 ?        Ssl    0:03 /var/ssh.conf
     3050 ?        Ss     0:00 /lib/systemd/systemd-logind
     3124 ?        Ssl    0:00 /usr/bin/.sshd
     3178 ?        Ssl    0:00 /usr/bin/pythno
     3235 tty1     Ss+    0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
     3417 ?        I      0:00 [kworker/0:0-ata_sff]
     3427 ?        I      0:00 [kworker/0:1-events_power_efficient]
     3428 ?        Ss     0:00 sshd: my_username [priv]
     3431 ?        Ss     0:00 /lib/systemd/systemd --user
     3432 ?        S      0:00 (sd-pam)
     3440 ?        S      0:00 sshd: my_username@pts/0
     3441 pts/0    Ss+    0:00 -bash
     3451 ?        Ss     0:00 sshd: my_username [priv]
     3457 ?        S      0:00 sshd: my_username@pts/1
     3458 pts/1    Ss     0:00 -bash
     3484 ?        I      0:00 [kworker/0:2-ata_sff]
     3590 ?        I      0:00 [kworker/u2:0-events_unbound]
     3629 pts/1    S+     0:00 ps ax
     3630 pts/1    S+     0:00 sh -c /usr/bin/dpkgd/ps ax
     3631 pts/1    R+     0:00 /usr/bin/dpkgd/ps ax
    
    
    Thanked by 1jar
  • raindog308raindog308 Administrator, Veteran

    /usr/bin/dpkgd/ps ax

    If this is a compromised binary, then you have no idea what was running.

  • pwnedpwned Member

    @raindog308 said:
    /usr/bin/dpkgd/ps ax

    If this is a compromised binary, then you have no idea what was running.

    It is--you're absolutely correct.

  • jarjar Patron Provider, Top Host, Veteran
    edited June 2020

    I guess any reasonable way you spin it if /var/log is empty it was rooted. It wasn’t booted into single user mode or anything since your screen session was alive. So even if they got a console they weren’t guessing your password.

    Interesting set of variables. I’m a bit curious too. Compromised SSH key really isn’t a standard thing for this kind of behavior, that’s too personal and specific where these attacks are most typically automated and not targeted (granted can’t rule out that this one was). Compromised password still possible I suppose but seems like you’d have a log of that in your screen session and it would probably mean you’re compromised somewhere that password was noted.

    Thanked by 1pwned
  • pwnedpwned Member

    I should clarify--I had screen running on my local box, ssh'd into the victim, with tail -F /var/log/auth.log running on the victim. When the victim rebooted I lost the connection. If it had been a ssh compromise, I'm pretty sure I would have seen at least the sssh connection start in tail before they had a chance to cover their tracks, right?

    Now if they somehow had a console, they could have booted single user at grub, installed the rootkit, then rebooted. I was able to ssh in after the break.

    If it was targeted, I would think the 8GB quad-core black friday special would have been a better choice than the 384 kB KVM-lite instance...

    Thanked by 1jar
  • that's scary.

  • Haven't run debian in years but doesn't it have a recovery option on boot up that auto logs in via root? A compromised vnc session might get in that way

    Thanked by 1pwned
  • pwnedpwned Member

    @sureiam said:
    Haven't run debian in years but doesn't it have a recovery option on boot up that auto logs in via root? A compromised vnc session might get in that way

    I'd have to check, but I'm pretty sure that recovery option hangs if there's no root account, as in this case.

    I've just about convinced myself to re-install as before just to see what happens.

  • lonealonea Member, Host Rep

    Most likely your VNC connection had no password on, and someone just vnc in and did a single mode boot.

    Thanked by 1pwned
  • DPDP Administrator, The Domain Guy

    Perhaps you should let your provider know what’s going on and your findings. Just an ‘FYI’ for them and if they’re willing to help or provide feedback, then they just might.

    Thanked by 1pwned
  • pwnedpwned Member

    @lonea said:
    Most likely your VNC connection had no password on, and someone just vnc in and did a single mode boot.

    If so, How can I prevent that in the future? I'm not sure, but I thought I'd turned VNC off at the control panel. Assuming it was on, doesn't that mean someone compromised the host to gain access to my control panel?

    I certainly don't run a VNC server on the VPS.

  • lonealonea Member, Host Rep

    No, they connected via the vps node's IP and VNC port, ip:5900, ip:5901, etc

    Try to see if you can connect to it.

    @pwned said:

    @lonea said:
    Most likely your VNC connection had no password on, and someone just vnc in and did a single mode boot.

    If so, How can I prevent that in the future? I'm not sure, but I thought I'd turned VNC off at the control panel. Assuming it was on, doesn't that mean someone compromised the host to gain access to my control panel?

    I certainly don't run a VNC server on the VPS.

    Thanked by 1pwned
  • pwnedpwned Member

    Someone asked for the files I grabbed. They're available at https://we.tl/t-8RuPUWz9t1

  • pwnedpwned Member

    @lonea said:
    No, they connected via the vps node's IP and VNC port, ip:5900, ip:5901, etc

    Try to see if you can connect to it.

    No connection. No process listening on those ports, and it's not open in iptables.

  • jarjar Patron Provider, Top Host, Veteran
    edited June 2020

    @pwned said: I should clarify--I had screen running on my local box, ssh'd into the victim, with tail -F /var/log/auth.log running on the victim. When the victim rebooted I lost the connection. If it had been a ssh compromise, I'm pretty sure I would have seen at least the sssh connection start in tail before they had a chance to cover their tracks, right?

    Now if they somehow had a console, they could have booted single user at grub, installed the rootkit, then rebooted. I was able to ssh in after the break.

    Ah ok, now that does sound plausible.

    @pwned said: No connection. No process listening on those ports, and it's not open in iptables.

    Makes me wonder if the host was compromised.

    Thanked by 1pwned
  • DPDP Administrator, The Domain Guy

    @jar said: Makes me wonder if the host was compromised.

    Hence why I advised to raise this to the provider as perhaps it could ring some bells or even light up bulbs on their end. They still need to be aware of what’s going on and be on standby in case other ‘neighbors’ are reporting the same.

    Thanked by 1pwned
  • pwnedpwned Member

    @thedp said:

    @jar said: Makes me wonder if the host was compromised.

    Hence why I advised to raise this to the provider as perhaps it could ring some bells or even light up bulbs on their end. They still need to be aware of what’s going on and be on standby in case other ‘neighbors’ are reporting the same.

    Will do. I was hesitant in case someone found an obvious mistake on my part (why waste the host's time in that case), and it's a black-friday-no-support deal, so this might cost me $5. But at this point I'm curious enough to accept the charge if it gets us an answer.

  • DPDP Administrator, The Domain Guy
    edited June 2020

    @pwned said:

    @thedp said:

    @jar said: Makes me wonder if the host was compromised.

    Hence why I advised to raise this to the provider as perhaps it could ring some bells or even light up bulbs on their end. They still need to be aware of what’s going on and be on standby in case other ‘neighbors’ are reporting the same.

    Will do. I was hesitant in case someone found an obvious mistake on my part (why waste the host's time in that case), and it's a black-friday-no-support deal, so this might cost me $5. But at this point I'm curious enough to accept the charge if it gets us an answer.

    It’s a security-related matter so it should go above everything 😊

    Thanked by 1pwned
  • PulsedMediaPulsedMedia Member, Patron Provider

    @pwned said: Will do. I was hesitant in case someone found an obvious mistake on my part (why waste the host's time in that case), and it's a black-friday-no-support deal, so this might cost me $5. But at this point I'm curious enough to accept the charge if it gets us an answer.

    whatever host you went with is asking $5 per ticket? :O
    Whatever the case is, you should raise a ticket with them just to be sure.

  • tetechtetech Member

    Sounds like a VirMach BF special. I'd agree to raise a ticket, and say that you're not looking for help setting up your VM but want them to be aware in case there is a vulnerability on the node.

    Thanked by 1pwned
  • @pwned said: why waste the host's time in that case

    Are we sure that we're on LET?


    I'd also guess the VNC part. Maybe your host has some VNC running and you don't need to have a port open to get that VNC to be connected. (It will even show you rebooting)

    I've seen a similar setup at some providers. They don't use your IP address but some IP address from the provider. It's possible that password was cracked and someone was able to login using VNC.

    Thanked by 1pwned
  • raindog308raindog308 Administrator, Veteran

    @pwned said: Someone asked for the files I grabbed. They're available at https://we.tl/t-8RuPUWz9t1

    You may find it interesting to run:

    strings .sshd

    If you're not familiar, the 'strings' command pulls out all strings (English and other words) in a binary.

    That .sshd binary has

    • a list of 188 IP addresses
    • references to many other binaries
    • The word "Google" which doesn't appear in my OpenSSH binary LOL
    • references to Mozille, AppleWebKit, etc.
    • The string "#!/bin/bash" and commands to put things in /etc/rc.d
    • C++ stuff (OpenSSH does not use C++)
    • mentions of many files and binaries on the system
    • the name and address of a guy in Denmark, though looking him up, I believe that's just an author credit from some binary packed inside it

    The IP addresses (just judging by the nslooked-up hostnames) are mostly DNS servers. Lots and lots of .cn there, as well as cnmobile.net, chinamobile.com, etc.

    Same IP list is in pythno, and since it's the same binary, ssh.conf (which is not a conf file but rather a binary). Also same IP list in my.conf (also a binary).

    Not really my specialty but I'm sure someone used to deconstructing binaries could learn more from these.

  • VirMachVirMach Member, Patron Provider

    We've worked with @pwned to resolve this issue.

    A total of 47 virtual servers could have potentially been affected by this, and we will send out communication soon. We did already patch everyone; we are just triple/quadruple checking everything to make sure we didn't miss anything before sending out more information. This was related to a specific fix for a problem that we discovered in regards to SolusVM's configuration for certain VMs which resulted in higher than normal idle CPU usage. We are also working on our own layer of security to avoid this from potentially being an issue in the future. We've informed SolusVM and have also requested they consider making some changes.

    I actually don't believe anyone else was affected to the same degree. This was just a combination of very specific and rare scenarios. It definitely could have been handled better -- our staff followed some direct instructions from SolusVM instead of questioning it. Said staff was instructed not to use the fix in that manner, but it was never reverted. The fix should have theoretically worked out, but it just wasn't ideal.

    We never pushed out this fix in this manner outside of a single node, but we are checking others as well. I'll return with more information later.

Sign In or Register to comment.