Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

What's up with Verelox being down?

What's up with Verelox being down?

bluesegabluesega Member
edited June 8 in Providers

Hello guys,

It seems @Verelox is down at the moment, checked this night (so about 9 hours ago) and it was not reachable (DNS seems down), and now from work (where DNS has still address in cache) it's still down, saying target network is not reachable. Does this happen for you as well? Please try https://verelox.com This does not only apply to their website, my server hosted there also seems to be down, can't reach it and they usually provide only a contact form to reach. I'll try PM them here, hope they're not dead or being under heavy attack or so.

Greetings

«1

Comments

  • bluesegabluesega Member

    @ewrek said: try this link: http://verelox.com/

    Does this work for you? It says connection closed here and ping says --- verelox.com ping statistics --- 9 packets transmitted, 0 received, +6 errors, 100% packet loss, time 8019ms

  • ewrekewrek Member

    @bluesega said:

    @ewrek said: try this link: http://verelox.com/

    Does this work for you? It says connection closed here and ping says --- verelox.com ping statistics --- 9 packets transmitted, 0 received, +6 errors, 100% packet loss, time 8019ms

    It says this to me:

    We’ll be back soon!

    Our network and website is currently under maintenance. We’ll be back shortly.

    — Verelox

  • bluesegabluesega Member

    Okay then. Can't tell about this because all browsers are redirecting to https:// always and then tell it's not reachable, probably it is with :80 and http://, but sadly can't tell. A mail from them would've been nice though. (Is a screenshot possible for you please?)

  • bluesegabluesega Member

    Thank you very much. Let's see how long this takes, and of course the reason for that. A > 14 hour maintenance is kinda unusual without prior announcement.

  • Did this ever happenned before? And is there any info whether this will take hours or days?

  • @bluesega said: Thank you very much. Let's see how long this takes, and of course the reason for that. A > 14 hour maintenance is kinda unusual without prior announcement.

    I hope you have your own backups.

    || Look for VPS Hosting Offers ||

  • @TheRealMakan said:

    @cyberpersons said: http://imgur.com/a/qgpa1

    That looks a bit familiar... Hmm..

    https://gist.github.com/pitch-gist/2999707

    I hope they will be back sometime soon.

    || Look for VPS Hosting Offers ||

  • sinsin Member

    I can't reach their control panel, it just times out.

  • bluesegabluesega Member

    @petarpan22 said: Did this ever happenned before? And is there any info whether this will take hours or days?

    No, it hasn't. I mean, it would've been fine if they just announced this by mail or on their social media channels if this was a problem on short notice. But I assume they got some problems they aren't to blame for (e.g. something at SmartDC, etc.) and have to solve this now or a mentioned attack.

    Can't really imagine they going offline because they've just recently opened a new location in US and also discussed with me about future plans there.

    @cyberpersons said: I hope you have your own backups.

    I do, but uploading about 300GB onto the server would take some time again :/

  • I do, but uploading about 300GB onto the server would take some time again :/

    If they don't come back, there is no other alternative, you should save backups to some offshore server, even if you are keeping them in your personal computer.

    || Look for VPS Hosting Offers ||

  • sinsin Member

    Well it came back up for me but then I just checked and now it's down again.

  • sinsin Member

    Here's what they posted on their Facebook

    We have a security issue with our network/website, we are currectly busy to resolve the problem.
    As expected some servers will be online in 2/4 hours.
    
    More updates coming up...
    
    During this downtime servers are not billed and all client accounts are freezed.
    
    Sorry for the inconvenience.
    
    Kindly Regards,
    
    Verelox Staff.
    
  • DextronoxDextronox Member
    edited June 9

    If you specify WWW, this shows up.

    First of all, we want to offer our apologies for any inconvenience.

    Unfortunately, an ex administrator has deleted all customer data and wiped most servers. Because of this, we took the necessary steps to temporarily take our network offline. We have been working hard to recover the data but this was not possible for all data that was lost.

    Our network and hosting services will be back this week with securty updates. Current customers who are still interested in our services will receive compensation for their services. If clients have important data please contact us at [email protected] We will try our best with our technical team to recover you data.

    Verelox

    Thanked by 1sin
  • sinsin Member
    edited June 9

    Dextronox said: If you specify WWW, this shows up.

    Wow that's nuts! Hope you're able to get your data somehow @bluesega :-(

  • MikePTMikePT Member

    Damn that is not nice... I wish @Verelox the best luck. Feel free to pm me if you need any help folks.

    European NOC legally registered - Providing Ticket Support/ Server Monitoring Solutions

    Thanked by 1inthecloudblog
  • tranzmediatranzmedia Member, Provider

    All the best @Verelox:)

    Tranzmedia Premium Web Hosting| Shared Hosting | Reseller Hosting | VPS Hosting | Dedicated Server
    Web hosting starting at $3.99/month 24x7 Support 99.9% Uptime Guarantee - Daily Backups
    https://tranzmedia.com

  • ClouviderClouvider Member, Provider
    edited June 9

    If they say the truth the cited announcement is missing a note about police involvement since in most jurisdictions the 'ex-admin' would have not only civil but potentially also a criminal case to answer.

    Hope they recover from this. Fingers crossed!

    Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

    Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

    Thanked by 2Bopie netomx
  • bluesegabluesega Member

    Not exactly sure what I should be thinking about this tbh. First of all, it's a real pity of course, and I now understand why they weren't able to tell us by mail, etc. about the stuff, due to the missing customer data. Also hoping that they'll resolve everything possible for them and that they'll be able to continue to operate services. But what also comes to my mind is that this "ex-administrator"-issue sounds...please forgive me my words, but it sounds childish. Like, I don't know if this guy made his way into the system after he was fired, gone, etc. via security breach, but it sounds like company structures are not clearly set (admin didn't know about his position, not being CEO and stuff) and basically they had some sort of fight with this guy, leading him to planning some sort of "revenge", with basically him being an sorry, idiot and they either continued having a fight with him or weren't realizing that (if it wasn't a security breach) you have to remove ex-members of the team's access completely. I've never heard this kind of issue in a real company, only in some sort of private webpages or projects from people, that's why I'm thinking like this. Again, please do not take this as an assault on Verelox, I just couldn't stop thinking about this regarding the reason they are stating.

    If this was a security breach, then please, just do everything in order for him taking legal consequences for this. He deserves it.

    That being said, I'm still planning to continue operating with them and waiting for them to come up online again. But I guess I'll focus some more on how they are going to trying to prevent this in the future. RAID 10 doesn't help if the one responsible for data loss is on the front side. Please understand my opinion about this due to it was my main server, basically operating 90% of my services and therefore being packed with customized configurations of which I'm not sure I have a up-to-date backup (which is my fault, yes, but still ...) and other reasons (not gonna state all, if I'll get the same IP again, it'll be fine though). Thank you for doing that, and @Verelox - best luck to you to get onto your feet again, I'll be waiting.

    Thanked by 1Falzo
  • bluesega said: That being said, I'm still planning to continue operating with them and waiting for them to come up online again.

    Whyyyyyyy

    Signatures are to identify who I am. I'm me. Who the hell are you?

  • bluesegabluesega Member

    @rajprakash said: Whyyyyyyy

    Because their services were very good and I liked it, but I guess it'll depend on how they'll perform from now on. If things stay as nice as they were before that faux pas, it'll be fine. They deserve a second chance.

  • sibapersibaper Member

    @bluesega do you believe the're competent enough to do their job? because they dont keep backup of their billing database.

    Thanked by 1k0nsl
  • @sibaper said: @bluesega do you believe the're competent enough to do their job? because they dont keep backup of their billing database.

    I'm not saying this in a way to defend them, but there's a chance that they had a backup which was also wiped by their ex-administrator. I have no idea how high up that employee was in their company, but there's always a chance that it was wiped by that staff member (although it could be questioned whether they could have done more to prevent staff deleting backups through some kind of muti-level backup system).

  • VereloxVerelox Member, Provider
    edited June 9

    Thanks everyone for all kind words. We have a status update, all dedicated servers located in The Netherlands should be online next hour. We are still working on a solution for the cloud servers.

  • Verelox said: online next hour

    Online with customer data? Or just online with fresh installs?

    Signatures are to identify who I am. I'm me. Who the hell are you?

  • TomTom Member

    @Verelox best of luck. Did you have backups & the ex-admin wiped these too?

  • VereloxVerelox Member, Provider

    @rajprakash said:

    Verelox said: online next hour

    Online with customer data? Or just online with fresh installs?

    All servers that are coming online have customer data. Servers that are wiped will not come online right now, we will e-mail those clients with more information.

    @Tom said: @Verelox best of luck. Did you have backups & the ex-admin wiped these too?

    We have a backup of the customer database. Right now we are using the information in this backup to bring up all dedicated servers and cloud servers. We are'nt sure yet if we are missing any data in our backup.

    Thanked by 1MikePT
  • TomTom Member

    @Verelox said:

    @rajprakash said:

    Verelox said: online next hour

    Online with customer data? Or just online with fresh installs?

    All servers that are coming online have customer data. Servers that are wiped will not come online right now, we will e-mail those clients with more information.

    @Tom said: @Verelox best of luck. Did you have backups & the ex-admin wiped these too?

    We have a backup of the customer database. Right now we are using the information in this backup to bring up all dedicated servers and cloud servers. We are'nt sure yet if we are missing any data in our backup.

    Good luck. Why did dedicated servers come down anyway? Did you take your whole infra down?

  • ihadpihadp Member

    @Tom said:

    @Verelox said:

    @rajprakash said:

    Verelox said: online next hour

    Online with customer data? Or just online with fresh installs?

    All servers that are coming online have customer data. Servers that are wiped will not come online right now, we will e-mail those clients with more information.

    @Tom said: @Verelox best of luck. Did you have backups & the ex-admin wiped these too?

    We have a backup of the customer database. Right now we are using the information in this backup to bring up all dedicated servers and cloud servers. We are'nt sure yet if we are missing any data in our backup.

    Good luck. Why did dedicated servers come down anyway? Did you take your whole infra down?

    Taking down the network isn't unusual or unwarranted when you have an attack on your infra and having trouble locating the source.

    Pull the plug on your border routers, cut off all public Access and begin diagnostics using your out of band network.

    www.whatuptime.com
    Microsoft Windows Templates for Online.net, Kimsufi, DigitalOcean, OVH, Vultr & Much More!

  • VereloxVerelox Member, Provider

    @ihadp said:

    @Tom said:

    @Verelox said:

    @rajprakash said:

    Verelox said: online next hour

    Online with customer data? Or just online with fresh installs?

    All servers that are coming online have customer data. Servers that are wiped will not come online right now, we will e-mail those clients with more information.

    @Tom said: @Verelox best of luck. Did you have backups & the ex-admin wiped these too?

    We have a backup of the customer database. Right now we are using the information in this backup to bring up all dedicated servers and cloud servers. We are'nt sure yet if we are missing any data in our backup.

    Good luck. Why did dedicated servers come down anyway? Did you take your whole infra down?

    Taking down the network isn't unusual or unwarranted when you have an attack on your infra and having trouble locating the source.

    Pull the plug on your border routers, cut off all public Access and begin diagnostics using your out of band network.

    This admin installed in our servers backdours when we add the servers online it will be big risk, thats why we reinstalling all our servers and import client data's.

    We expect some VM's will coming up in some hours.

    We are keep updating our websites if there is any server done with restoring.

  • Verelox said: backdours

    Now what kind? Or do you have a clue? How did he take control of the dedicated servers? Something is not adding up!

    Thanked by 1k0nsl
  • VereloxVerelox Member, Provider

    @PieNotEvenEaten said:

    Verelox said: backdours

    Now what kind? Or do you have a clue? How did he take control of the dedicated servers? Something is not adding up!

    All dedicated servers are back online. It was not only the servers in network (switch/router) everything hi got access. We have configure complete everything again.

  • sibapersibaper Member

    @PieNotEvenEaten said: How did he take control of the dedicated servers?

    gotcha! i'm really interesting to hear their answer

  • Verelox said: All dedicated servers are back online. It was not only the servers in network (switch/router) everything hi got access. We have configure complete everything again.

    I'm curious: Where is this ex-Administrator located? Have yours or his local police been notified?

    Thanked by 1Clouvider
  • bluesegabluesega Member

    Wow, even IT magazines now report on this. I'm also curious about the stuff people asked in posts before, but please, focus on restoring the network, etc. first. As soon as all is done, we are eager for you telling us your story. XD

  • FreakensteinFreakenstein Member
    edited June 10

    Whatever the reason or whatever Verelox did to this guy (or whatever he thinks they did to him), he's a massive a-hole. Not only is he causing damage to all the customers who never did anything to hurt him, but also his former co-workers are now having to work around the clock, 24x7, to try and clean up the mess of this spiteful little crybaby. I'm a sysadmin myself and no matter how mad I would get at my bosses, the idea I would screw over my co-workers by having them have to fix the damage I caused, would be one of the first reasons not to do something like this. That and knowing right from wrong of course, and trying to be a decent human being.

    It's almost impossible to keep high level admins from doing stuff like this, especially if he had time to prepare. Yes, you can disable his accounts, but he'll know passwords from administrator service accounts or will have created a second shadow admin account for himself. He could have copied all passwords of all priviliged users in advance. He could have installed a remote control tool in advance. He could have installed a keylogger on a workstation of a co-worker. As an admin, you'll know all the ways in and all the backdoors, and if no backdoors exist, you can create them. There's no company who will change all the passwords for all the privileged accounts every time an admin leaves the company.

    Best of luck to his former co-workers at Verelox! I know how you must feel, and hope that people will understand the situation you're in and appreciate the work you're doing. Pressure must be intense. :(

    Thanked by 1HyperSpeed
  • MikeAMikeA Member, Provider
    edited June 10

    @Freakenstein said: Whatever the reason or whatever Verelox did to this guy (or whatever he thinks they did to him), he's a massive a-hole. Not only is he causing damage to all the customers who never did anything to hurt him, but also his former co-workers are now having to work around the clock, 24x7

    That's why;
    A.) You don't give people control of your whole server/network without a means of removing access.
    B.) You make people sign legal contracts before giving them access to your whole business.

    I've never heard of them but it seems like a mistake a Minecraft server host would make.

    ExtraVM - DDoS Protected VPS - US, CA, FR, SNG

    Thanked by 1HBAndrei
  • Yeah, but you can only remove the access you know of. I can think of a dozen ways I could access the network of my company/customers or destroy their servers after they have disabled all my accounts. A legal contract won't stop someone like this. Even without a contract, this is completely illegal. He'll be prosecuted and probably go to jail. He'll also never get a job in his line of work again. Every tech company is going to avoid him like the plague and won't let him near any of their servers. If that didn't stop him, why would a piece of paper do the trick? Sure, they can fine him, but as we say in my country (and probably his): You can't pluck feathers from a bald chicken.

    Thanked by 1netomx
  • drivexdrivex Member
    edited June 10

    Some people are saying it's related to this: https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/?sort=confidence

    Don't know if it's true, but if it is the CTO should be fired yesterday. Horrible story anyway.

    Thanked by 1williclarkam
  • johnklosjohnklos Member

    @bluesega said: Okay then. Can't tell about this because all browsers are redirecting to https:// always and then tell it's not reachable, probably it is with :80 and http://, but sadly can't tell. A mail from them would've been nice though. (Is a screenshot possible for you please?)

    Probably HSTS: https://tools.ietf.org/html/rfc6797

  • @Freakenstein said: Whatever the reason or whatever Verelox did to this guy (or whatever he thinks they did to him), he's a massive a-hole. Not only is he causing damage to all the customers who never did anything to hurt him, but also his former co-workers are now having to work around the clock, 24x7, to try and clean up the mess of this spiteful little crybaby. I'm a sysadmin myself and no matter how mad I would get at my bosses, the idea I would screw over my co-workers by having them have to fix the damage I caused, would be one of the first reasons not to do something like this. That and knowing right from wrong of course, and trying to be a decent human being.

    It's almost impossible to keep high level admins from doing stuff like this, especially if he had time to prepare. Yes, you can disable his accounts, but he'll know passwords from administrator service accounts or will have created a second shadow admin account for himself. He could have copied all passwords of all priviliged users in advance. He could have installed a remote control tool in advance. He could have installed a keylogger on a workstation of a co-worker. As an admin, you'll know all the ways in and all the backdoors, and if no backdoors exist, you can create them. There's no company who will change all the passwords for all the privileged accounts every time an admin leaves the company.

    Best of luck to his former co-workers at Verelox! I know how you must feel, and hope that people will understand the situation you're in and appreciate the work you're doing. Pressure must be intense. :(

    I agree - I've personally seen colleagues visibly angry with managerial staff but I haven't seen any of them take it to the level of disrupting the service out of anger. Hope I never do, but if you did it leaves you liable to being sued etc so I don't think it's worth it for the weird sort of enjoyment this individual got out of it.

    Unfortunately, as you mentioned sysadmins have access to almost everything so there's a fair amount of trust within the role not to do any damage.

    Jr System Administrator

  • @drivex said: Some people are saying it's related to this: https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/?sort=confidence

    Don't know if it's true, but if it is the CTO should be fired yesterday. Horrible story anyway.

    Doubt it. That thread is 7 days old. The Verelox incident apparently happened 3 days ago. Besides, a single mistake can never delete all servers in different kinds of datacenters. According to the thread he just deleted a production database. But I agree. That CTO should not blame the new guy. It never should have been possible for this to happen. Who in heaven's name puts production admin credentials in a manual for setting up a dev enviroment?! That's waiting for an accident to happen. \0/ Wait, I'm wrong. It's not even waiting, it's rolling out the red carpet and sending an invitation on a golden platter. "You are cordially invited to destroy our production data. Formal attire not required."

  • bluesegabluesega Member

    Interestingly, nobody posted the update from the last days here, so I'll do that:

    It seems that there wasn't any huge data loss after all, can also confirm that my server is up again without any data loss (at least didn't notice one yet).

    Now we're up for your story, @Verelox :)

    https://www.verelox.com

    Thanked by 1netomx
  • sinsin Member
    edited June 12

    @verelox any idea what day the hosting/panel will be back? I wanted to start up a new ashburn server :-)

    -edit- nevermind, I see the new status update says the website and everything will be coming up next.

  • CoreyCorey Member, Provider

    If any sysadmin wanted to do some damage they can probably wreak havoc before being caught, it's just how much access the position needs to do work. You can't restrict a sysadmin too much or they wouldn't be able to do their job, especially if they aren't tier 1 phone support and do more advanced things on your systems.

    BitAccel - OpenVZ VPS / IRC,VPN,Anything Legal & Unrivaled Support!
    Thanked by 2netomx hostdare
Sign In or Register to comment.