New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
ChicagoVPS hacked, bunch of VPS customers offline
This discussion has been closed.
Comments
@Jack please remind me all your post here after chief taking back the thanks button..
and please keep updating us!
All I can say at this stage is thy haven't contacted us about it so I'm doubtful it's an actual exploit through SolusVM at all, however it's too early to tell at the minute.
--
Kind Regards,
Jason Smith
Management
Good work @Jack!
Can you explain what this means? Is there an exploit @ SolusVM or was there simply an issue @ chicagovps?
I just mean good work on getting some info out of SolusVM.
thy?
I cant accept that if an exploit like this was around that ChicagoVPS would be the first reported case, it would have happened already to somebody else.
I would be more willing to believe it if they didn't say it was an API bruteforce. The shear amount of computing power required to guess 2 completely random 100bit values at the same time is not cost effective to hack a provider only to impact a handful of nodes.
Exactly & it wouldn't have happened this quick.
So since this is a lighttpd related issue, why not switch to nginx?
Just a though?
We don't really know how the seemingly random api key is generated. If it is something as stupid as using a function seeded from time() then it is relatively easy to bruteforce it. If it gets it's random data from /dev/urandom - then brute forcing should not be possible.
And i don't have the source for SolusVM - i can only guess.
I'm assuming you mean server instead of website? Well, in that case it blows :P
On nginx you restrict the admin api to the whmcs host. If you want the client api too.
location ^~ /api/admin { allow xxx.xxx.xxx.xxx; deny all;include /etc/nginx/php.conf; fastcgi_index index.php; index index.html index.php; if (-f $request_filename) { fastcgi_pass 127.0.0.1:9000; } fastcgi_param SCRIPT_FILENAME /usr/local/solusvm/www$fastcgi_script_name; }Doh! Fixed.
Hey everyone, just a quick update before I head to bed to get some rest and finish up later today.
We worked very hard to get everyone back online. What we have done so far is just installed fresh installs of everyone's VPS to get them back online. Later today we will start to attempt to recover everyone's VPS. This is not a guarantee but its a shot.
We thank everyone for not freaking out. This was a really big event, little over 1000 VPS's lost. I think this was handled well, and quickly. If you have any concerns you can open a ticket but probably will not get answered for a few hours since we are all resting.
We will give appropriate credits to everyone. Sorry if my wording doesn't make sense as I am very tired ;-)
Thanks again.
Chris
@CVPS_Chris
Sure it was, however why wasn't solusvm notified of this exploit? Or was it CVPS trying to cover something up?
Still down here for me
From chi-vps29.chicagovps.net icmp_seq=1 Time to live exceeded
Hopefully data recovery is possible. I'll wait for that.
@kujoe has given us some good details, i don't see how chicagovps will cover up anything
They called, Jeremiah didnt answer as we are more focused on getting people online than go explain the problem. At this point, if it happens to someone else it happens to someone else. Im going to put my customers first.
Goodnight
I doubt this is some widespread exploit, more likely someone got pissed off at Crhis for his attitude and is trying to punish him personally.
+1
+2
But I don't agree with such tactics.
There always is some exploit that someone knows about but has no need to exploit.
Chris probably annoyed someone enough to make it worth their while to exploit.
Whatever it is is probably still there, and there are probably a few more that can be exploited.
The only real defence against this kind of events are regularly secure backups by customers as well as good recovery procedures on the part of the providers.
More free publicity for Chris as usual!
@CVPS_Chris and @jshinkle , if this is indeed a SolusVM security issue, you have an obligation to not only the LEB/LET community, but the internet community as a whole, to share any relevant information with @soluslabs and the rest of us so that anyone who is using SolusVM can take the proper precautions. You may see it as "Well, we got hacked and we don't give a shit if anyone else gets hacked, it's not our problem", but the fact of the matter is that by hiding this information, you're being a bad internet citizen.
I'd say webhosts being hacked might not be the best kind of free publicity, being a jerkoff to other providers is one thing but this would likely turn me off as a client if I hadn't been already.
I wonder who did it and if they got any pics of how badly CVPS nodes are oversold.
The bad guys clearly already have it.
Possibility to bruteforce undetected is something I'd consider a vulnerability - but I'm not quite sure I'd lay the blame with ChicagoVPS here.
Um, yes, they are. Bruteforcing is one of the most common techniques to get into something, and not having some kind of protection against that is straight-out negligence.
Huh. How does a software-specific intrusion have anything to do with a generic HTTPd? Or was it not the admin API after all?
And i don't have the source for SolusVM - i can only guess.
And another reason Ioncube is evil
His obligation is to his paying customers first and foremost, surely? Chris has already said that he'll provide more details once his business is sorted out, hardly the 'we got had and don't give a shit about anyone else' attitude you've just described.
Certainly, that's his first obligation. The time that he spent patting himself on the back for how he's handling the situation, though, would probably have been better spent talking to someone from Solus and posting the details here.
@CVPS_Chris
Even though I do not like your attitude - This is something I would not have wished to happen to you. All the best and my empathy concerning this shit!
They had to have been bruteforcing for hundreds of years to even have a remote chance of guessing both the random alphanumeric API username and password at the same time. While I am all for more security, it would be like adding kevlar to a tank.
In my opinion, that's an unnecessarily harsh criticism - a post providing an update to customers, explaining the situation, thanking people for their patience and with a few self-congratulatory words is hardly 'time spent patting himself in the back'.
I'll be the first to admit that I know next to nothing about SolusVM, and how much time it would take to discuss a potential issue with them, but I am prepared to bet it's a lot longer than the time it took to write that message.