All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
INIZ NY Migration
INIZ had a migration scheduled and my VPS is down for 3 days now. Got this note yesterday
"Hello,
We are continuing to work for a faster resolution for clients on NYC4KVM however we are running out of luck. All our cables have been reseated and controller flashed with latest update without any luck. A replacement has been ordered along with spares and we are already reviewing what spares were previously available in all our facilities to prevent such long outages in the near future.
In addition, we hope to avoid any data loss so are not attempting any unusual quick fixes that may work.
Few requests were submitted to our ticket system regarding compensation, we will start reviewing this shortly and send a further email once we have came to a decision regarding this however at this time our number one priority is to get clients back online as quick as possible.
Regards,
INIZ Management
Zeniva Limited"
and it is still down. Has anyone else faced this issue.
Comments
Shit happens, especially with migration. Maybe someone dropped the server, who knows...
As you probably know, the warranty is limited tot he money paid and compensation will be given for downtime, best you can do is pray and prepare your backups and a plan for redundancy, next time you wont have a downtime.
"The Cloud"
That to me says the storage system has failed, did they have the migration transferred before the issue?
Sounds like they are still working on it.
Have you raised a ticket?
I'm also offline because of this. Super annoying.
Well it shouldn't be "super" annoying, you were told beforehand that this move was going to happen, things can happen in the meantime..
Realistically it's your fault for not having some sort of failure plan..
@joereid said:
why not just use your backups and make a new instance elsewhere till everythings on?
I'm only using it as a dns server, and i have 3 others, so no need for temporary setups.
However it is still super annoying to be told to expect a few hours downtime and get a few days and counting. Especially with very little communication. I got the one email as OP but that's it. A status update or explanation after a few days might be nice.
I also heard about INIZ moving servers in LA too very soon. Hopefully that goes smoother...
They pre-warned about the move and issued a statement as above about the issues.
I don't know about you guys but when shit hits the fan I would rather have people trying to resolve the issue and solve it rather than sat in the ticket desk answering the same response to all users...
Also if they have suffered data loss I can understand why they are taking their time and giving it the due care and attention that kind of recovery needs rather than just shrugging it off and saying we lost the data, recreate the VM's from backups.
>
>
Stop jumping on my back. I posted that I'm annoyed by the situation, that's my personal opinion. I'm allowed to have one of those right? I'm not screaming scam or demanding compensation or publicly bashing INIZ and I haven't even opened a ticket with them wasting their time so they can fix the problem. I know what's going on, but it still annoys me.
By little communication I'm talking about AFTER they had problems. Sure they issued a statement, 12 hours later.
I'm annoyed by the "Murphy's Law" of it all. Of course the ONE node that won't come up is the one I'm on. Just like when they move their servers in LA I'm sure I'll be on the one node there that has problems too.
My ovz with them was back up same day no issues for me.
Don't! Nothing makes him happier and next time will come looking for you. I know him well.
To be honest, no communication was not at all bad. Iniz did send out info on the upcoming relocation weeks in advance and multiple times. In particular they also sent out a "last minute warning reminder".
And they had a status page online that was very well kept up to date.
Where things turned ugly was when there was a serious failure with (at least) on KVM node. Then the status updates stopped, everything turned quiet and a vague sounding guesstimate for friday was emailed to clients.
Even asking in tickets rendered no more than a "tomorrow" response.
From what I have understood, a raid card broke and a new one had to be ordered. This by itself it a bad sign and not indicating professionality. But I guess (again: this is just my personal guess and I might be wrong!) that they fell into the very raid trap that make responsible professional hosters a) have spare cards and b) the same version as installed.
My guess (see above disclaimer) is that they have seen the beast by now but are somewhat at a loss because the original plan "plug in new raid card, boot node an be done" didn't/doesn't work out because the new card doesn't read/work the old cards data.
Evidently "Sorry, we were stupid/negligent so all your data is lost" isn't an attractive option so I guess (see disclaimer above) they are right now under pressure (it's friday) to somehow save the old data, maybe by properly transferring another node to a new card so as to gain the old card for the broken node.
Kindly note that this is not a post against Iniz but rather a post pro good communications (which unfortunately stopped when it became the most needed). I have otherwise no anger against Iniz; they've provided good service and were friendly.
Wiki doesn't have any more updates as only one node down still. Those affected clients were emailed I believe twice in the past 24 hours assuming the SolusVM email is up to date then you would have received that. They were brief but we/I processed all the usual tests and debugs without much luck such as reseating etc. So the only option left is waiting of replacement part. Which of course is not unacceptable with the huge delay so it's being looked into.
Another email will go out when we have something to say that is constructive in anyway rather than just a general reply like "we're still looking into it..."
Thanks for being patience guys, we hope to be back up very soon
I can tell it was a hard day/night/day whatever. I know how it feels.
Sadly yup, I have heard some other providers whom I keep in touch with have had some bad luck in the past few days as well as you guys too
How many nodes is that? That sucks man.
He means all are up except one. But he is obviously tired and that monster came out.
Ha yes my bad, fixed that.
Oh lol, sorry
Update and finale:
It seems I was about right thinking that the new raid card wouldn't work.
Yesterday evening I got an email from Iniz telling me about my new KVM VPS on a new node.
In other words: All my data were gone, the old VPS was dead as a nailed parrot.
I was, of course, not at all pleased to read that and to have lost everything on my VPS.
Before continuing let me clearly state that my point isn't bashing Iniz. While they were obviously not well prepared, damn, shit just happens, and besides the obvious failure, Iniz did a good job and have always been a good provider for me (more than a year).
Also front up: I got a free month added and I didn't even ask for any compensation. One might discuss that whole thing back and forth but, hey, that's a pretty decent step from Iniz and I appreciate that token of them being sorry.
What was good and what was bad?
Definitely good was their initial open information and communication policy.
Less good was that policy rapidly degrading to emails being sent, and not always in a hurry it seems and getting more vague.
Definitely not good was them not having a version matching spare raid card and such "inviting" a desaster.
Not good was that in the end there was a multiple day down-time.
Definitely good was their "sorry" in the form of a free month added.
Also noteworthy and definitely good was their support. I can't remember (during these days) ever waiting longer than 15 mins. or so.
What is to be learned (besides the obvious spares issue)?
Guys, COMMUNICATE! Communicate early and communicate a lot!
Let me explain: We humans are wired to desire understanding what happens. Putting it bluntly: You providers can not stop clients from doing that. What you can do, however, is to decide whether you are somewhat in control or whether you want weird night-mare and conspiracy generators to fire off.
So: f-cking communicate! Don't be picky or shy! While most clients are, of course, p-ssed off by trouble, they do know quite well that bad things can happen even to the best. And they do know that everyone is human and sometimes does sh-t.
The decisive issue, once the sh-t hits the fan, is to communicate, to keep clients from getting afraid and worried or making up weird sh-t in their heads or, worse, spreading that sh-t and ruining your reputation. Tell them, tell everything, tell it right away. Doing that you can even WIN something in a bad situation.
And again, to avoid misunderstandings. I was a happy Iniz client and I will stay an Iniz client. All in all they at least didn't do worse than many others and they did at least communicate properly in the beginning. Moreover they did what's also important, they had their support staffed and react quickly to help clients to smooth that ugly ride.
P.S. And I'm talking as a really cheap client, in the 20-25$/year range for a KVM. Considering that Iniz looks even better. Not every company would have cared that much for the low end of the client ladder.
Yep, trying to hide anything is a bad decision, might work once on a sparsely populated node, but it will not work always. Once is enough to ruin all you built in years.
That being said, most of the time it is not provider's fault. You had a raid card fail? Well, that will keep happening once every few wiles, the more nodes you have, the more often. Multiple disks failure is also possible, especially in a power failure event.
Even if it is human error or cheap hardware, if you are open about it, people will not blame you, but those who are:
1. Without backups (data saved);
2. Without backups (stand-by spare(s), especially in a "multi-million dollar business" hosted on a 2$ random box.
Of course, if you advertise back-ups and those fail, there will be a lot of eggs on your face, even rotten tomatoes, but, in the end, that is a possible scenario too, even if you do take good care. We were just lucky pm14 just finished doing backups when power failed, otherwise would have been a very ugly situation since we only keep one round and incomplete backups interrupted by power failure would have most probably been unusable and, while we do not advertise backups except for selected Biz plans, people came to expect them for some reason, I was even offered bribes to restore some VM where the user screwed up or was hacked, thinking we do not restore it for some weird reason even if we have them...
So, do not rely on anyone, not even god, do your backups!
@Maounique
That's the one point I disagree (assuming you mean clients).
No matter the backup, having to set up the OS again is a PITA. Moreover, no matter how often it's preached, fact is that many, many people do not make backups (frequently enough, reliably enough, ...). Sure, you may tell them "Your fault, idiot!" but that certainly won't get them less p-ssed off.
Turn it as you like, most clients just expect things to work. To them, the hoster is "the professionals" and they expect him to do all the right things and to avoid all their own sins (like not backing up properly).
I agree, already included that, nobody likes failures, but when they will happen (because they will happen to everybody), it must be a lesson learned, not a blame game, I know what the customer expects, we are blamed for hacked servers too, because we did not care for security, but this is not the point here, an unmanaged service is, well, unmanaged, customers must know what to expect.
I am on NL node, and no problems for years. ;-) One of the most stable hosting ever. Thanks Patrick!