New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
If OP prefers, we can move him back on a server that'll probably continue to go down multiple times a week at this point with no support from the datacenter, but I'm not looking forward to those weekly threads.
The optional poll is unrelated to the emergency migration, for those servers, should we move them, customers will receive plenty of notice.
This is a situation that's been escalating and we've unfortunately had no space to move people sooner. We finally had some space after Amsterdam cooled off, but ColoCrossing essentially cancelled our expansion with them in Germany after we waited about a year, and Serverhub failed to fix servers with failing RAID controllers several times. On top of this, they've been having billing issues, one recently where they attempted to terminate our service without notice, and then after confirming everything was good, suspending it. Luckily we have backups either way, but this isn't a risk we're willing to take while the server is also throwing out hardware errors and having difficulty booting back up, with their support being unable to assist us in a timely manner (causing the day-long outages.) Serverhub has also randomly "lost" our servers for 3 days in the past, and there's been a lot of other documented issues with them. One of the servers we had with them for a year never went into production because they couldn't get the hardware working from the start.
Yes, as OP mentioned, other servers have been stable there for something like 5 years, however, after their lifecycle when they do start running into issues, at least in our situation, Serverhub has been abysmal at assisting us so when problems arise we just move them out.
The only server that will be left at this point is GERKVM6, as that one doesn't have immediate hardware issues. We will still most likely be moving that one soon (with a longer notice, assuming emergency situation does not arise.)
This is disappointing to us as well, but it's most likely just Maxmind being Maxmind. They sell their geolocation services at a price that's next to nothing so it's very popular for it to be used by all these "what's my IP" websites that just want to generate ad revenue. And all the conversations I've had with Maxmind, they don't sound too bright. We've got mixed answers before but what ends up happening is they essentially want us to report incorrect geolocations to them one at a time and then half the time they'll actually do something about it but they might just say that we're trying to spoof the IP locations the other half. They have nothing that really pings the IP, or does anything outside of just guessing it.
So what's happening when it goes to "Kansas" is that they have zero confidence in their own guess at that point, they just know the IPs were initially assigned to a US company, so they "average" the US into being Kansas, since that's the center of the US.
Use something more reputable like DB-IP (https://db-ip.com/192.210.175.4) and that'll show the correct geolocation. Only problem is, not a lot of people will use them because they're more expensive since they actually have to spend some money to gather their accurate data. As a side project, I've been working on a geolocation service that doesn't suck and would be pretty cheap so we'll see where that project goes and if we can get people using it so we're not stuck in this weird situation where most of these sites mislabel IP addresses. I believe ARIN also actually lets you use their API if it is for non-commercial purposes so if we make it free, and combine it with some other methods we could maybe even just provide it without a fee to download the database (which I believe is what Maxmind does to save having to serve it to people via API requests, this ends up causing more problems though because when Maxmind updates it on their database, people have to actually get the new one or the sites may still have the incorrect geolocation.)
When we're doing a lot of these together, we might miss a few where the function failed. You can still contact us for support at this point but its good to let people know that this is most likely what would fix it on their end, so they can do it more quickly themselves.
Yep, we do, it's just a backup message in case it failed.
I would much rather have to press buttons to redo my own stupid network config than have the provider bruteforce their way into my VM to do it themselves.
We didn't develop the software, but I believe what it does is power down the VM, mount the KVM image, and modify the pre-generated network configuration if it detects it. When you modify networking or some other default configuration enough, it usually says it can't detect it and fails. I also believe if you encrypted it, that it would fail, but again that's just an educated guess. It wouldn't have to bruteforce its way in there because it can just boot into grub a certain way without having to log in.
Correct, as I mentioned above, but I want to elaborate a little more on the second portion of your message.
We're not a "ColoCrossing shop" or at least that was not our intention. We always used more than one partner, but it was difficult to expand with Serverhub and then later on we ran into issues with them. The reason Frankfurt and Phoenix stayed in limbo is because ColoCrossing included it in an expansion, just for show, so we would expand with them further in other locations and then pulled the plug on Frankfurt after about a year and we finally had to cancel Phoenix because they just held our deposit for it from November 2019 until maybe a month ago when we cancelled it.
Frankfurt has a good chance of returning since there's demand there, from the surveys we sent out. Phoenix, not as much, since there's lower demand and we generally do not trust the server hands there from our experience with Serverhub (PhoenixNAP.)
The email we sent is us informing you, when we realized that the server was having catastrophic failure for GERKVM2. We included GERKVM1 because it also had a history of hardware problems, and it was about to continue having those billing issues I also mentioned above. We did not include GERKVM6 because it did not have hardware issues, as of now.
If it is urgent/emergency, it's a situation that cannot wait. I admit, we could have probably given an extra days notice for GERKVM1 but we needed to migrate those before GERKVM2 as GERKVM2 was offline, and if we waited until that's online before doing GERKVM1, we might have a situation where we have to move both together at the same time and that's not possible right now with the method we're using to reduce downtime.
With our other method, it's possible, but we don't use that method in a situation where the node may go offline in the middle of migration. We have to migrate these one VM at a time, which takes a much longer TOTAL time, but much less per VM, so lower downtime.
Yes, this is absolutely the situation.
Another issue when these outages happen is that backups break. That's a huge problem, backups being constantly interrupted. It leads to corrupt backups, no backups, or old backups for disaster recovery backups. So while someone may say they prefer having outages over being migrated, I'm sure they do not prefer potentially having data loss. It doesn't mean there will only be reboots, it means the situation will just get worse in most cases and there are other factors at play.
I'm not going to state that we'll completely move away from ColoCrossing, but that is an option that is on the table at this point in time due to various internal reasons I cannot fully elaborate on right now.
Right now we are in talks with Psychz, Dedipath/Inap, 365Datacenters, Hivelocity, and some others to do some quick initial expansions. We plan on adding a few locations, and focusing on getting our Windows packages back on track. Then, at some point in time we may move some customers over to these new servers as a part of a free or paid upgrade option. This is not planned on being an immediate move for everyone.
Well.. There you go... Let's see who is going to double the bandwidth. 😂
Don't know, but I hope no one is going to double your quote...
@VirMach of all those providers you are in talk with @HiVelocity is the best. TOP NOTCH 10/10
I love reading to your detailed replies (when it is not my shit that hit the fan though).
Got to give it to you for all the time and effort you put into to make things clear.
And the new softened tone is appreciatable.
I've found that VNC often doesn't work when the primary interface has changed. What I've often ended up having to do is use the rescue disk option most providers make available, verify the interface name with
ip link show, mount the disk, edit the interfaces file, then boot back to the OS.