Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

Serveraxis Miami is offline - Inception Hosting affected

Serveraxis Miami is offline - Inception Hosting affected

jvnadrjvnadr Member
edited March 2016 in General

Just got this from @AnthonySmith

Good Afternoon,
Late afternoon on the 29th February connectivity to all Inception Hosting servers in Miami was gradually lost over a period of a few hours. This included all remote and out of band access via iLO/IPMI equipment. All controls within the provider’s control panels also failed to respond to any requests. Requests for information and remote hands assistance was sent immediately following this.
Over the past 24 hours there have been precisely 3 responses from ServerAxis whom we trusted to provide service and networking within Miami-IX (datacentre). Initially they were looking in to it as at this point it was incorrectly assumed to be power related due to the nature of the failure. Despite numerous attempts to gain updates very little was forthcoming until confirmation was sent that it was a networking issue affecting all servers and services. Many further attempts were made to glean more information including around 100 phone calls, none of which were answered. A final update came from ServerAxis around 6 hours ago stating they are still checking.
Obviously, this level of service is completely unacceptable. As it turns out, from communications with others, this issue is not isolated to Inception hosting but to all ServerAxis customers. It was then discovered that the entire /20 subnet was dropped by Cogent (network upstream) and it is assumed that there is a financial dispute between Cogent or Miami Internet Exchange directly and ServerAxis waiting to be resolved. Offers have been made by Inception Hosting to pay any outstanding bills to restore service temporarily for the benefit of our customers however Cogent will not communicate with a third party and ServerAxis simply are not responding.
With this in mind and no predictable ETA in sight for getting things restored, I have been liaising with other hosts to get alternative hardware provisioned within Florida. The best time scale offered in Florida due to the hardware spec. required is 14 days which is obviously not acceptable. Therefore requests were sent further afield. One of our long term, trusted suppliers has been able to offer immediate interim hardware (within the next 4 hours) pending the delivery of the actual replacement hardware this weekend. It will take some time to get the interim hardware setup and configured, but please be assured that everything that can be done is being done. ServerAxis were initially selected due to their long record in business. This has come as a complete surprise and one that could not be predicted. It has left us, our customers and our customer’s customers in a bad and in some cases vulnerable position, with a completely unacceptable response. In light of this even when (hopefully a when and not an if) access to the servers is restored, Inception hosting will not continue service with ServerAxis beyond some access for recovering customer data. A migration was being planned for better network diversity anyway, but it was never expected to happen under these circumstances.
On a lighter note, the change will bring some strengths and advantages. The new platform will be KVM based rather than Xen, which will offer some significant flexibility advantages. OpenVZ services will remain the same but will be switched to SolusVM for management instead of Virtualizor. The new hardware will have newer, faster CPU’s as well as enterprise drives in raid 10 with 1TB of SSD cache on board the raid controller with cachevault licenses. Additionally, all inbound bandwidth will become unmetered. The only shortcoming is a lack of IPv6 initially however this will be in place within a few weeks of the new hardware delivery. If you are able to wait until the weekend you can go directly on the new hardware/network. If you require your server sooner please open a support ticket with ‘request to be placed on interim hardware’ in the subject line. These tickets will then be actioned in the order they are opened when the hardware is ready. Limited space is also available in the Netherlands and UK if you would rather not wait at all.
I would like to apologise again for the inconvenience this will no doubt cause to many people but due to the nature of the problem it is impossible to mitigate the impact in any other way.
Further instructions will follow where necessary.
Thanks.
Anthony Smith.

Hope Ant can gain quickly acces to his old infrastructure. In any case, he is one of the most trustworthy small players in this industry and his service are always top-notch despite their very low prices, he is extremely helpful and capable and I am very happy with him and inception hosting (in fact, he is one of the very few vps providers I am keeping after moving most of my services to dedicated servers).

• If a program actually fits in memory and has enough disk space, it is guaranteed to crash.
• If such a program has not crashed yet, it is waiting for a critical moment before it crashes.

Thanked by 2MikePT BeardyUnixGuy
«1

Comments

  • Are the servers collocated or rented? if collocated can't they be physically moved to another datacenter asap?

  • rm_rm_ Member

    Serveraxis Miami is offline - Inception Hosting infected

    Did you mean "affected"?

    Thanked by 2jvnadr netomx
  • rm_ said: Did you mean "affected"?

    haha, yes, probably I was somehow thinking again INception... Or I am too tired...

    • If a program actually fits in memory and has enough disk space, it is guaranteed to crash.
    • If such a program has not crashed yet, it is waiting for a critical moment before it crashes.

    Thanked by 1netomx
  • NyrNyr Member

    Wow.

    I feel bad for Anthony, I hope this can be resolved as quickly and easily as possible.

    And I would like to see a RFO from ServerAxis, should be fun.

  • ehabehab Member
    edited March 2016

    i can't read that,,, jvnadr make some words bold and i am try again.

    edit: please excuse me i was too sleepy to try last night, now i can read/understand it.

    "You know the price everyone wants."

  • This sucks for Anthony. He has always provide excellent services when I used his Miami location. One of the companies I work for had 2 nodes with Server Axis but we pulled out a month ago due to lack of response of tickets to get hardware replaced. That was a big no no. Well if Anthony you need anything just give a me a shout.

  • jvnadr said: Offers have been made by Inception Hosting to pay any outstanding bills to restore service temporarily for the benefit of our customers however Cogent will not communicate with a third party and ServerAxis simply are not responding.

    Props for Anthony for his good willing. @AnthonySmith, you've always had my respect, and will always have it. No one else would do that. Proves you really do care about your customers.

    Looking forward to need another VPS so I can buy it from you.

    If there's anything I can help you with, feel free to PM me and I'll stay awake the whole night in order to help you.

    European NOC legally registered - Providing Server/Cluster Management Solutions

    Design & IT Company - We deliver within the EU!

  • shovenoseshovenose Member, Provider

    Damn, that really sucks.

  • RadiRadi Member, Provider
    edited March 2016

    I feel bad for Anthony as well. I had a VM in all his locations and his services were just awesome. He provides a solid service and does a great job in customer service as well. Respect and kudos for being informative in the situation. I hope for the best in this case and it to be resolved soon.

    @AnthonySmith If I could help with anything, I would gladly do so.

    VikingLayer now offers VPS resource pools. Ask me about them today. :)

    Thanked by 1MikePT
  • Let me just put this here...... fuses

  • Was this sent to customers? I am an active Miami client and didn't get it.

  • @AnthonySmith wish you lots of engery these extra few hours you need them. Keep up the good job.

  • Man, that really sucks. Specially when you try to mend it yourself but get denied. Sounds like he's doing absolutely everything in his power to get back to a restored order. Good luck.

  • FranciscoFrancisco Top Provider

    People are reporting 24+ hours of downtime on WHT.

    Francisco

    BuyVM - Dedicated KVM Slices / Anycast Support! / Stallion Control Panel / Windows 2008, 2012, & 2016! / Unmetered Bandwidth!
    BuyShared - Shared & Reseller Hosting / cPanel + Softaculous + CloudLinux / Pure SSD! / Free Dedicated IP Address
    Thanked by 1netomx
  • AnthonySmithAnthonySmith Member, Provider

    I was just told about this thread, I have to admit I opened it with dread, thanks for surprising me, support is appreciated.

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • AnthonySmithAnthonySmith Member, Provider

    @Francisco said: People are reporting 24+ hours of downtime on WHT.

    Francisco

    It is probably more like 35 hours total from the first stuff starting to go down.

    http://cidr-report.org/

    Prefixes added and withdrawn by this origin AS in the past 7 days.

    - 64.188.32.0/20              Withdrawn
    

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • FranciscoFrancisco Top Provider

    @AnthonySmith said: - 64.188.32.0/20 Withdrawn

    Well, wasn't an SBL as far as I can tell.

    Could be power, but if it was I would assume they would be bringing it back up pretty quick. Really does sound possibly billing related.

    Francisco

    BuyVM - Dedicated KVM Slices / Anycast Support! / Stallion Control Panel / Windows 2008, 2012, & 2016! / Unmetered Bandwidth!
    BuyShared - Shared & Reseller Hosting / cPanel + Softaculous + CloudLinux / Pure SSD! / Free Dedicated IP Address
  • AnthonySmithAnthonySmith Member, Provider

    Francisco said: Could be power, but if it was I would assume they would be bringing it back up pretty quick. Really does sound possibly billing related.

    I got one random reply to a ticket which I am certain was not intended for me with some out of context text saying that "the van is having mechanical issues", like a accidental paste.

    That was about a month ago, I thought nothing of it as it was not relevant, and the ticket was related to an IPv6 issue, putting 2 and 2 together now.... could be a volumedrive style effort.

    Miami to Chicago in a van, 25 - 30 hours drive?

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • ClouviderClouvider Member, Provider
    edited March 2016

    Since they weren't Cogent singlehomed it seems pretty weird.

    While 64.188.32.0/20 isn't announced at the moment, the AS36252 (SERVERAXIS - Server Axis Corporation,US) continues to be, at least via HE and Steadfast. I can still see various subnets, like 208.82.96.0/21, still in the routing table.

    Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

    Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

  • AnthonySmithAnthonySmith Member, Provider

    @Clouvider indeed, as they still seem up in Chicago despite a minor blip yesterday, this is what makes me suspect the servers are actually in a van on the way to Chicago even more now.

    You would think they would have announced the range by now though if that was the case.

    I could be completely wrong, but with zero information forthcoming the mind wanders, I actually hope that is the case at this stage, at least access to data could be regained.

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • FranciscoFrancisco Top Provider

    That's a serious distance to drive.

    Francisco

    BuyVM - Dedicated KVM Slices / Anycast Support! / Stallion Control Panel / Windows 2008, 2012, & 2016! / Unmetered Bandwidth!
    BuyShared - Shared & Reseller Hosting / cPanel + Softaculous + CloudLinux / Pure SSD! / Free Dedicated IP Address
  • simonindiasimonindia Member
    edited March 2016

    @AnthonySmith said: Miami to Chicago in a van, 25 - 30 hours drive?

    If they can do that sry for saying this I would be impressed about their transport and driving skills

    but who would do this

    Make your choice on your own But i can help you to make them right.

  • FranciscoFrancisco Top Provider

    @simonindia said:

    Especially since Budget/etc trucks have a governor on them and cap out at 60 - 70 mph.

    Francisco

    BuyVM - Dedicated KVM Slices / Anycast Support! / Stallion Control Panel / Windows 2008, 2012, & 2016! / Unmetered Bandwidth!
    BuyShared - Shared & Reseller Hosting / cPanel + Softaculous + CloudLinux / Pure SSD! / Free Dedicated IP Address
    Thanked by 1simonindia
  • I'm in North Florida on the other side from Miami. 6 hour drive down 95 or 1hr 15mins via flight

    "Stop quoting laws, we carry weapons!" - Pompey the Great | Vultr aff link

    @gapper said: I don't like you.

  • FlamesRunnerFlamesRunner Member
    edited March 2016

    I got one of his $3/year shared hosting plans on FreecPanelSharedHosting :p

    It had magnificent uptime, this is the only downtime I have recorded so far.

    Has anyone requested migration already?


    Side note: @AnthonySmith, are you planning to in the near future start assigning IPv6 addresses to cPanel accounts?

    Benchmark: wget https://s.flamz.pw/dl/bench.sh && bash bench.sh

  • @FlamesRunner said: Side note: AnthonySmith, are you planning to in the near future start assigning IPv6 addresses to cPanel accounts?

    If I'm not mistaken I think @mikho was running that.

    Thanked by 1davidgestiondbi
  • AnthonySmithAnthonySmith Member, Provider

    FlamesRunner said: Has anyone requested migration already?

    A lot of the OpenVZ customers have requested migration to the UK or Netherlands, there is a small queue of people for PV/HVM.

    mikeyur said: If I'm not mistaken I think @mikho was running that.

    That is correct.

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • MikePTMikePT Member
    edited March 2016

    @AnthonySmith said: That is correct.

    So they won't even pick up the phone?

    I'll shoot them a sales email to test.

    Edit: Done, just curious to see if they'd reply with a Miami test ip, and asked for 10 servers. Should work if their sales are online? Who knows. Let's see. Can't do much for Anthony unfortunately.

    European NOC legally registered - Providing Server/Cluster Management Solutions

    Design & IT Company - We deliver within the EU!

  • Just tried to give them a call to their number in the WHOIS, didn't pick up, hm.

    European NOC legally registered - Providing Server/Cluster Management Solutions

    Design & IT Company - We deliver within the EU!

  • @mikeyur

    Whoops, made the assumption because when I click the VPS Hosting link on the bottom of the page, it linked me to https://inceptionhosting.com/ :)

    Benchmark: wget https://s.flamz.pw/dl/bench.sh && bash bench.sh

  • Sad things.

    This time the situation took me by surprise, because apparently I don't have the backups for all my data on my instance.

    So, to be honest, I am in the hopes to get my data back before getting a new server (HVM instance).

    And a big thank you to Anthony, as always, because he is keeping us, informed about these issues.

  • mikhomikho Member, Provider

    @FlamesRunner said: Has anyone requested migration already?

    There has been a few requests from the shared hosting clients as well.

    @FlamesRunner said: Side note: AnthonySmith, are you planning to in the near future start assigning IPv6 addresses to cPanel accounts?

    Soon (tm)

    @mikeyur said: If I'm not mistaken I think mikho was running that.

    If you send in a ticket, I will answer it.

    @FlamesRunner said: mikeyur

    Whoops, made the assumption because when I click the VPS Hosting link on the bottom of the page, it linked me to https://inceptionhosting.com/ :).

    The payment for the shared hosting has also been sent to Inception. This issue was also the starting point to seperate the two.
    Btw, I don't see any harm in sending the parent company some clients if I couldn't offer the product? ;) ;)

    ask me about Storage offers
    NAT VPS in Germany, USA, Bulgaria available, Order here! and some Shared Hosting offers
    Locations: Phoenix (US) | Amsterdam (NL) | Atlanta (US) | Milan (IT) | Kansas City (US) | Falkenstein (DE), | Sofia (BG)
  • As a current Miami customer, I'm happy to wait out the downtime if servers are expected to be restored (or otherwise recreated) in Florida, since I needed servers in that location.

    But if that end game is unknown, should I just request a transfer to Dallas?

  • I got one random reply to a ticket which I am certain was not intended for me with some out of context text saying that "the van is having mechanical issues", like a accidental paste.

    That was about a month ago, I thought nothing of it as it was not relevant, and the ticket was related to an IPv6 issue, putting 2 and 2 together now.... could be a volumedrive style effort.

    Miami to Chicago in a van, 25 - 30 hours drive?

    I'm slow. What is this actually implying?

    Favourite host in general: Ramnode (SSD10 for 10% off, affiliate link)
    Favourite host for hourly billing/custom ISOs: Vultr (affiliate link)

  • mikhomikho Member, Provider

    @hostnoob said: I'm slow. What is this actually implying?

    Server Axis also has a Chicago presence.

    ask me about Storage offers
    NAT VPS in Germany, USA, Bulgaria available, Order here! and some Shared Hosting offers
    Locations: Phoenix (US) | Amsterdam (NL) | Atlanta (US) | Milan (IT) | Kansas City (US) | Falkenstein (DE), | Sofia (BG)
  • mikhomikho Member, Provider

    @rpollestad said: As a current Miami customer, I'm happy to wait out the downtime if servers are expected to be restored (or otherwise recreated) in Florida, since I needed servers in that location.

    But if that end game is unknown, should I just request a transfer to Dallas?

    I don't know if Ant got a reply while sleeping but from the email sent out I wouldn't expect a Miami location, at least not with Server Axis.

    ask me about Storage offers
    NAT VPS in Germany, USA, Bulgaria available, Order here! and some Shared Hosting offers
    Locations: Phoenix (US) | Amsterdam (NL) | Atlanta (US) | Milan (IT) | Kansas City (US) | Falkenstein (DE), | Sofia (BG)
  • MSPNickMSPNick Member, Provider

    @hostnoob said: I'm slow. What is this actually implying?

    They could be up and moving but not telling anyone, aka I am sure that's what volume drive did...

  • lbftlbft Member

    Clouvider said: Since they weren't Cogent singlehomed it seems pretty weird.

    I was under the impression that the Miami location was pure Cogent, and that everything else was only in Chicago.

    Thanked by 1Clouvider
  • shovenoseshovenose Member, Provider
    edited March 2016

    @Francisco said: rental vans are governed

    Ah, but if they were discussing mechanical issues they more likely own it. In which case they could go 80+. Still a 17+ hour drive though if they had two people taking turns. Then again, if they were having issues, that's quite a drive for an older or high mileage vehicle.

  • AnthonySmithAnthonySmith Member, Provider

    shovenose said: Ah, but if they were discussing mechanical issues they more likely own it. In which case they could go 80+. Still a 17+ hour drive though if they had two people taking turns. Then again, if they were having issues, that's quite a drive for an older or high mileage vehicle.

    It has been 48 hours now since they went down, the road trip theory is out the window I think.

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • jvnadrjvnadr Member
    edited March 2016

    AnthonySmith said: It has been 48 hours now since they went down, the road trip theory is out the window I think.

    But what may be doing? Closing the whole Miami facility? Wouldn't this be extreme? Except if they deadpool as a company, but this will affect all of their points of presence. If not, what will they do with their infrastructure?
    IMO, providers that are affected by this situation, should start to think taking some legal mesures to protect their interests. Companies like serveraxis should not be let escape easily from the mess they created...
    As of the road trip, there is not just the road but unracking and racking again, routing etc. This could take days...

    • If a program actually fits in memory and has enough disk space, it is guaranteed to crash.
    • If such a program has not crashed yet, it is waiting for a critical moment before it crashes.

  • AnthonySmithAnthonySmith Member, Provider

    Well I suspect given that only around 5 hosts that I know of have made any public comment that they have been winding down Miami for a long time, for clarification ServerAxis were/are housed within Miami Internet Exchange, the physical DC itself is no doubt in full swing.

    ServerAxis was housed within Miami-IX with their own equipment, I have tried to reach out directly to Miami IX without any luck.

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • jvnadrjvnadr Member
    edited March 2016

    AnthonySmith said: ServerAxis was housed within Miami-IX with their own equipment

    Do you know what do they have there as infrastructure? Some colocation? A cabinet? A rented space with own elements? Or how big is their point of presence in Miami? If it's just a few servers, maybe they will abandon them there. But if we are talking about a larger number of nodes (let's say, over 30-35) with their routers, then we are talking about a serious asset that it is hard for anyone to just let it die there...

    • If a program actually fits in memory and has enough disk space, it is guaranteed to crash.
    • If such a program has not crashed yet, it is waiting for a critical moment before it crashes.

  • lbftlbft Member

    jvnadr said: then we are talking about a serious asset that it is hard for anyone to just let it die there...

    Depends if they own the hardware or lease it.

  • AnthonySmithAnthonySmith Member, Provider

    @jvnadr I really don't know the full extent of the presence, they did have their own networking equipment though that much I know.

    I trusted them because they had been around so long, 11 years iirc.

    What really has a pit in my stomach is that I just cannot even begin to put myself in the shoes of someone who would do this, businesses fail, things go wrong, disasters happen, we all accept that.

    But to handle it in this way shows a complete disregard for anyone, its not just their own customers, but the customers customers, and so on, 1000's of people having a really bad time now because someone is to much of a coward to come forward or to much of a psycho to even consider it necessary.

    My suspicion now is that they just shut down Miami because it was not making enough money, they did it late on the last day of the month so no one had time to react quick enough before they started charging everyone's cards the next day to squeeze the last drops out of people.

    Now they/he, plans on just staying quiet for a few months until it blows over and concentrate on Chicago, that said it would not surprise me at all if ServerAxis Chicago started falling apart in the very near future as these issues are obviously financial.

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

    Thanked by 2mpkossen jvnadr
  • vpsukvpsuk Member

    Hi, What are the chances to get my data back (Miami VPS @Inc.hosting), I have a backup from january but all my work during february is on the server, and all the configs which are necessary for me. Is there a chance to boot up the servers for a couple of hours to backup every valuable data or ServerAxis is gone forever with the hardwares?

  • I thought ServerAxis was in CoreSite Miami, I maybe wrong

  • AnthonySmithAnthonySmith Member, Provider

    vpsuk said: Hi, What are the chances to get my data back

    Honestly, getting lower with each passing hour, sadly the lack of information from the only people that can answer that question with any certainty is 99% of the problem.

    MarkTurner said: I thought ServerAxis was in CoreSite Miami, I maybe wrong

    Your right, I tend to write IX/coresite interchangeably as for around a year I thought it was the same place and it stuck.

    Inception Hosting Limited - NL, UK, Phoenix AZ USA, KVM and OpenVZ - | 50% OFF Phoenix AZ KVM. | 256mb & 512mb UK
    ServersNV a brand of Inception Hosting Limited - UK KVM and OpenVZ - | ^Use CODE: 50offphx4life | ^€8.00 p/year LIMITED

  • AnthonySmith said: Your right, I tend to write IX/coresite interchangeably as for around a year I thought it was the same place and it stuck.

    If you need something on the East coast let @mikeyur know, we have a good variety of large boxes we can get up in minutes.

Sign In or Register to comment.