Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Serveraxis Miami is offline - Inception Hosting affected
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Serveraxis Miami is offline - Inception Hosting affected

jvnadrjvnadr Member
edited March 2016 in General

Just got this from @AnthonySmith

Good Afternoon,
Late afternoon on the 29th February connectivity to all Inception Hosting servers in Miami was gradually lost over a period of a few hours. This included all remote and out of band access via iLO/IPMI equipment. All controls within the provider’s control panels also failed to respond to any requests. Requests for information and remote hands assistance was sent immediately following this.
Over the past 24 hours there have been precisely 3 responses from ServerAxis whom we trusted to provide service and networking within Miami-IX (datacentre). Initially they were looking in to it as at this point it was incorrectly assumed to be power related due to the nature of the failure. Despite numerous attempts to gain updates very little was forthcoming until confirmation was sent that it was a networking issue affecting all servers and services.
Many further attempts were made to glean more information including around 100 phone calls, none of which were answered. A final update came from ServerAxis around 6 hours ago stating they are still checking.
Obviously, this level of service is completely unacceptable. As it turns out, from communications with others, this issue is not isolated to Inception hosting but to all ServerAxis customers. It was then discovered that the entire /20 subnet was dropped by Cogent (network upstream) and it is assumed that there is a financial dispute between Cogent or Miami Internet Exchange directly and ServerAxis waiting to be resolved. Offers have been made by Inception Hosting to pay any outstanding bills to restore service temporarily for the benefit of our customers however Cogent will not communicate with a third party and ServerAxis simply are not responding.
With this in mind and no predictable ETA in sight for getting things restored, I have been liaising with other hosts to get alternative hardware provisioned within Florida. The best time scale offered in Florida due to the hardware spec. required is 14 days which is obviously not acceptable. Therefore requests were sent further afield. One of our long term, trusted suppliers has been able to offer immediate interim hardware (within the next 4 hours) pending the delivery of the actual replacement hardware this weekend.
It will take some time to get the interim hardware setup and configured, but please be assured that everything that can be done is being done. ServerAxis were initially selected due to their long record in business. This has come as a complete surprise and one that could not be predicted. It has left us, our customers and our customer’s customers in a bad and in some cases vulnerable position, with a completely unacceptable response. In light of this even when (hopefully a when and not an if) access to the servers is restored, Inception hosting will not continue service with ServerAxis beyond some access for recovering customer data. A migration was being planned for better network diversity anyway, but it was never expected to happen under these circumstances.
On a lighter note, the change will bring some strengths and advantages. The new platform will be KVM based rather than Xen, which will offer some significant flexibility advantages. OpenVZ services will remain the same but will be switched to SolusVM for management instead of Virtualizor. The new hardware will have newer, faster CPU’s as well as enterprise drives in raid 10 with 1TB of SSD cache on board the raid controller with cachevault licenses. Additionally, all inbound bandwidth will become unmetered. The only shortcoming is a lack of IPv6 initially however this will be in place within a few weeks of the new hardware delivery.
If you are able to wait until the weekend you can go directly on the new hardware/network. If you require your server sooner please open a support ticket with ‘request to be placed on interim hardware’ in the subject line. These tickets will then be actioned in the order they are opened when the hardware is ready. Limited space is also available in the Netherlands and UK if you would rather not wait at all.
I would like to apologise again for the inconvenience this will no doubt cause to many people but due to the nature of the problem it is impossible to mitigate the impact in any other way.
Further instructions will follow where necessary.
Thanks.
Anthony Smith.

Hope Ant can gain quickly acces to his old infrastructure. In any case, he is one of the most trustworthy small players in this industry and his service are always top-notch despite their very low prices, he is extremely helpful and capable and I am very happy with him and inception hosting (in fact, he is one of the very few vps providers I am keeping after moving most of my services to dedicated servers).

Thanked by 2MikePT BeardyUnixGuy
«134

Comments

  • Are the servers collocated or rented? if collocated can't they be physically moved to another datacenter asap?

  • rm_rm_ IPv6 Advocate, Veteran

    Serveraxis Miami is offline - Inception Hosting infected

    Did you mean "affected"?

    Thanked by 2jvnadr netomx
  • rm_ said: Did you mean "affected"?

    haha, yes, probably I was somehow thinking again INception... Or I am too tired...

    Thanked by 1netomx
  • NyrNyr Community Contributor, Veteran

    Wow.

    I feel bad for Anthony, I hope this can be resolved as quickly and easily as possible.

    And I would like to see a RFO from ServerAxis, should be fun.

  • ehabehab Member
    edited March 2016

    i can't read that,,, jvnadr make some words bold and i am try again.

    edit: please excuse me i was too sleepy to try last night, now i can read/understand it.

  • PieHasBeenEatenPieHasBeenEaten Member, Host Rep

    This sucks for Anthony. He has always provide excellent services when I used his Miami location. One of the companies I work for had 2 nodes with Server Axis but we pulled out a month ago due to lack of response of tickets to get hardware replaced. That was a big no no. Well if Anthony you need anything just give a me a shout.

  • MikePTMikePT Moderator, Patron Provider, Veteran

    jvnadr said: Offers have been made by Inception Hosting to pay any outstanding bills to restore service temporarily for the benefit of our customers however Cogent will not communicate with a third party and ServerAxis simply are not responding.

    Props for Anthony for his good willing. @AnthonySmith, you've always had my respect, and will always have it. No one else would do that. Proves you really do care about your customers.

    Looking forward to need another VPS so I can buy it from you.

    If there's anything I can help you with, feel free to PM me and I'll stay awake the whole night in order to help you.

  • shovenoseshovenose Member, Host Rep

    Damn, that really sucks.

  • RadiRadi Host Rep, Veteran
    edited March 2016

    I feel bad for Anthony as well. I had a VM in all his locations and his services were just awesome. He provides a solid service and does a great job in customer service as well. Respect and kudos for being informative in the situation. I hope for the best in this case and it to be resolved soon.

    @AnthonySmith If I could help with anything, I would gladly do so.

    Thanked by 1MikePT
  • Let me just put this here...... fuses

  • Was this sent to customers? I am an active Miami client and didn't get it.

  • cociucociu Member

    @AnthonySmith good luck , this is not a joke ... sucks situations

  • @AnthonySmith wish you lots of engery these extra few hours you need them. Keep up the good job.

  • Man, that really sucks. Specially when you try to mend it yourself but get denied. Sounds like he's doing absolutely everything in his power to get back to a restored order. Good luck.

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    People are reporting 24+ hours of downtime on WHT.

    Francisco

    Thanked by 1netomx
  • AnthonySmithAnthonySmith Member, Patron Provider

    I was just told about this thread, I have to admit I opened it with dread, thanks for surprising me, support is appreciated.

  • AnthonySmithAnthonySmith Member, Patron Provider

    @Francisco said:
    People are reporting 24+ hours of downtime on WHT.

    Francisco

    It is probably more like 35 hours total from the first stuff starting to go down.

    http://cidr-report.org/

    Prefixes added and withdrawn by this origin AS in the past 7 days.

    - 64.188.32.0/20              Withdrawn
    
  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @AnthonySmith said:
    - 64.188.32.0/20 Withdrawn

    Well, wasn't an SBL as far as I can tell.

    Could be power, but if it was I would assume they would be bringing it back up pretty quick. Really does sound possibly billing related.

    Francisco

  • AnthonySmithAnthonySmith Member, Patron Provider

    Francisco said: Could be power, but if it was I would assume they would be bringing it back up pretty quick. Really does sound possibly billing related.

    I got one random reply to a ticket which I am certain was not intended for me with some out of context text saying that "the van is having mechanical issues", like a accidental paste.

    That was about a month ago, I thought nothing of it as it was not relevant, and the ticket was related to an IPv6 issue, putting 2 and 2 together now.... could be a volumedrive style effort.

    Miami to Chicago in a van, 25 - 30 hours drive?

  • ClouviderClouvider Member, Patron Provider
    edited March 2016

    Since they weren't Cogent singlehomed it seems pretty weird.

    While 64.188.32.0/20 isn't announced at the moment, the AS36252 (SERVERAXIS - Server Axis Corporation,US) continues to be, at least via HE and Steadfast. I can still see various subnets, like 208.82.96.0/21, still in the routing table.

  • AnthonySmithAnthonySmith Member, Patron Provider

    @Clouvider indeed, as they still seem up in Chicago despite a minor blip yesterday, this is what makes me suspect the servers are actually in a van on the way to Chicago even more now.

    You would think they would have announced the range by now though if that was the case.

    I could be completely wrong, but with zero information forthcoming the mind wanders, I actually hope that is the case at this stage, at least access to data could be regained.

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    That's a serious distance to drive.

    Francisco

  • simonindiasimonindia Member
    edited March 2016

    @AnthonySmith said:
    Miami to Chicago in a van, 25 - 30 hours drive?

    If they can do that sry for saying this I would be impressed about their transport and driving skills

    but who would do this

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @simonindia said:

    Especially since Budget/etc trucks have a governor on them and cap out at 60 - 70 mph.

    Francisco

    Thanked by 1simonindia
  • I'm in North Florida on the other side from Miami. 6 hour drive down 95 or 1hr 15mins via flight

  • FlamesRunnerFlamesRunner Member
    edited March 2016

    I got one of his $3/year shared hosting plans on FreecPanelSharedHosting :p

    It had magnificent uptime, this is the only downtime I have recorded so far.

    Has anyone requested migration already?


    Side note: @AnthonySmith, are you planning to in the near future start assigning IPv6 addresses to cPanel accounts?

  • @FlamesRunner said:
    Side note: AnthonySmith, are you planning to in the near future start assigning IPv6 addresses to cPanel accounts?

    If I'm not mistaken I think @mikho was running that.

    Thanked by 1gestiondbi
  • AnthonySmithAnthonySmith Member, Patron Provider

    FlamesRunner said: Has anyone requested migration already?

    A lot of the OpenVZ customers have requested migration to the UK or Netherlands, there is a small queue of people for PV/HVM.

    mikeyur said: If I'm not mistaken I think @mikho was running that.

    That is correct.

  • MikePTMikePT Moderator, Patron Provider, Veteran
    edited March 2016

    @AnthonySmith said:
    That is correct.

    So they won't even pick up the phone?

    I'll shoot them a sales email to test.

    Edit: Done, just curious to see if they'd reply with a Miami test ip, and asked for 10 servers. Should work if their sales are online? Who knows. Let's see. Can't do much for Anthony unfortunately.

  • MikePTMikePT Moderator, Patron Provider, Veteran

    Just tried to give them a call to their number in the WHOIS, didn't pick up, hm.

Sign In or Register to comment.