Serveraxis Miami is offline - Inception Hosting affected
Just got this from @AnthonySmith
Late afternoon on the 29th February connectivity to all Inception Hosting servers in Miami was gradually lost over a period of a few hours. This included all remote and out of band access via iLO/IPMI equipment. All controls within the provider’s control panels also failed to respond to any requests. Requests for information and remote hands assistance was sent immediately following this.
Over the past 24 hours there have been precisely 3 responses from ServerAxis whom we trusted to provide service and networking within Miami-IX (datacentre). Initially they were looking in to it as at this point it was incorrectly assumed to be power related due to the nature of the failure. Despite numerous attempts to gain updates very little was forthcoming until confirmation was sent that it was a networking issue affecting all servers and services. Many further attempts were made to glean more information including around 100 phone calls, none of which were answered. A final update came from ServerAxis around 6 hours ago stating they are still checking.
Obviously, this level of service is completely unacceptable. As it turns out, from communications with others, this issue is not isolated to Inception hosting but to all ServerAxis customers. It was then discovered that the entire /20 subnet was dropped by Cogent (network upstream) and it is assumed that there is a financial dispute between Cogent or Miami Internet Exchange directly and ServerAxis waiting to be resolved. Offers have been made by Inception Hosting to pay any outstanding bills to restore service temporarily for the benefit of our customers however Cogent will not communicate with a third party and ServerAxis simply are not responding.
With this in mind and no predictable ETA in sight for getting things restored, I have been liaising with other hosts to get alternative hardware provisioned within Florida. The best time scale offered in Florida due to the hardware spec. required is 14 days which is obviously not acceptable. Therefore requests were sent further afield. One of our long term, trusted suppliers has been able to offer immediate interim hardware (within the next 4 hours) pending the delivery of the actual replacement hardware this weekend. It will take some time to get the interim hardware setup and configured, but please be assured that everything that can be done is being done. ServerAxis were initially selected due to their long record in business. This has come as a complete surprise and one that could not be predicted. It has left us, our customers and our customer’s customers in a bad and in some cases vulnerable position, with a completely unacceptable response. In light of this even when (hopefully a when and not an if) access to the servers is restored, Inception hosting will not continue service with ServerAxis beyond some access for recovering customer data. A migration was being planned for better network diversity anyway, but it was never expected to happen under these circumstances.
On a lighter note, the change will bring some strengths and advantages. The new platform will be KVM based rather than Xen, which will offer some significant flexibility advantages. OpenVZ services will remain the same but will be switched to SolusVM for management instead of Virtualizor. The new hardware will have newer, faster CPU’s as well as enterprise drives in raid 10 with 1TB of SSD cache on board the raid controller with cachevault licenses. Additionally, all inbound bandwidth will become unmetered. The only shortcoming is a lack of IPv6 initially however this will be in place within a few weeks of the new hardware delivery. If you are able to wait until the weekend you can go directly on the new hardware/network. If you require your server sooner please open a support ticket with ‘request to be placed on interim hardware’ in the subject line. These tickets will then be actioned in the order they are opened when the hardware is ready. Limited space is also available in the Netherlands and UK if you would rather not wait at all.
I would like to apologise again for the inconvenience this will no doubt cause to many people but due to the nature of the problem it is impossible to mitigate the impact in any other way.
Further instructions will follow where necessary.
Hope Ant can gain quickly acces to his old infrastructure. In any case, he is one of the most trustworthy small players in this industry and his service are always top-notch despite their very low prices, he is extremely helpful and capable and I am very happy with him and inception hosting (in fact, he is one of the very few vps providers I am keeping after moving most of my services to dedicated servers).
• If a program actually fits in memory and has enough disk space, it is guaranteed to crash.
• If such a program has not crashed yet, it is waiting for a critical moment before it crashes.