Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


DelimiterVPS does not restart server after maintenance
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

DelimiterVPS does not restart server after maintenance

loydloyd Member
edited June 2014 in General

A heads up to those who want to host with Delimiter.

Server was down this morning, ILO reported STANDBY. I raised a ticket and got "you should check our network issue page, there was scheduled maintenance posted 10 days ago".

For one, I don't have time for daily browsing of all my operators websites, if there is an event like this, I am expecting email alert.

Secondly, when you are done with mainenance, START MY BLOODY SERVER BACK UP !

«1

Comments

  • Yep I did the same thing, reported a ticket after about ~30 minutes of downtime. Told me to go check the page and was posted 10 days before.

    I told him an email or something would have been nice to know, no one checks the client area constantly.

  • ReeRee Member

    My server was rebooted just fine -- I'll bet yours would have been too, but if there was a large number of affected servers then it might have taken awhile to get to it.

    And ask about their incident list -- they subscribed me so I'll get notifications in the future. Kind of strange that it's opt-in instead of opt-out, but at least the option to receive notifications is there if you ask about it.

  • @Ree said:

    For sure, I told him it would have been nice to receive an email about it and he added me to the list.

  • loydloyd Member
    edited June 2014

    My server went down on 2014/6/14 23:10
    Learned about this on 2014/6/15 10:00

    Incident was closed at 00:34, 9.5hrs before I learned about my server being down.

    Reply to my ticket had no hint of apology, tech blamed me for not checking their network issues page (which actually promises the servers being restarted at the end of the maintenance) and closed the ticket, ignoring my subsequent comments about email alerts. I was not offered to be signed-up to the list.

    I would expect something like this from GreenValueHost but not Delimiter, now my impression of Delimiter is also 14yr old kid with minimum professional integrity.

    Good to know, time to look around.

  • @loyd said:
    My server went down on 2014/6/14 23:10
    Learned about this on 2014/6/15 10:00

    Incident was closed at 00:34, 9.5hrs before I learned about my server being down.

    Reply to my ticket had no hint of apology, tech blamed me for not checking their network issues page (which actually promises the servers being restarted at the end of the maintenance) and closed the ticket, ignoring my subsequent comments about email alerts. I was not offered to be signed-up to the list.

    I would expect something like this from GreenValueHost but not Delimiter, now my impression of Delimiter is also 14yr old kid with minimum professional integrity.

    Good to know, time to look around.

    Delimiter is a well known company, they're a smaller brand of what's bigger Yomura Holdings, Which has alot of datacenters

  • @loyd - what was the ticket #, I'd like to read what went on.

  • loydloyd Member

    @nexmark said:
    Delimiter is a well known company, they're a smaller brand of what's bigger Yomura Holdings, Which has alot of datacenters

    Obviously size and fame don't automatically mean professional conduct these days.

  • DylanDylan Member
    edited June 2014

    I cancelled my Delimiter X5150 earlier this month. The server itself was pretty good, once I got past the setup nightmare, but their rude and unresponsive support is a deal-breaker.

    Thanked by 1srvrpro
  • @loyd - I've asked for a copy of the ticket. Lets see what happened

  • Just sent in a ticket...

    My server is still down and I can't attempt to power cycle it or anything as for some reason it's listed as suspended in WHMCS... Bill's paid up and it's simply used as a staging server for development of our software so no TOS issues (and I haven't received any emails indicating that).

  • @lucashayes - Please PM me the IP of your server. When was it suspended and do you know why it was suspended?

  • loydloyd Member

    @MarkTurner said:
    loyd - I've asked for a copy of the ticket. Lets see what happened

    check your pm

  • @loyd - I know already received, as I said I am waiting for a copy of the ticket now from Delimiter

  • @MarkTurner said:
    lucashayes - Please PM me the IP of your server. When was it suspended and do you know why it was suspended?

    It has apparently been taken care of. It got suspended a few weeks ago because I forgot to pay the bill but apparently when I paid it they unsuspended the service but didn't mark it as unsuspended in WHMCS. Problem being that the power management button goes away when it's marked as suspended so I couldn't reboot it to get it back online.

  • @loyd, I've received the ticket and the maintenance notes showing that the results of a visual inspection of your blade showed it was already powered up and therefore did not require resetting. The engineers on-site walked the site three times over the course of the evening to make sure that all servers were powered, however it was noted that a few blades were reported down, despite showing a green health and green power light illuminated on the front of them.

    It seems that in some cases (7 blades) the power management system on the blade had simply crashed and left the indicators in their last state ie powered and running error free. The engineers were unaware there was a problem with these servers until tickets were opened and they were requested to check them.

    The blade management system showed that your blade was powered the whole time as well, so again there was no way for them to tell that there was an error or issue on your server.

    Its impractical in this instances to manually connect a monitor/keyboard to all 1440 blades server and confirm that they have loaded to their OS. We used to monitor customer servers with ICMP but almost all customers disable ICMP response on their servers these days.

    Unfortunately in this instance there was very little that could be done until this issue was reported. The physical indicators showed server was up, the management system confirmed the server was up. Their assumption therefore was the server was up.

    Thanked by 1loyd
  • @lucashayes - be aware that the Yomura companies don't use WHMCS for their real billing/support/status management, its just used a convenient front-end. All the suspension activity is handled on their internal billing system, which means sometimes the frontend (WHMCS) gets out of sync with the backend. So if you get suspended for non-payment and then unsuspended, make sure that WHMCS shows your server as unsuspended. If not just open a support ticket and have them do a manual update for you.

  • loydloyd Member
    edited June 2014

    The issue did not require support since I was able to start my server from ILO.

    Understandable, mistakes happens. Understandable, Delimiter does not use ILO API to verify actual server status. I am an engineer who knows things are not always ideal nor they go as planned. Simple answer "your blade's indicator incorrectly reported server as being up" to my ticket this morning would make a day and night difference, instead I was lead to believe it's my job to watch company website in order to receive service and my frustration exploded.

    I appreciate your work here and thank you for taking the time to dig through and write-up an explanation. As I understand you are not Delimiter staffer and don't represent them here, so Delimiter did not take ownership of their mistake. Noted. Case closed. Thanks.

    Thanked by 1daxterfellowes
  • MarkTurnerMarkTurner Member
    edited June 2014

    @loyd - Its actually our fault not theirs. Delimiter is a sales/support business thats sells Yomura's services. The datacentre your server is in is ours and staffed by our staff.

    We have two ways to determine blade status - the obvious the LED on the front and the chassis management system which gives us live data about the status of the servers: http://imgur.com/KLBAprM

    The servers were checked physically and virtually using the tool. More than likely in your case the watchdog on the ILO rebooted it after about an hour or two and so the ILO reported status of the blade reflected the fact it was down.

    The wording wasn't good from their side, it was somewhat terse. I've already highlighted that point internally so it can be handled better the next time it happens.

    Thanked by 1neroux
  • nerouxneroux Member
    edited June 2014

    MarkTurner said: Its actually our fault not theirs.

    Something you dont read often around here :)

    Kudos.

    Only suggestion as to this thread's topic, make email notifications for such announcements default.

    Thanked by 1netomx
  • @neroux - being advocating compulsory maintenance emails for a long time. The problem that kept happening is that customers received the notices, it got tagged as spam and then their invoice/ticket emails went into the spam bin.

    I am going to bring it up again on Monday. Delimiter also posts to Twitter with their announcements both in realtime and for future maintenance events.

  • nerouxneroux Member

    MarkTurner said: being advocating compulsory maintenance emails for a long time. The problem that kept happening is that customers received the notices, it got tagged as spam and then their invoice/ticket emails went into the spam bin.

    Just an idea, maybe a different email address/server for billing related emails?

  • ReeRee Member

    @MarkTurner said: The wording wasn't good from their side, it was somewhat terse. I've already highlighted that point internally so it can be handled better the next time it happens.

    Terse is a good way to describe my ticket responses as well. But to be honest I'm totally cool with terse if that's what allows them to respond so quickly. My tickets have been responded to in 3, 2, 3 and 9 minutes...for such a cheap service, this is awesome. And this is after marking them as low priority because...they were low priority!

    Thanked by 1MarkTurner
  • Mine went into standby. It looks like they updated the iLO firmware on mine and something went wrong. Was fixed very quickly though! Props to Delimiter!

    Thanked by 1MarkTurner
  • @Ree - There is terse and terse. Loyd's ticket was handled brashly.

    All of Yomura's support team are told to read the customer's enquiry, if it can be dealt with inside 5 minutes then deal with it and respond that its done. If it will take more than 5 minutes or requires escalation then notify the customer that its in hand and then follow the escalation. They are told to keep responses clear and straight to the point, avoid waffling, be explicit.

    Most of the time, I see their responses, no matter what the complexity of the issue as being words to the effect that 'its done' or 'this has been resolved'. If you need more data then just reply back to the ticket and ask them for an explanation.

  • talsittalsit Member

    I had the same problem with one of my three servers during the maintenance. Tech response was good and got me back up and running very shortly (~15 minutes) after I contacted them. Kudos.

    Thanked by 1MarkTurner
  • loydloyd Member

    There was a similar incident last night, this time email notice was sent and server was up and running in the morning. Is it early to state this thread was constructive and the service quality has improved? :) DelimiterVPS, thumbs up!

  • Yeah email was sent because 'As some customers did not see the notice last weekend that was posted in the Network Issues', more like no one seen it. But no issues at all last night, not even a bit of downtime.

  • nerouxneroux Member
    edited June 2014

    loyd said: Is it early to state this thread was constructive and the service quality has improved?

    I do not think so I am afraid. Only three days ago there was a failure of a network switch which made the iscsi mounted storage once more switch to read-only. No notification was sent in this case.

  • @Neroux - out of 400+ Atom servers, one rack was affected (44 Atoms). The switch for that rack failed. It was replaced within 30 minutes and servers rebooted.

  • @loyd - the irony is that we had four spam reports from that where live customers reported the notifications as spam.

    @Catalystium - there are definitely quite a number of people who saw it because there was a flurry of tickets and PM's to me on here about which rack their server was located.

Sign In or Register to comment.