Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


cloudshards.com - WHMCS / Hosting offline - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

cloudshards.com - WHMCS / Hosting offline

2

Comments

  • Awmusic12635Awmusic12635 Member, Host Rep

    GameBeatDotNet said: Yes. But wait... aren't you with Query Foundry / Fliphost? I've actually got a PM in my inbox from you back in 2013 asking me to stay with FH after a similar outage... LOL. :-)

    I was the original founder/owner of Fliphost. I know own Subnet Labs/Impact VPS / Impact Shared

  • TheLinuxBugTheLinuxBug Member
    edited January 2017

    A little birdy told me this is more likely the fact they didn't pay their DC bills and they have been shut off.

    I will not confirm or deny why I think this, but I am pretty confident this is the case.

    my 2 cents.

    Cheers!

  • @Awmusic12635 said:

    GameBeatDotNet said: Yes. But wait... aren't you with Query Foundry / Fliphost? I've actually got a PM in my inbox from you back in 2013 asking me to stay with FH after a similar outage... LOL. :-)

    I was the original founder/owner of Fliphost. I know own Subnet Labs/Impact VPS / Impact Shared

    Awesome, I like that you are available on LET! I really like VDR8... can you get me a good price on that? (Maybe we should take this to PMs)

  • edited January 2017

    @TheLinuxBug said:
    A little birdy told me this is more likely the fact they didn't pay their DC bills and they have been shut off.

    I will not confirm or deny why I think this, but I am pretty confident this is the case.

    my 2 cents.

    Cheers!

    Good lord I hope they get around to paying it.... very unprofessional if this is true. But I've seen it time and time again over the years and I've even been the victim of it before. I guess this is the price we pay for sticking to the bottom dollar. Harumph!

  • Awmusic12635Awmusic12635 Member, Host Rep

    GameBeatDotNet said: Awesome, I like that you are available on LET! I really like VDR8... can you get me a good price on that? (Maybe we should take this to PMs)

    Sure, pmed.

  • AnthonySmithAnthonySmith Member, Patron Provider

    TheLinuxBug said: I will not confirm or deny why I think this, but I am pretty confident this is the case.

    Would your little birdy (I assume the irc rumor mill) like to tell us how much is owed?

  • The picture being painted here is quite bleak. Personally, I'll wait 48 hours and if things aren't available, I'll cancel/move on.

  • "Unfortunately, it appears that the XCs between our network and colo pops through Alpheus are not passing enough light in DFW. (cont)"

    "Partial services in DFW have been restored (those of you in the Cologix POP), transport provider has been contacted to look into the rest."

    Oh gee, well don't put any pressure on them to 'look into' the rest.... I mean, you're fast approaching 36 hours of downtime here. No big deal, right?

    Sheesh.

  • @GameBeatDotNet said:
    "Unfortunately, it appears that the XCs between our network and colo pops through Alpheus are not passing enough light in DFW. (cont)"

    "Partial services in DFW have been restored (those of you in the Cologix POP), transport provider has been contacted to look into the rest."

    Oh gee, well don't put any pressure on them to 'look into' the rest.... I mean, you're fast approaching 36 hours of downtime here. No big deal, right?

    Sheesh.

    And this was not a problem before their hardware broke?

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    Jeez, talk about getting kicked while you're down. No pun intended.

    Best of luck to them on fixing things up :)

    Francisco

  • WSSWSS Member

    Proof that they're a bit dim, eh?

  • AnthonySmithAnthonySmith Member, Patron Provider

    Nothing in DFW has been restored, it simply is a stalling tactic, it is clear at this stage that it is an unpaid bill.

    Do not expect much folks, I am now in plan for the worst, hope for the best mode.

    i.e. assume your data is lost and server is never coming back, if you do not have backups, that is your fault do not project that part of your anger on to anyone but yourself and start to rebuild.

    If they do come up, move elsewhere immediately, it is a company on the brink of failure.

    Thanked by 1willie
  • Hold on AnthonySmith, I do blame myself for not having recent backups downloaded from there and not leaving there long time ago, but in all fairness. Where they providing this service as a charity and we should be grateful for what ever service we get? You buy this service because you assume(yes I know now) your data is safe in a DC. Unfortunately, did not factor in this risk.

  • @AnthonySmith said:
    12 hours to replace this and allowing it to keep literally everything down stinks of amateur.

    They've been amateur for years now.. it's amazing some of the shit they come up with.

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited January 2017

    Ranger said: Hold on AnthonySmith, I do blame myself for not having recent backups downloaded from there and not leaving there long time ago, but in all fairness. Where they providing this service as a charity and we should be grateful for what ever service we get? You buy this service because you assume(yes I know now) your data is safe in a DC. Unfortunately, did not factor in this risk.

    You are the only person to blame, not a single penny of what you pay them goes towards ensuring you have a constant data continuity flow in terms of backups, that is 100% your responsibility and absolutely no part of that should contribute towards your feelings regarding cloudshards.

    There are many situations that exist that can result in no access to your data, your raid could fail badly, your disks can and will fail at some point, none of this is cloudshards responsibility, it has been a fact since the dawn of the IT age that your data is yours, if it is important back it up.

    It is not for anyone except you to factor in as a risk, because it is your risk.

    While I am sympathetic to the fact that you are obviously new to this concept and most people learn this lesson the hard way (myself included) as you are now, it is never anyone's fault but your own if you lose more than 24 hours worth of data.

    You never assume your data is safe, DC's certainly not bargain basement ones like this have some sort of magic that prevents disk failures, raid failures, OS issues, and so on and so on.

    Are they useless for being down so long? Yes

    Are they handling it well? No

    Is it a fair way to treat customers? No

    Is the owner clearly a amateur who either does not care or is to scared to communicate? Yes.

    Did he give you any assurances at all implied or otherwise before you paid your hard earned money that he would guarantee your data? No.

    So like I said, at 48 hours, assume its not coming back and start rebuilding elsewhere.

    When a company went bankrupt on me in Miami and shut everything down with 0 notice I was moved within 24 hours, yet I was still getting emails from others 3 months later who seemed to think it was not impossible that they would be able to recover their data at some point, completely stupid.

    Suck it up and move on.

    acknowledgement: this was far more text than was needed and I wrote far to much.

  • The server is back online and was not rebooted. 38 hours downtime in total.

  • AnthonySmithAnthonySmith Member, Patron Provider

    and 2 minutes later, everything came back up.

    Backup your data, find your nearest exit .. and RUN!

  • AnthonySmith said: Suck it up and move on.

    acknowledgement: this was far more text than was needed and I wrote far to much.

    Thanks for your insight and no worries. What ever gets you through the day.
    In the meanwhile, I just copied the backups that were on their system( and I should have copied on a more regular bases) to a safe location. Lesson learnt.

    Thanked by 1AnthonySmith
  • AnthonySmithAnthonySmith Member, Patron Provider

    RFO in case anyone is interested:

    The failure was made up of 3 individual issues that acted as one problem which led to the complete
    
    loss of service. They are described below:
    
    1. Switch Fabric Module Failure in cr-0.clgx.dfw
     Query Foundry operates a fully redundant core router with backup line cards, routing
    engines and line cards. Unfortunately, both switch fabric modules (that are responsible for
    allowing various router components to communicate) abruptly failed on 17th January.
    
    ① Remedy – Both modules have been replaced and we have ensured to not pick them
    from the same assembly line to avoid the possibility of cascading failures in the future.
    
    2. Virtual Chassis Split
     Unfortunately, issue #1 led to our clustered switches using Juniper’s virtual chassis
    technology to end up becoming split, and they would not merge back together without
    onsite intervention. Unfortunately, our onsite tech was not available to assist. We lost all
    access to these switches and datacenter remote hands proved quite problematic when
    dealing with unfamiliar networking gear.
    
    ① Remedy – we were finally able to reset both devices and re-build the virtual chassis
    from the ground up using assistance from our partner datacenters. Huge thumbs up
    to the Viawest team for the assistance
    
    3. Transport unavailability
     Query Foundry’s POPs are designed in such a way that Network Connectivity is usually
    supplied from an well connected datacenter and the connectivity is then backhauled over
    protected dark fibers to our colocation POPs. In this instance, both of our inter-dc links
    proved to not be passing enough light for a successful Ethernet negotiation to happen.
    Failure of intra-building cross-connects like this is very rare, and caught us off guard.
    
    ① Remedy – The transport provider had to be contacted, and several cross-connect fiber
    pairs had to be re-wired. Unfortunately, this ended up taking the biggest amount of
    time as we had to coordinate with multiple datacenters who do not have a knowledge
    overlap. We have enabled optical monitoring on all of our core ports to avoid this in
    the future.
    
  • AnthonySmithAnthonySmith Member, Patron Provider

    "both switch fabric modules (that are responsible for
    allowing various router components to communicate) abruptly failed on 17th January"

    Seems highly unlikely, and they actually failed on the 16th January.

  • @AnthonySmith said:
    "both switch fabric modules (that are responsible for
    allowing various router components to communicate) abruptly failed on 17th January"

    Seems highly unlikely, and they actually failed on the 16th January.

    It seems stupid to build such an extensive lie, however, i suppose how will they get people to believe them, I suppose one big problem here is if this did all happen at once they should have had a backup plan in place to resolve the issue.

    If their claims are true then I feel their pain as a provider myself, however, from reading through all this even if their claims are true the way they handled this situation was poor, to say the least and shows that they have no care for their customers.

  • Bit of a worry given these guys also provide connectivity for @dediserve in Sydney.

    They had a terrible run in Sydney during 2015 / early 2016, but a lot better the last 9 months of 2016.

    It does seem like a lack of redundancy / clue / communication.

  • service was restored about 08:30 GMT today (20170118), having dropped
    at 19:30 GMT yesterday (20170117) according to our external
    monitoring.

    I note that the VPS we have there wasn't rebooted, it looks like it
    was just a network issue rather than someome temporarily pulling the
    plug on the entire operation.

    Thanked by 1ricardo
  • @AnthonySmith, do you have any further word on them going bankrupt?

  • AnthonySmithAnthonySmith Member, Patron Provider

    AlyssaD said: @AnthonySmith, do you have any further word on them going bankrupt?

    nope

  • @AnthonySmith said:

    AlyssaD said: @AnthonySmith, do you have any further word on them going bankrupt?

    nope

    So just a made up story from IRC that you threw around here for bashing sake?

  • AnthonySmithAnthonySmith Member, Patron Provider

    AlyssaD said: So just a made up story from IRC that you threw around here for bashing sake?

    did I ?

  • @AnthonySmith said:

    AlyssaD said: So just a made up story from IRC that you threw around here for bashing sake?

    did I ?

    You were very much convinced that they couldn't pay their bills without proper evidence.

  • Whatever happened almost 48 hours of downtime is absolutely inexcusable. It's unprofessional at the very minimum and is why I, after several years with them, am taking my business elsewhere.

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited January 2017

    PremiumN said: You were very much convinced that they couldn't pay their bills without proper evidence.

    I am still convinced, the absence of a smoking gun and any effective communication will only ever lead to assumptions based in what is known, and that is why I am convinced, give it a few months.

    I did not start the financial issues theme here, I did suspect it before anyone mentioned it though, I know more than is in this thread from a few independent sources, information gathered over a number of months not just the past 48 hours.

    If on the other hand you chose to believe that 2 independent fiber modules along with 2 other issues struck simultaneously and they had good reason for such poor communication and not even acknowledging the issue for 4 hours with their entire infrastructure down, then fair enough.

    I would encourage everyone to make up their own minds and act accordingly.

    side note: I have not been on IRC for about 2 years now, its a distraction at best.

Sign In or Register to comment.