Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


LeaseWeb USA , is Down
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

LeaseWeb USA , is Down

2 of my VPS on US Node are down.

> [Update October 12th 20:00 CEST]

Unfortunately the stabilizing of the platform is taking longer then we expected. 
Some of the instances running on this platform have been restarted due to the unexpected storage event.

We will update this post as soon as more information becomes available

[Original message]

Dear LeaseWeb customer,

We encountered an unexpected event on our storage platform, resulting in a failover to the second (redundant) storage head. This possibly leads to a temporary storage performance degradation of your instance(s).
Together with the supplier, our engineers are investigating the cause of this issue.

Please be assured that we will do our utmost to keep the impact as low as possible, but despite our efforts you might experience some performance issues. 

At this time the platform is stabilizing again and we expect performance to normalize in the coming minutes.

We will update you again once we have more information.

Sincerely,

LeaseWeb
http://www.leasewebnoc.com

http://leasewebnoc.com/en/networkstatus/possibly-degraded-performance-on-us-storage-platform

«13

Comments

  • Didn't they have the same drama on the NL storage too?

    Thanked by 44n0nx Shade bersy cassa
  • yes. I have started wondering, if leaseweb fails to deliver on "cloud" front whom do we trust ?

    Thanked by 14n0nx
  • Mridul said: yes. I have started wondering, if leaseweb fails to deliver on "cloud" front whom do we trust ?

    Leaseweb wouldn't come the top of my list for anything production grade but maybe I am just jaded.

    Are you buying HA cloud? Or just 'cloud' aka expensive VPS?

  • From what I observed, any "cloud" provider has more downtime than simple OpenVZ VPS's and especially dedicated servers..

    Maybe it is due to increased complexity? Maybe it's just my imagination... :)

    Thanked by 1jar
  • jarjar Patron Provider, Top Host, Veteran
    edited October 2015

    4n0nx said: From what I observed, any "cloud" provider has more downtime than simple OpenVZ VPS's and especially dedicated servers..

    Yeah, I notice that too. People spend so much money to eliminate single points of failure and end up creating more potential points of failure in the process. I remember a "high availability" provider a while back that was within LEB pricing, had SAN failures every week.

    I'll take two independent servers that replicate data with network level failover any day.

  • Jar said: Yeah, I notice that too. People spend so much money to eliminate single points of failure and end up creating more potential points of failure in the process. I remember a "high availability" provider a while back that was within LEB pricing, had SAN failures every week.

    Thats what redundant SAN and multipathing is for!

    Another brand runs HA corporate cloud services, zero downtime in 3 years. SAN is NetApp cluster so any head can be taken down for maintenance without disruption to storage. Servers have multiple connections to storage over redundant switches.

    There is nothing in that setup that isn't redundant, even location redundancy.

    But price point would make any normal person weep.

    Thanked by 1jar
  • 1 VPS is Online.

  • ClouviderClouvider Member, Patron Provider

    Unbelievably frequent issues they have...

  • Guess I was lucky picking the German location then, so far 100% uptime.

  • did they changed their NL storage to ssd and remove capping?

  • ru_tldru_tld Member, Host Rep

    NL storage has been already migrated from Nexenta to NetApp storage vendor.
    Block speed write on NL instances is currently stable 80mb\s.
    DE and US migration is still ongoing.

  • 2nd VPS is online as well.
    Can i Claim Credit ? ( im on the basic SLA they have )

  • ClouviderClouvider Member, Patron Provider

    @tr1cky said:
    Guess I was lucky picking the German location then, so far 100% uptime.

    Don't worry, it's somewhere in the queue for free downtime :).

    Jokes aside, fingers crossed you won't have similar problems.

  • @MarkTurner and this says the person who works for a company which can vanish with all data anytime soon AGAIN.

  • rm_rm_ IPv6 Advocate, Veteran
    edited October 2015

    MarkTurner said: Thats what redundant SAN and multipathing is for!

    All too often we still see the "redundant SAN" spectacularly come down, together with all its multipathing and whatnot. :D

    Really, a humble Kidechire with a locally attached old 160 GB HDD has proved leaps and bounds more reliable for me than REDUNDANT CLOUD VPSes by some providers from around here.

  • @rm_ said:
    Really, a humble Kidechire with a locally attached old 160 GB HDD has proved leaps and bounds more reliable for me than REDUNDANT CLOUD VPSes by some providers from around here.

    All hail the Kidechire!

  • @MarkTurner said:
    Leaseweb wouldn't come the top of my list for anything production grade

    And Delimiter is?

    Thanked by 2alexvolk GStanley
  • MaouniqueMaounique Host Rep, Veteran

    The added complexity does create more PoF, the software is not as stable as platforms tested for tens of years.
    We never had a SAN failure, still the orchestrator failed or went bezerk a few times moving VMs around without a reason.
    That being said, we have better uptime than Azure, no downtime for anyone on the HA platforms since early spring.

  • Maounique said: we have better uptime than Azure

    Everyone has better uptime than Azure :P

    Thanked by 1alexvolk
  • @Maounique said:
    The added complexity does create more PoF, the software is not as stable as platforms tested for tens of years.
    We never had a SAN failure, still the orchestrator failed or went bezerk a few times moving VMs around without a reason.

    The question is, did your system encounter situations where failover was actually used? Or were you just lucky not to have hardware failures?

    I mean, even my runabove instances have uptimes of ~200 days.

    I recall a time when BuyVM switched datacenters and encountered an outage 2 days after moving in because the datacenter staff forgot to switch the backup power switch from testing to operational. Of course, a power outage occured and the generators were running full power but were actually useless because the datacenter staff had bypassed the backup power system when running stress tests and forgot to re-enable it.

  • sinsin Member
    edited October 2015

    My US VPS has been up for a long time, one of the most stable vpses I have - they do plan on moving the US locations to the same platform they put the NL vpses on.

  • rm_ said: All too often we still see the "redundant SAN" spectacularly come down, together with all its multipathing and whatnot. :D

    Yea. The VPS.NET kinda thing, even more ironic considering they are like owned by OnApp and using the recommended storage backend.

  • Mridul said: We encountered an unexpected event on our storage platform, resulting in a failover to the second (redundant) storage head. This possibly leads to a temporary storage performance degradation of your instance(s).

    Sounds like another way to say disk failure and RAID rebuilding...

  • sinsin Member
    edited October 2015

    Mridul said: Can i Claim Credit ? ( im on the basic SLA they have )

    You could try...I remember canceling one of my Leaseweb vpses the other week (didn't need it anymore) and not only did their sales send me an email but they also called me and asked if there was anything they can do for me - thought it was pretty nice over a single $2.97 VPS.

  • MaouniqueMaounique Host Rep, Veteran

    @Shigawire said:
    I recall a time when BuyVM switched datacenters and encountered an outage 2 days after moving in because the datacenter staff forgot to switch the backup power switch from testing to operational. Of course, a power outage occured and the generators were running full power but were actually useless because the datacenter staff had bypassed the backup power system when running stress tests and forgot to re-enable it.

    It is not only that. We can perform maintenance transparently to the users, they are migrated without downtime.
    To answer your question directly, yes, we had twice a node failure (not physical, kernel panic for unknown reasons, perhaps related to clvm issues we had at that time) a few times more when the orchestrator thought the nodes are down and were not, one instance of an ARP storm due to a broken NIC where 1/3 of the nodes was marked as offline and the orchestrator began moving everyone.
    We learned from all instances, now the nodes have internal network connections to the orchestrator too, switched away from centos and clvm, using fewer nodes in a cluster to minimize impact in case something goes amok.
    These changes are visible, storage performance increased, deploy time fell dramatically as a result, uptime is flawless since early spring on the HA zones.

    Thanked by 2vimalware Shigawire
  • alexvolk said: and this says the person who works for a company which can vanish with all data anytime soon AGAIN.

    Your predictions have been completely wrong so far. You said that 3 years ago when we decimated the dedicated server market and again 2 years ago and so on.

    You just make yourself look stupid.

  • Jonchun said: And Delimiter is?

    Depends on the application and the selected product.

    If you're going to take a Delimiter Atom server and run a mission critical application on it, then I'd say you need your head examining.

    If you're going to take our cloud storage and run a mission critical application on it, then I'd say sure, its the same platform that we sell to corporates. Its fantastically reliable and redundant.

  • OnApp_TerryOnApp_Terry Member
    edited October 2015

    @William said:
    Yea. The VPS.NET kinda thing, even more ironic considering they are like owned by OnApp and using the recommended storage backend.

    To clarify, OnApp was spun out of UK2 Group in 2010. Since then, it has been a standard vendor/client relationship.

  • Does not matter in this case though - fact is they were owned by OnApp back then (or OnApp owned them, no idea) and 100% used the recommended the SW/HW, yet had massive SAN issues. This was the point i lost all trust in OnApp that you still did not gain back.

  • ru_tldru_tld Member, Host Rep
    edited October 2015

    LeaseWeb is going to migrate DE VPSes to SSD very soon.

    Migration of DE Virtual Servers to SSD Storage system begins 15 October 2015
    
    Dear LeaseWeb Customer,
    
    The general stability of the current storage platform for your virtual servers does not meet our company standards and the reliability that you may expect from LeaseWeb. We already started preparations last month to replace the current storage platform with a new, full-SSD storage platform. We are writing to you now to inform you we aim to start migrating servers in batches today.
    
    When your batch is about to be processed we will contact you 1 hour before the process begins. If you have multiple instances we will group your servers into the same batch wherever possible.
    Then, during a 4-hours window we will switch off your instance, perform the migration, verify it is successful and then return your instance to its original state. We will do everything we can to get you back to normal much sooner than the maximum of 4 hours.
    
    Once the migration is complete we will contact you again to confirm that your server is available once more. Any downtime should be kept to an absolute minimum and, following the migration, we look forward to giving you a more stable and high performing platform than ever.
    
    If you have any questions about the migration please contact [email protected] and we’ll give you the assistance you need.
    
    
    Kind regards,
    
    
    LeaseWeb Operations
    
    Thanked by 1Issam2204
Sign In or Register to comment.