Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

Trabia / AS43289 incl. all downstreams dropped off earth

Trabia / AS43289 incl. all downstreams dropped off earth

WilliamWilliam Member, Provider
edited April 20 in Outages

Entirely.

Try eg trace to 178.17.171.1

All ranges dropped from BGP, all their downstreams (eg. Mivocloud) reroute currently.

No one seems singlehomed Trabia so no further outages than that.

Anyone using them for protection, eg AS3281 (2Cloud, an ISP in Latvia) is now unprotected, and they have some customers like that:

https://bgpview.io/asn/43289#downstreams-v4

«1

Comments

  • MaherMaher Member

    What's the reason behind this though ?

    Rocketrix | Host it like a Rocket | https://rocketrix.com/

  • WilliamWilliam Member, Provider

    Maher said: What's the reason behind this though ?

    I would open a ticket but, well, website offline as same DC :)

    No phone number of Sven (the owner) and know no one local enough to verify.

    Last time they lost power entirely i think.

    Thanked by 1Maher
  • WilliamWilliam Member, Provider
    edited April 20

    No one transiting via Trabia, no routes on their AS:

    [email protected]:~# birdc -r "show route all filter { if bgp_path ~ [= * 43289 * =] then accept; reject; }"
    BIRD 1.6.3 ready.
    Access restricted
    [email protected]:~#
    
    [email protected]:~/lg/scripts# birdc -r "show route all filter { if bgp_path ~ [= * 43289 =] then accept; reject; }"
    BIRD 1.6.3 ready.
    Access restricted
    [email protected]:~#
    
  • WilliamWilliam Member, Provider

    My Romanian is bad/had someone help me, but searches for power outage in the city (eg. pană de curent în chisinau) get no new results.

    All others seem online, so Trabia only.

    Ironically they shut down the DC 23.11.2016 to switch to UPS/Genset upgrades lol:

    As announced August this year, we are currently in progress of upgrading our entire power infrastructure to give us about 5 times the power capacity we currently have.
    
    We performed large scale changes in the past 3 months, such as:
    
    - Installing new redundant main power feeds
    - Installing new redundant data center UPS
    - Installing new diesel generator
    - Installing new air conditioning distribuion
    - Installing new power communications inside the entire facility
    - Installing new power monitoring systems
    
    We finished all preparations and are ready to connect all PDUs within our KIV01 data center to the new electricity network.
    
    In order for all equipment to benefit from the upgrades, we will have to schedule a maintenance on which we will have to move of all power circuits to the new electricity network.
    This will result in a power loss for all active powered equipment within the data center for 5-30 minutes. In almost all cases the power loss will be less than 5 minutes.
    Services with dual power supplies are not going to loose power, however services with single power supply will be impacted by an reboot.
    We will perform soft reboots of all servers possible, however there may be cases on which a hard reset may occure.
    
    Please find the following maintenance window information:
    
    Maintenance Information
    Services: All active powered equipment located within the KIV01 data center facility
    Start: Thursday, November 24th, 2016, 23:00 o'clock UTC
    End: Friday, November 25th, 2016, 5:00 o'clock UTC
    Effect: Power loss for 5 minutes. Only in an extreme unpredictable case, the power loss might be up to 30 minutes.
    
    
    We will move circuits step by step, meaning not all the data center will be shut down at once but rather only on a per-circuit base.
    Our technicians will do everything to ensure a smooth and fast migration to the new electricity network in order to minimize the impact on your services.
    
    We will ensure that all servers are booting up normal with our monitoring system and manual verification.
    
    
    I apologize for the inconvenience this may cause and thank you for your understanding and your trust in us by being our customer.
    
    Do not hesitate to contact us if you have any further questions.
    
  • AlyssaDAlyssaD Member

    @william, can you link me a tutorial on how to use bird as your are for BGP lookups? I have looked for that for awhile, and haven't found anyway of doing it.

    Thank you.

  • Just notice this outage as well and was seeing no BGP, at first I though it was a flap but then noticed all their announcements just disappeared. Seems you have more info than I could find. Very curious to say the least..

    Cheers!

    Have an Allwinner H3 device? Want Android? Check out H3Droid! | Lichee Pi Zero - The 6$ SBC
    18+ Years IT Experience in Linux/Windows Hosting and Administration

  • WilliamWilliam Member, Provider
    edited April 20

    AlyssaD said: @william, can you link me a tutorial on how to use bird as your are for BGP lookups? I have looked for that for awhile, and haven't found anyway of doing it.

    You need to have a BGP session, not my config and i doubt we used any tutorials. @fusl ?

    A lot of functionality is available via mtr.sh public.

  • IshaqIshaq Member
    [BudgetNode] DDoS Protected. 5 Locations (US/EU). Check out our latest offer!
  • AlyssaDAlyssaD Member

    How do you even get a BGP session? What does it require? Do you need you own AS?

  • WSSWSS Member

    SMFH

    I've been asked to stop derailing threads. So, stop enticing more-interesting topics!

  • WilliamWilliam Member, Provider
    edited April 20

    AlyssaD said: How do you even get a BGP session? What does it require? Do you need you own AS?

    Not really right thread/place for that, but yes you need an AS, private or public. No IP space required though.

    @ishaq that is pointless, he uses himself as upstream at home as far as i know.

  • TheLinuxBugTheLinuxBug Member
    edited April 20

    Crap old thread, on mobile, didn't notice till I posted.

    Cheers!

    Have an Allwinner H3 device? Want Android? Check out H3Droid! | Lichee Pi Zero - The 6$ SBC
    18+ Years IT Experience in Linux/Windows Hosting and Administration

  • LordSpockLordSpock Member, Provider

    As far as I know, bad weather in Chișinău has been damaging things such as powerlines. Maybe their power is not as reliable as they say.

  • WilliamWilliam Member, Provider

    LordSpock said: As far as I know, bad weather in Chișinău has been damaging things such as powerlines. Maybe their power is not as reliable as they say.

    Google and local news sites do not report power loss and the cheaper DCs are online.... either they failed HARD or there is a major fiber problem.

  • sinsin Member
    edited April 21

    I didn't even notice my Trabia VPS was down, bought it yesterday and it was so much slower then my old VPSes with them (and they wouldn't move it) I just decided to let it idle.

    -edit- my VPS is back up with them and same with their website

  • xaocxaoc Member

    @LordSpock said: As far as I know, bad weather in Chișinău has been damaging things such as powerlines. Maybe their power is not as reliable as they say.

    Yea it is probably weather related cause it's been snowing here since yesterday morning and the snow cover is thicker than 10cm. Lots of trees fell down in Chisinau as far as i know. Nobody was prepared for this kind of weather in April...

    So Say We All

  • WilliamWilliam Member, Provider
    edited April 21

    Back online. Opening ticket now.

    xaoc said: Yea it is probably weather related cause it's been snowing here since yesterday morning and the snow cover is thicker than 10cm. Lots of trees fell down in Chisinau as far as i know. Nobody was prepared for this kind of weather in April...

    Does not explain why no other, much crappier, datacenter was affected. Zero of them. And some are basements with not even UPS.

    Thanked by 1sin
  • WilliamWilliam Member, Provider

    Ticket open, SLA coverage requested (ToS 7.3).

    ToS say SLA only applicable if you report downtime - was impossible with website offline so if they insist on that point... there will be problems.

  • I gotta ask - what's so special with Trabia? I mean, apart from it going black-hole every once in a while.

  • BG32BG32 Member

    Who?

    Thanked by (44)

  • bsdguybsdguy Member
    edited April 21

    Being at "strange things at providers": You know that the cia was deep inside Ikoula, do you?

    My favourite prime number is 42. - In memory of WSS: "Our house. In the middle of our street. That's madness! Madness, I tell you!"

  • WSSWSS Member

    @bsdguy said: Being at "strange things at providers": You know that the cia was deep inside Ikoula, do you?

    Sounds HOT.

    I've been asked to stop derailing threads. So, stop enticing more-interesting topics!

    Thanked by 1deadbeef
  • bsdguybsdguy Member
    edited April 21

    Yep. See TheShadowBrokers files.

    P.S./Note: This information is by no means intended to smear @Ikoula! In fact, it's quite possible that they became a target of the us of a spooks because they did the-right-thing (tm), e.g. by supporting Wikileaks. Moreover it should be noted that this information does not in any way suggest Ikoula being less secure than other providers! From what is known the nsa/cia tools would have worked against almost every other provider too. (edit: tagged Ikoula so as to give them notice of being mentioned and to give a statement if they please to to so.)

    My favourite prime number is 42. - In memory of WSS: "Our house. In the middle of our street. That's madness! Madness, I tell you!"

  • ClouviderClouvider Member, Provider

    @bsdguy said: Yep. See TheShadowBrokers files.

    P.S./Note: This information is by no means intended to smear @Ikoula! In fact, it's quite possible that they became a target of the us of a spooks because they did the-right-thing (tm), e.g. by supporting Wikileaks. Moreover it should be noted that this information does not in any way suggest Ikoula being less secure than other providers! From what is known the nsa/cia tools would have worked against almost every other provider too. (edit: tagged Ikoula so as to give them notice of being mentioned and to give a statement if they please to to so.)

    Tags in post edits do not flag @Ikoula

    Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

    Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

  • bsdguybsdguy Member

    Oops, sorry. How to make sure that Ikoula sees this?

    My favourite prime number is 42. - In memory of WSS: "Our house. In the middle of our street. That's madness! Madness, I tell you!"

  • ClouviderClouvider Member, Provider

    @bsdguy said: Oops, sorry. How to make sure that Ikoula sees this?

    They should now see mention in my post, you could also PM them :)?

    Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

    Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

  • bsdguybsdguy Member

    OK. Thank you.

    My favourite prime number is 42. - In memory of WSS: "Our house. In the middle of our street. That's madness! Madness, I tell you!"

  • WilliamWilliam Member, Provider
    edited April 21

    deadbeef said: I gotta ask - what's so special with Trabia? I mean, apart from it going black-hole every once in a while.

    Nothing. It's Moldova (rather rare), cheap (rather rare, there) and generous BW (rare) with protection free mostly (rare).

    In this case the entire BGP drop was the weird thing, DCs down are not unusual or upstreams not reachable (but via IX); entire blackout is strange.

    Thanked by 1deadbeef
  • sinsin Member

    William said: In this case the entire BGP drop was the weird thing, DCs down are not unusual or upstreams not reachable (but via IX); entire blackout is strange.

    Have they mentioned anything in your ticket?

  • WilliamWilliam Member, Provider

    sin said: Have they mentioned anything in your ticket?

    Answer "In Progress" and set to On-Hold.

  • MacPacMacPac Member

    deadpool

  • trabiatrabia Member, Provider
    edited April 21

    Hi guys,

    We are currently suffering from severe weather conditions.

    Starting from the night of 19th to 20th April a cold weather front hit us resulting in large snow and rain. Last night, 20th to 21st April, the weather worsened a lot more that even traveling within the city became hard.

    During this past 2 days, fiber optic providers, such as we, are facing issues with a lot of fiber cuts, some ISPs work round-the-clock to solve fiber cuts. Eventually during the night, even though we have multiple fiber optic cables on multiple paths and our teams were working already on repairing affected paths, we were hit that hard that we suffered a total blackout.

    We solved the situation temporary on alternate paths and our teams are continuously working to solve the issue entirely to re-establish the old paths.

    Currently the government released a warning, encouraging people not to leave their homes. Public transportation isn't working, driving or walking through the city is very hard, in some regions even impossible, the government closed several streets entirely.

    The government further announced that one district of the city will be taken down of electricity. We aren't affected by this, however we are either way protected by our autonomous power through UPS and Diesel Generators.

    I apologize deeply for the issues, be assured that we are working hard to fight this weather conditions and keep your services up and running smooth.

    Cheers, Sven

  • rds100rds100 Member

    So are these fiber optic cables underground, or hanging on the trees?

    -

    Thanked by 2bersy Wirrkopf
  • dedicadosdedicados Member
    edited April 21

    this mus be bad because you really look worried... @William

    Some VPS promos: MyAffLinks | NodeStats ❤ Monitor your servers for FREE!! | lowend.rocks!! LG from all providers, Promos and more!
    Thanked by 1netomx
  • WilliamWilliam Member, Provider
    edited April 21

    rds100 said: So are these fiber optic cables underground, or hanging on the trees?

    Mostly pole fiber. As in Romania.

    Sven/Trabia also still failed to explain me why much shittier DCs had zero issues while there are only very few cables going in/out of MD.

    No mention of my SLA credit/entirely ignored, ticket was closed.

    dedicados said: this mus be bad because you really look worried... @William

    i'm very used to downtime with them, by now i don't care anymore.

    None of our own fiber optic cables were direct impacted, however every all our fiber optic providers in Moldova and in Romania are suffering from fiber cuts.

    Lie. Moldtelecom personally, just a ~15min ago, told me while there are issues that no international connectivity is affected.

    Thanked by 2doughmanes quick
  • AlexeiAlexei Member
    edited April 21

    @William said: Lie. Moldtelecom personally, just a ~15min ago, told me while there are issues that no international connectivity is affected.

    Moldtelecom is a national state provider and basically monopolized underground communications left after USSR.

    So, serious datacenters should have interconnection with Moldtelecom even if it is pricey.

  • trabiatrabia Member, Provider
    edited April 21

    @William said: Mostly pole fiber. As in Romania.

    It's mixed, a part of our fiber paths are in sewers others on poles. Mostly in sewers already, but not yet exclusively.

    Sven/Trabia also still failed to explain me why much shittier DCs had zero issues while there are only very few cables going in/out of MD.

    I'd guess: bad luck. All providers, without exception, suffered fibercuts todsy. Half of the capital was down, partially still is, such as urban areas. The government officially declared state of emergency in the morning. We have a few paths going to the borders and from the borders to our POPs in Iasi and Bucharest. If a cable that has 48 fibers breaks, depending the way it breaks, it does not mean thst all 48 fibers become unusable. There are many factors and cables separate and merge depending how they have to go.

    No mention of my SLA credit/entirely ignored, ticket was closed.

    This is declared as force major, which does not fall into an SLA. However, it was stated that you will receive a separate email later.

    Lie. Moldtelecom personally, just a ~15min ago, told me while there are issues that no international connectivity is affected.

    We do not take connectivity from any local Moldovan ISP. We do lease only dark fibers or wavelengths from various different providers that have cables crossing the borders, therefor it is impossible for Moldtelecom to know our infrastructure nor what worked and what did not, they can speak only for themselves. All statements provided were accurate. In Romania, CFR and Telekom struggled hard with fiber cuts with which as well many Moldovan providers had to struggle with, and still do.

    Cheers, Sven

  • WilliamWilliam Member, Provider
    edited April 21

    trabia said: This is declared as force major, which does not fall into an SLA. However, it was stated that you will receive a separate email later.

    Nice try, but no:

    10.1 clearly states "were not reasonably foreseeable" - you were very well aware of the weather and issues before already (you stated this above also personally), so you broke this.

    Thus i am eligible for compensation per 7.3.

    EDIT: received compensation for the downtime, but will not accept it as force majeure case, as it simply is not. I want Trabia to state it was not.

  • bsdguybsdguy Member

    @William said:

    trabia said: This is declared as force major, which does not fall into an SLA. However, it was stated that you will receive a separate email later.

    Nice try, but no:

    10.1 clearly states "were not reasonably foreseeable" - you were very well aware of the weather and issues before already (you stated this above also personally), so you broke this.

    Thus i am eligible for compensation per 7.3.

    EDIT: received compensation for the downtime, but will not accept it as force majeure case, as it simply is not. I want Trabia to state it was not.

    Calamities due to force of nature out of the common fall under force majeur in pretty every legislation I know. Government declaring emergency puts the official seal on the current situation being uncommon force of nature.

    Well noted I do not take a side here and I do not know whether trabias statements are true. But if they are true then that's a classical case of force majeur.

    My favourite prime number is 42. - In memory of WSS: "Our house. In the middle of our street. That's madness! Madness, I tell you!"

    Thanked by 2trabia chrisp
  • bsdguybsdguy Member

    @trabia said:

    @bsdguy said: But if they are true then that's a classical case of force majeur.

    https://www.facebook.com/sven.wiese/posts/1596230870396500

    Enjoy the pictures :)

    a) I can't see anything on facebook as I have no account there. b) my point wasn't to say you are a liar but not to take sides and to make clear that what you described clearly is a case of force majeur.

    My favourite prime number is 42. - In memory of WSS: "Our house. In the middle of our street. That's madness! Madness, I tell you!"

    Thanked by 1trabia
  • trabiatrabia Member, Provider
    edited April 21

    @bsdguy said: a) I can't see anything on facebook as I have no account there.

    Not everyone needs to have an account. :)

    @bsdguy said: b) my point wasn't to say you are a liar but not to take sides and to make clear that what you described clearly is a case of force majeur.

    And I understood it in this way and that's the most correct way!

    Cheers, Sven

  • TomTom Member

    @bsdguy said:

    a) I can't see anything on facebook as I have no account there.

    There's 35 pictures there, but a select few: here here and here

  • rds100rds100 Member

    A lot of fiber there :) I guess that explains the problems.

    image

    -

    Thanked by 1trabia
  • WilliamWilliam Member, Provider
    edited April 21

    bsdguy said: Government declaring emergency puts the official seal on the current situation being uncommon force of nature.

    This was after the outage (you can also see this on Svens FB and the official gov site if you speak Romanian or Russian):

    trabia said: Currently the government released a warning

  • bsdguybsdguy Member

    Doesn't matter. The government said it regarding that current situation which is still not over..

    My favourite prime number is 42. - In memory of WSS: "Our house. In the middle of our street. That's madness! Madness, I tell you!"

  • trabiatrabia Member, Provider

    @rds100 said: A lot of fiber there :) I guess that explains the problems.

    It's pretty bad. There are still and will be for the next days, a lot of techs going around the country fixing fiber optic cables. The only Tier 1 provider present in the country, Cogent, is entirely down since about 22+ hours already and they have their own 3 paths to two different countries (Ukraine and Romania) and they still do not have an ETR yet. We handled the situation as fast and best as possible.

    A lot of people faced damages, on buildings, cars, companies with services, etc.

    I hope for everyone that it will be over soon and we will have a better spring after cleaning up :)

    Cheers, Sven

  • WilliamWilliam Member, Provider
    edited April 21

    bsdguy said: Doesn't matter. The government said it regarding that current situation which is still not over..

    After i sent in and after the outage. This is clear (and verifiable) - no such thing as retroactive emergency state. This ain't Turkey or Saudi Arabia.

    Analogy with criminal - If you smoke weed today, and the gov tomorrow decides it is illegal, they cannot jail you (unless you live in a shithole of a dictatorship).

    Point stands - days bad weather, NO gov emergency state, outage, THEN emergency state. At the time of outage there was none and the issues were predictable to them, so the ToS point is valid and this is not force majeure (that a Romanian speaker should be able to write correctly at least.).

  • @William Governments don't exactly declare emergency on every bad weather. They wait till shit goes to hell first, for the obvious reason that state of emergency -> more money spent.

    Thanked by 1jvnadr
  • WilliamWilliam Member, Provider
    edited April 21

    Correct. Thus it was none until then and they cannot claim this.

This discussion has been closed.