Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › Dacentec.com down - Page 4
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Dacentec.com down

12467…13

Comments

  • jbilohjbiloh Administrator, Veteran

    Sounds pretty rough. Hope for a speedy and complete restoration of service for the customers impacted.

    Looks like they had power issues which led to ongoing network problems?

    Thanked by 1MikePT
  • boerndboernd Member 
    Date
Tuesday January 26, 2016 

Impact 
Data Center Location(s): 
801 Main Street NW, Lenoir, NC  

Service:
Network
UPS

Details
Dacentec will be enabling a redundant internet provider to restore network redundancy to the facility. During the maintenance window customers may experience brief periods of instability as routes fail over to alternate devices and re-converge throughout the internet.  Further to our power incident earlier, we will be also be performing emergency maintenance to our UPS systems.

Support
Dacentec’s support team will be available for assistance 24/7. For questions, concerns, or to report a problem, please contact our Operations Center:
    Thanked by 1arminds
  • PrincessOfCatsPrincessOfCats Member
    edited January 2016

    @boernd said:
    Date
    Tuesday January 26, 2016

    Impact
    Data Center Location(s):
    801 Main Street NW, Lenoir, NC

    Service:
    Network
    UPS

    Details
    Dacentec will be enabling a redundant internet provider to restore network redundancy to the facility. During the maintenance window customers may experience brief periods of instability as routes fail over to alternate devices and re-converge throughout the internet. Further to our power incident earlier, we will be also be performing emergency maintenance to our UPS systems.

    Support
    Dacentec’s support team will be available for assistance 24/7. For questions, concerns, or to report a problem, please contact our Operations Center:

    • cough * Hurricane Electric * cough *
  • boerndboernd Member

    Level3 it seems

     6. 84.116.140.174                                                                                                                        0.0%    13   86.5  87.2  85.8  88.6   0.9
 7. us-nyc01b-ri2-et-4-1-0-0.aorta.net                                                                                                    0.0%    13   87.3  86.7  85.8  87.4   0.6
 8. lag-5.ear3.NewYork1.Level3.net                                                                                                        0.0%    13   99.6  89.7  86.4  99.6   4.2
 9. ???
10. ae-56-56.ebr2.Washington12.Level3.net                                                                                                 0.0%    13  102.7 103.7 102.3 109.3   1.9
11. ae-1-100.ebr1.Washington12.Level3.net                                                                                                 0.0%    13  102.5 102.7 101.8 104.0   0.6
12. ae-101-101.ebr3.Washington12.Level3.net                                                                                              91.7%    13   92.7  92.7  92.7  92.7   0.0
13. ???
14. ???
15. CENTRILOGIC.ear1.Atlanta2.Level3.net                                                                                                  0.0%    12  102.8 103.8 101.9 110.2   2.3
16. dct-cr01--v51.dacentec.com                                                                                                            0.0%     7  107.8 108.8 106.8 115.7   3.2
17. dct-ds42-ve33.dacentec.com                                                                                                            0.0%     7  109.2 112.3 109.1 121.4   4.7
18. 192.254.78.202                                                                                                                        0.0%     6  106.6 106.9 106.4 107.3   0.4
19. 104.171.XXX.XX                                                                                                                        0.0%     6  111.0 110.0 109.2 111.3   0.9
  • PrincessOfCatsPrincessOfCats Member
    edited January 2016

    Woah.

    They enabled level3 as well now, a few months ago, I had level3 going to the server, it eventually switched over to telia (even when going through level3 looking glass), and now it is at HE.

    Absolutely no idea what they are doing.

    Thanked by 1netomx
  • miTgiBmiTgiB Member

    boernd said: Level3 it seems

    Indeed

                                 My traceroute  [v0.75]
escapepad (0.0.0.0)                                    Tue Jan 26 19:26:16 2016
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. hostigation-gw.caro.net           0.0%    13    0.5   0.5   0.5   0.7   0.1
 2. te-3-1.car2.Charlotte1.Level3.ne  0.0%    12    0.6   0.6   0.6   0.6   0.0
 3. CENTRILOGIC.ear1.Atlanta2.Level3  0.0%    12    5.2   6.7   5.1  20.7   4.5
 4. dct-cr01--v51.dacentec.com        0.0%    12   13.9  13.9  13.9  13.9   0.0
 5. mirror.dacentec.com               0.0%    12   13.8  13.8  13.7  14.1   0.1

    Why they went to Atlanta instead of Charlotte is a head scratcher

  • PrincessOfCatsPrincessOfCats Member

    @miTgiB said:
    Why they went to Atlanta instead of Charlotte is a head scratcher

    I thought they had backhauls to atlanta only, though I've been seeing telia going directly to charlotte

  • miTgiBmiTgiB Member

    ALinuxNinja said: I thought they had backhauls to atlanta only

    Maybe, but there is a great metro fiber network in Charlotte that extends to Lenoir

  • PrincessOfCatsPrincessOfCats Member
    edited January 2016

    nvm about going through HE.

    The network I'm on (AS549) only has HE and Cogent, that must be why it is doing that.

  • IThinkUFailedIThinkUFailed Member

    Yeah honestly, with all these constant downtimes and network "upgrades" that get done I'm leaving. Why is it every time a network issue occurs we then hear about them adding in redundancy... why hasn't it been done already?

    Power outage too? Where are the backup generators etc? Honestly I'm not impressed and will be sending my cancellation in once I have a replacement box up and running.

    Thanked by 1arminds
  • PrincessOfCatsPrincessOfCats Member
    edited January 2016

    Anyone here have a view on what networks they are announcing from now?

  • catalystiumcatalystium Member

    @mikeyur

    So any tasty stuff incoming soon? ;-)

  • ManofServerManofServer Member

    @Infinity580 said:
    Our technicians have resolved the network and power issues that we have been experiencing at our Lenoir, North Carolina data center. All services have now been restored, and we are continuing to monitor our infrastructure to ensure further customer impact. Further information will be shared with all customers once all analysis is complete

    To report any additional issues, please submit a ticket to [email protected].

    Lol didn't knew you were a Dacentec employee. Why didn't you guys take more precaution?

  • mike0000mike0000 Member

    @catalystium said:
    mikeyur

    So any tasty stuff incoming soon? ;-)

    Whatcha looking for? :-)

  • KuJoeKuJoe Member, Host Rep

    I feel so dumb. Spent way too much time troubleshooting the RTO server problem and realized the port on the switch was turned off.

    Note to self: Copy running configs to start-up configs regularly in case there's a power outage. :(

  • miTgiBmiTgiB Member

    KuJoe said: Copy running configs to start-up configs regularly in case there's a power outage.

    copy run start on every change and write to offsite backup, you can always roll back since you do keep a history of backups, otherwise I'll need to have your wife collect your nerd card for the rest of the month

  • dacentecdacentec Member, Host Rep

    Full network redundancy is back online. We have our UPS manufacturer on site now performing work to restore full redundancy to the facility. While we are staffed 24x7x365 anyway, we have increased staff on site until the issue is fully remedied.

    Thanked by 4Silvenga PrincessOfCats boernd k0nsl
  • PrincessOfCatsPrincessOfCats Member

    @dacentec said:
    Full network redundancy is back online. We have our UPS manufacturer on site now performing work to restore full redundancy to the facility. While we are staffed 24x7x365 anyway, we have increased staff on site until the issue is fully remedied.

    Hi,

    Is there any official RFO atm?

  • dacentecdacentec Member, Host Rep

    @ALinuxNinja said:
    Is there any official RFO atm?

    Not at the moment but we will certainly make one available.

  • KuJoeKuJoe Member, Host Rep

    @miTgiB said:
    copy run start on every change and write to offsite backup, you can always roll back since you do keep a history of backups, otherwise I'll need to have your wife collect your nerd card for the rest of the month

    I don't do that because if I break something I like the option of rebooting and fixing it since I am not local and I don't trust them to console into my switches.

  • AutoSnipeAutoSnipe Member
    edited January 2016

    By chance are any of the extra staff you have there answering tickets?

    Still got a server down, ticket unreplied for a record 7hrs or so.

    Ticket 586951

  • vpsGODvpsGOD Member, Host Rep
    edited January 2016

    @autosnipe I am waiting for 10 hours to get a reply for inprogress ticket reply

    Again gone down (came up)

  • armindsarminds Member, Host Rep

    @dacentec said:
    Full network redundancy is back online. We have our UPS manufacturer on site now performing work to restore full redundancy to the facility. While we are staffed 24x7x365 anyway, we have increased staff on site until the issue is fully remedied.

    I am unable to login the KVM you attached hours ago. No responses for hours. The authentication failed. My server is still down because of your issues and you weren't ready with enough KVMs?

    What is the benefit of increasing staff if you don't have resources to help customers fix their servers after your careless power outage?!!!!

    I am totally frustrated!

  • NateN34NateN34 Member
    edited January 2016

    How does the power outage kill our servers?

    I'm guessing the budget servers aren't UPS backed?

  • KuJoeKuJoe Member, Host Rep

    @NateN34 said:
    How does the power outage kill our servers?

    Shouldn't there be battery powered UPS and then the generators kick in?

    The announcement about the emergency maintenance to the UPS makes me think the UPS failed to switch over properly. I'm interested in a real RFO to see exactly what happened, what could have been prevented, and what steps they are taking to prevent it in the future.

  • JacobJacob Member

    Do you expect the $20 servers to run through a UPS?

    @NateN34 said:
    How does the power outage kill our servers?

    I'm guessing the budget servers aren't UPS backed?

    Thanked by 1netomx
  • lmnotranlmnotran Member
    edited January 2016

    Jacob said: Do you expect the $20 servers to run through a UPS?

    I do.

  • armindsarminds Member, Host Rep

    @Jacob said:
    Do you expect the $20 servers to run through a UPS?

    You are telling me that may be out colocated owner servers also are not run through UPS?

  • vpsGODvpsGOD Member, Host Rep

    @arminds i am also on the same boat.
    You were lucky to get the KVM. I am till waiting.

    I dont understand why they need to replace the server. OMG

    @dacentec

  • mike0000mike0000 Member

    @vpsGOD said:
    I dont understand why they need to replace the server. OMG

    Maybe a fried PSU, or worse? Can you ask them to pull the drives and put them in a new box?

This discussion has been closed.