New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
These are 2 different things.
Prometeus, essentially, runs from own DC, Salvatore is the CTO for the DC, has a whole team there if things get f-ed up and we did have some issues with our IPv6 last year when it was out a few hours, routes flapping and cards failing with some carriers even like this. I know I am monitoring even the temperatures in the DC in multiple points.
In Dallas and Pune, however, we have been down multiple times, in Dallas because the DC failed to push correctly our network and refused to aknowledge the mistake blaming it on cogent or other carriers, while in Pune there is packet loss a few times a month due to DDoS and the low and expensive bandwidth in India. When you do not control the DC, it is out of your hands many times. I lost count how often the disks in our backup solutions with leaseweb in DE failed, once they even pulled the wrong disk, this is the reason we try to keep things in Milano for our premium products, even though IWStack has a small zone in US too.
TL;DR, being on site and having own team there all the time is the only way to keep uptime over certain thresholds. It may work for months, even years, without permanent residence there, until shit happens. Once it will be fixed, will go again for many months at a time, especially since there will be backup routers.
sometimes its cathartic to know the techs are within punching-distance
How did they handle this?
By saying sorry a lot it would seem...
And by giving $0.06 credit when you ask for compensation.
I think @Francisco mentioned he is going to Cali this week on vacation and might spend a day in DC.
BRB, just raising a ticket.
A lot of raging. A LOT.
As of right now we got a temp solution that mostly works. We rolled an additional autonull that helps with one of the problem spots but it isn't ideal by any means.
Credit has ranged from a day to a month, on a case by case basis. Most people see a week I figure.
I purchased a brocade MLX-4 that will be installed tomorrow evening. I get into Vegas at 8AM, DC by 8:30 AM. By about 9:00 AM FEDEX will be by dropping it off. Due to how simple our configuration is, I figure it'll take me only 1 - 2 hours to get the whole configuration tested and ready for the night time swap.
Installation will be pretty fast. Power off & derack old router, rack new one, plug cables, power on & should be routing without issue. My hope is that Rob from Fiberhub will be around at that time (he usually is it seems) so he can lend me a hand incase something acts up.
Francisco
I should probably add that this isn't a case of 'Fran did it again'.
There's some pissed off Brazilian guy that ran a super sketchy IRC on our network that
finally got to the point of pissing us all off so we sent him packing.
He proceeded to throw 16 gigs at us and a ton of other interesting combo's. Staminus
did a good job handling 99% of it but that 1% that leaks is what causes the GRE to interrupt storm itself into the ground.
Sunday evening I was at my hotel quickly jotting together another autonull just to handle that all. It works well, but our other autonull was giving us attitude due to the platform change.
It's all a big pain in the ass and one that I don't want to repeat again. I bought
enough 10gig ports on the MLX that I can just setup a mirror port and continue
with autonull like nothing has changed.
Francisco
It would be cool if hosts offered bonus resources instead of $ back on the SLA.
I'd much rather get a bit more storage or RAM than a few cents back. Over time, loyalty at staying through the bad times would have definite benefits.
If you have service with us you can bring that up and Aldryic will think it through.
Either way, the router installs tomorrow evening.
Francisco
>
>
Good to hear, just recently moved one of my sites over to try out the BuyShared plans :-).
Since i haven't played with brocade gear and out of curiosity, does this new hardware router do GRE in hardware @Francisco ? I hope you check this in advance, to avoid any surprises.
@Francisco I really hope FedEx will not give you a surprise tomorrow.
I foresee a router with a bootprint on the side and an angry Fran.
Either that or a painless install + config like you guys deserve.
First thing I checked
Rob from Fiberhub helped me shop for what was the best option.
I considered an RX-4 at the start but they have serious ARP issues past a few thousand entries.
I looked at Juniper but the pricing was too high and we didn't need any of the special things they added. I paid $5500 for the brocade for what would have cost me $20k on the juniper. Cash wasn't the issue, it was just that I couldn't justify the $14k difference for no gains.
We got a nice unit. 2GB of RAM just for routes so if we start getting our own drops I can store a million+ routes. NULL0 is handled in hardware, GRE is, all switching is, etc. It's a really nice unit and will likely be a good base line for future deployments.
Francisco
The company we bought from is the same company Rob buys from so we know we're going to get a well packed, running, platform. Latest firmwares, nice long warranty, etc.
Really, it was a steal of a deal that I got on it in the end.
Francisco
I have only used next day once. It should arrive 9am, and arrived 2PM. When I asked the guy, he said: I made you a favor delivering this hour...
Seriously?!
But you're in Mexico.... :P
Francisco
Should have gone with an XMR for future proofing.
Considered but couldn't justify the need.
There's no point getting support for 10M routes when I see no reason to mix my own network in LV. The blend Rob has for us is solid and he has TELIA probably coming in.
I couldn't come close to this blend for the price he charges us so there's no real perks of that all.
If I ever roll another large setup (Europe? Central US?) then we'll look at at XMR if the inhouse brands are meh.
Francisco
Cross fingers
BTW, I'm gonna be there (LV) on June 16
I mentioned it on the IRC channel. I have the cheapie annual KVM.
MLXe = 2 million
XMR = 1 million
MLX = 512K
Point still stands, I need like...100 routes total
I could simply upgrade the MR if we wanted to start blending but as I said, I couldn't match the pricing that Rob gives me so why bother?
Francisco
Wasn't arguing with you, just wanted to make sure you were clear on the specs.
Nevermind. It's up now.
That was the scheduled maintenance for the Brocade
It was completed in less time than we scheduled for the most part. IPV6 was fixed just a few minutes ago.
The biggest problem was getting the old router out of the rack so I could put the brocade there. The walls of the cage are a tiny bit too close so you can't extend the server out 100% and pull it off the rails, you have to slide the rail extensions in a tiny bit to give you wiggle.
Once I had that done I installed the grouber rails, hoisted it into place, ran the cables, & fixed the ASN numbers (I left the testing ones on, woops).
Francisco
Looks like there is a lot of hard work going on behind the scenes.
I wonder, do they do DDoS protected IPs for their shared/reseller hosting? Curious.