New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
looks like it's down again
I spoke too soon
Looks like we are down in a big way now in Buffalo, @Francisco!
I figured ...
I have no clue on which node I am on, but the monitoring for Node 1-3 says ugliness:
ovz-ny-01 38m
ovz-ny-02 23m
ovz-ny-03 9m
Those are color coded RED indicating DOWN.
cant wait for them to be online in vegas
If Francisco is traveling with the big truck at the moment i doubt he can do much to mitigate the DoS right now.
It is nice that they communicated it so well to their customers. I removed my 4 BuyVM's from my DNS a few days ago, added 4 other VPS's to the cluster, deployed a new version of the website, and done.
Only forgot to schedule the downtime in my nagios, had 4 mails this morning...
which DNS service do you use?
@jcaleb Trans IP's DNS service. They host my domain ass well.
@Raymii thanks boss. I ask because I am interested to learn high availability
I'm not hosting my own DNS for raymii.org. I do have some DNS servers under my control for a few companies, they all run BIND (except for 3 servers at offices, they run NSD for caching and the office computers DNS).
HA DNS requires an investment in time and servers, and the first thing is the thing I don't have much more.
I have received a couple dozen emails about one of my NY VPS' being down. I thought this move was for SJ and had nothing to do with Buffalo?
@luma Someone started DDOSing Buffalo pretty much right as SJ went down.
Buyvmstatus.com seems to be down now.
Getting less packet loss in NY, but at 5% or so.
So, ongoing issues.
server is up, website is down (looks like it is hosted by Ram Host in Atlanta)
RAM Host provides Quality hosting, even for their competitors! :P
I wonder how's their roadtrip
ny dc: stable 142 ms ping for me from eu..
0% packet loss, time 13015ms
They said Fran wasn't driving, So he is probably sat there on his Andriod/Iphone trying to do what he can.
Shit, I forgot to shut down my storage KVM in advance :-/
Anyone able to PM me the IRC IP as well? Thanks
Done
Thanks for the feedback, however buyvmstatus.com "is not operated by, sponsored by or affiliated with BuyVM in any way." (from their site).
We haven't had any outages on our services anytime recently.
No CC's guys were off duty at the time it seems.
There was no fault on there part, rather a misconfiguration on my end in regards to thresh levels.
There was some burps through-out the move as there was anywhere from 500 - 1Gbit/sec floods coming in. They null quickly, but CC's junipers take 2 - 5 seconds to kick fully.
I may take Jon's offer up on the 2nd gig port just for breathing room during those few moments of time.
I had things in NY settled pretty quickly. I didn't get informed about the issues for about 10 minutes as it's hard getting reception in MPT. Once I got somewhere that worked, Anthony rung me like mad to get on the IPMI and check it out.
The guys continued to derack the gear while I mopped up the drool left over by the retard infestation.
Francisco
Not true.
We set the limits a lot lower so the nullroutes were kicking in automatically.
Ponynull is a beautiful batch of code and only improves with each revision.
There was nothing more I could have done sitting infront of a workstation the whole time compared to just letting ponynull do its job.
Ponynull does tallies every second and runs through looking for anything that goes over many different water marks we set (some are set based on pps, some bytes/sec and some are a combo of both). When it finds an entry that's offending it'll issue off a nullroute and continue chewing through the lists.
This might sound heavy but in reality the whole thing eats like 20% of a single core for a few gbit/sec of traffic. I'm fairly sure if I compiled it with a *RING based deployment it'd be more like ~3%.
Francisco
Correct, the availability issues buyvmstatus had were on my side, ramhost wasn't to blame. (OOM ftw)
Ponynull sounds pretty darn cool.
Love this sentence
@Francisco
Hello, do you have any status for the Storage node? Because I am unable to boot it, use VNC, or even ping the IP to the VNC server (same as the node IP I guess). The server came backup since long time ago according to buyvmstatus but I can't connect.
log a ticket in a few minutes for anthony