All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How could this traffic spike have been caused?
OK, strange one here. I got a new VPS a couple of days ago from a recommended provider on LEB. On Friday night I went out and came back around 2AM to find the server had been suspended due to exceeding the traffic limit. Strange, I thought, as I'd only moved a very low traffic test blog onto it and had been out all night so wasn't using up any bandwidth via SSH or whatever.
Here's a graphic of the traffic spike:
http://i.imgur.com/gPfO5.png
So it looks like transfer was over 200mb/s (edit:MB/s) and the entire 250+GB usage allowance was used up in a very short space of time.
I asked support what could have caused it and they said it must have been (a) me running test files or (b) a DDoS attack. Ignoring for the time being that I know (a) not to be true and I think (b) extremely unlikely, would it even be possible to get such high speed transfer via either?
My other thought was maybe the server was hacked, but I wonder what would a hacker do to use all that data?
So ... anyone have any theories on how this could have been caused? If you wanted to make such a spike happen, how could you?
Thanks!
Leon
Comments
I have seen a single customer exceed 1Gbps outbound. 200Mbps is more than possible.
Not saying it's not possible, I'm just interested to know how.
Is that a solusvm graph? If so, I thought those were in megabytes/second not megabits/second
Yes it's SolusVM. Probably is Megabytes - 30 mins, 284 GB used up so that would be around 160 megabytes a second on average, which seems about right looking at that graph. (Should I have written MB/s in capitals?)
Just to point out, LEA has stated at least oncce that none of the providers listed are "recommended providers". He just lists offers that comes his way and allows us to pull them apart or make comments or what have you.
Having said that, I'm very surprised that the provider hasn't stated specifically what caused the issue. (Unless it's one of those providers that are just trying to cover their butts with a couple new clients that they quickly ToS just to get their money in an attempt to cover their bills. We've had a number of those.)
If it's a test blog, one thing that comes to mind is that an image was hotlinked and used by a main stream site. I do shared hosting and we've had that happen a couple of times where a cough low life ABC news station has used a image from a client's site while still on their site and quickly run up a couple of gigs within minutes. One specific station did that a couple of times until we filed an arrest warrant for their station manager. (And that was after filing the FCC complaint.)
Having said that, the first thing i would be doing is pulling the logs to see what caused the issue. The provider should have done that already and told you what the issue was. Considering that they probably already wiped the account, the only real thing you can do is to go back to their sales thread and raised hell about their poor customer service and lack of an explanation. Doubt it will get you anything though.
Good luck with your next provider.
B is for bytes
b is for bites.
Big difference there.
I suppose that suspended != wiped
I'll admit that I was skimming. To be honest, after no mention of logs being viewed or referenced, it sounds fishy to me. Not sure from what side though. Could be a provider trying to cover a bill with a couple of ToS'es, a provider without a clue, the OP trying to cover their butt after being caught doing something that they shouldn't have or one provider trying to bad mouth another provider. (No offense to anyone here. Being honest.) edit: Or one of a hundred other things.
And at least in the shared hosting world, suspended does quite often equal wiped. Depends on the host, the circumstances, etc.
AFAIK, when you finish your bw on a vps is normal that the account is suspended until next month.
I think that d60eba wasn't speaking about a punitive suspend.
I did have a look at the nginx access log but it was only just under a meg in size and had around 2500 server hits in total from a handful of I.Ps. Basically no traffic. That's why this is so weird - I've gone through every log I can think of (nginx access, nginx error, messages) and I can find no trace of what caused the spike. The provider says they have no way of knowing either and that the SolusVM bandwidth counting is very accurate, so (pretty much) it must have been me downloading test files. Basically they don't believe me when I say "it wasn't me" which is understandable I suppose.
So .. hence this thread. I wanted to compile a list of things that could possibly have caused this. List so far:
Umm, err. I can't think of anything else but I'm by no means very experienced. So I thought you guys might be able to come up with some scenarios, no matter how outlandish. A fun game ;-)
Incidentally, the VPS is still not wiped and they even unsuspended it and gave me a couple of extra GBs so I could log in and have a look at the logs. I moved the blog back to a different provider. No problems there.
@drmike thanks for the suggestions! I should add this one to the list -
(5) I was doing something I shouldn't have. (I'd be interested to hear what "naughty" users get up to that could cause data transfer like this)
I deliberately haven't mentioned what provider it is. I don't really blame them. They seem to genuinely be reacting to what their system is telling them. But that just adds to the mystery.
@d60eba: I think that you should consider, in any case, the option 3 (that mean: reinstall os).
However:
2) Do you have a location that disable access log in your nginx config files?
Like this:
thanks @Steve81. The nginx log isn't disabled - it shows a steady trickle of traffic. The "hacked server" angle is interesting, but I'm asking myself - "what were they possibly doing?" I've had servers hacked before and they've used it to store warez and the traces were obvious. But this server is empty - just a bootstrapped LEB with a Wordpress blog. No suspicious files. So what was this hacker doing?
The only clue I have to work with is the data usage pattern and so far I can't find anything that fits it...
I have seen solusvm's graphs show me as transferring at 50 petabytes a second, so it's probably another solusvm glitch.
Have your host check the port traffic on their server/switch/router to confirm they were actually pushing 1.6Gbps.
@d60eba: the line that i show to you didn't disable log completely, but only on some kind of files.
Well, excluding a bad provider or a bug in solus, remain the dDoS option. Maybe your ip was a target before you got it.
This isn;t wordpress, is it? I see some sort of image hack just came down within the last few days. The pattern doesnt fit as this looks more like a one time spike but I did want to mention it.
I'm going home.
Thanks for the comments guys, very helpful. A couple of questions ...
@drmike, you say
I'm curious, as a provider if you actively wanted to create such a spike on a client's account, how could you go about doing it? Or is there a "mistake" you could make that would cause this?
@Steve81
Do you think a DDos could generate those transfer speeds without targeting HTTP (as the nginx logs show nothing) and without overloading the CPU on this LEB?
Many thanks!
@d60eba: Yes, AFAIK it could without targetting HTTP (for example a distribuited reflection attack/udp flood/amplification attack). About the cpu overload: I see, in the image that you've posted, a spike in the load at the same time of the bw spike.
I'd just log in as the client, pull down the last Harry Potter, and say "Lookie at this!"
The "Could be a provider without a clue" comment was more about the poster not getting anything definite from the provider saying what had occurred. If I ToS a client, they know why.
I've also had server hacked before. The first thing they did is to install some perl scripts to DoS some Russian site, pushing around 100Mbps continuously...
You might want to check whether there is any suspicious process. Check whether you have strange files in world-writeable directories (
/tmp,/var/tmp,/dev/shmfor example).It could have also been an unsecure email relay as well. Maybe an older version of exim that was left open?
Could really be one of a hundred things.
It would be nice if the provider stepped up and explained what did occur. Learning and teaching opportunity here.
Assuming you can still access it, give it a scan with rkhunter etc, see if there's anything stands out.
Smells like a bug though, sustained 160MB/sec outbound? That's ~1.5gbit/sec. I've never seen a VPS with an uplink greater than 1gbit, and even then you'd have a hard time maxing that out due to drive and network IO.
@drmike - could you pull down the latest Harry Potter at 160MB/s though? (Good suggestion of getting the provider involved. I wanted to go to the head honcho and bypass the support guy. I think I have enough suggestions here now to do that.)
@Gary - My confusion with the whole MBs/mbs thing strikes again, so thanks for putting me on track here. The port is advertised as 1Gbps. A web conversion calculator tells me that equals 125.00 MB/s. So really, there's no way I could have gotten the transfer speeds shown on the graph?
Sure, pick up some usenet accounts
One thing for sure, looking at the graph, is that the download/upload was not done through just one (or few) connections or probably any normal http connecting and downloading, but it was done through a high priority application/process, hence you can see CPU graph when downloading started it jumped 4 times as much as the highest previous CPU usage.
However the RAM went down to 0MB!!!! Which makes the previous assumption not very likely (that, someone hacked and ran an abusive shell/script taking a lot of CPU, and killed all other processes).
I guess among the possibilities you made most likely are:
1) Someone hacked your VPS and ran something abusive that caused CPU usage spike as well, and also killed all other process! Therefore RAM usage dropped suddenly to 0 MB. And it did some extensive abuse to BW and killed your whole allocated BW!
2) There was a problem in either SolusVM/Node/Main node, and that is actually a very possible assumption. If a node is having troubles and mostly in hardware, sometimes all RAM/BW/CPU data and bandwidth display in the VPS panel are effected. I had this once in HyperVM and was fixed when the node problem was repaired through datacenter. However, that caused my graphs to be similar to ONLY your RAM graph! ===> Meaning RAM/CPU/BW all at 0 usage all the time, I had no "false spikes" I mean if that was what you got. Probably you can ask them to check the total node status CPU/RAM/BW graphs during exact same period that had your "spikes" and check their "total node graphs" against it. Maybe they can discover/notice something.
I didn't read all posts yet so this could've been suggested already, anyway I hope that helps!
243 megabytes = 1944 megabits
Unless your provider has a 2Gbps link on their server for some strange reason - the bandwidth usage isn't very accurate.
1G up 1G down FD
And if they tells you SolusVM is the most accurate ...etc
Just tell them IT'S the buggiest control panel in that matter at many times, if you refresh your page right after a VPS reboot, at times you will see the whole node RAM usage displayed as your own VPS RAM.
So, you'd be better informing them that SolusVM bug I mentioned as a one good example. They have to check things from their side as well and from other clients side, and not depend solely on what they think "SolusVM proven accuracy"!
Just noticed this follow up.
Um, some of my associates reported to me that the last Harry Potter was available at some very high speeds at the time. And they in fact have to limit their assigned to rtorrent bandwidth so not use their whole monthly allocation and/or kick off their provider.
:whistle: