New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Aggregate Nginx access logs and combine them
I have multiple VPS servers running Nginx. All of them proxy cache content from the same origin so the file structure is the same everywhere.
My end goal is to get the total number of hits from all servers for the last 30 days for all files under a directory.
Something like /files/project/* = 2000hits
What are your recommendations on the whole system?
I am looking into https://logentries.com to stream there my access logs, but not sure yet if I can do everything I need with that.
What do you think?
Comments
Sumologic has probably one of the better filtering / aggregation short of rolling your own.
I would set up all of the nginx servers to log to a syslog server, and configure the syslog server with a MySQL or Postgres backend. That way, you can do all kinds of analytics on it.
To combine Nginx logs, use "logresolvemerge.pl" from Awstats. Then you can do analysis. Refer to:
http://awstats.sourceforge.net/docs/awstats_tools.html
(Scroll down to find description of that tool)
On Ubuntu it is located at /usr/share/awstats/tools/logresolvemerge.pl. To install Awstats just run
apt-get install awstats.@nickm I see here that only Nginx Plus supports remote syslog http://nginx.com/products/
@hwdsl2 For this I would need to somehow sync all access logs to a single server, right?
@tchen SUmologic is an enterprise product. Once the free trial is over I am afraid how much they will ask.
We use Loggly (www.loggly.com) this will allow you to aggregate/search your syslog data
You can shove in 500MB per day of logs when free.
I tried loggly too but so far have no idea how to aggregate it in a way to get the number of hits per day. logentries.com has a COUNT function that I think I can use, waiting for support on that.
@tchen 7 days of retention is not good enough for me.
I am also looking into Piwik https://github.com/piwik/piwik/tree/master/misc/log-analytics
How about Logstash and Kabana
Or something like OpenWebAnalytics
Yes that is correct. You can schedule a hourly rsync via cron, for example.
Try splunk storm, the most generous offer for free tier.