Tracking bandwidth per virtual host - nginx

Jonchun

Hey,

Question is pretty self explanatory. Looking for a log parser or something similar that will let me view bandwidth per virtual host on nginx.

What I've looked at: Munin, Luameter
Don't like either because Munin is a little too full-featured for my needs, whereas luameter is paid. If I can't find anything else, I will go with Luameter.

What I want (or if someone can guide me in the right direction to coding this myself)

Write a bash script that uses logtail/awk or something to do the following:
./viewbandwidthbyvh.sh timestamp1 timestamp2 (default withotu timestamp should give total all-time values)

and it will give me the output of all virtual hosts and the bandwidth used during that time period.

I am looking for a free preexisting solution if possible, but am willing to try scripting this myself if anyone provides pointers.

Thanks!
Jonchun

Nomad

What about old classics such as Webalizer/AWStats

eva2000

@Jonchun check out nginx-module-vts which I have integrated into my Centmin Mod LEMP stack installer https://centminmod.com/nginx.html#vhoststats

stats don't persist on nginx restart IIRC - haven't followed latest developments closely as I should.

just waiting on Nginx TCP stream stats support https://github.com/vozlt/nginx-module-vts/issues/22

JustAMacUser

Things like AWStats will certainly do this, including pseudo-pretty graphs.

You can make shift something in bash. I found this article online that talks about optimizing PHP-FPM, but in it you can see how the author parses his web logs to get average transfer, etc. It's a starting point.

eva2000

seems https://github.com/vozlt/nginx-module-vts now has json format too for stats output, so you can probably take that output and feed it into other web apps/charting tools

Jonchun

@JustAMacUser said:
Things like AWStats will certainly do this, including pseudo-pretty graphs.

You can make shift something in bash. I found this article online that talks about optimizing PHP-FPM, but in it you can see how the author parses his web logs to get average transfer, etc. It's a starting point.

Will take a read. thanks!

@eva2000 said:
seems https://github.com/vozlt/nginx-module-vts now has json format too for stats output, so you can probably take that output and feed it into other web apps/charting tools

Saw this, but a little too full-featured for my needs. Would prefer something simpler with little to no configuration.

SplitIce

With OpenResty / the lua module you can use log_by_lua_* to build your own system easy enough (as little as maybe 30 lines of code) if you desire.

eva2000

Jonchun said: Saw this, but a little too full-featured for my needs. Would prefer something simpler with little to no configuration.

yeah not aware of anything else that does it per vhost though

SplitIce said: With OpenResty / the lua module you can use log_by_lua_* to build your own system easy enough (as little as maybe 30 lines of code) if you desire.

nice nginx lua !

centmin mod's nginx 1.11 branch by default also has optional lua module support too and i use some nginx lua for centminmod.com site itself. Need to look at log_by_lua_*

edit: log_by_lua_block is the new way https://github.com/openresty/lua-nginx-module#log_by_lua

WARNING Since the v0.9.17 release, use of this directive is discouraged; use the new log_by_lua_block directive instead.

Jonchun

@SplitIce said:
With OpenResty / the lua module you can use log_by_lua_* to build your own system easy enough (as little as maybe 30 lines of code) if you desire.

Bookmarked. This definitely looks interesting. Will have to learn some lua because 30 lines of code sounds perfect to me.

SplitIce

@Jonchun

I could probably open source what we use, but I'd have to extract the dependencies from the system (x4b.timer, x4b.syslog)

local M = {}
local timer = require "x4b.timer"
local syslog = require "x4b.syslog"

local log_facility = syslog.***
local log_severity = syslog.***
local log_tag = "usage"
local buffer = {}

local function get_length()
    return tonumber(ngx.var.bytes_sent), tonumber(ngx.var.request_length)
end

function M.bill(server_addr)
    local bytes_in, bytes_out = get_length()

    local b = buffer[server_addr]
    if b == nil then
        b = {bytes_in = 0, bytes_out = 0}
        buffer[server_addr] = b
    end

    b.bytes_in = b.bytes_in + bytes_in
    b.bytes_out = b.bytes_out + bytes_out
end

local function dotimer_push()
    local local_buffer = buffer
    buffer = {}

    local sum = 0
    for server_addr, traffic in pairs(local_buffer) do
        syslog.syslog(log_facility, log_severity, log_tag, "[".. server_addr .."] "..traffic.bytes_in.." "..traffic.bytes_out)

        sum = sum + traffic.bytes_in + traffic.bytes_out
    end
end

function M.init()
    timer.repeatat(300, dotimer_push, "accounting")
end

return M

call accounting.bill(ngx.var.server_addr or http_host or whatever) in log_by_lua

we push to syslog in the background so .init in init_by_worker_lua_*

Jonchun

@SplitIce will be looking into this when i get some time. Thanks for the snippet!