Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Anyone using web server log analyzers instead of Piwik/Google Analytics?
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Anyone using web server log analyzers instead of Piwik/Google Analytics?

rm_rm_ Member
edited February 2017 in General

If so, what are some good/modern ones that you use?

Two most widely known ones seem to be http://www.awstats.org/ and http://www.webalizer.org/

Even doesn't have to be modern, really, e.g. this output generated by software from almost 20 years ago http://www.webalizer.org/sample/usage_199905.html
is way more than I personally could ever want from Piwik. And this is simple, small and doesn't require to lug MySQL around with it all the time (would love to get rid of MySQL entirely). As a bonus, this kind of analytics can't be blocked client-side with ad/tracker blockers.

Do you think log analyzers can replace full blown web-based trackers? What information do you actually use from Piwik/GA that cannot be obtained from web server logs?

Thanked by 3Blazing Yura varwww

Comments

  • Used webalyzer and home cooked scripts back in the day. Never felt a need for anything like piwik. My manager insisted on putting chartbeat in our site for a while and watched the monitoring page obsessively, but it was like seeing a TV addict. After the fact log analysis was always good enough for me, and let us pull out stuff that piwik etc never could (by writing custom scripts),

    Thanked by 1doghouch

    #lexit spread the word.

  • rm_ said: Do you think log analyzers can replace full blown web-based trackers?

    Pretty much everyone made do with them before Google turned log file analyser Urchin into Google Analytics (which had to be JS-based because they didn't have access to your logs).

    Sure there's plenty of information that can't be gathered that way, but it depends whether you care.

    As a bonus, this kind of analytics can't be blocked client-side with ad/tracker blockers.

    For a general audience, this is basically irrelevant. For some kinds of specific audience you might even need to care about people with JS disabled.

  • rm_ said: What information do you actually use from Piwik/GA that cannot be obtained from web server logs?

    try getting these metrics from web server logs :)

    • average page load time
    • average document interactive time
    • average server connection time
    • average server response time
    • average mobile page load time
    • average tablet page load time

    and being able to break down these times by

    • by continent
    • by city
    • by specific page url
    • by browser
    * Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
    * Centmin Mod LEMP Stack Quick Install Guide
  • I use Weblog Expert: http://www.weblogexpert.com/ for my logfiles.

    Thanked by 2Yura flatland_spider

    Kenneth Myhre
    WindowsTemplate.com - free Windows templates for OVH/Hetzner/Kimsufi/Online.net

    Powered by Hetzner.com, backed up by OVH, Kimsufi and VULTR.com

  • because analytics and metrica that im using right now doesn't count bot, i can see user session and how many percent my ads blocked by visitor.

    so i can do data mining for my website purpose. :)

  • So do I. Nice tool and very much like that I'm not bound to use a web interface (which always opens yet another security hole). Plus it's light.

    Thanked by 1sin

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • @bsdguy said:

    So do I. Nice tool and very much like that I'm not bound to use a web interface (which always opens yet another security hole). Plus it's light.

    +1 for goaccess, I don't need detailed tracking and mainly just want to know some stats like location of the visitors, time of day with the most traffic and what amount of traffic per vhost I have.

    It now also has a mode that allows realtime stats in a browser, I haven't tried it though. For the most part the CLI and HTML reports are all I ever use.

    Thanked by 1sin
  • Anybody used GoAccess on cPanel? It looked good until I realized I couldn't install it on a remote server and have it act like Piwik/GA

    How to clean up a questionable reputation: throw the kids some BF/CM offers.

  • @doughmanes said:
    Anybody used GoAccess on cPanel? It looked good until I realized I couldn't install it on a remote server and have it act like Piwik/GA

    Being that it builds through a shell and requires ncurses, you will likely need shell access. I can see someone possibly building it statically for use in specific scenarios, but you've still got libc issues/et al. I might look into this later, but I don't deal with cPanel on a daily basis anymore (thank god).

    I won't be back until @bsdguy is released.

  • datanoisedatanoise Member
    edited February 2017

    Seems fantastic, thanks guys for sharing. Seems like it even can generate static html files like webalizer / awstats. Do you know which one would be the more efficient (memory / cpu) for static web pages generation?

    don't buy what you don't need: you'll save money and will end up able to grab a quality VPS when really needed.

  • @myhken said:
    I use Weblog Expert: http://www.weblogexpert.com/ for my logfiles.

    Weblog Expert is a nice package. It's Windows only, so MacOS or Linux users are out.

  • flatland_spider said: It's Windows only, so MacOS or Linux users are out.

    Yea, thats true. I have it running on a server at home, there I download all log files each night, then Weblog Expert do it's job, and then the data is transferred to one of my production servers, there I and my customers can look at it.
    But if OP is looking for some software to use at home/on a RDC server for him self, it's just perfect.

    Kenneth Myhre
    WindowsTemplate.com - free Windows templates for OVH/Hetzner/Kimsufi/Online.net

    Powered by Hetzner.com, backed up by OVH, Kimsufi and VULTR.com

  • Is it possible to hide Google Analytics from adblockers by hosting GA code yourself or something like that?

  • mfsmfs Banned, Member

    @Yura: https://github.com/igrigorik/ga-beacon (but it has severe limitations; you may as well use your server logs, if you can get them/if you're the one hosting it)

    Thanked by 1Yura

    I've left LET since February 2019, account made inactive on request.

  • YuraYura Member
    edited February 2017

    @mfs said:
    @Yura: https://github.com/igrigorik/ga-beacon (but it has severe limitations; you may as well use your server logs, if you can get them/if you're the one hosting it)

    Thanks for the link, it would work in a pinch, but tracking pixel is suboptimal due to severe limitations as you said.

    Piwik is simply not as powerful or fast as GA. Of course it is theoretically possible to extract some data from logs, analyze it, abstract it away and display on a good presentation level anything GA or other analytics service does. But that's exactly what GA does and they spent millions of dollars and hundreds of man-hours on exactly that. Logs are good for admin, marketing needs analytics. These are different realms.

    I think I read about self-hosted GA somewhere, but can't remember now. That would be good solution, especially if possible to alter it im such a way as to make it unrecognizable for adblockers.

  • JustAMacUserJustAMacUser Member
    edited February 2017

    I've used AWStats. Charts are ugly, but it works.

    If I need more client data (the stuff JavaScript in a browser can provide), I go with Piwik. It's nice, albeit heavy.

  • I have replaced GA with goaccess and am happy with it. I don't really need those advanced features like conversion or page heatmap so goaccess is enough for me.

  • One of the main reasons I started using Piwik was due to sites being hosted across multiple servers in high availability setups.

    It seemed easier than doing the global mashup of local log files needed to analyze the data from all servers.

    Or so I thought at the time ... maintaining Piwik (which I also have running in high availability) in this setup can still be a PITA when there is an update.

    I think it might depend on who the data is for:

    • if just for you, a log analyzer fits the bill just fine
    • if it needs to be shared with management or customers, Piwik can be quite sexy and offer useful conversion info
    Thanked by 1BeardyUnixGuy
  • ELK can take / store a lot of your data for you to make variable based dashboards. It's really handy once you get the hang of it.

    As for the actual load time testing I'd consider looking at automated web testing type software.

  • @Yura said:
    Is it possible to hide Google Analytics from adblockers by hosting GA code yourself or something like that?

    You can use their php SDK to do that. however, as far as i know, u will have to code it (send data). I don't think anyone has created a library where u can just include a php or use a namespace that will auto do it.

    Thanked by 1Yura
  • rm_rm_ Member
    edited February 2017

    Just found that Piwik can collect stats from the server logs too: https://piwik.org/log-analytics/
    No need to use the always-on Javascript tracking.
    Could run Piwik offline on some local VM, and feed it new logs downloaded from the web server from time to time. Take a look at the stats, then close/shutdown it. No need to keep a public VM with Piwik & MySQL running at all times. Maybe this will be the solution for me.

  • WilliamWilliam Member, Provider

    rm_ said: Just found that Piwik can collect stats from the server logs too: https://piwik.org/log-analytics/

    It does that actually pretty good, it's what i use primarily - just cronjob sync the logs to a central box where piwik crons run.

    Not idiot simple to configure (one cronjob per website etc.) but an ok solution.

  • Thanks @rm_! I didn't realize Piwik could offline log analytics either.

    You can send your dogecoin "thanks" to: DNhnwKWR5vm8ddbWPpWfrpGR8atXH5ZFeP

  • Awesome, thank @rm_ for the tip about Piwik log collection - I'm going to try that out!

  • zilch said: here is (goaccess)

    Thanks zilch :) I'm quite liking goaccess for weblog analysis. The old stalwart Webalizer seems dead as project, and awstats is heavy and slow when run against multiple domain logs.

    I've wrote a bash script that rsyncs logs, massages then a bit with cat & sed, then uses goaccess to produce a last-7-days report plus a previous-month report. It walks though 25 domains with ridiculously low CPU and RAM usage.

    Thanked by 1inthecloudblog
Sign In or Register to comment.