Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Netdata memory leak
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Netdata memory leak

SplitIceSplitIce Member, Host Rep

Before I deep dive myself or goto Github I figured I would ask on the LowEndSupport desk. Netdata is pretty popular here so maybe someone here has experienced this.

Has anyone experienced a major kernel memory leak when running Netdata with the eBPF collector (with out of box defaults for all other settings)?

I have noticed gcc running regularly so I guess it's loading new eBPF bytecodes regularly. Possibly due to changes being detected? Perhaps leaking.

Comments

  • servarica_haniservarica_hani Member, Patron Provider

    We did install it as test in few servers and it did cause some segfaults that resulted in whole server being rebooted

    We stopped using it as it was nice to have and not critical ( we didnt investigate further)
    this is the logs usually we see

    Aso another issue we suffered from was this https://github.com/netdata/netdata/issues/8001

    just noticed it was fixed

    the segfault error looks like this
    451755.058785] INFO: PLUGIN[proc][12872]: segfault at 7c ip 00000000004b01f8 sp 00007f5e29943bf0 error 4 in netdata[400000+109000]

  • BrianHarrisonBrianHarrison Member, Patron Provider
    edited May 2020

    @SplitIce Yep, we've seen the same thing, across many different servers -- certainly looks like a memory leak to me. We've worked around the problem by configuring netdata to automatically restart every two weeks. If we don't do that, then the memory footprint of netdata will grow and grow til the point where the server starts swapping.

  • SplitIceSplitIce Member, Host Rep

    I was looking into it as a Zabbix replacement down the line. Maybe not any more.

    @BrianHarrison netdata process usage or kernel?

    New version of netdata or old?

  • BrianHarrisonBrianHarrison Member, Patron Provider

    @SplitIce said:
    I was looking into it as a Zabbix replacement down the line. Maybe not any more.

    @BrianHarrison netdata process usage or kernel?

    New version of netdata or old?

    Netdata process usage and a recent version of netdata, maybe 4-5 months old.

  • I've tried Netdata and found it to be "not quite there". I too experienced a memory leak, albeit not as severe as what you all describe.

    The reason I uninstalled, is I hated their plugin system. It was a chore, instead of being fun, to setup new monitor services.

Sign In or Register to comment.