New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Netdata memory leak
Before I deep dive myself or goto Github I figured I would ask on the LowEndSupport desk. Netdata is pretty popular here so maybe someone here has experienced this.
Has anyone experienced a major kernel memory leak when running Netdata with the eBPF collector (with out of box defaults for all other settings)?
I have noticed gcc running regularly so I guess it's loading new eBPF bytecodes regularly. Possibly due to changes being detected? Perhaps leaking.
Comments
We did install it as test in few servers and it did cause some segfaults that resulted in whole server being rebooted
We stopped using it as it was nice to have and not critical ( we didnt investigate further)
this is the logs usually we see
Aso another issue we suffered from was this https://github.com/netdata/netdata/issues/8001
just noticed it was fixed
the segfault error looks like this
451755.058785] INFO: PLUGIN[proc][12872]: segfault at 7c ip 00000000004b01f8 sp 00007f5e29943bf0 error 4 in netdata[400000+109000]
@SplitIce Yep, we've seen the same thing, across many different servers -- certainly looks like a memory leak to me. We've worked around the problem by configuring netdata to automatically restart every two weeks. If we don't do that, then the memory footprint of netdata will grow and grow til the point where the server starts swapping.
I was looking into it as a Zabbix replacement down the line. Maybe not any more.
@BrianHarrison netdata process usage or kernel?
New version of netdata or old?
Netdata process usage and a recent version of netdata, maybe 4-5 months old.
I've tried Netdata and found it to be "not quite there". I too experienced a memory leak, albeit not as severe as what you all describe.
The reason I uninstalled, is I hated their plugin system. It was a chore, instead of being fun, to setup new monitor services.