New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
New KVM hypervisor using a lot of ram?
Hi,
So I just got a new KVM hypervisor to host my own virtual machines, and for some reason, ~15GB of RAM has been consumed out of nowhere. I have no VMs running either.
[root@lax-kvm1 ~]# free -m
total used free shared buffers cached
Mem: 32101 15589 16511 0 148 472
-/+ buffers/cache: **14969** **17132**
Swap: 8191 2 8189
Looking at the processes show no heavy memory usage.
[root@lax-kvm1 ~]# ps aux --sort -vsz | less USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20806 0.1 0.0 894284 26660 ? Sl 19:39 0:00 libvirtd --daemon root 1322 0.0 0.0 249468 4096 ? Sl 2012 0:21 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 solusvm 20725 0.0 0.0 212796 5492 ? S 19:38 0:00 /usr/bin/php-cgi solusvm 20727 0.0 0.0 212788 5592 ? S 19:38 0:00 /usr/bin/php-cgi solusvm 20707 0.0 0.0 211976 6636 ? Ss 19:38 0:00 /usr/bin/php-cgi solusvm 20721 0.0 0.0 211976 3068 ? S 19:38 0:00 /usr/bin/php-cgi solusvm 20722 0.0 0.0 211976 6640 ? Ss 19:38 0:00 /usr/bin/php-cgi solusvm 20723 0.0 0.0 211976 3072 ? S 19:38 0:00 /usr/bin/php-cgi solusvm 20724 0.0 0.0 211976 6640 ? Ss 19:38 0:00 /usr/bin/php-cgi solusvm 20726 0.0 0.0 211976 6640 ? Ss 19:38 0:00 /usr/bin/php-cgi
Any ideas?
Comments
You highlighted buffers/cache, i.e. that Linux ate your RAM. It's not "used" for anything.
My assumption was that the free value under -/+ buffers/cache denote the actual "free" memory usage?
14969 is used
17132 is free
So Linux ate 14969 of your RAM and this is the 15GB you are talking about?
Yes; this server has been running for 15 days with no VMs currently running... libvirtd and SolusVM running, but it doesn't explain where the memory went...
It's used in buffers/cache. It means it's used for caching. It'll be freed up when you need it.
-/+ buffers/cache already takes that into account, as in minus buffers/cache in the used column, and plus buffers/cache in the free column. Read about it here: http://thoughtsbyclayg.blogspot.com/2008/09/display-free-memory-on-linux-ubuntu.html
Also, I rebooted the server and the memory usage is back done to 490MB now. I'm going to be monitoring /proc/meminfo for a few days to see what's happening.
If anyone have any other ideas, please feel free to let me know!
http://www.linuxatemyram.com/
Let's format this a little closer to what we're all used to.
total used free shared buffers cached Mem: 32101 15589 16511 0 148 472 -/+: 14969 17132 Swap: 8191 2 8189If there are no VMs running this doesn't seem right. I'm seeing roughly 17GB of roughly 32GB of memory available. I'm only seeing 472MB cached.
What happens if you do
echo 3 > /proc/sys/vm/drop_cachesAnd then try a free -m again ?
If you read the "How do I see how much free ram I really have?" section, you know that that's not the problem I described in the OP. I had only 17 GB of free memory (including caches and buffers) when no heavy-in-ram processes were running on the server. I'm still trying to figure out why
That's exactly what I was seeing, which is why I thought it was weird... and prompted me to post this question here.
I already rebooted the server, as I had to update the kernel. I'll definitely try that if it happens again.
The server now, with 2 VM's running (4GB + 1 GB VMs):
[root@lax-kvm1 ~]# free -m total used free shared buffers cached Mem: 32101 2406 29694 0 1 18 -/+ buffers/cache: 2386 29715 Swap: 8191 4 8187which looks normal with 2386MB used and 29.7GB free.
Thanks everyone for the input!
If you have traffic shaping and IP/arp protection on disable both and reboot the node. They don't work with RHEL6 and SolusVM.
Is that what's causing this?
90% sure it will be. Give it a try!
In my experience the symptons of this involve your memory increasing rather rapidly, like 512MB an hour etc. Also, what guest cache method are you using?
I'm experiencing around 512MB of RAM eaten up every 2 hours or so. How would I go into disabling both traffic shaping and IP/arp protection? I'm not using traffic shaping on neither of my VMs.
I currently have default set as the guest cache method, which I assume I'm using writethrough?
been a bug in solusvm since v1.3.02 iirc, it actually caused Xen nodes to hang completely, KVM was hardly supported if at all back then so I guess its the old bug showing its face again, I have had both switched off and replaced with my own systems ever since as 3 claimed 'fixes' later the issue was still present.
This isn't really a solution to this issue, but I feel compelled to say it. If you're hosting KVM for yourself, Proxmox really does a great job of it. I don't like it much for clients, but for myself I use it locally and on servers. Just one less cost, and if Solus is going to drop the convenience factor for you on this, +1 in favor of a free solution that works.
Does Proxmox support CentOS? Otherwise, I'll have to reload the OS to Debian.
I believe you'd have to install it directly or on top of a Debian install. If the annoyance builds, keep that option in your back pocket at least
It certainly was the dentries/inodes that was causing this that are not accounted for in free's buffer/cache.
From the previous:
Slab: 1225008 kB
SReclaimable: 1181252 kB
SUnreclaim: 43756 kB
to:
Slab: 54828 kB
SReclaimable: 11104 kB
SUnreclaim: 43724 kB
There's two ways to install Proxmox
Either using A: Their ISOs
or B: Installing Debian and then installing it using their instructions here: http://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Squeeze
Note: Proxmox does not support RHEL