New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
NFS Storage Pooling/Clustering
Caveman122
Member
in General
So say I had a few identical storage servers (not physically close) and I would like to pool the resources on each server together and have another server see it as a pool of storage. The protocol used have to be as efficient as NFS (supports random r/w and non-blocking I/O) and somewhat flexible in future expansion of storage.
Is there any solution exist for what I described?
Comments
glsuterfs
Ceph?
Awesome! Any personal experience with it? How's fault tolerance for glsuterfs? (a member goes offline, network distribution etc.)
It seems both glusterfs and ceph are acquired by Red Hat. I will try glusterfs and its distributed volume first, seems like exactly what I need.
It's like doing raid1. Should be working fine but expect some small problems if you are not familiar with it.
Gluster is fine, but you really need predictable network conditions.
Are these servers all in the same datacentre? If not I wouldn't contemplate Gluster or Ceph. You'll end up with continual storage splits.
Same DC, solid network and uptime but no RAID on individual nodes, it's expected and ok to lose some files but not ok if the entire cluster is corrupted/lost due to 1 node. What's your thought on distributed glusterfs volume for this type of situation?
What is the latency between machines?
Gluster is /ok/ and you can get up replication to avoid dataloss. Last year I built a 16 node gluster cluster, performance was nothing special, simulated network issues caused disruption.
But for large scale distributed applications where everything is setup on a low/no latency network (read infiniband, etc) you get better reliability and for heavily distributed applications where you have enough nodes you can get better performance than to a single node but every application is different.
Its easy enough to setup so give it a go, but break it and learn how to put it back together before you trust it with data.
True, the ping is less than 0.5ms but not infinband, DC internal is capped at 100mbps. It's a small cluster with just a few nodes, I am interested in an easy way to expend capacity while keeping my applications sort of oblivion to the file system changes besides capacity.
0.5ms is ok - but 100Mbps port is going to be painful.
What is the server-to-server performance in Mbps?
Here's an interesting one -- https://github.com/rozofs/rozofs
Uses erasure coding instead of replication to provide fault tolerance without requiring (N)x of storage space.
Consistently 100Mbps sustained it seems. Thanks for the answers by the way.
Try it - maximum theoretical performance is 100Mbps (12.5MB/s) more likely going to be 8MB/s. You may need to tweak Gluster's timeouts and keep a close eye on local latency but for cheap storage it maybe good, but still think it will be cheaper to upgrade your server to add second disk.
how about Tahoe-LAFS? Would that be an option ? http://www.lowendguide.com/3/networking/how-to-set-up-your-own-distributed-redundant-and-encrypted-storage-grid-in-a-few-easy-steps-tahoe-lafs/