Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Distributed Storage I/O Speed
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Distributed Storage I/O Speed

randvegetarandvegeta Member, Host Rep

I'm testing some highly distributed storage clustering with ISCSI, NFS and S3 support. I'm using only older hardware that is really of no more use in production, such as Core2Duo servers with <4GB RAM, Gbit network cards, and servers distributed throughout the DC in multiple racks, with inter-rack connectivity limited to 1Gbit. No caching, either!

So a small (5 node) test cluster was setup, and I've decided to test Xen HVM performance using this storage cluster over ISCSI. The performance is actually surprisingly good all things considered, but still of course no where near what you may expect from local storage.

The write speed averages at about 40MB/s, which is by no means fast, and you probably wouldn't want to put a DB on here. But it should be fast enough for cold/warm storage.

The cluster does offer a high degree of redundancy / reliability and with XenServer, HA is easily configured. Any use for a highly available, but 'low' performance server over a regular VPS with no HA?

I already have a high performance cluster, so I'm looking to build this new cluster ONLY with what is already in place, sticking with GBit NICs and no caching. Wondering if it is worth pursing or not.

Comments

  • WSSWSS Member

    Explain your network hardware. RealTek garbage (of the time) will nowhere near provide the same throughput of the Intel offerings.

  • randvegetarandvegeta Member, Host Rep

    WSS said: Explain your network hardware. RealTek garbage (of the time) will nowhere near provide the same throughput of the Intel offerings.

    The old machines have onboard NICs of various brands (some Realtek) but we have quite a few machines with PCIe Intel cards too. Ones that can be bonded if we stick more cards in.

  • randvegeta said: I'm testing some highly distributed storage clustering with ISCSI, NFS and S3 support

    That sounds like GlusterFS. Use Ceph, kick NFS entirely and drop the ISCSI for Ceph volumes. This also has native Proxmox support then which iSCSI is only limited, especially security wise.

    randvegeta said: The cluster does offer a high degree of redundancy / reliability and with XenServer, HA is easily configured

    Yea, they all say that, until you actually pull a node offline...

    randvegeta said: The write speed averages at about 40MB/s, which is by no means fast, and you probably wouldn't want to put a DB on here. But it should be fast enough for cold/warm storage.

    You are ultimately limited by the Gbit NIC in each server and by the Gbit NIC in the source you will either end up writing at full 1G to a cache node which will use the same 1G link to spread it out or you end up writing N times from each node on this Gbit link.

    Since a usual single HDD can do only 3/4th Gbit it is not the obvious limit but by topology it ends up to be on some side.

    Thus, yes you can keep the Gbit to the storage but the VM nodes need (this is not negotiable in any way, or 5 users will kill the backend entirely by just installing OS) 10G to a switch that runs ideally 1G to each rack you use for storage (better 10G but yea, the costs); then a switch there for the storage nodes.

    Overall ceph will run fine on near anything and more nodes with just few HDDs are generally the recommended config but 1G will limit at some point.

  • ClouviderClouvider Member, Patron Provider

    William said: Yea, they all say that, until you actually pull a node offline...

    Yeah, you should have seen my face when we first deployed our Cloud platform with HA storage and I actually did the 'plug test' during commissioning of the platform before we started selling services. That was a mess. I was so unhappy : (

    Thanked by 1vimalware
  • RazzaRazza Member
    edited July 2017

    Clouvider said:
    Yeah, you should have seen my face when we first deployed our Cloud platform with HA storage and I actually did the 'plug test' during commissioning of the platform before we started selling services. That was a mess. I was so unhappy : (

    It's a good thing you did the plug test before you put the setup into production or it's could of been a lot more of a mess.

    I take it work's fine nowadays.

    Thanked by 1iRepko
  • ClouviderClouvider Member, Patron Provider
    edited July 2017

    @Razza said:

    Clouvider said:
    Yeah, you should have seen my face when we first deployed our Cloud platform with HA storage and I actually did the 'plug test' during commissioning of the platform before we started selling services. That was a mess. I was so unhappy : (

    >

    It's a good thing you did the plug test before you put the setup into production or it's could of been a lot more of a mess.

    I take it work's fine nowadays.

    Yeah, I always test all HA features actually work before going into production.

    It works now very well, the road to this point took quite a bit of time though and delayed GA :).

    Thanked by 1Razza
  • randvegetarandvegeta Member, Host Rep

    William said: That sounds like GlusterFS. Use Ceph, kick NFS entirely and drop the ISCSI for Ceph volumes. This also has native Proxmox support then which iSCSI is only limited, especially security wise.

    Actually it's Virtuzzo (7) Storage. I'm testing out their new 'erasure encoding' as opposed to the 'replica' version I've been using for the past few years. Doing 'replica' based storage is great for overall performance and reliability but prohibitive expensive to do for any kind of 'budget' service. The 'erasure encoding' system is much more like RAID, so the overheads are much lower (as low as 18%).

    I've not tested pulling the plug on this yet, and I indeed I am aware that 1G NICs will be the biggest bottleneck. But I think there should still be some good use for low powered storage VPS.

    The lower overhead on the storage means that for each 100GB of usable space, we would only need 118GB physical space, which is very attractvie considering our current requirement of 300GB physical per 100GB usable. So we can put all of our old servers with old drives to good use, and spin up nodes as and when needed should any other node go offline.

    I'm really not keen on upgrading the entire network to 10G as that raises the cost quite a lot, but I could theoretically do NIC teaming, upto 4G. Put 1-2 nodes in each rack and that should be a pretty reliable storage cluster. The trick would be keeping the read/write operations to less than the link speed..

    I'll be doing a pull the plug test soon. So far running just a few VMs on a 2 node XEN cluster, it works well. I'll have to fill up the nodes, and have those nodes all active, doing something, and then pull the plug at various locations to see how well it all holds up at each stage.

    The question is.. 40MB/s too slow for any practical use or as a cold/warm storage is it still acceptable?

  • @randvegeta said:
    The question is.. 40MB/s too slow for any practical use or as a cold/warm storage is it still acceptable?

    Main problems I see with it is the delays (scaleway network-ssd was significantly slower than a real one) and having to share those 40mbyte/s over multiple users.

  • pbgbenpbgben Member, Host Rep

    I too have just done a bunch of testing with distributed storage, have had to choose local as speed is key :/

  • randvegetarandvegeta Member, Host Rep

    teamacc said: was significantly slower than a real one) and having to share those 40mbyte/s over multiple users.

    When I was testing raw read/write speed, I was getting full line rate. The 40Mbyte/s was average speed seen in the VMs themselves. The actual speeds varied from as slow as 27MB/s to 90MB/s. But when testing directly on the cluster, I was getting 100-120MB/s, which is basically the full speed of the NIC.

    Now I think it would make sense no matter what to have 2x NIC on each of the storage nodes. 1 NIC for the actual storage network, and the other NIC for other servers to connect to to access the cluster. So the replication can be done at full line rate and access can be full line rate too.

    If running VMs on the cluster is impractical, perhaps it would be most useful for simple FTP storage...

  • randvegetarandvegeta Member, Host Rep
    edited July 2017

    pbgben said: have had to choose local as speed is key :/

    I've no issue getting good performance out of distributed storage. But my existing cluster has quite high cost. Data is replicated 3 x and we use 10G switch with dual 10G NICs, Enterprise HDDs and Enterprise SSDs for cache. The result is actually pretty good, but it isn't cheap. And so we are now experimenting with purely commodity or surplus hardware.

  • ClouviderClouvider Member, Patron Provider

    Storpool is great. They will design it to your requested rated Mbps or IOPS.

Sign In or Register to comment.