OVH/SYS failover IP bridging

tehdan

Has anyone with an ovh/sys server managed to get VMs working on bridged failover IPs using a normal linux distribution (Ubuntu in my case) on the host? I need the host to run a standard distro (not proxmox).

I've registered a virtual mac, assigned it to the VM etc etc. Looked at all the documentation but traffic just won't flow. It sounds simple in theory, I've got everything bridged like the documentation says, set up static routes on the guests but the gateway refuses to reveal its MAC to ARP requests which I can see going out eth0. Is there some magic step I'm missing?

Any pointers more than gratefully received, really need to do some actual work on this machine tomorrow

William

sounds like you need arp_proxy enabled?

tehdan

@william thanks, I have tried that - also rp_filter=2 and 0 which a lot of people seem to claim works...

GoodHosting

Welcome to OVH.

Also, beware the claim in their marketing / website that they "We are the only Service Provider to support your Virtual Machines!" as they don't even know what KVM is, and even their "SWAT" team doesn't know what "Bridging" is and why you need it.

I was disgusted with the level of service we have been shown, but I am very happy with their network otherwise, as well as their hardware selection and what they do support.

---

Also worth noting, you can't use the Primary IP [ eth0 /32 ] as well as a vSwitch KVM Bridge [ eth1 /whatever ] simultaneously. Their routers will kill your server for broadcasting on the wrong nets. Also, bridging doesn't [currently] respect custom routing [routes-eth0] under CentOS 6.5 latest.

You're going to have to invent some pretty nasty ARP rules, back-filtration and proxying, just to get their network to play nice with you. You're going to have a huge problem if you want to use their IPv6 (stuck on eth0) with your normal failover IP ranges (stuck on eth1) due to the aforementioned bridging / arping hell.

Good luck!

tehdan

Thanks - the SYS servers don't have eth1 connected, so assume this is for their non-budget servers? I've got my VMs bridged to eth0, and I can see broadcast traffic on the VM but their router never speaks directly to the VM, even if I enable proxy arp and explicitly create static arp entries set to 'publish'.

Did know what I was getting into with OVH, but you'd think they'd at least write a simple document somewhere that explained their network weirdness such that someone who knows the difference between bridged and routed networking can actually set it up.

Profforg

GoodHosting said: Also, beware the claim in their marketing / website that they "We are the only Service Provider to support your Virtual Machines!" as they don't even know what KVM is, and even their "SWAT" team doesn't know what "Bridging" is and why you need it.

KVM is not panacea. VMWare is correct solution for their case, because they know how to work with it. OpenVZ is the only way for Linux-only cheap machines and it's cool, if you know what is cgroups.
They have correct business model and i am sure they'll provide better service than you ever can.

Rather than blame good provider, go to read manuals.

tehdan said: Did know what I was getting into with OVH, but you'd think they'd at least write a simple document somewhere that explained their network weirdness such that someone who knows the difference between bridged and routed networking can actually set it up.

Try to read French documentation. They usually provide much more detailed information in their native language.

GoodHosting

@Profforg said:
KVM is not panacea. VMWare is correct solution for their case, because they know how to work with it. OpenVZ is the only way for Linux-only cheap machines and it's cool, if you know what is cgroups. They have correct business model and i am sure they'll provide better service than you ever can.

Rather than blame good provider, go to read manuals.

My comment was not baseless, I've had servers with them for well over three years now, and have learned how to work "around" the staff to get the answers I need. OVH constantly gives me ranges that aren't even routed to my rack, ranges with IP addresses containing pre-existing firewall / mitigation rules, ranges on the wrong VLAN / vSwitch, .... lots of issues.

As per the documentation, the only documentation they have on bridging or KVM at all, as well as how to use their silly /32 IPv4 ranges is located on this rather old link:

http://help.ovh.co.uk/BridgeClient

However, their own staff do not recommend the use of this guide / manual anymore, as it is outdated and only works "sometimes". It was what I needed to get the /32 working in the first place, so I could download mod_bridging, then I did the bridging, and the /32 never worked again. [ Oh well, I don't need 1x IPv4 that badly. ]

Their marketing / website clearly states that they support "Virtual Machines". It does not say that they specifically only support "VMWare" , else I might agree with this bit of your statement [ as VMWare is what they use. ] They should update their site / advertisements if they intend to only support VMWare. I managed to get 30% off my services for life because of this marketing blunder on their part, and they still didn't bother correcting it [ so I guess they don't care that much. ]

Oh, and to top it all off; here's my bandwidth graph: two points if you can figure out what's wrong with it, OVH staff couldn't figure it out!

Hint: Those values are shown in bytes per second.

Shoaib_A

@GoodHosting KVM can be easily set up on OVH. I have been doing it for a few years for non-profit use & Phil from mycustomhosting has been doing it for his company.Only thing you need to understand is how bridging/networking works on OVH servers through their manuals & rest is easy just like on other hosts' networks.

If you want I can write a tutorial so that anyone who is having problems can benefit from it.

MCHPhil

@K2Bytes said:
GoodHosting KVM can be easily set up on OVH. I have been doing it for a few years for non-profit use & Phil from mycustomhosting has been doing it for his company.Only thing you need to understand is how bridging/networking works on OVH servers through their manuals & rest is easy just like on other hosts' networks.

If you want I can write a tutorial so that anyone who is having problems can benefit from it.

Could not agree more. MAC addresses are big. Make sure they match and you have a proper bridge setup.

OVH provides an unmanaged service so yes I expect them to tell me I'm on my own if I asked a question concerning software or etc.

GoodHosting

@K2Bytes said:

I'd be curious to see if I was missing anything obvious.

@MCHPhil said:

Here's the fun part about KVM bridging in a smart environment like OpenNebula: OpenNebula is smart enough to have the ability to tag each Virtual Machine's traffic with an individualized vLan identification [ QinQ ] . OVH's network does not support this.

OpenNebula falls back to MAC-based IP allocation when vLan identification and smart switching is not supported [ i.e.: when the host network does not support Vyetta, or does not have Open vSwitch listening, etcetera. ]

OVH's vSwitch will happily listen to any MAC address on any IP address, on any vLan [ a huge security issue I've brought up with SWAT, and they are currently looking into how to solve. ] However, their public network [ eth0 ] does not support this whatsoever.

So what happens when [for example] 00:02:4b:6a:ce:9f tries to speak for 2607:5300:60:2517::1 on the public network [ eth0 ] ? OVH's network will drop all incoming packets, but allow outgoing packets just fine [???] . This is also something that the SWAT is still looking into, as they are sure that it should either allow all or block all; not work half and half.

This would all easily be solved if:

• You could use your "failover" IPs on the public network [ eth0 ] -OR-
• You could use your Primary IPs on the "private" network [ eth1 ] -OR-
• The "private" network supported IPv6 whatsoever [ eth1 ] -OR-
• Any of OVH's network supported QinQ anywhere.

At least one of the four above qualifying factors [ or full Vyetta support ] can be found with most other hosts [ even ColoCrossing's network supports QinQ... somehow. ] I'm not sure why OVH's network isn't able to support one of these.

---

SWAT's responses on the matter so far are humorous:

What I could tell you though, is that our traffic counters are taking straight from the switch port your server[0] is connected to. They are taking in a 5min basis frequency, and is shown to the client in EDT time zone.

Unfortunately, it is not possible to use an IPv6 block on the vRack. We also do not have an ETA on when or if this feature will be available.

---

Nobody has commented on the broken bandwidth graph yet.

Shoaib_A

@MCHPhil said:
OVH provides an unmanaged service so yes I expect them to tell me I'm on my own if I asked a question concerning software or etc.

Their sales & tech support is handled by same guys with just basic knowledge only but I cannot blame them for it because they provide decent quality server with high specs with lower pricing as compared to competitors.You helped me when I needed help with setting up SolusVM & IPV6 properly on OVH's network there are lot of guys out there who do have problems & OVH support would do nothing except providing you with links to their guides.

MCHPhil

Sorry @Goodhosting I do not use the vRack. I assumed you are talking about vRack as you mention VLAN's as that would be the only way I know OVH to support this. If you'd like to send me your ifcfg-eth0 and ifcfg-br0 or likewise, interfaces if debian etc etc etc. I will compare things with my setup and provide any assistance I may :P PM me. Response time is not guaranteed but neither is OVH As we both know.

Likely the reason for the troubles is the classless routing involved with everything, they do this to avoid having to spare X IP's per block etc. It makes sense to me at least.

Also please note the OVH support staff are NOT the ones creating the scripts. As a programmer I can tell you this will cause a bit of an issue translating to bugreports.

Shoaib_A

From what I have heard the most popular cloud platforms like VMware & Parallels work very well with OVH, don't know what is wrong with OpenNebula though.

tehdan

K2Bytes said: If you want I can write a tutorial so that anyone who is having problems can benefit from it.

Please, please do! Extracts from a working config would be more than enough.

Shoaib_A

@tehdan said:
Please, please do! Extracts from a working config would be more than enough.

Alright will work on it very soon, may be tonight or tomorrow.

tehdan

Awesome, thanks.

I'm beginning to suspect something is broken on OVH's side. I resorted to assigning the FO IP's to the host, and NATing them through to VMs. This works fine, but only on IPs that have never had a virtual mac assigned. The ones I've tried to bridge, and removed the virtual mac from are all failing to route to the host, even after a few hours.

Any tips on getting your problem in front of someone competent at OVH?

tehdan

Typically - just as I get my NAT setup mostly working, got an email from OVH saying 'the switch is fixed' (which it actually is) and my bridged IPs sprang to life. Not sure if this is a win or not.

@K2Bytes would still be interested to see your tutorial. Since I am apparently capable of basic networking let me know if I can do anything to help. There really is no non proxmox/vmware based tutorial out there that isn't largely wrong, so I think this would be a good thing to contribute to the community...

William

GoodHosting said: even ColoCrossing's network supports QinQ... somehow

Yea...unlikely, it just looks like that in a L2 design or is a misconfiguration.

GoodHosting

@K2Bytes

@MCHPhil

@William

@tehdan

This is our current setup, which magically works; but requires our customers to do some really fancy overrides in their Guest Operating System; and only works in Some Operating Systems [ doesn't work in Windows at all. ]

It makes absolutely no sense, but it's the only way we've managed to have both the guest and host machine's routing with the Primary IPv4 and Primary IPv6 working.

---

• 1) Disable the vSwitch and remove the IP addresses and servers from it, it's useless.
• 2) One by one, assign each IP block to the server [using "Assign the failover IP" option]
• 3) IMPORTANT Delete all "Virtual MAC" delegations, mitigation, and firewalls TEDIOUS
• 4) On the host machine, configure the following semi-sane files [ CentOS 6.5 example ]

ifcfg-eth0

DEVICE=eth0
TYPE=Ethernet

ONBOOT=yes

## Thanks to MCHPhil for the following two lines. [ required for IPv6 to work ]
IPV6INIT=yes
IPV6_AUTOCONF=yes

## Your physical MAC, call OVH if you can't find it.
## This has to be the MAC for eth0.
HWADDR=xx:xx:xx:xx:xx:xx

BRIDGE=br0

ifcfg-br0

## Make sure bridge-utils is installed, in Minimal it is not!
DEVICE=br0
TYPE=Bridge

ONBOOT=yes
BOOTPROTO=static

## The following is really really stupid, blame OVH's /32
NETWORK=1.2.3.0 ## Your primary IP ending in .0
IPADDR=1.2.3.4 ## Your full primary IP
GATEWAY=1.2.3.254 ## Your primary IP ending in .254
BROADCAST=1.2.3.255 ## Your primary IP ending in .255
NETMASK=255.255.255.0 ## Yeah... /24 == /32 ?  ....

## Thanks to MCHPhil for the following two lines. [ required for IPv6 to work ]
DELAY=0
IPV6INIT=yes

• 5) You're going to love this step The following disgustingly confusing files have to be created in the CentOS 6.5 guest, haven't got it working on any other OS yet.

Guest ifcfg-eth0

DEVICE=eth0
TYPE=Ethernet

ONBOOT=yes

IPADDR=9.8.7.2 ## Failover IP [ IP of VPS ]
GATEWAY=1.2.3.254 ## Your primary IP ending in .254
## Note, no broadcast, no netmask, guest thinks it's a /24 .

Note that this gateway makes absolutely no sense, and watch this fun:

Guest route-eth0 [MUST CREATE THIS FILE]

1.2.3.4 dev eth0 ## YOUR PRIMARY IP [WTF??]
default via 1.2.3.4 dev eth0 ## YOUR PRIMARY IP [WTF??]

This makes absolutely no sense, but was the only way I could actually get it working.

I'm waiting for OVH to comment on this, so far they've taken five ticket replies to even realize I was using KVM, and which files were on the host and which were on the guest.

---

Workaround amalgamation taken from: http://help.ovh.co.uk/BridgeClient

---

Huge caveat: DHCP can't force that route-eth0 file onto guest machines, therefore it can't be done through DHCP. DHCPd also requires you give a subnet, gateway, network, etcetera; which breaks the configuration of the guest as well.... so this all has to be done manually, have fun.

GoodHosting

If you want an IPV6 VPS to work, replace a few steps below:

---

Guest ifcfg-eth0:

DEVICE=eth0
TYPE=Ethernet

ONBOOT=yes

IPV4_FATAL=no # Don't fail if IPv4 doesn't work

IPV6ADDR=2607:5302::26/128 ## IPv6 of VPS
IPV6GW=2607:5302::ff:ff:ff:ff ## Your primary IPv6 ending in ff:ff:ff:ff

Guest route-eth0:

2607:5302:60::1 dev eth0 ## YOUR PRIMARY IPv6 [WTF??]
default via 2607:5302:60::1 dev eth0 ## YOUR PRIMARY IPv6 [WTF??]

---

So be it IPv4 or IPv6 , your host machine turns into a router forwarding packets, due to how their network is set up to only allow certain MAC addresses to speak for each individual IP address. It's bloody ludicrous.

In both cases, the host machine should be set up to forward packets.

MCHPhil

I don't think you are using your gateway IP correctly. The IP's are routed to the host nodes gateway. If you use any other IP for your gateway OVH will block your IP after XX hours or days.

Please look back over the config I showed you, as that configuration will work if you place your values into the spots. All IP's route to your host nodes gateway. Using another gateway will cause very bad ARP issues. OVH blocks for this.

Setting gateway in the network configure while also setting up static routes? That is not right. Debian will automatically fail the network if you do so. Comment them out. Set the gateway manually via routes. Do not your your primary node IP. Use the gateway IP and you will have to setup vMAC addresses in OVH manager. Use type OVH and generic other information. If MAC on VM is not correct with MAC in manager. Bad day.

GoodHosting

@MCHPhil

What gets me:

• The whole point of a router is to route things. Once the router isn't routing things...

This means that your guest machines:

" Route " to your host machine BUT use the /24 gateway of your host machine. They're also REQUIRED to ARP themselves, which breaks a lot of things [ both on OVH's network, and on mine. ]

The above steps are literally what I was told to do during my 2 hour call with them today.

MCHPhil

Don't worry about what they say, I have multiple nodes and 100's of KVM's online in OVH that use this setup and it's just classless static routing. It's a pita but it's nothing wrong or invalid. Please be familiar with http://tools.ietf.org/html/rfc3442

GoodHosting

@MCHPhil said:
Don't worry about what they say, I have multiple nodes and 100's of KVM's online in OVH that use this setup and it's just classless static routing. It's a pita but it's nothing wrong or invalid. Please be familiar with http://tools.ietf.org/html/rfc3442

I'm rather curious, what would an exmaple dhcpd ruleset look like for that?

We've got the following at the moment, which worked on their vSwitch , but doesn't work on the public network [ obviously. ]

authoritative;

subnet 1.2.3.0 netmask 255.255.255.0 { # Primary .0
  option domain-name-servers 8.8.8.8, 8.8.4.4;
  option broadcast-address 1.2.3.255; # Primary .255
  option routers 1.2.3.4; # Primary IP

  default-lease-time 1200;
  max-lease-time 14400;
}

host 1-7-8-9.reverse-dns.goodhosting.co {
  hardware ethernet 02:00:09:08:07:01;
  fixed-address 9.8.7.1;
}

MCHPhil

Please look at the link I pasted and it provides all you want to know about configuring DHCP with classless static routes.

GoodHosting

Found it:

    option rfc3442-classless-static-routes 24, 9, 8, 7, 1, 2, 3, 254 ;
    option ms-classless-static-routes 24, 9, 8, 7, 1, 2, 3, 254 ;

Absolutely confusing.

GoodHosting

Sorry, even worse; is that you must specify it in special speak:

option 121 18:C0:A8:7B:0A:22:48:2A ;

MCHPhil

That is IT sir. It's pretty straightforward in my eyes. No magic needed

You should not have any hex characters. Notice the examples they gave has no hex values?