Trouble with Proxmox and Hetzner IPv6

Lunar

I've bee trying to configure IPv6 for guests and I just can't get it working. IPv4 works fine. Looking for any kind of help as I've been baffled for weeks. I also followed all of the Hetzner documentation and tried lots of other configs. This is my current config:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address  5.x.xxx.44
    netmask  255.255.255.255
    gateway  5.x.xxx.33
        pointopoint  5.x.xxx.33

iface eth0 inet6 static
    address  2a01:xxx:xxx:xxxx::2
    netmask  128
    gateway  fe80::1

auto vmbr0
iface vmbr0 inet static
    address  5.x.xxx.44
    netmask  255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    up ip route add 144.xx.xx.248/32 dev vmbr0
    up ip route add 144.xx.xx.249/32 dev vmbr0
    up ip route add 144.xx.xx.250/32 dev vmbr0
    up ip route add 144.xx.xx.251/32 dev vmbr0
    up ip route add 144.xx.xx.252/32 dev vmbr0
    up ip route add 144.xx.xx.253/32 dev vmbr0
    up id route add 144.xx.xx.254/32 dev vmbr0
    up ip route add 144.xx.xx.255/32 dev vmbr0  

iface vmbr0 inet6 static
    address  2a01:xxx:xxx:xxxx::2
    netmask 64

Falzo

did you do this on the hostnode:

sysctl -w net.ipv6.conf.all.forwarding=1

for using an IPv6 subnet you are missing the gateway like

gateway fe80::1

are we talking about OVZ / LXC / KVM guest? how are you assigning the IPv6 there? via the control panel? what OS are your guests?

Lunar

@Falzo said:
did you do this on the hostnode:

sysctl -w net.ipv6.conf.all.forwarding=1

for using an IPv6 subnet you are missing the gateway like

gateway fe80::1

are we talking about OVZ / LXC / KVM guest? how are you assigning the IPv6 there? via the control panel? what OS are your guests?

I have enabled IPv6 forwarding at boot. I also added the gateway to vmbr0 it didn't make a difference. I've been testing it with LXC and I'm using the control panel to assign IPv6 there. Here is a screenshot of my config. Also keep in mind I've tried using fe80::1 as the gateway here and it made no difference.

Falzo

I had a further look into this, as I probably need to use IPv6 soon enough ;-)

first to mention I enabled ipv4/ipv6 forwarding in /etc/sysctl.conf

here is my /etc/network/interfaces I ended up with by now - http://pastebin.com/EGV2ecCz

don't mind the three bridges in there, out of which vmbr2 is the one for routed setup as you do above, and vmbr1 is used for natted IPv4 only.

vmbr0 would be used for bridging IPs (vMAC for IPv4). this is why vmbr0 kind of replaces eth0 as it is directly bridging so no need for defining anyting else around eth0.

vmbr0 gets a single IPv6 assigned and sets the default route (as you do above for your eth0), for the other bridges I do use smaller blocks of IPv6 as can be seen.

I also tested to assign IPv6s to lxc containers either for the natted or routed IPs within the gui, choosing the correct bridge and keeping the netmask for the client, so for routed it would be vmbr2 , IPv6/CIDR looks like 2a01:X:X:X:200::111/100 and for the gateway 2a01:X:X:X:200::2

this works for me without problems. sadly I can't break my whole config down to match and try yours but I'd suggest for your case leaving eth0 as it is and assigning a somewhat smaller block to vmbr0.

also have a look at 'sysctl -a|grep disable_ipv6' as on my server ipv6 for eth0 is disabled but doesn't hurt in my setup... might be a problem in yours?

are you able to use IPv6 on the hostnode itself?

also if this might help here is the output of my 'sysctl -a |grep ipv6' - http://pastebin.com/EU1zA6Dt

it also took me a while to figure it out and get it working like above, I saw restarting the network didn't always took all changes made in account so I needed to reboot the host more or less often...

hope this helps in some way ;-)

Lunar

@Falzo said:
I had a further look into this, as I probably need to use IPv6 soon enough ;-)

first to mention I enabled ipv4/ipv6 forwarding in /etc/sysctl.conf

here is my /etc/network/interfaces I ended up with by now - http://pastebin.com/EGV2ecCz

don't mind the three bridges in there, out of which vmbr2 is the one for routed setup as you do above, and vmbr1 is used for natted IPv4 only.

vmbr0 would be used for bridging IPs (vMAC for IPv4). this is why vmbr0 kind of replaces eth0 as it is directly bridging so no need for defining anyting else around eth0.

vmbr0 gets a single IPv6 assigned and sets the default route (as you do above for your eth0), for the other bridges I do use smaller blocks of IPv6 as can be seen.

I also tested to assign IPv6s to lxc containers either for the natted or routed IPs within the gui, choosing the correct bridge and keeping the netmask for the client, so for routed it would be vmbr2 , IPv6/CIDR looks like 2a01:X:X:X:200::111/100 and for the gateway 2a01:X:X:X:200::2

this works for me without problems. sadly I can't break my whole config down to match and try yours but I'd suggest for your case leaving eth0 as it is and assigning a somewhat smaller block to vmbr0.

also have a look at 'sysctl -a|grep disable_ipv6' as on my server ipv6 for eth0 is disabled but doesn't hurt in my setup... might be a problem in yours?

are you able to use IPv6 on the hostnode itself?

also if this might help here is the output of my 'sysctl -a |grep ipv6' - http://pastebin.com/EU1zA6Dt

it also took me a while to figure it out and get it working like above, I saw restarting the network didn't always took all changes made in account so I needed to reboot the host more or less often...

hope this helps in some way ;-)

Somehow It's working now. I just reinstalled the server and went through the Proxmox setup again. After installation of Proxmox I enabled IPv6 forwarding in 3 configs.

/etc/sysctl.conf
/etc/sysctl.d/99-hetzner.conf
/etc/sysctl.d/99-sysctl.conf

Somehow that made the difference. Thanks for all your help!

Lunar

Not sure what happened but it hasn't been working for quite some time.

I will post the output of ip -6 route and ip -6 neigh of the host and a VM. Hopefully someone can help me out here. I really have no idea what I'm doing wrong...

Host:

ip -6 route

2a01:4f8:xxx:xxxx::2 dev eth0  proto kernel  metric 256  pref medium
fe80::/64 dev vmbr1  proto kernel  metric 256  pref medium
fe80::/64 dev eth0  proto kernel  metric 256  pref medium
fe80::/64 dev vmbr0  proto kernel  metric 256  pref medium
default via fe80::1 dev eth0  metric 1024  pref medium

ip -6 neigh

fe80::1 dev eth0 lladdr 0c:86:10:f5:c0:a9 router STALE
fe80::90bd:95ff:feac:a456 dev vmbr0 lladdr 92:bd:95:ac:a4:56 STALE
fe80::b4a7:16ff:fe9d:cbfa dev vmbr0 lladdr b6:a7:16:9d:cb:fa STALE

VM:

ip -6 route

2a01:4f8:xxx:xxxx::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
default via 2a01:4f8:xxx:xxxx::2 dev eth0  metric 1024

ip -6 neigh

2a01:4f8:xxx:xxxx::2 dev eth0  FAILED

Lunar

Here's my sysctl -p

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.accept_dad = 0
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.all.accept_dad = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.all.accept_ra_defrtr = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 0
net.ipv6.conf.all.accept_ra_pinfo = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_redirects = 0

Lunar

Hey @Falzo could you please post what you have in your /etc/sysctl.conf and the /etc/sysctl.d directory?

Falzo

@Lunar said:
Hey @Falzo could you please post what you have in your /etc/sysctl.conf and the /etc/sysctl.d directory?

sure (sorry it took me so long)

this are the only (uncommented) things in /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

in /etc/sysctl.d I have this:

 10-no-icmp-redirects.conf
 99-hetzner.conf
 99-sysctl.conf -> ../sysctl.conf
 pve.conf
 README.sysctl

I am pretty sure I tried with and without that hetzner.conf, but didn't change anything at least on my short play around...

that 99-sysctl.conf is just a link to sysctl.conf and should be processed after that hetzner.conf, because 99-s is after 99-h. so settings in sysctl.conf should always override settings of the hetzner config at least if there are the same options in them at all.

that pve.conf contains

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
fs.aio-max-nr = 1048576

if I remember correctly I didn't change the contents of those pve or hetzner conf at all.

what do you get on

sysctl -a|grep disable_ipv6

?

Lunar

@Falzo said:

@Lunar said:
Hey @Falzo could you please post what you have in your /etc/sysctl.conf and the /etc/sysctl.d directory?

sure (sorry it took me so long)

this are the only (uncommented) things in /etc/sysctl.conf

net.ipv4.ip_forward=1
> net.ipv6.conf.all.forwarding=1
> 

in /etc/sysctl.d I have this:

 10-no-icmp-redirects.conf
>  99-hetzner.conf
>  99-sysctl.conf -> ../sysctl.conf
>  pve.conf
>  README.sysctl
> 

I am pretty sure I tried with and without that hetzner.conf, but didn't change anything at least on my short play around...

that 99-sysctl.conf is just a link to sysctl.conf and should be processed after that hetzner.conf, because 99-s is after 99-h. so settings in sysctl.conf should always override settings of the hetzner config at least if there are the same options in them at all.

that pve.conf contains

net.bridge.bridge-nf-call-ip6tables = 0
> net.bridge.bridge-nf-call-iptables = 0
> net.bridge.bridge-nf-call-arptables = 0
> net.bridge.bridge-nf-filter-vlan-tagged = 0
> fs.aio-max-nr = 1048576
> 

if I remember correctly I didn't change the contents of those pve or hetzner conf at all.

what do you get on

sysctl -a|grep disable_ipv6

?

Hmm, 99-sysctl.conf link isn't necessary is it? I deleted that. Could you also paste what you had in 99-hetzner.conf? I deleted that too. Thank you so much for helping me out btw.

This is what I get when I do sysctl -a | grep disable_ipv6

http://pastebin.com/bMGAdrzJ

Falzo

Lunar said: Hmm, 99-sysctl.conf link isn't necessary is it? I deleted that. Could you also paste what you had in 99-hetzner.conf? I deleted that too. Thank you so much for helping me out btw.

hmm, I guess you are right, as /etc/sysctl.conf should be the last one applied, so the link to it shouldn't matter at all. with sysctl --system you can see all conf files getting applied, their content and the order of it being done:

# sysctl --system
* Applying /etc/sysctl.d/10-no-icmp-redirects.conf ...
net.ipv4.conf.all.send_redirects = 0
* Applying /etc/sysctl.d/99-hetzner.conf ...
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.accept_dad = 0
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.all.accept_dad = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.all.accept_ra_defrtr = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 0
net.ipv6.conf.all.accept_ra_pinfo = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.all.forwarding = 0
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
* Applying /usr/lib/sysctl.d/pve-firewall.conf ...
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/pve.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
fs.aio-max-nr = 1048576
* Applying /etc/sysctl.conf ...
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

I am sorry to say, that's pretty much it.

in your ip -6 route you posted above, there is no IPv6 for any of your bridges, I'd say that is the point to start with. did you change anything lately with your bridges through proxmox e.g. added one or something like that? this could have overwritten other changes you did to /etc/network/interfaces ...