New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Help Proxmox bridged container no internet
Hi.
What I have:
- A VPS with 5 IPv4 xxx.xxx.20.93-xxx.xxx.20.97, same subnet and 1 /64 IPv6.
- Proxmox latest, 6.0 VE: Debian 10. Container used Ubuntu 16 template.
- I can make IPv4 and IPv6 work on Proxmox host.
- Provider doesn't restrict IP's to virtual MAC ID's, so bridged mode is recommended.
- No Nest Virtualization so only linux container.
What I want:
- 1 IPv4 for Proxmox host.
- Assign 4 IPv4 and 4 IPv6 for 4 Containers.
- Fancy stuff: private LAN network for containers like 10.14.43.xxx so containers can talk to each other if needed.
What is my situation now:
- IPv4 and v6 work on Proxmox host.
- Try routed mode, assign IP for container: Can ping host from container but cannot ping internet, like 8.8.8.8. There is packet arrived vmbr0, but no tx packet from vmbr.
- Try bridged mode, assign IP for container: Can not ping anything.
What I have tried:
- blog.no-panic.at/2016/08/09/proxmox-on-debian-at-hetzner-with-multiple-ip-addresses/
- sysorchestra.com/proxmox-5-on-hetzner-root-server-with-ipv4/
- And countless other. Both route and bridge.
Since my provider told me bridged mode is recommended I will pursuit this direction. Here is my config:
Proxmox /etc/network/interfaces:
auto lo
iface lo inet loopback
iface ens3 inet manual
auto vmbr0
iface vmbr0 inet static
address xxx.xxx.20.93
netmask 255.255.254.0
gateway xxx.xxx.20.1
bridge-ports ens3
bridge-stp off
bridge-fd 0
My config for a container /etc/network/interfaces: (nothing in /etc/network/interfaces.d)
auto lo
iface lo inet loopback
iface lo inet6 loopback
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
auto eth0
iface eth0 inet static
address xxx.xxx.20.94
netmask 255.255.254.0
gateway xxx.xxx.20.1
The whole weekend was spent, no hope ;(. I am really depressed now.
Comments
Probably there is no routes for vmbr0.
Can you explain? The config above is for bridged mode so I think it has nothing to do with route in
/etc/network/interfaces
file. Or you mentionedip route
?https://www.kiloroot.com/proxmox-kimsufi-ovh-soyoustart-ipv6-host-multiple-containers-and-virtual-machines-on-a-single-kimsufi-server-using-ipv6-and-proxmox/
I use CSF so things are a bit different but I do setup manual routing.
Here's my host setup, from a live environment with IPs setup like yours..
auto lo
iface lo inet loopback
iface enp1s0 inet manual
allow-vmbr1 int1
iface int1 inet static
address 192.168.xxx.2
netmask 255.255.255.0
ovs_type OVSIntPort
ovs_bridge vmbr1
auto vmbr0
iface vmbr0 inet static
address 66.xxx.xxx.18
netmask 255.255.255.248
gateway 66.xxx.xxx.17
bridge-ports enp1s0
bridge-stp off
bridge-fd 0
iface vmbr0 inet6 static
address 26xx:xxx0:12xx:1::2
netmask 64
gateway 26xx:xxx0:12xx:1::1
auto vmbr1
iface vmbr1 inet static
address 192.168.xxx.1
netmask 255.255.255.0
ovs_type OVSBridge
ovs_ports int1
default via 66.xxx.xxx.17 dev vmbr0 onlink
66.xxx.xxx.16/29 dev vmbr0 proto kernel scope link src 66.xxx.xxx.18
192.168.xxx.0/24 dev int1 proto kernel scope link src 192.168.xxx.2
192.168.xxx.0/24 dev vmbr1 proto kernel scope link src 192.168.xxx.1
I'm assuming you already set up forwarding, however, looks like you're missing an SNAT.
I assume you have a typo in your post?
I understand that SNAT is necessary only for routed mode, am I correct?
Usually, but where do you assume the typo?
I originally just copied some snippets from my notes but I changed them for the interfaces and routing table which are obscured live ones.
Indeed, however, i'm seeing that your container example config is pointing to the provider's gateway instead of the proxmox public host ip, if you want to set up this as per your config, you'll need an open bridge which interfaces to all container interfaces, so it works as a proper bridge.
In my configs i avoid trying to set up public interfaces for anything behind the host, and instead use private IP's directly assigned with a minimal and restricted DHCP server, and then setup DNATs/SNATs correspondingly for any additional public IPs, this reduces management complexity as the public IP setup is done on one place only, and i can control firewall rules on a single place, and i benefit from the fact that i know that there is only one place where the network failures can be traced.
The method i use is to set up a bridge only for the local lan, and leave virtual interfaces for any additional public IPs i want to use properly restricted with firewall / forwarding rules. (this has minimal overhead, and, doesn't cause total network connectivity loss on interface de(re)-activation if i need to flush rules or reload network.)
I have the VM (KVM in this case) gateway the same as the host gateway.
That is what I see:
But then do you need to deal with the problem such that app in container never knows the true IP where the packet come from (i.e in Wordpress's log all request come from route IP like 10.14.43.xxx).
Update: It suddenly works.
Sounds of a Leonard Cohen classic ring out from the galleries. :-p
No shit man. The same procedure, same host. It suddenly works after a few hour.
Oh, if i'm using specific services (like a web server) i'll use mod_rpaf for apache, or real_ip with nginx.
If i don't care about that, i'll simply DNAT/SNAT the whole IP address.