Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Need someone to test IPsec on their boxes
New on LowEndTalk? Please Register and read our Community Rules.

Need someone to test IPsec on their boxes

Hello.
I'm trying to configure IPsec in OpenVZ containers. The tunnel itself works fine, but SNAT doesn't work at all. I need someone to test it on their boxes, especially with OpenVZ, because exactly the same configuration works fine on my dedicated server. It will take only 3-5 minutes of your time.

You need 64-bit Debian 7 or jessie.

% aptitude install strongswan libcharon-extra-plugins

Add to the bottom of /etc/ipsec.conf

conn rw
    left=%any
    leftsubnet=0.0.0.0/0
    leftauth=psk
    right=%any
    rightsourceip=10.3.0.0/24
    rightdns=8.8.8.8
    rightauth=psk
    rightauth2=xauth
    auto=add

Add to the bottom of /etc/ipsec.secrets

: PSK "psk"
test : XAUTH "test"

% iptables -t nat -I POSTROUTING -s 10.3.0.0/24 -j MASQUERADE
% service ipsec restart

Now try to connect to your IPsec tunnel (I do this from my Android smartphone). Use "IPsec Xauth PSK" profile, "psk" as preshared key and test/test as username and password.

Expected result:
You can access internet on your smartphone with server IP address

Actual result:
You cannot access internet on your smartphone, while you can ping server ip address from smartphone and smartphone ip (10.3.0.1) from server.

I highly appreciate any testing results.

Comments

  • Here is exactly the same issue in StrongSWAN maillist
    https://lists.strongswan.org/pipermail/users/2014-February/005822.html

  • I didn't test this, but what's the output of "ip x p s"?

  • ValdikSSValdikSS Member
    edited August 2014

    @agentsmith said:
    I didn't test this, but what's the output of "ip x p s"?

    [email protected]:~# ip xfrm state
    src serv_ip dst my_ip
        proto esp spi 0x0ebf3dd9 reqid 1 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha1) 0xe61a8df63c8c5e2e717e8451f63e083a1e751909 96
        enc cbc(aes) 0x8fb14095f17184ed6069689c3f61590c
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
    src my_ip dst serv_ip
        proto esp spi 0xc40e0ce1 reqid 1 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha1) 0xd2d6cde7752bd1ad2813362f4337c7a074544c48 96
        enc cbc(aes) 0x8c4270e2bc5e6efe2ad1ef6d69b73426
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
    
    [email protected]:~# ip xfrm policy
    src 10.3.0.1/32 dst 0.0.0.0/0 
        dir fwd priority 2947 ptype main 
        tmpl src my_ip dst serv_ip
            proto esp reqid 1 mode tunnel
    src 10.3.0.1/32 dst 0.0.0.0/0 
        dir in priority 2947 ptype main 
        tmpl src my_ip dst serv_ip
            proto esp reqid 1 mode tunnel
    src 0.0.0.0/0 dst 10.3.0.1/32 
        dir out priority 2947 ptype main 
        tmpl src serv_ip dst my_ip
            proto esp reqid 1 mode tunnel
    src 0.0.0.0/0 dst 0.0.0.0/0 
        socket in priority 0 ptype main 
    src 0.0.0.0/0 dst 0.0.0.0/0 
        socket out priority 0 ptype main 
    src 0.0.0.0/0 dst 0.0.0.0/0 
        socket in priority 0 ptype main 
    src 0.0.0.0/0 dst 0.0.0.0/0 
        socket out priority 0 ptype main 
    src ::/0 dst ::/0 
        socket in priority 0 ptype main 
    src ::/0 dst ::/0 
        socket out priority 0 ptype main 
    src ::/0 dst ::/0 
        socket in priority 0 ptype main 
    src ::/0 dst ::/0 
        socket out priority 0 ptype main
    
  • iptables -t raw -A PREROUTING -i venet0 -p udp --dport 4500 -j TRACE
    

    With one packet to your host and one to the internet

  • @agentsmith said:
    iptables -t raw -A PREROUTING -i venet0 -p udp --dport 4500 -j TRACE
    With one packet to your host and one to the internet

    I'll ask provider to load ipt_LOG, but I'm afraid it won't be loaded.

  • You can also add some ACCEPT in filter table and watch the packet counter. Does the packet go out? Can you check with tcpdump?

  • @agentsmith Can I do this with nfnetlink_log?

  • You need to know in which chain the packet is stopped. So having Trace would be ideal. Don't know whether you can log each chain with nfnetlink_log.

  • @agentsmith, asked hoster to load iptable_raw and xt_TRACE. Would this be sufficient?

  • agentsmithagentsmith Member
    edited August 2014

    would be a great help. whats with

    tcpdump -lnvi venet0 icmp
    

    when you ping an external host.

  • ValdikSSValdikSS Member
    edited August 2014

    @agentsmith, just built strongswan with kernel-libipsec, so now ipsec works in userspace, and nat works fine. I'll install Debian 7, since jessie doesn't have openswan, and will test with openswan. Maybe it's a strongswan issue.
    Waiting for hoster. Thank you a lot for your effort.

  • ValdikSSValdikSS Member
    edited August 2014

    @agentsmith said:
    would be a great help. whats with

    tcpdump -lnvi venet0 icmp
    

    when you ping an external host.

    31.220.5.43 is the server IP.

    [email protected]:~# tcpdump -lnvi venet0 icmp
    tcpdump: listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
    17:44:11.294011 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 2355, seq 1, length 64
    17:44:12.304503 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 8.8.4.4: ICMP echo request, id 2355, seq 2, length 64
    17:44:12.304532 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 2355, seq 2, length 64
    17:44:13.305043 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 8.8.4.4: ICMP echo request, id 2355, seq 3, length 64
    17:44:13.305080 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 2355, seq 3, length 64
    17:44:14.304637 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 8.8.4.4: ICMP echo request, id 2355, seq 4, length 64
    17:44:14.304672 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 2355, seq 4, length 64
    

    There is no response on ping

    % ip a a 1.2.3.4/32 dev venet0
    
    17:45:14.715232 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 1.2.3.4: ICMP echo request, id 3635, seq 1, length 64
    17:45:15.714504 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 1.2.3.4: ICMP echo request, id 3635, seq 2, length 64
    17:45:16.724553 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 1.2.3.4: ICMP echo request, id 3635, seq 3, length 64
    

    There is a response.
    I suppose venet might be the culprit, but literally no hosters provide veth.

  • agentsmithagentsmith Member
    edited August 2014

    @ValdikSS
    Really strange...packet seems to go out but doesn't. Do you see any unusual in

    ip r s c
    ip r s t all
    

    ?

  • ValdikSSValdikSS Member
    edited August 2014

    @agentsmith, i'm even not sure why the hell the first record in the first tcpdump listing is from 31.220.5.43 (server's IP), and THEN from roadwarrior IP. It's not buffering issue or like i've started tcpdump in the middle of the ping, i've double checked that. My dedicated box shows packet flow as you might expect (it works fine there):

    [email protected]:~# tcpdump -lnvi eth0 icmp
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
    01:25:17.252501 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 8.8.8.8: ICMP echo request, id 17461, seq 1, length 64
    01:25:17.252596 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        195.154.104.228 > 8.8.8.8: ICMP echo request, id 17461, seq 1, length 64
    01:25:17.258208 IP (tos 0x0, ttl 50, id 0, offset 0, flags [none], proto ICMP (1), length 84)
        8.8.8.8 > 195.154.104.228: ICMP echo reply, id 17461, seq 1, length 64
    01:25:18.250658 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 8.8.8.8: ICMP echo request, id 17461, seq 2, length 64
    01:25:18.250722 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        195.154.104.228 > 8.8.8.8: ICMP echo request, id 17461, seq 2, length 64
    01:25:18.256378 IP (tos 0x0, ttl 50, id 0, offset 0, flags [none], proto ICMP (1), length 84)
        8.8.8.8 > 195.154.104.228: ICMP echo reply, id 17461, seq 2, length 64
    01:25:19.252937 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.1 > 8.8.8.8: ICMP echo request, id 17461, seq 3, length 64
    01:25:19.253000 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        195.154.104.228 > 8.8.8.8: ICMP echo request, id 17461, seq 3, length 64
    01:25:19.258601 IP (tos 0x0, ttl 50, id 0, offset 0, flags [none], proto ICMP (1), length 84)
        8.8.8.8 > 195.154.104.228: ICMP echo reply, id 17461, seq 3, length 64
    

    By the way, it seems like tcpdump (or, more likely, kernel itself) can't capture non-encrypted traffic, which has ipsec tunnel destination.
    http://seclists.org/tcpdump/2011/q1/107

    Do you see any unusual in

    No, nothing unusual. Can't test right now, I've reinstalled that test box. I'm setting up openswan for tests.

  • ValdikSSValdikSS Member
    edited August 2014

    @agentsmith, installed and configured racoon, it's just the same. I can ping client from server and server from client, but can't access the internet.
    I hope someone could try it on BuyVM VPS since BuyVM has all the modules loaded as their wiki states.

  • @ValdikSS said:
    % iptables -t nat -I POSTROUTING -s 10.3.0.0/24 -j MASQUERADE

    Can you use MASQUERADE on OpenVZ? I thought it had to be SNAT.

  • @ValdikSS
    Very first packet is missing (seq 1). Does tcpdump tell you it dropped one when exiting?
    Also interesting to see the other direction:

    iptables -t nat -A PREROUTING -d 31.220.5.43 -p tcp --dport 80 -j DNAT --to 10.3.0.1
    

    or any other service and try to access it from outside.
    Routing cache would be interesting to see whether venet0 is listed there for all packets we use.

    @bertan
    I think that was true for 2.6.18 kernel, but anyway I prefer SNAT since venet0 has at least 2 IP addresses.

  • @agentsmith, yes, thanks for the clarification. I don't know why it is missing. I wait some seconds to make sure tcpdump started, but always get the same missing packet.

    [email protected]:~# tcpdump -lnvi venet0 icmp
    tcpdump: listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
    04:01:09.503220 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 44634, seq 1, length 64
    04:01:10.513275 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.2 > 8.8.4.4: ICMP echo request, id 44634, seq 2, length 64
    04:01:10.513306 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 44634, seq 2, length 64
    04:01:11.493521 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.2 > 8.8.4.4: ICMP echo request, id 44634, seq 3, length 64
    04:01:11.493558 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 44634, seq 3, length 64
    04:01:12.513753 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.3.0.2 > 8.8.4.4: ICMP echo request, id 44634, seq 4, length 64
    04:01:12.513788 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        31.220.5.43 > 8.8.4.4: ICMP echo request, id 44634, seq 4, length 64
    ^C
    7 packets captured
    8 packets received by filter
    0 packets dropped by kernel
    

    This is on ubuntu with racoon now.

  • ValdikSSValdikSS Member
    edited August 2014

    @agentsmith, no, dnat doesn't work either.

    nc -l -p 4455 on client, works fine from ipsec server.
    
    % iptables -t nat -A PREROUTING -d 31.220.5.43 -p tcp --dport 4455 -j DNAT --to 10.3.0.3
    Trying to send some data from another connection to 31.220.5.43:4455
    
    [email protected]:~# tcpdump -lnvi venet0 port 4455
    tcpdump: listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
    04:20:38.304469 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        10.3.0.3.4455 > 31.220.5.43.52758: Flags [S.], cksum 0x3a28 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2254924 ecr 14261521,nop,wscale 6], length 0
    04:20:38.304502 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        31.220.5.43.4455 > 95.215.45.33.52758: Flags [S.], cksum 0xb735 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2254924 ecr 14261521,nop,wscale 6], length 0
    04:20:39.287656 IP (tos 0x0, ttl 58, id 56121, offset 0, flags [DF], proto TCP (6), length 60)
        95.215.45.33.52758 > 31.220.5.43.4455: Flags [S], cksum 0x909a (correct), seq 3471722025, win 29200, options [mss 1373,sackOK,TS val 14261822 ecr 0,nop,wscale 7], length 0
    04:20:39.344109 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        10.3.0.3.4455 > 31.220.5.43.52758: Flags [S.], cksum 0x39c0 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255028 ecr 14261521,nop,wscale 6], length 0
    04:20:39.344142 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        31.220.5.43.4455 > 95.215.45.33.52758: Flags [S.], cksum 0xb6cd (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255028 ecr 14261521,nop,wscale 6], length 0
    04:20:39.495692 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        10.3.0.3.4455 > 31.220.5.43.52758: Flags [S.], cksum 0x39b0 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255044 ecr 14261521,nop,wscale 6], length 0
    04:20:39.495724 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        31.220.5.43.4455 > 95.215.45.33.52758: Flags [S.], cksum 0xb6bd (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255044 ecr 14261521,nop,wscale 6], length 0
    04:20:41.294579 IP (tos 0x0, ttl 58, id 56122, offset 0, flags [DF], proto TCP (6), length 60)
        95.215.45.33.52758 > 31.220.5.43.4455: Flags [S], cksum 0x8e40 (correct), seq 3471722025, win 29200, options [mss 1373,sackOK,TS val 14262424 ecr 0,nop,wscale 7], length 0
    04:20:41.365595 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        10.3.0.3.4455 > 31.220.5.43.52758: Flags [S.], cksum 0x38f5 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255231 ecr 14261521,nop,wscale 6], length 0
    04:20:41.365626 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        31.220.5.43.4455 > 95.215.45.33.52758: Flags [S.], cksum 0xb602 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255231 ecr 14261521,nop,wscale 6], length 0
    04:20:41.693183 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        10.3.0.3.4455 > 31.220.5.43.52758: Flags [S.], cksum 0x38d4 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255264 ecr 14261521,nop,wscale 6], length 0
    04:20:41.693216 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        31.220.5.43.4455 > 95.215.45.33.52758: Flags [S.], cksum 0xb5e1 (correct), seq 2270620501, ack 3471722026, win 28960, options [mss 1460,sackOK,TS val 2255264 ecr 14261521,nop,wscale 6], length 0
    ^C
    12 packets captured
    14 packets received by filter
    0 packets dropped by kernel
    

    It missed some packets again.
    I get no packets from my side (from where i've tried to send data).

  • @ValdikSS
    At least packets are reaching your client (it answers SYNACK), but packet disappears in egress :-( Can you show me routing cache and tables after trying to send a packet?

  • ValdikSSValdikSS Member
    edited August 2014

    @agentsmith

    [email protected]:~# ip r s c
    client_ip from 31.220.5.43 dev venet0 
        cache  mtu 1500 advmss 1460 hoplimit 64
    42.120.111.2 from 10.3.0.1 dev venet0  src 31.220.5.43 
        cache <src-direct,redirect>  mtu 1500 advmss 1460 hoplimit 64 iif venet0
    175.102.15.68 from 31.220.5.43 dev venet0 
        cache  mtu 1500 advmss 1460 hoplimit 64
    37.1.152.13 from 31.220.5.43 dev venet0 
        cache  ipid 0x488b mtu 1500 advmss 1460 hoplimit 64
    77.88.8.7 from 10.3.0.1 dev venet0  src 31.220.5.43 
        cache <src-direct,redirect>  mtu 1500 advmss 1460 hoplimit 64 iif venet0
    local 31.220.5.43 from 37.1.152.13 dev lo  src 31.220.5.43 
        cache <local,src-direct>  iif venet0
    8.8.4.4 from 10.3.0.1 dev venet0  src 31.220.5.43 
        cache <src-direct,redirect>  mtu 1500 advmss 1460 hoplimit 64 iif venet0
    95.215.45.33 from 31.220.5.43 tos lowdelay dev venet0 
        cache  mtu 1500 advmss 1460 hoplimit 64
    local 31.220.5.43 from 175.102.15.68 dev lo  src 31.220.5.43 
        cache <local,src-direct>  iif venet0
    local 31.220.5.43 from 95.215.45.33 dev lo  src 31.220.5.43 
        cache <local,src-direct>  iif venet0
    local 31.220.5.43 from client_ip dev lo  src 31.220.5.43 
        cache <local,src-direct>  iif venet0
    54.225.251.145 from 10.3.0.1 dev venet0  src 31.220.5.43 
        cache <src-direct,redirect>  mtu 1500 advmss 1460 hoplimit 64 iif venet0
    77.88.8.3 from 10.3.0.1 dev venet0  src 31.220.5.43 
        cache <src-direct,redirect>  mtu 1500 advmss 1460 hoplimit 64 iif venet0
    [email protected]:~# ip ru   
    0:  from all lookup local 
    32766:  from all lookup main 
    32767:  from all lookup default 
    [email protected]:~# ip r 
    default dev venet0  scope link 
    [email protected]:~# ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    2: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
        link/void 
        inet 127.0.0.2/32 scope host venet0
        inet 31.220.5.43/32 brd 31.220.5.43 scope global venet0:0
        inet 1.2.3.4/32 scope global venet0
    

    this is after pinging 8.8.4.4 from client.

  • ValdikSSValdikSS Member
    edited August 2014

    That's interesting. Using another OpenVZ VPS with strongSwan, I now get ICMP Destination Port Unreachable pinging 8.8.4.4 or any other public address from client.
    EDIT: Sorry for misinformation, that was because of FORWARD rule. After deleting it it's just the same.

  • rm_rm_ Member
    edited August 2014

    Just cancel all OpenVZ already, and forget it as a horrible nightmare. :)

  • @rm_, it't not like I can't manage it to work at all. strongSwan can be configured with kernel-libipsec, which is userspace ipsec implementation, and it works that way on openvz as I tested yesterday. Now it's just the matter of understanding how the hell the packet flow is going and what's the matter of this issue.

  • btw: "ip r" only shows main table.
    It seems to be a bug...
    http://forum.openvz.org/index.php?t=tree&goto=39937&&srch=ipsec#page_top
    https://bugzilla.redhat.com/show_bug.cgi?id=1081804
    Maybe you can try playing around in /proc/sys/net/ipv4/conf/

  • @agentsmith thanks for the links, although it's not very useful. I wrote a message to Pavel from Redhad, who wrote he's aware of IPsec problems in OpenVZ. Let's hope we'll get response from him.

Sign In or Register to comment.