EdgeRouter: Dual WAN + Hair Pin+ Multiple networks + OpenVPN site-to-site VPN

By | April 1, 2016

My setup becomes more and more complicated bellow there are the updated steps of my configurations with references to the original posts containing the partial configurations.

STEP 1 EdgeRouter : Dual WAN with Hair Pin
Initially I started with a dual-wan configuration with some extra setting to exclude from load balancing the hair pin connections: EdgeRouter : Dual WAN with Hair Pin

STEP 2 LAN-to-LAN Exclusion

Then because I added some new sub-networks (a wifi AP ) and I started to have issues due to the dual WAN setup I discovered that I have to exclude the local networks from the dual WAN setup.

After the modify rule was added at Step 1 (to exclude the hair pin connections) the configuration (from CLI) looked like:


ubnt@ubnt:~$ configure
[edit]
ubnt@ubnt# show firewall modify
 modify balance {
     rule 10 {
         action modify
         destination {
             group {
                 address-group ADDRv4_eth0
             }
         }
         modify {
             table main
         }
     }
     rule 20 {
         action modify
         modify {
             lb-group G
         }
     }
 }
[edit]

As instructed by the ubnt forum I had to add another rule to exclude also from the load-balancing the LAN to LAN traffic.
See the updated tutorial from ubnt for this EdgeMAX – Dual WAN Load-Balance Feature

LAN-to-LAN Exclusion

From CLI do the following:

Lower the priority of the existing rules:


configure
edit firewall modify balance
rename rule 20 to rule 30
commit
edit firewall modify balance
rename rule 10 to rule 20
commit
save
exit

Add LAN-to LAN traffic exclusion from load balancing by adding a new modify rule that has a bigger priority than the load balancing rule.
Also create a group where all the sub-nets are added as members.


configure
set firewall group network-group LAN_NETS network 192.168.1.0/24
set firewall group network-group LAN_NETS network 192.168.2.0/24
set firewall group network-group LAN_NETS network 192.168.9.0/24
commit

set firewall modify balance rule 10 destination group network-group LAN_NETS
set firewall modify balance rule 10 action modify
set firewall modify balance rule 10 modify table main
commit

Note that 192.168.9.0/24 is added here because is going to be used later when we will add the openvpn site-to-site VPN.

After this the final setup looks like the following:


ubnt@ubnt:~$ configure
[edit]
ubnt@ubnt# show firewall modify
 modify balance {
     rule 10 {
         action modify
         destination {
             group {
                 network-group LAN_NETS
             }
         }
         modify {
             table main
         }
     }
     rule 20 {
         action modify
         destination {
             group {
                 address-group ADDRv4_eth0
             }
         }
         modify {
             table main
         }
     }
     rule 30 {
         action modify
         modify {
             lb-group G
         }
     }
 }
[edit]

STEP 3 EdgeRouter: OpenVPN site-to-site VPN
Another complication was added to the configuration when I added a second site to my network. This remote site has an EdgeRouter Lite so I decided to make a permanent OpenVpn site-to-site permanent VPN between sites
The setup is described in the following post: EdgeRouter: OpenVPN site-to-site VPN

STEP 4 VPN Exclusion (not yet implemented in the current firmware)
After STEP 3 was done I still had a strange issue. My VPN connection was dropping packets. I discovered that if I disabled the secondary WAN suddenly everything was OK. So I concluded that the cause of this unstable VPN was the load-balancing. In fact it was the same behaviour I noticed before STEP 2 was done when I was ping-ing between machines from different LANs.
Note that this is not yet implemented in the current firmware (1.8) but this is how I expected that should work as a solution. It was confirmed by ubiquity support that this is one of the solutions that is taken into account in a future firmware update.
See STEP 5 for the current partial solution.

From CLI do the following:

Lower the priority of the existing load-balance rule:


configure
edit firewall modify balance
rename rule 30 to rule 40
commit
save
exit

Add an exclusion rule for the vtun0 site-to-site VPN interface just before the load-balance rule.


configure
edit firewall modify balance
set rule 30 modify table main
set rule 30 destination group address-group ADDRv4_vtun0
commit
save
exit

The final configuration then looked like:


ubnt@ubnt# show firewall modify
 modify balance {
     rule 10 {
         action modify
         destination {
             group {
                 network-group LAN_NETS
             }
         }
         modify {
             table main
         }
     }
     rule 20 {
         action modify
         destination {
             group {
                 address-group ADDRv4_eth0
             }
         }
         modify {
             table main
         }
     }
     rule 30 {
         destination {
             group {
                 address-group ADDRv4_vtun0
             }
         }
         modify {
             table main
         }
     }
     rule 40 {
         action modify
         modify {
             lb-group G
         }
     }
 }
[edit]

Then I enabled the second WAN from eth1 on my main site.

Restarted the openVPN:


reset openvpn interface vtun0

Force the reload of the load-balancing configuration by killing the process. This is a trick from UBNT-stig 🙂

Hmm “WAN Out” is empty. I wonder if you ran into a bug I fixed the last week. Can you try doing “sudo pkill ubnt-util” from the CLI and then see if “WAN Out” gets filled in?

Check if load-balance is up and acting as fail-over:


ubnt@ubnt:~$ show load-balance status
Group G
  interface   : eth0
  carrier     : up
  status      : active
  gateway     : 84.117.103.1
  route table : 201
  weight      : 100%
  flows
      WAN Out : 44
      WAN In  : 52
    Local Out : 172

  interface   : eth1
  carrier     : up
  status      : failover
  gateway     : 192.168.0.1
  route table : 202
  weight      : 0%
  flows
      WAN Out : 0
      WAN In  : 0
    Local Out : 136

Checked the status of the connection:


show openvpn status site-to-site

STEP 5 Current Issue still pending
The main issue I have now is caused by the load-balancing setup.
When both WANs are active any connection through the vtun0 is loosing half of the packages.
This is because the logic solution from STEP 4 is not yet supported by the current firmware.

After asking on the Ubiquity forums I got the following explanation in the topic I opened http://community.ubnt.com/t5/EdgeMAX/Dual-WAN-failover-OpenVPN-site-to-site/m-p/1524860/highlight/false#M104986:

UBNT-stig wrote:

I think the problem is likely that load-balance uses a separate routing table for each WAN and those routing tables only include the default gateway.  Normally the way around that is to create your own routing table:


ubnt@ubnt# show protocols 
 static {
     interface-route 192.168.9.0/24 {
         next-hop-interface vtun0 {
         }
     }
     table 1 {
         interface-route 192.168.9.0/24 {
             next-hop-interface vtun0 {
             }
         }
         route 0.0.0.0/0 {
             next-hop  {
             }
         }
     }
 }
[edit]

And then tell load-balace to use that table.

set load-balance group G interface eth0 route table 1

Unfortunately your gateway is from dhcp and may change over time.  I've been thinking of ways to work around that similar to ADDRv4_eth0, but can't use firewall groups in the routing table.

So my problem is that there is a partial solution that I cannot apply due to the fact gateway cannot be specified as DHCP in other than the default routing table.

I am still waiting for a solution that will come from Ubiquity most probably in a future firmware update. There is a thread in fact requesting for this feature http://community.ubnt.com/t5/EdgeMAX-Beta/DHCP-route-in-other-routing-table/m-p/1525360#M15702

3 thoughts on “EdgeRouter: Dual WAN + Hair Pin+ Multiple networks + OpenVPN site-to-site VPN

  1. David Biers

    I wrote a similar article but based on the OpenVPN configuration and package that IPFire and other web application firewalls will provide you with (https://dbiers.me/ipfire-sitetosite-vpn-edgeouter-vyatta/).

    My issue is (and you may even have the same issue, here) that setting up the site-to-site VPN on the edgerouter which is using 2 WAN links in Load balance mode (not fail-over) causes VPN packets to source from both WAN links.

    This poses an issue to OpenVPN as control packets come from the wrong address of the initial connection and therefore cannot establish a handshake reliably:

    “`

    Aug 22 15:35:55 fw01 n2nVPN[21583]: TLS: Initial packet from [AF_INET]72.191.29.250:1201, sid=b3c98442 31b0bae1
    Aug 22 15:35:55 fw01 n2nVPN[21583]: TLS Error: Received control packet from unexpected IP addr: [AF_INET]104.1.45.150:1024
    Aug 22 15:35:55 fw01 n2nVPN[21583]: TLS Error: Received control packet from unexpected IP addr: [AF_INET]104.1.45.150:1024
    Aug 22 15:35:58 fw01 n2nVPN[21583]: TLS Error: Received control packet from unexpected IP addr: [AF_INET]104.1.45.150:1024
    Aug 22 15:36:02 fw01 n2nVPN[21583]: TLS Error: Received control packet from unexpected IP addr: [AF_INET]104.1.45.150:1024

    “`

    I’m trying to find a way to “lock” that traffic to the IP address/WAN but I also want it to re-initiate and maintain a connection if it fails over.

    I’m not as experienced at network equipment as you seem to be, especially router configurations — but maybe you have an idea of how that can be resolved. Perhaps this would be something for your next article.

    Let me know!

    Reply
    1. voinageo Post author

      If you look at step 5 there is a solution. If you define a static gateway for table 1 routing table that is assigned to the load balance group it works. My final solution was to add an ER-8 in front of the EdgeRouter POE. In this way the ER-8 has a dynamic gateway from the ISP and I have a static gateway 192.168.7.1 that I can specify in the config done on EdgeRouter POE.

      Reply
  2. Pingback: EdegRouter: Policy Based Routing for OpenVPN when Load Balancing dual WANs | blog.voina.org

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.