EdgeRouter: Dual WAN + Hair Pin+ Multiple networks + OpenVPN site-to-site VPN

My setup becomes more and more complicated bellow there are the updated steps of my configurations with references to the original posts containing the partial configurations.

STEP 1 EdgeRouter : Dual WAN with Hair Pin
Initially I started with a dual-wan configuration with some extra setting to exclude from load balancing the hair pin connections: EdgeRouter : Dual WAN with Hair Pin

STEP 2 LAN-to-LAN Exclusion

Then because I added some new sub-networks (a wifi AP ) and I started to have issues due to the dual WAN setup I discovered that I have to exclude the local networks from the dual WAN setup.

After the modify rule was added at Step 1 (to exclude the hair pin connections) the configuration (from CLI) looked like:

ubnt@ubnt:~$ configure
[edit]
ubnt@ubnt# show firewall modify
 modify balance {
     rule 10 {
         action modify
         destination {
             group {
                 address-group ADDRv4_eth0
             }
         }
         modify {
             table main
         }
     }
     rule 20 {
         action modify
         modify {
             lb-group G
         }
     }
 }
[edit]

As instructed by the ubnt forum I had to add another rule to exclude also from the load-balancing the LAN to LAN traffic.
See the updated tutorial from ubnt for this EdgeMAX – Dual WAN Load-Balance Feature

LAN-to-LAN Exclusion

From CLI do the following:

Lower the priority of the existing rules:


configure
edit firewall modify balance
rename rule 20 to rule 30
commit
edit firewall modify balance
rename rule 10 to rule 20
commit
save
exit


Add LAN-to LAN traffic exclusion from load balancing by adding a new modify rule that has a bigger priority than the load balancing rule.
Also create a group where all the sub-nets are added as members.


configure
set firewall group network-group LAN_NETS network 192.168.1.0/24
set firewall group network-group LAN_NETS network 192.168.2.0/24
set firewall group network-group LAN_NETS network 192.168.9.0/24
commit

set firewall modify balance rule 10 destination group network-group LAN_NETS
set firewall modify balance rule 10 action modify
set firewall modify balance rule 10 modify table main
commit


Note that 192.168.9.0/24 is added here because is going to be used later when we will add the openvpn site-to-site VPN.

After this the final setup looks like the following:

ubnt@ubnt:~$ configure
[edit]
ubnt@ubnt# show firewall modify
 modify balance {
     rule 10 {
         action modify
         destination {
             group {
                 network-group LAN_NETS
             }
         }
         modify {
             table main
         }
     }
     rule 20 {
         action modify
         destination {
             group {
                 address-group ADDRv4_eth0
             }
         }
         modify {
             table main
         }
     }
     rule 30 {
         action modify
         modify {
             lb-group G
         }
     }
 }
[edit]

STEP 3 EdgeRouter: OpenVPN site-to-site VPN
Another complication was added to the configuration when I added a second site to my network. This remote site has an EdgeRouter Lite so I decided to make a permanent OpenVpn site-to-site permanent VPN between sites
The setup is described in the following post: EdgeRouter: OpenVPN site-to-site VPN

STEP 4 VPN Exclusion (not yet implemented in the current firmware)
After STEP 3 was done I still had a strange issue. My VPN connection was dropping packets. I discovered that if I disabled the secondary WAN suddenly everything was OK. So I concluded that the cause of this unstable VPN was the load-balancing. In fact it was the same behaviour I noticed before STEP 2 was done when I was ping-ing between machines from different LANs.
Note that this is not yet implemented in the current firmware (1.8) but this is how I expected that should work as a solution. It was confirmed by ubiquity support that this is one of the solutions that is taken into account in a future firmware update.
See STEP 5 for the current partial solution.

From CLI do the following:

Lower the priority of the existing load-balance rule:


configure
edit firewall modify balance
rename rule 30 to rule 40
commit
save
exit


Add an exclusion rule for the vtun0 site-to-site VPN interface just before the load-balance rule.


configure
edit firewall modify balance
set rule 30 modify table main
set rule 30 destination group address-group ADDRv4_vtun0
commit
save
exit


The final configuration then looked like:

ubnt@ubnt# show firewall modify
 modify balance {
     rule 10 {
         action modify
         destination {
             group {
                 network-group LAN_NETS
             }
         }
         modify {
             table main
         }
     }
     rule 20 {
         action modify
         destination {
             group {
                 address-group ADDRv4_eth0
             }
         }
         modify {
             table main
         }
     }
     rule 30 {
         destination {
             group {
                 address-group ADDRv4_vtun0
             }
         }
         modify {
             table main
         }
     }
     rule 40 {
         action modify
         modify {
             lb-group G
         }
     }
 }
[edit]

Then I enabled the second WAN from eth1 on my main site.

Restarted the openVPN:

reset openvpn interface vtun0

Force the reload of the load-balancing configuration by killing the process. This is a trick from UBNT-stig 🙂

Hmm “WAN Out” is empty. I wonder if you ran into a bug I fixed the last week. Can you try doing “sudo pkill ubnt-util” from the CLI and then see if “WAN Out” gets filled in?

Check if load-balance is up and acting as fail-over:

ubnt@ubnt:~$ show load-balance status
Group G
  interface   : eth0
  carrier     : up
  status      : active
  gateway     : 84.117.103.1
  route table : 201
  weight      : 100%
  flows
      WAN Out : 44
      WAN In  : 52
    Local Out : 172

  interface   : eth1
  carrier     : up
  status      : failover
  gateway     : 192.168.0.1
  route table : 202
  weight      : 0%
  flows
      WAN Out : 0
      WAN In  : 0
    Local Out : 136

show openvpn status site-to-site

UBNT-stig wrote:

I think the problem is likely that load-balance uses a separate routing table for each WAN and those routing tables only include the default gateway.  Normally the way around that is to create your own routing table:


ubnt@ubnt# show protocols 
 static {
     interface-route 192.168.9.0/24 {
         next-hop-interface vtun0 {
         }
     }
     table 1 {
         interface-route 192.168.9.0/24 {
             next-hop-interface vtun0 {
             }
         }
         route 0.0.0.0/0 {
             next-hop  {
             }
         }
     }
 }
[edit]

And then tell load-balace to use that table.

set load-balance group G interface eth0 route table 1

Unfortunately your gateway is from dhcp and may change over time.  I've been thinking of ways to work around that similar to ADDRv4_eth0, but can't use firewall groups in the routing table.