Troubleshooting Multi-WAN Load Balancing
This document gives troubleshooting steps that are specific for load balancing failures, focusing strictly on issues of traffic not properly distributed across various WAN connections. The principal symptom is one where all your outbound traffic, in a multiple-WAN set-up, just uses one single WAN interface; this defeats the purpose of doing load balancing where the idea was to distribute it across multiple WANs for high bandwidth, redundancy, and other performance improvements. These steps will help you verify the configuration, diagnose routing problems, and ensure the multi-WAN service is functioning correctly.
Troubleshooting Steps
Verify Current Load Balancing Configuration:
Log into the CE device and gain root access:
sudo su -
Display the last applied load balancing configuration in a readable format
cat /tmp/last_config_response.json | jq .multiWanV2
Example Response
root@manual-test:/tmp# cat last_config_response.json | jq .multiWanV2
{
"enable": true,
"mode": "LOAD_BALANCE",
"notificationEmails": [],
"wanInterfaces": null,
"wanInterfacesConfig": {
"pppoe0": {
"interfaceName": "pppoe0",
"targetIps": [
"8.8.8.8",
"4.2.2.2"
],
"failureInterval": 5,
"recoveryInterval": 5,
"pingInterval": 5,
"pingTimeout": 2,
"multiWANMetric": 3,
"multiWANWeight": 2,
"enable": false
},
"eth1": {
"interfaceName": "eth1",
"targetIps": [
"8.8.8.8",
"4.2.2.2"
],
"failureInterval": 5,
"recoveryInterval": 5,
"pingInterval": 5,
"pingTimeout": 2,
"multiWANMetric": 1,
"multiWANWeight": 2,
"enable": true
},
"eth0": {
"interfaceName": "eth0",
"targetIps": [
"8.8.8.8",
"4.2.2.2"
],
"failureInterval": 5,
"recoveryInterval": 5,
"pingInterval": 5,
"pingTimeout": 2,
"multiWANMetric": 2,
"multiWANWeight": 2,
"enable": true
},
"wlm0": {
"interfaceName": "wlm0",
"targetIps": [
"8.8.8.8",
"4.2.2.2"
],
"failureInterval": 5,
"recoveryInterval": 5,
"pingInterval": 5,
"pingTimeout": 2,
"multiWANMetric": 2,
"multiWANWeight": 2,
"enable": false
}
}
}
Please carefully check that all the weight values assigned to each WAN interface are correctly set. These weights represent the portion of traffic routed through each connection, and incorrect weights can lead to imbalanced traffic distribution.
The network configuration file also plays a role:
cat /etc/config/network
Example Response
root@manual-test:/tmp# cat /etc/config/network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals
option packet_steering '1'
config interface 'eth0'
option device 'eth0'
option proto 'dhcp'
option metric '1'
option ip4table '1'
option peerdns '0'
option default_wan '1'
list dns '8.8.8.8'
list dns '4.2.2.2'
option disabled '0'
option mtu '1500'
config interface 'eth2'
option device 'eth2'
option disabled '0'
option mtu '1500'
option proto 'static'
option ipaddr '172.1.30.3'
option netmask '255.255.255.0'
config rule
option priority '901'
option lookup 'main'
config interface 'eth1'
option disabled '0'
option device 'eth1'
option proto 'dhcp'
option metric '2'
option ip4table '2'
option mtu '1500'
config interface 'wlm0'
option disabled '1'
option proto '3g'
option pppname 'wlm0'
option device 'ttyUSB0'
option apn 'comgt'
option ipv6 '0'
option delegate '0'
option metric '3'
option ip4table '3'
config route '4f8253ad3b144cfca9f81e3223664117'
option target '172.1.30.3'
option netmask '255.255.255.0'
option gateway '172.1.30.1'
option metric '0'
option proto 'static'
option interface 'eth2'
The file will define the network interface and its associated setting, which would be important to how muli-WAN would act. Ensure the interface used in the multi-WAN would act. Ensure the interface used in the multi-WAN configuration is correctly defined here.
The details of the mwan3 configuration can be checked in the following files.
/etc/config/mwan3
Example Response
root@manual-test:~# cat /etc/config/mwan3
config globals 'globals'
option mmx_mask '0x3F00'
option local_source 'lan'
option mode 'LOAD_BALANCE'
option enabled '1'
config rule 'DEFAULT_HTTPS'
option family 'ipv4'
option sticky '1'
option proto 'tcp'
option dest_ip '0.0.0.0/0'
option dest_port '443'
option use_policy 'LOAD_BALANCE'
config rule 'DEFAULT_ANY'
option family 'ipv4'
option dest_ip '0.0.0.0/0'
option use_policy 'LOAD_BALANCE'
config interface 'eth1'
option enabled '1'
list track_ip '8.8.8.8'
list track_ip '4.2.2.2'
option interval '5'
option timeout '2'
option failure_interval '5'
option recovery_interval '5'
option down '1'
option up '3'
option initial_state 'online'
option track_method 'ping'
option reliability '1'
option count '1'
option size '56'
option max_ttl '60'
option check_quality '0'
config member 'eth1_m1_w1'
option interface 'eth1'
option metric '1'
option weight '1'
config policy 'FAIL_OVER'
list use_member 'eth1_m1_w1'
list use_member 'eth0_m2_w1'
config interface 'eth0'
option enabled '1'
list track_ip '8.8.8.8'
list track_ip '4.2.2.2'
option interval '5'
option timeout '2'
option failure_interval '5'
option recovery_interval '5'
option down '1'
option up '3'
option initial_state 'online'
option track_method 'ping'
option reliability '1'
option count '1'
option size '56'
option max_ttl '60'
option check_quality '0'
config member 'eth0_m2_w1'
option interface 'eth0'
option metric '2'
option weight '1'
config interface 'wlm0'
option enabled '0'
list track_ip '8.8.8.8'
list track_ip '4.2.2.2'
option interval '5'
option timeout '2'
option failure_interval '5'
option recovery_interval '5'
option down '1'
option up '3'
option initial_state 'online'
option track_method 'ping'
option reliability '1'
option count '1'
option size '56'
option max_ttl '60'
option check_quality '0'
config member 'eth1_m1_w2'
option interface 'eth1'
option metric '1'
option weight '2'
config policy 'LOAD_BALANCE'
list use_member 'eth1_m1_w2'
list use_member 'eth0_m1_w2'
config member 'eth0_m1_w2'
option interface 'eth0'
option metric '1'
option weight '2'
This file defines the multi-WAN rules, interface, and policies. Verify:
- enabled '1' in the globals section.
- Right measurement and weight value for each interface.
- Your rules usage policy matches your defined policies.
- The track_ip addresses are reachable from each WAN interface.
Validate WAN Selection with Traceroute:
Utilize the traceroute command to identify what actual path traffic travels on the way to the destination. This is helpful to verify which WAN interface is currently being used by way of sending out requests. Perform a traceroute to some known external IP address, and replace x.x.x.x with an appropriate IP, such as a public DNS server.
traceroute -n x.x.x.x
The WAN interface the traceroute keeps showing up for is only one, even after configuring the load balancing, which indicates a problem in the load balance rules or routing configuration.
System Log Inspection:
The system logs are always a good source of information in the operation of the Multi-WAN service. Errors related to interface configuration or routing issues are some of the problems that could be affecting the load balancing by reviewing the logs.
Check the system logs for errors related to Mwan3
logread
Investigate the log output for error messages, warnings, or unusual activity. Such logs can highlight specific issues with load balancing.
mwan3 Service Status and Control
Check the status of the mwan3 service:
/etc/init.d/mwan3 status
Example Response
root@manual-test:/tmp# /etc/init.d/mwan3 status
running
This will output the status of the mwan3 service, whether it is running, and any details that may have been encountered. Look for error or failure indications.
If the service is not running, start it:
/etc/init.d/mwan3 start
Example Response:
root@manual-test:/tmp# /etc/init.d/mwan3 start
running
This command starts the mwan3 service, thus beginning the process of load balancing based on configuration.
For troubleshooting, you can stop and restart the service:
/etc/init.d/mwan3 stop
/etc/init.d/mwan3 restart
Example Response
- stop
root@manual-test:/tmp# /etc/init.d/mwan3 stop
running
- Restart
root@manual-test:/tmp# /etc/init.d/mwan3 restart
running
Stopping and restarting the mwan3 service sometimes solves transient issues or applies configuration changes.
mwan3 Command-Line Interface
The mwan3 command provides additional information and control:
mwan3
Example Response
root@manual-test:~# mwan3
Syntax: mwan3 [command]
Available commands:
start Load iptables rules, ip rules and ip routes
stop Unload iptables rules, ip rules and ip routes
restart Reload iptables rules, ip rules and ip routes
ifup <iface> Load rules and routes for specific interface
ifdown <iface> Unload rules and routes for specific interface
interfaces Show interfaces status
policies Show currently active policy
connected Show directly connected networks
rules Show active rules
status Show all status
internal <ipv4|ipv6> Show internal configuration <default: ipv4>
use <iface> <cmd> Run a command bound to <iface> and avoid mwan3 rules
This command with no arguments shows helpful information about the mwan3 command and all of its available subcommands.
Check the current mwan3 status:
mwan3 status
Example Response:
root@manual-test:~# mwan3 status
Interface status:
interface eth1 is online 00h:25m:17s, uptime 00h:25m:24s and tracking is active
interface eth0 is online 00h:25m:25s, uptime 00h:25m:37s and tracking is active
interface wlm0 is offline and tracking is down
Current ipv4 policies:
FAIL_OVER:
eth1 (100%)
LOAD_BALANCE:
eth0 (50%)
eth1 (50%)
Current ipv6 policies:
FAIL_OVER:
unreachable
LOAD_BALANCE:
unreachable
Directly connected ipv4 networks:
172.1.30.255
192.168.1.0
192.168.1.255
172.1.30.3
172.30.4.165
127.0.0.0/8
192.168.1.4
127.0.0.0
172.1.30.0
172.30.4.0
172.30.4.255
127.0.0.1
172.1.30.0/24
224.0.0.0/3
127.255.255.255
Directly connected ipv6 networks:
Active ipv4 user rules:
185 11100 S DEFAULT_HTTPS tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 443
163 12612 - LOAD_BALANCE all -- * * 0.0.0.0/0 0.0.0.0/0
Active ipv6 user rules:
This provides a detailed overview of the mwan3 service's status, including interface states, connection statuses, and routing information. It can be more informative than the basic service status check.
Test connectivity through a specific interface using ping:
mwan3 use eth0 ping -4 google.com
Example Response
root@manual-test:~# mwan3 use eth0 ping -4 google.com
could not find family for eth0. Using ipv4.
Running 'ping -4 google.com' with DEVICE=eth0 SRCIP=192.168.1.4 FWMARK=0x3f00 FAMILY=ipv4
PING google.com (142.250.76.206): 56 data bytes
64 bytes from 142.250.76.206: seq=0 ttl=60 time=14.012 ms
64 bytes from 142.250.76.206: seq=1 ttl=60 time=13.390 ms
64 bytes from 142.250.76.206: seq=2 ttl=60 time=14.927 ms
64 bytes from 142.250.76.206: seq=3 ttl=60 time=14.979 ms
64 bytes from 142.250.76.206: seq=4 ttl=60 time=12.831 ms
64 bytes from 142.250.76.206: seq=5 ttl=60 time=14.128 ms
^C
--- google.com ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 12.831/14.044/14.979 ms
This command forces the ping to use eth0 and isolates WAN link connectivity issues. Examine the output for errors or unexpected behavior. If a ping fails on an interface, even if all the configurations appear correct, there may be a WAN connection issue.