Single arm troubleshooting
This is usually a single-arm configuration, where CE is connected to PE onone interface. Troubleshooting usually involves verification of gateway and networking interface correct configurations, routing, and communication toward hiCLOUDS. Common symptoms include failure to reach destinations beyond the local network, connectivity drops or registration failures with hiCLOUDS. This guide will walk you through the essential troubleshooting steps.
Troubleshooting Steps
- Cloud
- UCI
- Run-Time
- Testing
- Log
Cloud Configuration Verification
Access the CE Terminal
Log in to the CE (Customer Edge) terminal and gain superuser privileges:
sudo su -
Checking Gateway Configuration:
This command is used to check the gateway configuration. For example, when the WAN/Internet connection is not working and the last received configuration of the CE needs to be verified, this command can be used.
cat last_config_response.json | jq .gatewayConfig
The given one is just an example output; when this command is run, it will show something like this.
Example Response
[
{
"mode": "CE_PEGW",
"translationAddress": "masquerade",
"ceDeviceIp": "100.100.0.9/30",
"sourceSubnet": [
"192.168.1.0/24",
"192.168.10.0/24"
],
"zone": "in",
"localSeqId": 2,
"peDeviceIp": "100.100.0.10/30",
"id": 1,
"gatewayMode": "GLOBAL",
"singleArm": true,
"cid": 38
}
]
This command will extract the gateway configuration from last_config_response.json using jq, a JSON processor. Look at the output to make sure the gateway IP address and other pertinent settings are set up correctly.
Q:1 What information does the gateway configuration output provide?
Q:2 What should I do if the configuration does not match the expected settings?
Q:3 What is jq and why is it used here?
UCI Configuration Verification
uci show network | grep CGW is used to check the name, IP, gateway etc. details of CGW interface in Single Arm structure.
uci show network | grep cgw
The given one is just an example output; when this command is run, it will show something like this.
Example Response
network.br38_cgw=interface
network.br38_cgw.ipaddr='100.100.0.93'
network.br38_cgw.netmask='255.255.255.252'
network.br38_cgw.device='@br38'
network.br38_cgw.proto='static'
network.br38_cgw.metric='3'
network.br38_cgw.ip4table='3'
network.br38_cgw.gateway='100.100.0.94'
Checking CGW Firewall Rules
The uci show firewall | grep CGW command is used to check the CGW firewall zone and its rules
uci show firewall | grep CGW
The given one is just an example output; when this command is run, it will show something like this.
Example Response
firewall.CGW_ALLOWED_IPADDRESS=ipset
firewall.CGW_ALLOWED_IPADDRESS.name='CGW_ALLOWED_IPADDRESS'
firewall.CGW_ALLOWED_IPADDRESS.match='dst_net'
firewall.CGW_ALLOWED_IPADDRESS.storage='hash'
firewall.CGW_ALLOWED_IPADDRESS.enabled='1'
firewall.CGW_BLOCKED_IPADDRESS=ipset
firewall.CGW_BLOCKED_IPADDRESS.name='CGW_BLOCKED_IPADDRESS'
firewall.CGW_BLOCKED_IPADDRESS.match='dst_net'
firewall.CGW_BLOCKED_IPADDRESS.storage='hash'
firewall.CGW_BLOCKED_IPADDRESS.enabled='1'
firewall.CGW_ALLOWED_IPSUBNETS=ipset
firewall.CGW_ALLOWED_IPSUBNETS.name='CGW_ALLOWED_IPSUBNETS'
firewall.CGW_ALLOWED_IPSUBNETS.match='dst_net'
firewall.CGW_ALLOWED_IPSUBNETS.storage='hash'
firewall.CGW_ALLOWED_IPSUBNETS.enabled='1'
firewall.CGW_BLOCKED_IPSUBNETS=ipset
firewall.CGW_BLOCKED_IPSUBNETS.name='CGW_BLOCKED_IPSUBNETS'
firewall.CGW_BLOCKED_IPSUBNETS.match='dst_net'
firewall.CGW_BLOCKED_IPSUBNETS.storage='hash'
firewall.CGW_BLOCKED_IPSUBNETS.enabled='1'
firewall.CGW_ALLOWED_DOMAIN=ipset
firewall.CGW_ALLOWED_DOMAIN.name='CGW_ALLOWED_DOMAIN'
firewall.CGW_ALLOWED_DOMAIN.match='dst_net'
firewall.CGW_ALLOWED_DOMAIN.storage='hash'
firewall.CGW_ALLOWED_DOMAIN.enabled='1'
firewall.CGW_BLOCKED_DOMAIN=ipset
firewall.CGW_BLOCKED_DOMAIN.name='CGW_BLOCKED_DOMAIN'
firewall.CGW_BLOCKED_DOMAIN.match='dst_net'
firewall.CGW_BLOCKED_DOMAIN.storage='hash'
firewall.CGW_BLOCKED_DOMAIN.enabled='1'
firewall.CGW_ALLOWED=ipset
firewall.CGW_ALLOWED.name='CGW_ALLOWED'
firewall.CGW_ALLOWED.match='dst_set'
firewall.CGW_ALLOWED.storage='list'
firewall.CGW_ALLOWED.enabled='1'
firewall.CGW_ALLOWED.entry='CGW_ALLOWED_IPADDRESS' 'CGW_ALLOWED_IPSUBNETS' 'CGW_ALLOWED_DOMAIN'
firewall.CGW_BLOCKED=ipset
firewall.CGW_BLOCKED.name='CGW_BLOCKED'
firewall.CGW_BLOCKED.match='dst_set'
firewall.CGW_BLOCKED.storage='list'
firewall.CGW_BLOCKED.enabled='1'
firewall.CGW_BLOCKED.entry='CGW_BLOCKED_IPADDRESS' 'CGW_BLOCKED_IPSUBNETS' 'CGW_BLOCKED_DOMAIN'
Q:1 What is the purpose of CGW_ALLOWED and CGW_BLOCKED ipsets?
Q:2 What does each parameter in the CGW network output mean?
Run time Configuration Verification
Check openvpn Service Status
To check the status of the openvpn network, run the following command.
/etc/init.d/openvpn status
When this command is run, an output similar to the example below appears.
Example Response
running
To manually start or stop the network service, use the following commands.
/etc/init.d/openvpn start
/etc/init.d/openvpn stop
Q:1 Does checking OpenVPN service status instantly solve network issues?
Q:2 Does checking OpenVPN service status instantly solve network issues?
Q:3 How to verify the status after running the OpenVPN service start/stop command?
Testing Verification
Verify Traffic Routing Based on CGW Mode
Global Mode
In global mode, all traffic from the CE device should be routed through the CGW except for traffic destined for the local domain. To test this, perform the following traceroutes:
traceroute -n x.x.x.x
Expected Result: The traceroute to the global IP address should show hops traversing the CGW. The traceroute to the local DNS server IP should not traverse the CGW.
Problem: The configuration is correct if traffic to the global IP address does not pass through the CGW or if traffic from the local domain does pass through the CGW.
Full Mode
In full mode, all traffic by global, selective (if set up), and local domain should be sent over the CGW. Use these commands:
traceroute -n x.x.x.x
Expected Result: Both traceroutes should contain hops crossing over the CGW.
Problem: If any of the traffic passes the CGW, either the mode selection or routing rule needs adjustment.
Selective Mode
Selective mode routes traffic according to defined rules, such as allowed/blocked domains, IPs, and subnets. To test, and execute traceroutes to both allowed and blocked destinations:
traceroute -n x.x.x.x
Expected Result: Traffic to the allowed destinations must be routed over the CGW. Traffic to the blocked destinations should not traverse the CGW.
Problem: If blocked IPs/domains can be reached via the CGW, or allowed traffic is routed around the CGW, then the filtering rules need to be investigated. Selective mode relies on dnsmasq for its functionality.
Q:1 How does the traceroute command work?
Q:2 How to check the current status of the OpenVPN service?
Log Verification
Check System Logs
Run the following command to check the system log for any CGW-related errors or warnings.
logread | grep cgw
Example Response
Jun 24 07:00:01 manual-testing netifd: Interface 'br38_cgw' is enabled
Jun 24 07:00:01 manual-testing netifd: Interface 'br38_cgw' has link connectivity
Jun 24 07:00:01 manual-testing netifd: Interface 'br38_cgw' is setting up now
Jun 24 07:00:01 manual-testing netifd: Interface 'br38_cgw' is now up
Jun 24 07:00:02 manual-testing mwan3-hotplug[19424]: hotplug called on br38_cgw before mwan3 has been set up
Jun 24 07:00:05 manual-testing firewall: Reloading firewall due to ifup of br38_cgw (br38)
Jun 24 07:00:07 manual-testing firewall: Reloading network due to ifup of br38_cgw (br38)
Jun 24 07:00:10 manual-testing mwan3-hotplug[21818]: Execute ifup event on interface br38_cgw (br38)
Jun 24 07:00:13 manual-testing mwan3track[22215]: Check (ping) success for target "100.100.0.94" on interface br38_cgw (br38). Current score: 0
Jun 24 07:00:13 manual-testing mwan3track[22215]: Interface br38_cgw (br38) is connecting
Jun 24 07:00:19 manual-testing mwan3track[22215]: Check (ping) success for target "100.100.0.94" on interface br38_cgw (br38). Current score: 1
Jun 24 07:00:25 manual-testing mwan3track[22215]: Check (ping) success for target "100.100.0.94" on interface br38_cgw (br38). Current score: 2
Jun 24 07:00:25 manual-testing mwan3track[22215]: Interface br38_cgw (br38) is online
Jun 24 07:00:25 manual-testing mwan3-hotplug[23371]: Execute connected event on interface br38_cgw (br38)
Jun 24 07:00:53 manual-testing mwan3track[22215]: Stopping mwan3track for interface "br38_cgw". Status was "online"
Jun 24 07:00:54 manual-testing mwan3-hotplug[28815]: Execute ifup event on interface br38_cgw (br38)
Jun 24 07:00:56 manual-testing mwan3track[29193]: Check (ping) success for target "100.100.0.94" on interface br38_cgw (br38). Current score: 0
Jun 24 07:00:56 manual-testing mwan3track[29193]: Interface br38_cgw (br38) is connecting
Jun 24 07:01:02 manual-testing mwan3track[29193]: Check (ping) success for target "100.100.0.94" on interface br38_cgw (br38). Current score: 1
Jun 24 07:01:08 manual-testing mwan3track[29193]: Check (ping) success for target "100.100.0.94" on interface br38_cgw (br38). Current score: 2
Jun 24 07:01:08 manual-testing mwan3track[29193]: Interface br38_cgw (br38) is online
Jun 24 07:01:08 manual-testing mwan3-hotplug[30395]: Execute connected event on interface br38_cgw (br38)
Analyze logs to identify problems such as policy misconfigurations, connectivity errors, or authentication failures.