Routing Troubleshooting
This document follows a structured method for troubleshooting route configuration-related connectivity issues on the CE (Customer Edge) devices. It is designed to be used step-by-step in order to solve the issue correctly.
Troubleshooting Steps
Issue: Route Configuration Not Working
Symptom
The routes are not configured as expected on the CE device. and connectivity tests (e.g., ping) fail to reach the target IP.
- Cloud
- UCI
- Run-Time
- Testing
- Log
Cloud Configuration verification
Access the CE Terminal
- Log in to the CE terminal with superuser privileges to perform diagnostic checks.
sudo su -
Check Route Configuration File
This command is used to review the configuration file of a CE device (for example, the file is shown below, although the actual location of the file may vary). The file and format will depend on the CE device's operating system and configuration method.
cat /tmp/last_config_response.json | jq .protocols.staticRoutes
The given one is just an example output; when this command is run, it will show something like this.
Example Response
[
{
"address": "50.50.50.1",
"netmask": "255.255.255.248",
"gatewayIp": "50.50.50.1",
"announce": false,
"routeId": "e3489fc76ac743bc8aace7cc55276da7",
"deviceId": "64cafee7f2366e57b7b0d141",
"deviceLabel": null,
"routeType": "user",
"interfaceName": "vti56_1_3",
"overlayIp": "100.100.1.6",
"priority": 0
},
{
"address": "192.168.254.0",
"netmask": "255.255.255.0",
"gatewayIp": "172.30.3.141",
"announce": true,
"routeId": "3758a055c59b4650aae8f947fe6ca821",
"deviceId": "64e353e7f2366e57b7b3e0ba",
"deviceLabel": null,
"routeType": "custom",
"interfaceName": "eth3",
"overlayIp": null,
"priority": 1
},
{
"address": null,
"netmask": null,
"gatewayIp": null,
"announce": true,
"routeId": "64cafee7f2366e57b7b0d141",
"deviceId": "64cafee7f2366e57b7b0d141",
"deviceLabel": "Device2",
"routeType": "peers",
"interfaceName": "vti56_1_3",
"overlayIp": "100.100.1.6",
"priority": 0
},
{
"address": null,
"netmask": null,
"gatewayIp": null,
"announce": true,
"routeId": "64d9cc4ff2366e57b7b2432c",
"deviceId": "64d9cc4ff2366e57b7b2432c",
"deviceLabel": "Device1",
"routeType": "peers",
"interfaceName": "vti56_1_3",
"overlayIp": "100.100.1.1",
"priority": 0
}
]
Carefully look at the contents of the file and identify any misconfigurations, typos, or omissions regarding the problematic routes.
Q:1 How to check the route configuration files of a CE device?
To check the route configuration files on a CE device, you must log in with superuser privileges and inspect the cloud‑pushed configuration stored on the device. The recommended method is: Access the CE terminal with root privileges :- sudo su - , 2. View the last applied route configuration :- cat /tmp/last_config_response.json | jq .protocols.staticRoutes. This command displays all static routes received from the cloud, including: Route address and netmask, Gateway IP, Interface name , Route type (user, custom, peers) , Priority, Overlay IP. By reviewing this JSON output, you can verify whether the CE received the correct route configuration and identify any missing or incorrect entries.
Q:2 Which places should be checked if gatewayIp is null during troubleshooting
1. Cloud configuration → Verify if the route was created with a valid gateway IP in the cloud UI. 2.Peers routes → Check if the route is auto‑generated by VPN/overlay; in that case, gatewayIp null is normal. 3. User/custom routes → If it’s a custom route, null gateway is wrong; update with a valid gateway IP. 4. Sync status → Confirm the CE device has received and applied the updated configuration.
UCI Configuration Verification
Verify Netwrok Configuration Details
Use the command uci show network | grep route to view the static routes configured on the CE device (such as a router). This command displays the UCI network configuration for a Route.
uci show network | grep route
The given one is just an example output; when this command is run, it will show something like this.
Example Response
network.746905bb4d2a41ec921b7405b9b2d19e=route
network.161c2013f1a44f5d8f0ff2086758bcb1=route
network.bffa3cfac3314a11b7465d00093a9e47=route
Q:1 Which command can be used for troubleshooting routes?
The command uci show network | grep route is very useful because it helps you: Confirm route entries received by the CE device, Identify missing or duplicate routes, Verify route IDs in the UCI system, Detect inconsistencies between cloud‑pushed routes and runtime configuration
Run time Configuration Verification
Verify Current Routes
IP route show is used to display the current routing table and confirm the presence and correctness of expected routes.
ip route show
- Analyze output. If missing, incorrect, or does not exist, continue on to the next step.
The ' ip route list' command displays the current routing table.
ip route list
The given one is just an example output; when this command is run, it will show something like this.
Example Response
27.61.151.50 via 27.115.36.65 dev eth0 proto static metric
1
50.50.50.0/29 nhid 440 via 100.100.1.2 dev br56 proto bgp m
etric 20
100.100.1.0/24 dev br56 proto kernel scope link src 100.100
.1.1
106.213.242.138 via 27.115.36.65 dev eth0 proto static metr
ic 1
140.206.96.166 via 27.115.36.65 dev eth0 proto static metri
c 1
172.30.1.0/24 dev eth5 proto kernel scope link src 172.30.1
.1 linkdown
172.30.3.0/24 nhid 444 via 100.100.1.3 dev br56 proto bgp metric 20
192.168.254.0/24 via 100.100.1.3 dev br56 proto static metric 1
223.238.226.155 via 27.115.36.65 dev eth0 proto static metric 1
Check Network Service Status
To check the status of the route network, run the following command.
/etc/init.d/network status
When this command is run, an output similar to the example below appears.
Example Response
running
To manually start or stop the network service, use the following commands.
/etc/init.d/network start
/etc/init.d/network stop
Q:1 Which command is used to see the route list?
The command is ip route list, which shows the routing table entries of the system.
Q:2 Where does “linkdown” appear and what does it mean?
The term “linkdown” appears in the output of the runtime routing table, typically when running: ip route show or ip route list. It is shown next to an interface—for example: 172.30.1.0/24 dev eth5 proto kernel scope link src 172.30.1.1 linkdown. What it means: “linkdown” indicates that the physical interface is not active. This usually happens when: The Ethernet cable is unplugged, The upstream device (switch/modem/router) is powered off, The interface is administratively disabled, There is a hardware or port failure. When an interface is linkdown, routes associated with that interface cannot forward traffic, which often causes ping failures or unreachable networks.
Test Verification
Use the ping command to test the connectivity to your target IP address.
ping <target-ip>
Replace target-ip with your destination IP. Successful pings mean that your route is active. IF there are no replies to your pings, skip to the next step.
Q:1 What to do if Ping reply is not received?
If you do not receive a ping reply, it means the CE device cannot reach the target IP. You should: 1. Verify the route exists Check the routing table using: ip route show. If the expected route is missing or incorrect, the CE will not know where to send the traffic. 2. Check interface status Ensure the interface used for the route is UP and not showing linkdown. 3. Confirm gateway reachability Try pinging the gateway IP. If the gateway is unreachable, the route cannot work. 4. Check UCI configuration Make sure the route is correctly configured in UCI: uci show network | grep route. 5. Review logs Look for routing or interface errors: logread | grep route, logread | grep eth. 6. Restart network service :- /etc/init.d/network restart. If the ping still fails, the issue is likely with the route configuration, gateway, or upstream network.
Q:2 What is Test Verification?
Test Verification is the step where you validate connectivity after configuring or troubleshooting routes. It confirms whether the CE device can reach the target IP using the expected route. By running: ping <target-ip> you check: Whether the route is active, Whether the next hop is reachable, Whether the CE is forwarding traffic correctly. If the ping succeeds, the routing configuration is working. If it fails, you continue with deeper troubleshooting steps.
Log Verification
Check Logs for Errors
- Review system logs for any errors related to routing or connectivity:
logread | grep route
The given one is just an example output; when this command is run, it will show something like this.
Example Response:
Jun 24 06:47:04 manual-testing dnsmasq-dhcp[1]: IPv6 router advertisement enabled
Jun 24 06:47:05 manual-testing dnsmasq-dhcp[1]: router advertisement on eth3
Jun 24 06:47:05 manual-testing dnsmasq-dhcp[1]: IPv6 router advertisement enabled
Jun 24 06:47:08 manual-testing dnsmasq-dhcp[1]: router advertisement on eth3
Jun 24 06:47:08 manual-testing dnsmasq-dhcp[1]: IPv6 router advertisement enabled
Q:1 Why should you check the log?
Checking the logs helps you identify hidden issues that are not visible in the routing table or configuration files. Logs reveal: Errors during route creation or application, Interface‑related problems (e.g., link flaps, DHCP renewals, IPv6 advertisements), Conflicts between routes or services, System‑level events that may affect routing. In short, logs provide real‑time diagnostic information that helps pinpoint why a route is not working as expected.
Q:2 Which command should be used for route related errors?
To check for route‑related errors, use: logread | grep route. This filters the system logs and shows only entries related to routing