DNS Troubleshooting
This document outlines the steps required to troubleshoot DNS resolution issues on CE (Customer Edge) devices. It covers accessing the terminal, examining configuration files, verifying network settings, and monitoring system logs.
Troubleshooting steps
- Cloud
- UCI
- Run-Time
- Testing
- Log
Cloud Configuration Verification
Access the CE Terminal
Before starting the troubleshooting process, ensure the necessary permissions are in place to run the commands. To do this, run the following command.
sudo su -
Verify the Last Applied Configuration
This command is used to view information about the last successfully applied DNS configuration. Here, this command is used to retrieve the DNS server settings information and help identify the problem from the stored configuration response JSON file.
cat /tmp/last_config_response.json | jq '.service.dns'
This command will display the contents of the /tmp/last_config_response.json file. Carefully inspect the output and look for DNS server IP addresses or errors.
The configuration done on the cloud will be shown here. The output provided here is only for example purposes and displays the currently applied configuration
The given one is just an example output; when this command is run, it will show something like this.
Example Result:
{
"cacheSize": 8000,
"disable": false,
"listenAddress": [
"0.0.0.0"
],
"listenOn": [],
"domains": [],
"nameserver": [
"117.186.234.100",
"103.78.41.247"
],
"allowFrom": [
{
"address": "0.0.0.0",
"netmask": "0.0.0.0",
"cidrSignature": "0.0.0.0/0",
"subnet": "0.0.0.0"
}
]
}
In the example above, nameserver shows the configured DNS server IP addresses.
Q:1 Does this command show real-time changes in the cloud?
No. The command: cat /tmp/last_config_response.json | jq '.service.dns'. does not show real-time changes in the cloud. Instead, it displays the last successfully applied DNS configuration that was stored locally on the CE device. It reflects the configuration pushed from the cloud at the time of the last update. Any new changes made in the cloud will only appear here after they have been applied and synced to the device. This makes it useful for verifying what configuration is currently active, but not for monitoring live updates
Q:2 Which command should be used to verify the last applied configuration?
To verify the last applied DNS configuration, use: cat /tmp/last_config_response.json | jq '.service.dns'. This command reads the JSON file /tmp/last_config_response.json and filters the DNS section. It shows details such as cache size, listen address, nameservers, and allowed sources. Reviewing this output helps confirm whether the DNS settings were applied correctly and highlights any misconfigurations.
UCI Configuration Verification
This section focuses on verifying the DNS configuration stored in the UCI system.
Verify Network Configuration Details
The command uci show | grep dns is used to check the static DNS entries set in the network configuration file. This command helps verify the manually configured DNS servers on the Ethernet (LAN/WAN) interface.
uci show | grep dns
This example output shows the entries for the manually configured DNS on the (LAN/WAN) interface.
Example Result:
network .eth0 .peerdns='0'
network.eth0.dns='8.8.8.8' '4.2.2.2'
Q:1 What does the uci show | grep dns command do?
The command: uci show | grep dns. searches the UCI (Unified Configuration Interface) system for DNS-related entries. It specifically filters and displays any static DNS server settings configured on the CE device’s network interfaces (LAN/WAN). Example output: network.eth0.peerdns='0', network.eth0.dns='8.8.8.8' '4.2.2.2' This shows that DNS servers 8.8.8.8 and 4.2.2.2 are manually set for the eth0 interface, and peer DNS (from ISP) is disabled.
Run time Configuration Verification
Check dnsmasq Service Status
The command is used to view the dnsmasq services running on DNS.
/etc/init.d/dnsmasq status
This command is used to view dnsmasq services that stop and start on DNS.
/etc/init.d/dnsmasq start
/etc/init.d/dnsmasq stop
Q:1 When to use the service start/stop command?
Start (/etc/init.d/dnsmasq start) Use this when the dnsmasq service is not running or has been stopped. Starting the service enables DNS resolution and DHCP functionality on the CE device. Stop (/etc/init.d/dnsmasq stop) Use this when you need to temporarily disable DNS services, apply configuration changes, or troubleshoot issues by restarting the service. In practice, the start/stop commands are used during troubleshooting, configuration updates, or when dnsmasq needs to be reset to clear errors and reload settings
Q:2 What information is provided by the service status command?
The command: /etc/init.d/dnsmasq status. provides the current operational state of the dnsmasq service. It shows whether the service is running or stopped. Confirms if dnsmasq is actively handling DNS queries and DHCP assignments. May also indicate errors or warnings if the service failed to start properly. This status check is essential for verifying that the DNS service is active and functioning as expected on the CE device.
Testing Verification
This section provides steps for testing DNS resolution and inspecting system logs for further insights.
Test DNS Resolution with nslookup
The nslookup google.com command is used to check whether the IP address of a hostname is resolved by querying the DNS server.
nslookup google.com
The output shown here is an example. When the command is executed, output similar to this is displayed.
Example Response
Server: 127.0.0.1
Address: 127.0.0.1:53
Non-authoritative answer:
Name: google.com
Address: 142.251.42.78
Non-authoritative answer:
Name: google.com
Address: 2404:6800:4009:803::200e
View Current DNS Nameserver Configuration
The cat /etc/resolv.conf command displays the DNS nameserver configuration. It shows which DNS servers are currently being used by the system.
cat /etc/resolv.conf
Example Response
The output shown here is an example. When the command is executed, output similar to this is displayed.
search lan
nameserver 127.0.0.1
nameserver ::1
Q:1 Which command should be used to verify DNS resolution?
To verify DNS resolution, use the nslookup command: nslookup google.com This command queries the configured DNS server to resolve the hostname (google.com) into its IP address. The output shows: The DNS server being queried (e.g., 127.0.0.1:53). The resolved IP addresses (IPv4 and/or IPv6). If the command successfully returns IP addresses, DNS resolution is working. If it fails, it indicates a problem with the DNS configuration, connectivity, or the upstream nameserver.
Log Verification
Checking logs can help you diagnose specific issues, such as failed authentication attempts or service errors.
System Log Inspection
Check for any DNS-related errors or warnings by running this command to inspect the system logs:
logread | grep dns
The given one is just an example output; when this command is run, it will show something like this.
Exmaple Response
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: started, version 2.90 cachesize 1000
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: DNS service limited to local subnets
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: UBus support enabled: connected to system bus
Jun 30 09:33:29 hiCLOUDS dnsmasq-dhcp[1]: DHCP, IP range 192.168.254.101 -- 192.168.254.200, lease time 12h
Jun 30 09:33:29 hiCLOUDS dnsmasq-dhcp[1]: DHCPv6, IP range ::1000 -- ::ffff, lease time 12h, template for eth3
Jun 30 09:33:29 hiCLOUDS dnsmasq-dhcp[1]: DHCPv4-derived IPv6 names on eth3
Jun 30 09:33:29 hiCLOUDS dnsmasq-dhcp[1]: router advertisement on eth3
Jun 30 09:33:29 hiCLOUDS dnsmasq-dhcp[1]: IPv6 router advertisement enabled
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for test
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for onion
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for localhost
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for local
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for invalid
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for bind
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for lan
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: reading /tmp/resolv.conf.d/resolv.conf.auto
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using nameserver 8.8.8.8#53
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using nameserver 4.2.2.2#53
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for test
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for onion
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for localhost
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for local
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for invalid
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for bind
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: using only locally-known addresses for lan
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: read /etc/hosts - 12 names
Jun 30 09:33:29 hiCLOUDS dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 2 names
Jun 30 09:33:29 hiCLOUDS dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: exiting on receipt of SIGTERM
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: started, version 2.90 cachesize 1000
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: DNS service limited to local subnets
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: UBus support enabled: connected to system bus
Jun 30 09:33:30 hiCLOUDS dnsmasq-dhcp[1]: DHCP, IP range 192.168.254.101 -- 192.168.254.200, lease time 12h
Jun 30 09:33:30 hiCLOUDS dnsmasq-dhcp[1]: DHCPv6, IP range ::1000 -- ::ffff, lease time 12h, template for eth3
Jun 30 09:33:30 hiCLOUDS dnsmasq-dhcp[1]: DHCPv4-derived IPv6 names on eth3
Jun 30 09:33:30 hiCLOUDS dnsmasq-dhcp[1]: router advertisement on eth3
Jun 30 09:33:30 hiCLOUDS dnsmasq-dhcp[1]: IPv6 router advertisement enabled
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for test
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for onion
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for localhost
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for local
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for invalid
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for bind
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for lan
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: reading /tmp/resolv.conf.d/resolv.conf.auto
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using nameserver 8.8.8.8#53
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using nameserver 4.2.2.2#53
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for test
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for onion
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for localhost
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for local
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for invalid
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for bind
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: using only locally-known addresses for lan
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: read /etc/hosts - 12 names
Jun 30 09:33:30 hiCLOUDS dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 2 names
Jun 30 09:33:30 hiCLOUDS dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
Jun 30 10:46:19 hiCLOUDS hiclouds_config.sh[7704]: configuring dns
Jun 30 10:46:26 hiCLOUDS hiclouds_config.sh[7704]: restarting dnsmasq
Jun 30 10:46:27 hiCLOUDS dnsmasq[1]: exiting on receipt of SIGTERM
Jun 30 10:46:32 hiCLOUDS dnsmasq[1]: started, version 2.90 cachesize 8000
Jun 30 10:46:32 hiCLOUDS dnsmasq[1]: DNS service limited to local subnets
Jun 30 10:46:32 hiCLOUDS dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
Jun 30 10:46:32 hiCLOUDS dnsmasq[1]: UBus support enabled: connected to system bus
Jun 30 10:46:32 hiCLOUDS dnsmasq-dhcp[1]: DHCP, IP range 192.168.254.101 -- 192.168.254.200, lease time 12h
System logs are invaluable for identifying errors, warnings, and informational messages that can shed light on DNS resolution problems. The exact location and format of system logs can vary across different CE device models and firmware versions.
Q:1 What is dnsmasq and what does it mean in the logs?
dnsmasq is a lightweight service that provides DNS caching and DHCP functionality on CE devices. In the logs, entries related to dnsmasq indicate: Startup messages – showing the version, cache size, and compile-time options. DNS activity – which nameservers are being used (e.g., 8.8.8.8#53, 4.2.2.2#53). Scope of service – whether DNS is limited to local subnets or extended. DHCP activity – IP ranges, lease times, and router advertisements. Events – such as service restarts (exiting on receipt of SIGTERM) or configuration reloads. In short, dnsmasq log entries confirm that the device’s DNS and DHCP services are running, show which servers are being queried, and highlight any configuration or operational changes.