SSH Troubelshooting
This document provides steps to troubleshoot SSH Proxy related issues. When users are unable to connect to SSH Proxy, possible causes include speed issues, resource access failure, or complete connectivity failure. Troubleshooting requires checking the configuration on both CE and PE and checking the connection status and logs.
Troubleshooting Steps
- Cloud
- UCI
- Run-Time
- Testing
- Log
Cloud Configuration Verification
This section details how to check the configurations made in the cloud. The commands below can be used to verify the cloud configuration.
Check Last Cloud Config Response for SSH
This command checks the latest configuration in JSON format and highlights any lines related to SSH. It's helpful for fixing sync issues between the cloud and a CE device.
cat /tmp/last_config_response.json | jq | grep -i socks
The given one is just an example output; when this command is run, it will show something like this.
Example Response
"socksProxy": true,
"socksProxyType": "SSH",
"socksProxyPrivateKey": "LS0tLS1CRUdJTiBPUEVOU1NIIFBSSVZBVEUgS0VZLS0tLS0KYjNCbGJuTnphQzFyWlhrdGRqRUFBQUFBQkc1dmJtVUFBQUFFYm05dVpRQUFBQUFBQUFBQkFBQUJGd0FBQUFkemMyZ3RjbgpOaEFBQUFBd0VBQVFBQUFRRUFyRUx5T20xUDltZ21XNzFsbWZWNTlhdS9GRERmOXRSQk5yZ2hmY0RpTlgxK0JVcGpZNGU5CjdNRHBraUNqU0QvVXhNUnVlc3ZPckc4b1FNeEh2YSt5TmRtT1orRmNzSktsUlVoSTUvT0JJdGFhN3IrQmJYMmt6amEyWjYKVUpNZ3ozeEEzcDZ0NytmOXBRYWdVd3JJNmpRTzZvSC9DQkl3dkdhZXB3K1lhb216TUdWNkJ6aGN4bTBCSVFpeXRwRjl2RApDYWs4VmNhZ0MzdndDMnFTOXp0RXJjZEEzdG4yd0hMTTN2ei9ldllzS1d0MXdJYm9BZjNCWmY3a1RFcmZ0VXpHOWxRM010CjFMNU1SVUk4QUdOOUhLNVRvRzc5UWNTTEdpN0MyVDA5UkJlWU4yeHFhbnRXV2RaSWlObDFYOGtDaE1xc1A0L0hpWDA2UHUKeFVuUVovdkI3UUFBQThobk9qVUVaem8xQkFBQUFBZHpjMmd0Y25OaEFBQUJBUUNzUXZJNmJVLzJhQ1pidldXWjlYbjFxNwo4VU1OLzIxRUUydUNGOXdPSTFmWDRGU21Oamg3M3N3T21TSUtOSVA5VEV4RzU2eTg2c2J5aEF6RWU5cjdJMTJZNW40Vnl3CmtxVkZTRWpuODRFaTFwcnV2NEZ0ZmFUT05yWm5wUWt5RFBmRURlbnEzdjUvMmxCcUJUQ3NqcU5BN3FnZjhJRWpDOFpwNm4KRDVocWliTXdaWG9IT0Z6R2JRRWhDTEsya1gyOE1KcVR4VnhxQUxlL0FMYXBMM08wU3R4MERlMmZiQWNzemUvUDk2OWl3cAphM1hBaHVnQi9jRmwvdVJNU3QrMVRNYjJWRGN5M1V2a3hGUWp3QVkzMGNybE9nYnYxQnhJc2FMc0xaUFQxRUY1ZzNiR3BxCmUxWloxa2lJMlhWZnlRS0V5cXcvajhlSmZUbys3RlNkQm4rOEh0QUFBQUF3RUFBUUFBQVFFQW9HR0FGd2ZaVUFndDFuSWkKMjFIaVorbXNjSWNFcmxGMkdoVWJFWEw0NldaYVkyQ2I0bm5xU1RoZS9xOG1kc2ovTnVmUUdNSVNQUmNNK2k5M2g5TEI4RQpNS3hBcmRlb1ErSGFBc1NOQzg1WXU3R2hmd3o5a1FEUEtCcDBQUHFPckw3NXdvc29WRm5CSE5scmdZVmtCREhOeFdhYzVJCjNURkVYUzVxV3RlVnVWQWtOTlY4VWZtT0QyU0hWOVpBaURLMUQ5LzZ3WUtMSkdvK2NQSUhCYVNyaGptZTR4aEtWTjlqT2sKQ0JqVUFaOE9mMGJ0bTBsTmgwSW44QW1TQTh6ZVZoSG0rT29MRU01ZXRLR2tqK3EveUR6R0RxR0VOTStGeTZjc2JxZ0tqMQpzUjdwN29zaU9UK3ExeklJU2VzYXZJTzE3b1FyQjNMN3NteGhsZm43WXJoU1lRQUFBSUF0YU4zaXdJcFVsb0ZTTG5KZDd0CnVzODNYcTU5RHl3YnByVGhLT1ByUy9icHJtcFFXcXB1ckFrN0JSM1UyN2ZyZTIvNTMyWDkvSVMxb1BDWHFsUHZNd3JOUXcKcnFCOWxpSlQ4Qi9zSXozdmg5V2lHallxTVNNMnd2U3V5dTBIVzJab1M3bktiMXFhb01NUVlHUXZYZGQwWDJLVGUrTm40cgo3ZlNORUNIOStXMFFBQUFJRUExbFNFNEszU3RzVWRLeTRzUm9FMjNmaEFXZ2FuYWJCOCtpMG5tWTg2eFd5RU9kTkpTRnE3CnZVR2NGY2V1bmpXM1JOMlJ0NFBmRlhUcWovNTU1SWpXZkVOVlBIRjlhM3dYN3dkbEs1aEt2cUZscEJ0WHpsaWxOL0N5eW0KVEtCaS9PVlVWZDVDaitXY2NzQk8xRnF1dHhrczBJM05DU0t0d1NQTzNHbHBZZ3JCVUFBQUNCQU0zQW5pNDFDN2xsR0hpYwo3ekRDVnZ5TE8waTdwOGgvMFpUNXI1d2NkRFNjdjlwbHRjVzNMRU5wRFlHU2Fic014L2VsdjQ5Z2Y4TFE2bTJOWVMvUGZmCmczSmh4MUNzQWxMTEhoL0doNWo2cWVQYVJmUi9uK2wzQTFONkNpUDdwRUZrL3VXeERhcFkwSWdneG1rWGpZRS9xTVNtWUEKa1lIUEViSmVkY3h2UDd4NUFBQUFEbWhwTFdOc2IzVmtjMEJsWkdkbEFRSURCQT09Ci0tLS0tRU5EIE9QRU5TU0ggUFJJVkFURSBLRVktLS0tLQo=",
"socksProxyPublicKey": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDc1F2STZiVS8yYUNaYnZXV1o5WG4xcTc4VU1OLzIxRUUydUNGOXdPSTFmWDRGU21Oamg3M3N3T21TSUtOSVA5VEV4RzU2eTg2c2J5aEF6RWU5cjdJMTJZNW40Vnl3a3FWRlNFam44NEVpMXBydXY0RnRmYVRPTnJabnBRa3lEUGZFRGVucTN2NS8ybEJxQlRDc2pxTkE3cWdmOElFakM4WnA2bkQ1aHFpYk13WlhvSE9GekdiUUVoQ0xLMmtYMjhNSnFUeFZ4cUFMZS9BTGFwTDNPMFN0eDBEZTJmYkFjc3plL1A5Njlpd3BhM1hBaHVnQi9jRmwvdVJNU3QrMVRNYjJWRGN5M1V2a3hGUWp3QVkzMGNybE9nYnYxQnhJc2FMc0xaUFQxRUY1ZzNiR3BxZTFaWjFraUkyWFZmeVFLRXlxdy9qOGVKZlRvKzdGU2RCbis4SHQgaGktY2xvdWRzQGVkZ2UK",
"description": "6426ea7e83066500241c5f3a@hiclouds.io ( IND - test-shadowsocks ) ( vtun38_2 ) ",
"socksProxy": false,
"socksProxyType": null,
"socksProxyPrivateKey": null,
"socksProxyPublicKey": null,
Verify CE Configuration
Log into the CE and inspect the SSH configuration file. The location may be different but common locations include /etc/config# cat /etc/config/socks. Run the following command.
/etc/config# cat /etc/config/socks
The given one is just an example output; when this command is run, it will show something like this.
Example Response
config ssh 'vtun_50024'
option localport '50024'
option key_file '/etc/dropbear/vtun38_2.key'
option user 'edge'
option server '34.93.120.41'
option server_port '25321'
option enabled '1'
This command shows the configuration file contents. Carefully check the settings, especially the server address, port, and password.
Verify OpenVPN Configuration
This command is used to view OpenVPN-related configurations. It shows the OpenVPN-related configurations.
cat /etc/config/openvpn
The given one is just an example output; when this command is run, it will show something like this.
Example Response
option socks_proxy '127.0.0.1 50024'
option socks_proxy_retry '30'
Check for Configuration Errors
Review both configuration files for syntax errors, typos, or wrong characters. Even a tiny error can keep ssh from working correctly. For the PE's JSON configuration, validate the file with a JSON validator, either online or command line tools. Invalid JSON will prevent ssh from starting.
Q:1 The JSON shows "socksProxy": false even though I enabled SSH in cloud — why?
Q:2 Where can I check OpenVPN-related configuration on the CE?
Q:3 Can I validate the JSON configuration manually?
UCI Configuration Verification
This section focuses on verifying the ssh configuration stored in the UCI system.
Check Network UCI Configuration
To check network-related UCI configurations for "socks," use the following command:
uci show netwrok | grep socks
The given one is just an example output; when this command is run, it will show something like this.
Example Response
socks.vtun_50024=ssh
socks.vtun_50024.localport='50024'
socks.vtun_50024.key_file='/etc/dropbear/vtun38_2.key'
socks.vtun_50024.user='edge'
socks.vtun_50024.server='34.93.120.41'
socks.vtun_50024.server_port='25321'
socks.vtun_50024.enabled='1'
Q:1 How to verify the key file path?
Q:2 What to do to apply Network UCI changes?
Run Time Configuration Verification
This section details how to check the run-time status and manage the ssh and firewall services.
Check Running ssh Process
ps | grep ssh
The given one is just an example output; when this command is run, it will show something like this.
Example Response
21900 root 6860 S ssh -N -D 50024 -o StrictHostKeyChecking=no -i /etc/
23656 root 6832 S /usr/bin/ssh -L 35048:127.0.0.1:35048 -R 35048:127.0
23801 root 820 S /usr/sbin/autossh -R 50047:localhost:25321 -o Strict
23802 root 820 S /usr/sbin/autossh -R 50048:localhost:443 -o StrictHo
23806 root 6832 S /usr/bin/ssh -L 35047:127.0.0.1:35047 -R 35047:127.0
Check ssh Service Status
To check the current status of the ssh service:
/etc/init.d/socks status
View Start ssh Service
To start the ssh service if it's not running:
/etc/init.d/socks start
View Stop ssh Service
To stop the SocksProxy service:
/etc/init.d/socks stop
Q:1 What to do if the SSH process is not running?
Q:2 How to know the current status of SSH service?
Q:3 In what order should I run the commands to troubleshoot SSH service?
Testing Verification
Check the Connection Status on the CE
Use netstat on the CE to check the connection status. The following command can determine whether the CE tried to connect the PE on the configured port, such as port 50024:
netstat -ntualp | grep 50024
The given one is just an example output; when this command is run, it will show something like this.
Example Response
tcp 0 0 127.0.0.1:50024 0.0.0.0:* LISTEN 21900/ssh
tcp 0 0 127.0.0.1:52548 127.0.0.1:50024 ESTABLISHED 22531/openvpn
tcp 0 0 127.0.0.1:50024 127.0.0.1:52548 ESTABLISHED 21900/ssh
tcp 0 0 ::1:50024 :::* LISTEN 21900/ssh
This command filters the netstat output for lines containing the word "50024". Use this command to find Established connections from the CE that successfully connected to the PE.
Log Verification
Checking logs can help you diagnose specific issues, such as failed authentication attempts or service errors.
Check ssh Logs
The ssh logs on the CE and PE for errors or warnings. The log location varies depending on the system. For systems that use logread, run the following command:
logread | grep socks
The given one is just an example output; when this command is run, it will show something like this.
Example Response
Jun 27 06:54:31 manual-testing procd: /etc/rc.d/S99shadowsocks-libev: ss-rules: Skipping ipv6. Requires ip6tables-mod-nat
Jun 27 06:57:26 manual-testing hiclouds_config.sh[18916]: execute post config command "/etc/init.d/socks start vtun_50024"