Kubectl get nodes error! Unable to connect to the server: dial tcp i/o timeout

I recently had an issue with an Azure Container Service Kubernetes cluster. When I would try to run commands directly from the master, or from another system, I would receive a ‘dial tcp i/o timeout’ error. This post will run through the rabbit hole I traveled down to fix it.

Here you can see the error I was receiving: “Unable to connect to the server: dial tcp i/o timeout. If you look at my second command, I attempted to ping another agent node and I was getting a “Destination Host Unreachable” error.

Then I decided to see if I can ssh to the agent, just in case ping was turned off. As you can see, I got a new error: “ssh: connect to host port 22 No route to host”

I also checked if I could see my pods and services, which I could, so this led me to believe the issue was networking related.

I checked to see if systemd-networkd-wait-online.service was active and discovered it was infact inactive (dead). I tried to restart it and then found another clue: “A dependency job for systemd-networkd-wait-online.service failed. See ‘journalctl -xe’ for details.

When I ran the journalctl -xe command, I found the Network Service wasn’t active at all and, again, all attempts to restart it were also unsuccessful.

While I’m not quite sure what caused the systemd-networkd to fail, I found this Stack Overflow post, which helped. To summarize, run the following commands:


You’ll notice the status will likely report “inactive” until you reboot, but after, you should be almost good to go.

I said almost because I then found that while the systemd-networkd service was finally reporting as active and could process requests once again, the network service was still having trouble resolving its own hostname.

So the final part of this rabbit hole is to edit your hosts file (/etc/hosts) and ensure you have the hostname found in /etc/hostname also available under I.E. if you type:

You should get the output of your hostname back. Example:

You should also see the same hostname in your hosts file:

If you do not see it, you will have to add it. After adding it, restart systemd-networkd again:

The “sudo: unable to resolve host [host-name]” error message should be gone!

You can then restart kubelet and etcd, which should successfully restart now as it couldn’t before with the hostname missing/mismatch. Then, if you run your “kubeclt get nodes” command, you should properly get your list of nodes.

Hope this helps someone else!

Reminder: All of my posts are provided "AS IS", imply no warranties, and confer no rights or special privileges. Use of included postings, code samples and other works are subject to the terms specified at Microsoft. For more information, click here.

Leave a Reply

Your email address will not be published. Required fields are marked *