I’ve always been fascinated by networking and how all these computers, services and devices can talk to each other. It’s really the single most important component to, well, everything. Without any form of networking, the Internet and all it’s delights wouldn’t really exist or be useful. The core fundamentals of networking haven’t changed much in the last quarter century as well, and yet consistently I find people lack that fundamental understanding, or that don’t know how to troubleshoot connectivity issues.
I’ve kicked around the idea of diving into networking and have been sitting on the topic for the last couple of weeks. It’s time to start writing something. There are lots of good resources online explaining concepts such as the OSI model, and while this is very useful to know (especially if you want to blame it on the firewall), I hope to develop this series and make it practical with examples and an eye towards troubleshooting connectivity between two nodes. If you understand the basics, you can more easily diagnose a problem.
For this post we’re going to start by looking at the simplest possible configuration – a client and a server on the same network. The goal will be to add complexity by introducing additional networking components to the diagram in later posts.
- VNET-TEST is a Virtual Network with address block 10.10.0.0/16
- SUBNET-VMS is a subnet with address block 10.10.10.0/24
- VM1 is our client VM at IP 10.10.10.5
- SERVER01 is our server at IP 10.10.10.4
Some additional assumptions – this environment is in Azure, and the virtual machines are Windows-based (11 for the Client and Server 2022 for the server). If you would like to deploy the above in your own lab environment, click here for the source code.
Let’s assume that SERVER01 runs some sort of web server, so it’s listening on TCP port 80, and let’s assume you want to prove that it’s listening and prove you can connect to it. How would you do that?
(Note: I’m using a very specific example in this post, “reaching TCP port 80”, but this can apply to a number of connectivity scenarios between a client and a server. Some of the techniques described here can apply to a broad range of connectivity-related issues, and some are more specific. The intent is to provide tips, tricks, tools, and general flow on how to troubleshoot. Knowing how to collect, interpret, and act on data will help you in 100% of troubleshooting situations)
There’s no right or wrong way to troubleshoot any problem. There are perhaps better approaches to the issue. For example, if we knew that SERVER01 was supposed to be listening on TCP 80, but VM1 couldn’t reach that, would your first course of action to be to delete and rebuild SERVER01? Probably not. At least I’d hope not. Would you rip out the virtual network and attach the VMs to a different one? Again, probably not. So where do we begin…..?
"Data, data, data. I can not make bricks without clay," - Arthur Conan Doyle, The Adventure of the Copper Beeches - a Sherlock Holmes Short Story
The key to solving any problem is collecting data and identifying symptoms. Just like a good Sherlock Holmes mystery, data is key to solving the mystery.
A decent first step would be to try and scope the problem. In the diagram above we only have a single client, but in the real world you’ll probably have multiple client machines available. Can any of them reach the server on TCP 80, or is it just a single machine have problems? Identifying the scope of a problem gives you clues as to how widespread the issue potentially is. If only a single machine is having problems then you wouldn’t necessarily jump straight to troubleshooting the server (although the issue may still live in the server, it can’t be ruled out completely at this early stage).
For this first example let’s assume none of our client VM’s can hit TCP 80 on our server. What’s that tell us? Should we continue looking on our client VM’s? Should we check to see if our user account is locked out? Is there paper in the printer upstairs? No, we should probably pivot and investigate the server.
The first thing I’d do on the server is check the event logs. Yes, always always always start with the event logs. This is part of the basics. A good server administrator / application owner will become familiar with the event logs on the servers you are responsible for.
I’m a bit old school, and you may think so too, but I like to open Computer Management on a server I’m troubleshooting. Right click your start button and go up to Computer Management.
A good place to start here is the Application log under the Windows Logs folder. This gives you a list of all of the events. Usually Information-type events are not helpful. At least not initially. They could be, depending on the type of data you’re gathering. But in this case we really want to look for any errors or warnings that could tell us perhaps why clients can’t reach TCP 80. Now, you could scroll down this list….and keep scrolling and scrolling. But I’m lazy, and there are tools to make this easier. If you only want to see critical, errors or warnings, use the built-in filter:
Behold! The cleanest Application Log you’ll ever see. It’s all downhill from here….(no, really, it is)
This log only has a single error message and a single warning, and reading through the details of each, we don’t have any information that can help. The WinLogon message references a failed event notification, and the COMRuntime error talks about marshalling an unmarshalled marshall to an object thing…
I don’t know what this means, but nowhere in that message does it say “network”, “tcp”, “80”, or any other key words that pique my interest. So we leave the event log empty handed. That’s fine – part of data gathering is gathering no useful data. But what if we had other errors in the logs? What if we had other errors that indicated other, unrelated issues that need addressing? A good Server/Application Admin will keep tabs on the health of their servers. Unless you know for a fact that someone else is looking after your servers’ health, you should always assume it’s your responsibility (and even then, the “trust, but verify” mantra should come into play — after all, if your server blows up at 3am on Christmas Day it’s you they’re going to call, right?)
So with the easy bits out of the way, the next thing to check is whether the server really is listening. Easiest way to do this is with an old tool called netstat. This tool has been around in the Windows toolkit for a very long time, and it’s a tool you should have at least a basic familiarity with. Check here for Microsoft’s documentation on its usage.
I’ve used the command netstat -anp tcp, which means to display all active connections (-a), displayed completely numerically (-n) and only the TCP protocol (-p tcp). I’ve obviously combined the -a, -n and -p to read -anp (hey, we aren’t born with this knowledge).
If you’re not sure how to read the output above, within the red squiggly line you see a list of local addresses and ports with their corresponding state showing as LISTENING. As you might correctly assume, this shows you exactly which TCP ports this server is listening on. We see port 135, port 445, even port 49665. What we don’t see, however, is TCP port 80.
So here we have one break point. We’ve proven that a server that should be listening on TCP 80 is in fact not. What’s this mean? How do we fix it? Well, “it depends” is going to be a common answer. What application should be running and listening on port 80? Is it simply an IIS server? A JVM? Some archaic, custom hand-hackery process? That’ll be up to you, the server/application owner to decide (application-specific troubleshooting is out of scope. For now…).
But…..what if it is listening on port 80? What would that look like in our netstat output? Well, it would look like this:
This shows TCP port 80 in a LISTENING state on the local 10.10.10.4 address. If your clients are still unable to reach the server, where do we go from here? A logical next step would be the Windows Firewall. On all modern Server Operating Systems, the Windows Firewall is enabled by default. Things like Group Policy or Desired State Configuration (DSC) can disable it or make modifications to it. It would be a good idea to perform a quick check of it’s status and whether an allow rule exists. Search for “Firewall” in the start menu to find it under the Windows Security page:
In this example you can see that the firewall is turned on for all three domains. In your environment it may not be possible to simply “turn off” the Windows Firewall to test connectivity, and if that’s the case you’ll want to click the Advanced Settings and scroll through the lengthy list of firewall rules and make sure that a rule matching TCP 80 (or simply HTTP) is enabled.
Creating or configuring Windows Firewall Rules is out of scope for this, but Microsoft provides excellent documentation on the firewall overview, as well as best practices for configuring it. Both worth a quick glance. It could be as simple as creating a new inbound allow rule for TCP port 80 – this is a very common solution to this type of problem (especially if connectivity to this server has never worked on port 80).
For basic troubleshooting from the server-side, this is where this post ends. In part 2, we look at from the client side to determine what tool(s) we have there to test and confirm connectivity, and what that resultant output looks like. In further posts, we start adding additional complexity to our network (Network Security Groups, User Defined Routes, and maybe even a Load Balancer in front of our server!). Stay tuned!
Well done.