Collecting Data Remotely

The last two weeks I’ve been involved in assisting with an interesting problem. Initially it was reporting that application FOO was crashing for a few workstations at a remote location. The ask was, “can you help us collect logs”. Sure! And yes, I know there are a number of COTS products that can collect, analyze, and even do correlation. Let’s just assume for the sake of argument that those products are “turned off”. This will be a “quick-n-dirty” method of remotely collecting logs for “offline” analysis, as well as performing additional remote tasks.

Crawl – “What logs?”

In the spirit of crawl –> walk –> run, the first thing I asked what what type of logs do we need? The answer at first was Event Log data.

Everyone will have their preference for how to do this, but my quick and easy way to simply look at Windows Event Logs remotely is Computer Management. If you didn’t know, you can open up Computer Management (or even just the Event Viewer) on your local machine, right click the top and select “Connect to another computer”

As I move forward with the rest of this post, there’s one major assumption I’m making – that you have local administrator rights on the workstations you target. It’s hard to do any of this without the right permissions.

The problem with this method though is that it’s fine for looking at one, maybe two, remote Event Logs. But this doesn’t scale very well at all.

Walk – “oh look, a script!”

I love PowerShell. I love the possibilities. But I’m not a developer. I love that in true Battle Faction fashion, I can visualize doing something and then bang out some code to make it a reality.

In that spirit, knowing that I had a very specific Event Log error that I was looking for, and knowing that I had a list of workstations at the remote office, I whipped up the following bit of code:

$inputFile = Import-Csv -path "c:\temp\path_to_workstations.csv"
$outputFile = "c:\temp\outputFile.csv"

foreach ($pc in $inputFile) {
   $tmpListOfEvents = Get-EventLog -ComputerName $pc.Name -logname Application -Instanceid 1023 -Source ".NET Runtime" -Message "*application failed*" -Newest 10
   if ($null -neq $tmpListOfEvents) {
      foreach ($obj in $tmpListOfEvents) {
         "$($pc.Name),$($obj.TimeGenerated),$($obj.Message)" | Out-File $outputFile -Append
      }
   }
}

While I won’t be winning any coding awards for the above snippet, it did the trick. And in my use-case I actually added some checks to see if I can even hit the computer before attempting to get the events (using Test-Connection) with some console logging as well. Super simple.

What this does is iterate through the list of computers and looks for the newest 10 instances of events matching certain criteria. In this case we’re looking for event ID’s of 1023, from the .NET Runtime source, with “application failed” in the message (a generic example).

For each instance it finds, it’s then writing the computer name, the time of the event and the message to a common output file (for offline analysis).

The problem with the above is that it’s abysmally slow if the network is complex between where you’re running the code and where the endpoints are. There could be all sorts of inspection that could cause the script to run slowly. There must be a better way…

Run – “It was that easy!?”

By this point, I’d used the above script to target a couple of workstation at other remote offices and we found that application FOO was having problems in multiple locations. So the next question was, “how widespread is this really?” Remember, our COTS product that collects, analyzes, and correlates this information was “powered off” at the time, and since the issue is pretty visible I just wanted to get the data by any means.

I knew that my script above was running slow as it remotely queried the event logs looking for my event. I wondered if I might be able to copy those event logs over to my local workstation for “offline” parsing. Sure enough, that’s not only possible, but that allowed me to collect some interesting data.

First, unless your organization has done something strange, the default location for all of the event logs is in the Windows directory under \System32\WinEvt\Logs. If you’re unsure, right click one of the logs and choose properties. Here you can see the default:

At this point I just need a way to collect all of the application logs for all of the workstations I’m interested in. And how many workstations was that? All of them. I was interested in all of them.

With a list of every workstation at every remote location, I used the following bit of code to copy all of the application logs from said workstations locally:

$inputFile = Import-Csv -path "c:\temp\all_workstations.csv"
$logPath = "c$\windows\system32\winevt\logs\Application.evtx"
$localPath = "c:\temp\allLogs"

foreach ($pc in $inputFile) {
   Copy-Item -Pat "\\$($pc.Name)\$logPath" -Destination "$localPath\$($pc.Name).evtx" -ErrorAction SilentlyContinue
}

For the sake of fun, let’s just say I had a list of 1,000 workstations. This could reach out and copy the application event logs, renaming them to match their workstation names once copied locally. And this copy process takes less than an hour (using an average log size of 32MB-ish). But this meant I could scale my initial query and find out just how widespread the issue is.

The last step was to do just that. With this bit of code, which is similar to the first block, I queried all the offline event logs:

$allLogFiles = Get-ChildItem "c:\temp\allLogs"
$outputFile = "c:\temp\allOutput.csv"

foreach ($eventLog in $allLogFiles) {
   $eventsOfInterest = Get-WinEvent -FilterHashTable @{
      Path = $eventLog.FullName
      ProviderName = ".NET Runtime"
      Id = 1023
   }
   if ($null -neq $eventsOfInterest) {
      foreach ($event in $eventsOfInterest) {
         if ($event.properties.value -match 'application failed') {
            "$($eventLog.Name),$($event.TimeCreated),$($event.properties.value)" | Out-File -Append
         }
      }
   }
}

Before writing and using the above, I’d known about Get-WinEvent and it’s eventual replacement of Get-EventLog but I’d found it’s use….cumbersome. It’s just not as intuitive. But, it does have more power.

So in the above, the outer foreach loop is simply cycling through each of the offline event logs. Then, we scan it for events of interest. This is where my limited expertise came in. I’m getting all events that match both the ProviderName and the Id. If that list is not empty, then I scan the specific events that matched and look for the “application failed” message.

Lastly, for each of the events that match the criteria it dumps the relevant data to the output file for later analysis.

In the case of the event I’m still actively working on, it helped identify how widespread the issue really is, which helps in the troubleshooting effort. Knowing that it’s multiple workstations at multiple remote offices, we can rule out some things.

Bonus! Fun with PSEXEC

If you’ve not heard of the SysInternals Suite, shame on you! Just kidding….unless you’ve been in IT for a decade and still don’t know…

During the course of initial troubleshooting on this, there was an ask to get Wireshark and ProcMon logs on a couple workstations and try to capture application FOO having its issue. The easiest way to accomplish that is psexec, which can run commands against a remote machine (if you have administrative privileges on the workstation).

To run procmon remotely, I used the following to start it

c:\temp\psexec.exe -sd \\remotePC c:\temp\procmon.exe /accepteula /backingfile c:\temp\procmonoutput.pml /ringbuffer /ringbufferLen 5 /quiet

A few assumptions are made. The first is assuming locally, that the psexec.exe is located at c:\temp. Next, everything after \\remotePC is being run at the remote machine. So there’s an assumption that in the remote machine’s c:\temp directory we have the procmon.exe file.

The next thing we’re doing is ensuring a ring buffer of 5 minutes. If you’ve worked with procmon before, you know how chatty it can be. This ensures the size of the file doesn’t get too out of hand. It’s only capturing 5 minutes of data.

The way I used this was I’d run the command and periodically refresh the remote Event Viewer (in the crawl phase), watching for the event. When I’d found the event, I issued this command to stop the remote procmon:

c:\temp\psexec.exe -sd \\remotePC c:\temp\procmon.exe /accepteula /terminate /quiet

The last bit of fun was running a remote Wireshark capture on the workstation. Again, I used psexec to initiate a packet capture, sending some flags to enable the capture to not grow to be too big:

c:\temp\psexec.exe -sd \\remotePC "c:\Program Files\Wireshark\tshark" -a filesize:1000000 -s 128 -b files:2 -i "Ethernet" -w c:\temp\capture.pcap

With Wireshark, psexec will return the process ID that’s running remotely, and it’s that process ID that we feed to pskill (another SysInternals utility) for ending the capture (process ID 12345 in this example):

c:\temp\psexec.exe -sd \\remotePC c:\temp\pskill.exe -accepteula -nobanner -t 12345

So there’s lots of fun you can do remotely. I used psexec to do a number of things remotely, like run nslookups, pings, trace routes etc all from the remote workstation. I’d output the results to a text file local to that machine for review.

All of this, though, has helped collect a mountain of data that we’ve used. Now if only we can get that COTS product powered on…….