![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image.png\")
<\/figure>\n\n\n\nAzure let’s you drill down a little bit further and get more information on that specific error:<\/p>\n\n\n\n In this example, we have a server where replication is running successfully but there is no app-consistent recovery point. Microsoft gives us some possible causes and possible recommendations. So….how would you troubleshoot this?<\/p>\n\n\n\n Troubleshooting is part art, part science. It’s a repeatable process – gather data, create hypothesis, act, check. In nearly all cases, there’s no obvious right or wrong answer for beginning the troubleshooting process. Some places may make more sense than others. For instance, I wouldn’t jump to restarting the server first. For issues with servers, restarting it is the one thing you want to avoid at all costs.<\/p>\n\n\n\n Who hasn’t seen this meme in their career?<\/p>\n\n\n\n Don’t. Reboot. Servers.<\/p>\n\n\n\n In this case, we did start with Azure Site Recovery. We disabled replication, re-enabled replication and waited to see what happened. There was no harm in that, didn’t require a reboot, was a fairly quick thing to do. We wanted to rule out the possibility that it was the ASR agent that was somehow not cooperating. But alas, same result.<\/p>\n\n\n\n This is the first layer down the rabbit hole where I’ll sometimes see people stop. The old joke “I’ve tried nothing and I’m out of ideas” comes to mind. But in troubleshooting, we need more data in order to make an informed hypothesis. For this situation we need to get onto the server and start looking around.<\/p>\n\n\n\n The most important thing when troubleshooting a problem is data. The more data you have around the symptoms, the environment, the scope, the scale, etc. the better. When troubleshooting a problem on a Windows server, the first place you should always look is the Event Log. In this instance, we’re looking for either Azure Site Recovery or VSS errors. And here we’ve found one:<\/p>\n\n\n\n I’m going to pause briefly here and talk about the Volume Shadow Copy Service (VSS). Third party applications use it to make snapshots for replication purposes or, more commonly, backup. Microsoft has some fantastic, and in depth, documentation that you should be aware of if you want to get a deeper understanding of how it works.<\/p>\n\n\n\n Overview of Processing a Backup Under VSS – Win32 apps | Microsoft Learn<\/a><\/p>\n\n\n\n Working left to right you have a Requestor that talks to Backup Components. The Backup Components talk to Writers, and the Writers talk to back-end Providers. In the error above we see that it speaks very specifically about a provider not having some components registered. This is where we roll our sleeves up.<\/p>\n\n\n\n If you’re not familiar, Windows has a built-in tool called vssadmin <\/a>which lets you collect some potentially useful information. By running From doing some digging online, we find that the second and third providers are built-in, native. Microsoft File Share Shadow Copy provider looks to have been introduced as part of an enhancement to Server 2012 R2 (according to this older Microsoft blog post<\/a>).<\/p>\n\n\n\n In this case, I’m left with the The registry can be a scary place. It’s very easy to cause significant damage to your system if you don’t know what you’re looking at. For VSS providers, luckily it’s pretty straightforward. You can see the location above under the HKEY_LOCAL_MACHINE hive. What I did in this case was right click the provider with the matching GUID and export it (saving it to the desktop). Then removed the key. Next, run In a few cases, this worked. In others, it didn’t. In those that didn’t, we need to keep collecting data.<\/p>\n\n\n\n In one oddball case, we found VSS errors relating to a specific SQL Writer. If you run This shows three writers (there are usually about 10 or so in a default installation). The middle writer is clearly a problem here, as the State shows as “Failed” and Last Error shows as “Non-retryable error”. As it turns out, this has an application installed called “VSS Writer for Server 2016” installed – so this is almost certainly interfering with the operation of VSS.<\/p>\n\n\n\n But let’s continue down another tunnel in the rabbit hole…<\/p>\n\n\n\n When poking around, you may come across something that just doesn’t make sense. In a number of other servers we investigated, we found an interesting registry key in the VSS providers which was completely empty:<\/p>\n\n\n\n This key, To see precisely how many servers were affected, I wrote the following script. The output was a CSV file that I could then filter and get the exact list of servers.<\/p>\n\n\n\n You will want to have an input and output file defined, but this gave me a list of all servers and all the providers. I could then sort \/ filter on the empty one we identified above, and then with the following code snippet was able to remove the key and restart the VSS service cleanly.<\/p>\n\n\n\n![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-12.png\")
<\/figure>\n\n\n\nTier 1 – The Basics – “Is it plugged in?”<\/h2>\n\n\n\n
![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-1.png\")
<\/figure>\n\n\n\nTier 2 – “Data, data, data. I cannot make bricks without clay!”<\/h2>\n\n\n\n
![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-4.png\")
<\/figure>\n\n\n\n vssadmin list providers<\/code> on the server we get three items:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-5.png\")
<\/figure>\n\n\n\nHyper-V IC Software Shadow Copy Provider<\/code>. As of the writing of this post, I’ve no idea where this provider comes from. I’ve seen this provider on multiple servers with issues, and have verified that no other backup product has been installed. But the provider ID in the output above matches the provider ID in the Event Log error and we know isn’t used. The next step in this was to remove that provider and test. To do that, we need to venture into the registry.<\/p>\n\n\n\n![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-6.png\")
<\/figure>\n\n\n\nnet stop vss && net start vss<\/code> in order to restart the VSS Service and test.<\/p>\n\n\n\nvssadmin list writers<\/code> you’ll get a list of all the registered VSS Writers on the system:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-7.png\")
<\/figure>\n\n\n\nTier 3 – “Knee deep in the dead”<\/h2>\n\n\n\n
![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-8-1024x495.png\")
<\/figure>\n\n\n\n3eb85257-e0b0-4788-a080-d39aa6a515af<\/code>, was seen on a number of machines that included vague VSS errors in the event logs. The only thing we could guess was that an older version of the Azure Site Recovery agent had at some point been removed and some artifacts were left. But this was speculation. On one test server, we found that removing that empty key and restarting the VSS service cleared up some VSS errors, and actually caused ASR app-consistent backups to be taken properly.<\/p>\n\n\n\n$allServers = Import-Csv $pathToServerList
$array = @()
foreach ($server in $allServers) {
try {
$result = Invoke-Command $server.Name {Get-ChildItem -Path 'HKLM:\\SYSTEM\\CurrentControlSet\\Services\\VSS\\Providers'}
} catch {
}
$array += $result
}
foreach ($item in $array) {
\"$($item.Name),$($item.property),$($item.PSComputerName)\" | Out-File $pathToOutputFile -Append
}<\/code><\/p>\n\n\n\n$allServers = Import-Csv $pathToServerList
foreach ($server in $allServers) {
Try {
Invoke-Command $server.Name {
Remove-Item -Path 'HKLM:\\SYSTEM\\CurrentControlSet\\Services\\VSS\\Providers\\{3eb85257-e0b0-4788-a080-d39aa6a515af}' -Confirm:$false
} -ErrorAction SilentlyContinue
} catch {
}
}<\/code><\/p>\n\n\n\n
![\"\"](\"https:\/\/www.dumpsterfirecomputing.com\/wp-content\/uploads\/2023\/04\/image-11-1024x78.png\")
<\/figure>\n\n\n\n