On The Way to Rapid Recovery

The newly released version of Data Protection | Rapid Recovery ( introduced quite a few new features and overall enhancements. Among these is the ability to agentlessly protect VMware virtual machines (VMs). The aim of this ability is to reduce Rapid Recovery administrative effort and to avoid interruptions to production tasks since installing the agent software on protected machines requires a reboot.

There’s little need to state the difference between the ideal world and real world. Its synthetic indicator could be called “cost of support.” When paired with real-world environments, agentless protection introduces some extra administrative effort on the VMware administration side. The new tasks don’t occur often and aren’t difficult, per se, but addressing them requires some knowledge that isn’t normally applied in day-to-day administration of VMware environments.

Moreover, if you don’t execute a task on a regular basis, how long will it be until you forget how to perform it? What’s to be done to allow customers to deal with such issues in minutes without having to go through a knowledge refresh/cramming/high adrenaline consumption? Obviously, there’s a solution to propose (thus, this blog articles series). However, let’s not anticipate!

Let’s Take a Real-life Example

Some backups for agentless protected VMs fail. The customer (let’s call him Cx) identifies the issue correctly. Changed Block Tracking (CBT) is the most likely culprit and needs to be reset. Just in case you want to refresh your knowledge, CBT is explained on TechTarget.

Cx searches the VMware Knowledge Base (KB) to address the issue, finds KB article #2139574 (which shows up first in his search) and it looks pretty difficult to apply, notes that the VMs need to be powered off (which he can’t do with production machines), and calls Rapid Recovery support for help.

The discussion that follows proved to be very helpful for both of us.

Yes, it is possible to reset CBT without powering off the VM. (Of course, there are VMware articles explaining how to do it, but they show lower in the search results list.)

No, it can’t be done this way from the VMware GUI, you need PowerCLI.

What is PowerCLI? Well, a kind of PowerShell module collection — it has its own console. Too bad the ISE console doesn’t work with PowerCLI. (Does anybody know a way?)

How do I install PowerCLI, and what version do I need? I couldn’t find the download page. (That’s an easy fix; the latest version is 6.0 release 3.)

Can I access PowerCLI from the regular PowerShell console? (Cx runs a few PowerShell scripts for Rapid Recovery already.) Yes, it is possible, but some work needs to be done. I can even script this.

Do I need to enter the credentials for my vCenter and stand-alone ESXi hosts and connect every time I run into an issue I want to address via PowerShell? Yes, and no. This can be scripted, as well.

I can’t afford to have the vCenter/ESXi hosts’ credentials exposed. Not a problem, I can encrypt and password protect them — the password won’t be stored anywhere in the script. However, if you forget it, you must remove the stored information and add it back using a new password. (You can change the password, as well; it won’t affect the VMware credentials.)

The discussion continued much longer. In the end, we came up with two action items:

  1. Reset CBT (and explain how it works) — this would solve the current issue.
  2. Script a framework with the following properties:
  3. The script would run from the familiar PowerShell console and not from the PowerCLI one.
  4. When the script runs for the first time, it allows entering the connection information to one or multiple vCenters/ESXi hosts in an encrypted form protected by a password that can be changed at a later date. This list can be edited at a later time, as well.
  5. If PowerCLI and any other prerequisites aren’t present, the script would land the user on a menu that allows downloading and installing them. This menu can be revisited at a later time, if necessary.
  6. The framework script would connect (and disconnect) automatically to the VMware environment.
  7. It would contain prepackaged modules, each running a common task needed to solve possible Rapid Recovery issues. New modules could be added (and obsolete ones removed) at any time.
  8. Any task performed by a prepackaged module could be performed on a set of VMs or other VMware objects.

And the list may go on. The cool part is that as ambitious as it seems, this project is in a functional phase. More polishing is needed, but each of these functions will be presented in future blog posts.

Meanwhile, let’s see how to reset CBT with the following steps:

Phase 1:

  1. Disable CBT via PowerCLI.
  2. Take a snapshot of the VM.
  3. Delete the snapshot previously taken.

Phase 2:

  1. Enable CBT via PowerCLI.
  2. Take a snapshot of the VM.
  3. Delete the snapshot previously taken.

The same result can be obtained via stunning/unstunning (quescing) the VM instead of creating/deleting snapshots. However, I prefer snapshots.

Since the operations performed are repetitive with just a value that differs in each phase, it makes sense to create a function that would be called with a different parameter for enabling and disabling CBT.

Assuming that PowerCLI is Already Installed, Let’s Put Everything Together:

  • Collect the name/IP address of the VMware host and credentials (in a safe object).

$vmwarehost = Read-Host "Resolvable name or IP address of the VMWare Host"
$cred = get-credential -Message "Enter VMware Host Credentials:"

  • Mitigate for invalid certificates.

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -confirm:$false

  • Connect to VMware host.

Write-Host "`nAttempting to connect to $($vmwarehost)" -f Yellow; 
connect-VIserver -Server $vmwarehost -cred $cred -erroraction Stop | select-objectname,port,version,build;
Write-Host "Done!"
}catch{write-Host "Error connecting to $($vmwarehost)"}

  • Select the VMs to process.

$vmset=get-VM -name * | select-object name,id | Out-GridView -PassThru -Title "Select VMs to process"

  • Process CBT by disabling/enabling it.

process-cbt -vmxs $vmset -op $false;
process-cbt -vmxs $vmset -op $true;

  • Close all connections.

Write-Host "`nClosing all Connections" -ForegroundColor Yellow; 
disconnect-viserver -Server * -Force -confirm:$false -ErrorAction stop
}catch{Write-Host "No ESXi or Vcenter connections found"};

As you see, there are a lot of generic actions (hence the validity of the framework mentioned above), and the CBT reset job is performed by running the process-cbt function twice — once to disable and the second to enable CBT.
This is how the process-cbt function works:

  • Set the parameters:

$vmxs — The VM object (name and ID) created in the “select the VMs to process” phase above
$op — The operation to enable/disable CBT; it can be $true or $false

  • Send a message to indicate the executed operation.

$tmpx="Attempting to disable CBT"
if($op){$tmpx="Attempting to enable CBT"}

  • Create an object to indicate the CBT status (on or off).

$cs= New-Object VMware.Vim.VirtualMachineConfigSpec
$cs.ChangeTrackingEnabled = $op

  • Get each VM to process and reconfigure it.

$vm = get-vm -id $vmx.id 
($vm | get-view).reconfigVM($cs)

  • Create and remove a snapshot.

$snap=New-Snapshot $vm -Name "$($VM.name)-CBT-$([GUID]::NewGuid())"
$snap | remove-snapshot -Confirm:$false

That is all! The full code is shown below. Please note that some code lines may be broken because they are too long. Just copy them into Notepad and remove the unwanted carriage returns.

function process-cbt{
#message to indicate the executed operation
$tmpx="Attempting to disable CBT"
if($op){$tmpx="Attempting to enable CBT"}

# Create an object to indicate the CBT status
$cs= New-Object VMware.Vim.VirtualMachineConfigSpec
$cs.ChangeTrackingEnabled = $op

foreach($vmx in $vmxs){
Write-Host "`nProcessing $($vmx.name) `[ID=$($vmx.id)`]: $tmpx" -ForegroundColor Yellow
$vm = get-vm -id $vmx.id 
($vm | get-view).reconfigVM($cs)
$snap=New-Snapshot $vm -Name "$($VM.name)-CBT-$([GUID]::NewGuid())"
$snap | remove-snapshot -Confirm:$false
Write-Host "Done!"
}catch{Write-Host "Error!" -ForegroundColor Red}

write-Host "Resetting CBT`n-------------`n" -ForegroundColor Green
#    Collect the name/IP address of the VMWare host and credentials (in a safe object)
$vmwarehost = Read-Host "Resolvable name or IP address of the VMWare Host"
$cred = get-credential -Message "Enter VMware Host Credentials:"

# Mitigate for “invalid” certificates
Write-Host "`nMitigating for the certificate issues`n" -ForegroundColor Yellow
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -confirm:$false

# Connect to the VMWare host
try{Write-Host "`nAttempting to connect to $($vmwarehost)" -f Yellow; connect-VIserver -Server$vmwarehost -cred $cred -erroraction Stop | select-object name,port,version,build;Write-Host "Done"
}catch{write-Host "Error connecting to $($vmwarehost)"}
Write-Host "`nSelecting VMs to process:"

# Select the VMs to process
$vmset=get-VM -name * | select-object name,id | Out-GridView -PassThru -Title "Select VMs to process"

# Process CBT by disabling/enabling it
process-cbt -vmxs $vmset -op $false;
process-cbt -vmxs $vmset -op $true;
# Close all connections
Write-Host "`nClosing all Connections" -ForegroundColor Yellow; 
try{disconnect-viserver -Server * -Force -confirm:$false -ErrorAction stop}catch{Write-Host "No ESXi or Vcenter connections found"};

Next time, we will go step by step through (part of) creating the framework environment mentioned above, where connecting to the VMware environment is done automatically.

If you would like to see some snapshots, here you go.

Launch the script. 

Enter the VMware host and credentials.

Connect to the VMware host.

Select VMs to process.

Process the VMs.

All done. Close the connections.