++ VM on second Hyper-V Cluster Node not backing up - Failed to find VM ++

Hello,
we are using NetVault 12.1 to backup all virtual machines from our hyper-v 2012R2 cluster.
vms01 (node1) vms01 (node2) vmscl (cluster). Ping, nslookup/ptr, check access, firewall test is running successfully.

For some weeks we see the following messages in the backup log.

Job Message    2018/10/29     18:00:02  71        Data Plugin       VMS01 Number of VMs selected for backup: 20

Error         2018/10/29     18:00:05          71        Data Plugin       VMS01 VM [BI01]: Failed to find VM on cluster 'VMSCL'
Error         2018/10/29     18:00:06          71        Data Plugin       VMS01 VM [BI01]: Backup failed
Error         2018/10/29     18:00:10          71        Data Plugin       VMS01 VM [CLOUD]: Failed to find VM on cluster 'VMSCL'
Error         2018/10/29     18:00:10          71        Data Plugin       VMS01 VM [CLOUD]: Backup failed

Here is what I did to get it running again:
- Removed all agents on vms02, restart
- Deleted virutal client
- Created new virtual client with the cluster name "vmscl" (agents are installed on vms02 successfully)
- Restarted
- Hyper-V Cluster license is looking good
- Running the following commands

Import-Module FailoverClusters
Import-Module ServerManager

Remove-ClusterNode -Cluster vmscl vms02
Restarted vms02
Add-ClusterNode -Cluster vmscl vms02

Get-ClusterNode -Cluster vmscl
Name            ID    State
----                 --    -----
VMS01          2     Up
VMS02          1     Up

Virtual machines running on vms02 are still not backing up / shown in the create backup selection section.

Parents
No Data
Reply
  • Hello,

    it seems that we have some kind of connection issue within out hyper-v cluster. I am using the following script found in: C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2

    Content of the script:

    ##############################################################################
    #
    #  GetVM.ps1 
    #
    #    This script will fetch a bunch of info for a specific VM on the specified
    #    machine if a name is given, or all VMs if no VM name is given. We'll use
    #    this in the case of populating our VM nodes under a server node as well
    #    as any time we just need state info on a specific VM.
    #
    #    Usage:
    #         GetVM.ps1 [ComputerName] [-isCluster] [-unclusteredOnly] [VmName]
    #
    #    ComputerName - name of the host or cluster machine to get VM info from.
    #
    #    isCluster - specifies that the input computer name is a cluster and
    #                the desire is to get a list of VMs that resides inside that cluster
    #
    #    unclusteredOnly - only list VMs on inputted host that are not part of the local cluster
    #                      This option implies you are querying for VMs on the inputted
    #                      host (ComputerName) and is *not* compatible with the isCluster option
    #
    #    VmName - name of a specific VM. This is optional - if not specified, 
    #             info for all VMs is returned.
    #
    ##############################################################################
    param 
    (
       [string]$machine,
       [switch]$isCluster,
       [switch]$unclusteredOnly,
       [string]$vmname,
       [switch]$ignoreRPCError,
       [String[]]$InclusiveVMList,
       [String[]]$ExclusiveVMList

    )

    $ScriptDir = Split-Path -parent $MyInvocation.MyCommand.Path
    Import-Module $ScriptDir\UtilModuleV2
    Import-Module $scriptDir\..\UtilModule

    $error.clear()

    # -----------------------------------------------------------------------------
    # PrintVm
    #   - prints out VM's information
    # -----------------------------------------------------------------------------
    Function PrintVm
    {
    param
    (
    $myVM
    )

    write-host ("<vm>")
    write-host ("<Name>"+$myVM.VMId.ToString().ToUpper()+"</Name>")
    write-host ("<Element Name>"+$myVM.Name+"</Element Name>")
        if ($vm.State -eq "Off")
        {
        write-host ("<Enabled State>3</Enabled State>")
        }
        else
        {
        write-host ("<Enabled State>2</Enabled State>")
        }
    write-host ("<Host Name>"+$myVM.ComputerName+"</Host Name>")
    write-host ("</vm>")

    }

    Function GetSingleVm
    {
        if ($isCluster -eq $true)
        {
            foreach ($node in $nodes)
            {
                $vm = Get-VM -ComputerName $node $vmname
                if ($vm -ne $null)
                {
                    $vm
                    break
                }
            }
        }
        else
        {
        Get-VM -ComputerName $machine $vmname
        }
    }

    Function GetVmList
    {
        $vmlist = New-Object System.Collections.ArrayList

        $iExecuteQuery = 1

        if ([string]::IsNullOrEmpty($InclusiveVMList) -and [string]::IsNullOrEmpty($ExclusiveVMList))
        {
            $iExecuteQuery = 0
            if ($isCluster -eq $true)
            {
                foreach ($node in $nodes)
                {
                    $vmList.AddRange(@(Get-VM -ComputerName $node | where {$_.IsClustered -eq $true}))
                }
            }
            else
            {
                $vmList = @(Get-VM -ComputerName $machine)
            }
        } 
           

        if ($iExecuteQuery -eq 1)
        {
    if (-not [string]::IsNullOrEmpty($InclusiveVMList))

    $iFlag = 1
    $inclusiveQuery = ""
    foreach ($vmPattern in $InclusiveVMList)
    {
    if ($iFlag)
    {
    $iFlag = 0
    }
    else
    {
    $inclusiveQuery = $inclusiveQuery + " -or "      
    }
    $inclusiveQuery = $inclusiveQuery + '$_.Name' + " -clike `"" + $vmPattern + "`""         
    }
    }

    if (-not [string]::IsNullOrEmpty($ExclusiveVMList))

    $iFlag = 1
    $exclusiveQuery = ""
    foreach ($vmPattern in $ExclusiveVMList)
    {
    if ($iFlag)
    {
    $iFlag = 0
    }
    else
    {
    $exclusiveQuery = $exclusiveQuery + " -and "      
    }            
    $exclusiveQuery = $exclusiveQuery + '$_.Name' + " -cnotlike `"" + $vmPattern + "`""
    }
    }

    if (-not [string]::IsNullOrEmpty($InclusiveVMList))

    if ([string]::IsNullOrEmpty($ExclusiveVMList)) 
    {             
    $query = $inclusiveQuery     
    }
    else
    {
    $query = $inclusiveQuery + " -and " + $exclusiveQuery
    }         
    }    
    else
    {
                $query = $exclusiveQuery
    }

    if ($isCluster -eq $true)
    {
    foreach ($node in $nodes)
    {
    $command = "Get-VM -ComputerName $node"
    $cluster = "{"+'$_.IsClustered'+" -eq 'true'"+"}"
    $vmListCommandOnCluster = $command +" | where "+ $cluster
    $vmListCommandOnQuery = $vmListCommandOnCluster+" | where "+ "{" +$query +"}"
    write-host ("<vmListCommandOnQuery>"+$vmListCommandOnQuery+"</vmListCommandOnQuery>")
    $vmListCommandOutput = $null
    $vmListCommandOutput = Invoke-Expression $vmListCommandOnQuery          
        if (-not [string]::IsNullOrEmpty($vmListCommandOutput))
    {
    $count = $vmListCommandOutput.Count
    if ($count -ne 0)
    {
    if ($count -eq "1")
    {
    $vmList.Add($vmListCommandOutput) 
    }
    else
    {
    $vmList.AddRange($vmListCommandOutput)
    } 
    }
    }
    }
    }
    else
    {
    $command = "Get-VM -ComputerName localhost"
    $vmListQuery = "$command | where { $query" + "}"
    $vmList = Invoke-Expression $vmListQuery
    }
        }

    foreach ($vm in $vmlist)
        {
            if (($unclusteredOnly -eq $true) -and ($vm.IsClustered -eq $true))
            {
                continue
            }

            [void]$vmlist_out.Add($vm)
        }
    }

    # -----------------------------------------------------------------------------
    # START
    # -----------------------------------------------------------------------------

    # verify machine parameter has been specified
    if ([string]::IsNullOrEmpty($machine))
    {
      LogError "Missing argument.  Host or cluster machine name required."
      exit
    }

    # if isCluster, . won't work since it's always the host's name
    #   Not the cluster name even if we're on the cluster manager
    if (($isCluster -eq $true) -and (($machine -eq ".") -or ($machine -eq "localhost")))
    {
    LogError "Invalid arguments.  Cluster queries cannot use `".`" for machine name."
    exit
    }

    # FIXME:  if isCluster, how do we make sure the input is in fact a cluster name?


    if ($isCluster -eq $true)
    {
    # specifying a cluster name, but requesting 
    #   VMs *not* in a cluster makes no sense
    if ($unclusteredOnly -eq $true)
    {
    LogError "Invalid arguments.  Cannot specify unclusteredOnly for a Cluster query."
    exit
    }
    }
    else
    {
    if ($unclusteredOnly -eq $true)
    {
    $cluster = $(Get-Cluster).Name
    if ([string]::IsNullOrEmpty($cluster))
    {
    $unclusteredOnly = $false
    $machine = gc env:computername
    }
    }
    if ((($machine -eq ".") -or ($machine -eq "localhost")))
    {
    $machine = gc env:computername
    }
    }

    $error.clear()

    # get list of cluster nodes
    $nodes = Get-ClusterNode -Cluster $machine

    # initialize array of vms to be outputted
    $vmlist_out = New-Object System.Collections.ArrayList
    $vmlist_out.clear()

    # get the vm(s)
    if (-not [string]::IsNullOrEmpty($vmname))
    {
        $vmlist_out = GetSingleVm

    #  Log an error if a specific VM was requested, but not found
    if ($vmlist_out.count -eq 0)
    {
    LogError ("Couldn't Find Requested VM: " + $vmname)
    exit
    }
    }
    else
    {
        GetVmList
    }

    # sort the list of vm's
    $vmlist_out = $vmlist_out | Sort-Object Name

    write-host ("<start>")
    if (($vmlist_out -ne $null) -and ($vmlist_out.Count -ne 0))
    {
    # first print count of VMs outputted
    write-host ("<VmCount>"+$vmlist_out.Count+"</VmCount>")

    # now loop through our list 
    foreach ($vm in $vmlist_out)
    {
    PrintVm($vm)
    }
    }
    else
    {
    write-host ("<VmCount>0</VmCount>")
    }
    write-host ("<stop>")

    --------------------------------------------------------------------------------------
    --------------------------------------------------------------------------------------

    Node 1 - VMS01  Node 2 - VMS02  Cluster - VMSCL

    Running the script shows only the maschines on the current node and shows a error for the maschines hostet on node two.
    --------------------------------------------------------------------------------------
    --------------------------------------------------------------------------------------

    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "APP01" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>C2648FA1-B83B-4CC5-BC0E-CCCC7B8A4EF7</Name>
    <Element Name>APP01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "ARC01" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>8F77588D-9B15-4E58-9294-0340289517B2</Name>
    <Element Name>ARC01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "BI01" -isCluster
    Get-VM : Ein Parameter ist ungültig. Ein virtueller Computer mit dem Namen "BI01" konnte von Hyper-V nicht gefunden
    werden.
    In C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2\GetVms.ps1:77 Zeichen:19
    +             $vm = Get-VM -ComputerName $node $vmname
    +                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidArgument: (BI01:String) [Get-VM], VirtualizationInvalidArgumentException
        + FullyQualifiedErrorId : InvalidParameter,Microsoft.HyperV.PowerShell.Commands.GetVMCommand

    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>C04FEFDA-5F30-4480-94C0-1C5D55050374</Name>
    <Element Name>BI01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS02</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CLOUD" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>6B8E7B4F-E61C-48D3-A6BA-7B33314B6EF9</Name>
    <Element Name>CLOUD</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CTI" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>4ACE33FB-1E93-4C05-80DC-055171178A60</Name>
    <Element Name>CTI</Element Name>
    <Enabled State>3</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CTI01" -isCluster
    Get-VM : Ein Parameter ist ungültig. Ein virtueller Computer mit dem Namen "CTI01" konnte von Hyper-V nicht gefunden
    werden.
    In C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2\GetVms.ps1:77 Zeichen:19
    +             $vm = Get-VM -ComputerName $node $vmname
    +                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidArgument: (CTI01:String) [Get-VM], VirtualizationInvalidArgumentException
        + FullyQualifiedErrorId : InvalidParameter,Microsoft.HyperV.PowerShell.Commands.GetVMCommand

    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>B44260B6-3156-4600-85E1-58168A3CC3D5</Name>
    <Element Name>CTI01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS02</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CX" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>1200962C-E875-45F9-B4D9-E7BBA2BBBCFD</Name>
    <Element Name>CX</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "DC01" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>16A20330-7ABA-4694-A068-FCCA3D7E1A6F</Name>
    <Element Name>DC01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>

Children
No Data