This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

++ VM on second Hyper-V Cluster Node not backing up - Failed to find VM ++

Hello,
we are using NetVault 12.1 to backup all virtual machines from our hyper-v 2012R2 cluster.
vms01 (node1) vms01 (node2) vmscl (cluster). Ping, nslookup/ptr, check access, firewall test is running successfully.

For some weeks we see the following messages in the backup log.

Job Message    2018/10/29     18:00:02  71        Data Plugin       VMS01 Number of VMs selected for backup: 20

Error         2018/10/29     18:00:05          71        Data Plugin       VMS01 VM [BI01]: Failed to find VM on cluster 'VMSCL'
Error         2018/10/29     18:00:06          71        Data Plugin       VMS01 VM [BI01]: Backup failed
Error         2018/10/29     18:00:10          71        Data Plugin       VMS01 VM [CLOUD]: Failed to find VM on cluster 'VMSCL'
Error         2018/10/29     18:00:10          71        Data Plugin       VMS01 VM [CLOUD]: Backup failed

Here is what I did to get it running again:
- Removed all agents on vms02, restart
- Deleted virutal client
- Created new virtual client with the cluster name "vmscl" (agents are installed on vms02 successfully)
- Restarted
- Hyper-V Cluster license is looking good
- Running the following commands

Import-Module FailoverClusters
Import-Module ServerManager

Remove-ClusterNode -Cluster vmscl vms02
Restarted vms02
Add-ClusterNode -Cluster vmscl vms02

Get-ClusterNode -Cluster vmscl
Name            ID    State
----                 --    -----
VMS01          2     Up
VMS02          1     Up

Virtual machines running on vms02 are still not backing up / shown in the create backup selection section.

  • Hi is it 2012 r2 core? what hyper v plugin are you using? can you check the configuration in change settings -> client settings -> vms02 -> Network Manager you should have in preferred ip the ip for the host only and in barred ip addresses any other if you have iscsi, etc (cluster ip should be nowhere in this configuration on none of vms02 or vms01)

    After all this steps that you have done have you tried to delete virtual client, remove hyper-v plugin from both hosts, recreate the virtual client and retry?

    Looking forward

    Regards,

  • Hi,

    it is Server 2012 R2 with GUI, using Hyper-V plug-in version 12.0.0.7.
    NetVault doesn't care about preferred ip address entered or not. Result is the same.


    Now I did:
    - Remove all agents on vms01 + vms02, restart

    - Deleted virutal client
    - Created new virtual client with the cluster name "vmscl" (agents are installed on vms01 + vms02 successfully)
    - Restarted

    No change.

  • It was working before? if yes, what has changed?

    Can you try to browse the vm's under the hyper-v plugin using the host instead of the virtual client?

    Run in powershell in both hosts:

    Get-Cluster from both hosts

    Get-vm from both hosts

    Does that give you the correct info?

  • Good morning,
    sure, it was working for years. I guess I installed some windows updates.

    VMS01:
    PS C:\> Get-Cluster
    Name
    ----
    VMSCL


    PS C:\> get-vm

    Name State CPUUsage(%) MemoryAssigned(M) Uptime Status
    ---- ----- ----------- ----------------- ------ ------
    AP1 Running 0 20480 4.13:12:10 Normaler Betrieb
    AR1 Running 0 8096 4.10:14:41 Normaler Betrieb
    CLD Running 0 2048 6.00:25:16 Normaler Betrieb
    CI Off 0 0 00:00:00 Normaler Betrieb
    C Running 0 512 5.05:02:59 Normaler Betrieb
    D1 Running 0 6144 4.15:25:49 Normaler Betrieb
    D1 Running 0 6144 5.05:56:42 Normaler Betrieb
    FI1 Running 0 12228 3.08:35:31 Normaler Betrieb
    R1 Running 0 8096 4.16:18:15 Normaler Betrieb
    R01 Running 4 24576 3.02:24:44 Normaler Betrieb
    R02 Running 4 24576 3.02:01:31 Normaler Betrieb
    TIC Running 0 4096 5.21:26:38 Normaler Betrieb

    VMS02:
    PS C:\> Get-Cluster
    Name
    ----
    VMSCL

    PS C:\> get-vm

    Name State CPUUsage(%) MemoryAssigned(M) Uptime Status
    ---- ----- ----------- ----------------- ------ ------
    B1 Running 0 16384 3.01:15:26 Normaler Betrieb
    C01 Running 0 6144 2.12:01:59 Normaler Betrieb
    D01 Off 0 0 00:00:00 Normaler Betrieb
    GET Running 0 4096 2.15:45:47 Normaler Betrieb
    M01 Running 3 24576 2.15:44:23 Normaler Betrieb
    NP Running 0 6144 2.15:47:51 Normaler Betrieb
    R3 Running 3 24576 2.15:48:18 Normaler Betrieb
    R1 Running 0 8192 2.15:47:56 Normaler Betrieb
    TE Running 0 4096 2.15:45:07 Normaler Betrieb
    Wer Off 0 0 00:00:00 Normaler Betrieb
    WS Running 0 6144 3.01:14:33 Normaler Betrieb

     
    The results are correct!

  • Odd, I will recommend you to open a Service Request with NetVault Support to get debugging files and work closer to this.

  • Indeed! A support case is already open. As far as I have a solution Im going to update this thread. 

  • Looking forward for the resolution!

  • Hello,

    it seems that we have some kind of connection issue within out hyper-v cluster. I am using the following script found in: C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2

    Content of the script:

    ##############################################################################
    #
    #  GetVM.ps1 
    #
    #    This script will fetch a bunch of info for a specific VM on the specified
    #    machine if a name is given, or all VMs if no VM name is given. We'll use
    #    this in the case of populating our VM nodes under a server node as well
    #    as any time we just need state info on a specific VM.
    #
    #    Usage:
    #         GetVM.ps1 [ComputerName] [-isCluster] [-unclusteredOnly] [VmName]
    #
    #    ComputerName - name of the host or cluster machine to get VM info from.
    #
    #    isCluster - specifies that the input computer name is a cluster and
    #                the desire is to get a list of VMs that resides inside that cluster
    #
    #    unclusteredOnly - only list VMs on inputted host that are not part of the local cluster
    #                      This option implies you are querying for VMs on the inputted
    #                      host (ComputerName) and is *not* compatible with the isCluster option
    #
    #    VmName - name of a specific VM. This is optional - if not specified, 
    #             info for all VMs is returned.
    #
    ##############################################################################
    param 
    (
       [string]$machine,
       [switch]$isCluster,
       [switch]$unclusteredOnly,
       [string]$vmname,
       [switch]$ignoreRPCError,
       [String[]]$InclusiveVMList,
       [String[]]$ExclusiveVMList

    )

    $ScriptDir = Split-Path -parent $MyInvocation.MyCommand.Path
    Import-Module $ScriptDir\UtilModuleV2
    Import-Module $scriptDir\..\UtilModule

    $error.clear()

    # -----------------------------------------------------------------------------
    # PrintVm
    #   - prints out VM's information
    # -----------------------------------------------------------------------------
    Function PrintVm
    {
    param
    (
    $myVM
    )

    write-host ("<vm>")
    write-host ("<Name>"+$myVM.VMId.ToString().ToUpper()+"</Name>")
    write-host ("<Element Name>"+$myVM.Name+"</Element Name>")
        if ($vm.State -eq "Off")
        {
        write-host ("<Enabled State>3</Enabled State>")
        }
        else
        {
        write-host ("<Enabled State>2</Enabled State>")
        }
    write-host ("<Host Name>"+$myVM.ComputerName+"</Host Name>")
    write-host ("</vm>")

    }

    Function GetSingleVm
    {
        if ($isCluster -eq $true)
        {
            foreach ($node in $nodes)
            {
                $vm = Get-VM -ComputerName $node $vmname
                if ($vm -ne $null)
                {
                    $vm
                    break
                }
            }
        }
        else
        {
        Get-VM -ComputerName $machine $vmname
        }
    }

    Function GetVmList
    {
        $vmlist = New-Object System.Collections.ArrayList

        $iExecuteQuery = 1

        if ([string]::IsNullOrEmpty($InclusiveVMList) -and [string]::IsNullOrEmpty($ExclusiveVMList))
        {
            $iExecuteQuery = 0
            if ($isCluster -eq $true)
            {
                foreach ($node in $nodes)
                {
                    $vmList.AddRange(@(Get-VM -ComputerName $node | where {$_.IsClustered -eq $true}))
                }
            }
            else
            {
                $vmList = @(Get-VM -ComputerName $machine)
            }
        } 
           

        if ($iExecuteQuery -eq 1)
        {
    if (-not [string]::IsNullOrEmpty($InclusiveVMList))

    $iFlag = 1
    $inclusiveQuery = ""
    foreach ($vmPattern in $InclusiveVMList)
    {
    if ($iFlag)
    {
    $iFlag = 0
    }
    else
    {
    $inclusiveQuery = $inclusiveQuery + " -or "      
    }
    $inclusiveQuery = $inclusiveQuery + '$_.Name' + " -clike `"" + $vmPattern + "`""         
    }
    }

    if (-not [string]::IsNullOrEmpty($ExclusiveVMList))

    $iFlag = 1
    $exclusiveQuery = ""
    foreach ($vmPattern in $ExclusiveVMList)
    {
    if ($iFlag)
    {
    $iFlag = 0
    }
    else
    {
    $exclusiveQuery = $exclusiveQuery + " -and "      
    }            
    $exclusiveQuery = $exclusiveQuery + '$_.Name' + " -cnotlike `"" + $vmPattern + "`""
    }
    }

    if (-not [string]::IsNullOrEmpty($InclusiveVMList))

    if ([string]::IsNullOrEmpty($ExclusiveVMList)) 
    {             
    $query = $inclusiveQuery     
    }
    else
    {
    $query = $inclusiveQuery + " -and " + $exclusiveQuery
    }         
    }    
    else
    {
                $query = $exclusiveQuery
    }

    if ($isCluster -eq $true)
    {
    foreach ($node in $nodes)
    {
    $command = "Get-VM -ComputerName $node"
    $cluster = "{"+'$_.IsClustered'+" -eq 'true'"+"}"
    $vmListCommandOnCluster = $command +" | where "+ $cluster
    $vmListCommandOnQuery = $vmListCommandOnCluster+" | where "+ "{" +$query +"}"
    write-host ("<vmListCommandOnQuery>"+$vmListCommandOnQuery+"</vmListCommandOnQuery>")
    $vmListCommandOutput = $null
    $vmListCommandOutput = Invoke-Expression $vmListCommandOnQuery          
        if (-not [string]::IsNullOrEmpty($vmListCommandOutput))
    {
    $count = $vmListCommandOutput.Count
    if ($count -ne 0)
    {
    if ($count -eq "1")
    {
    $vmList.Add($vmListCommandOutput) 
    }
    else
    {
    $vmList.AddRange($vmListCommandOutput)
    } 
    }
    }
    }
    }
    else
    {
    $command = "Get-VM -ComputerName localhost"
    $vmListQuery = "$command | where { $query" + "}"
    $vmList = Invoke-Expression $vmListQuery
    }
        }

    foreach ($vm in $vmlist)
        {
            if (($unclusteredOnly -eq $true) -and ($vm.IsClustered -eq $true))
            {
                continue
            }

            [void]$vmlist_out.Add($vm)
        }
    }

    # -----------------------------------------------------------------------------
    # START
    # -----------------------------------------------------------------------------

    # verify machine parameter has been specified
    if ([string]::IsNullOrEmpty($machine))
    {
      LogError "Missing argument.  Host or cluster machine name required."
      exit
    }

    # if isCluster, . won't work since it's always the host's name
    #   Not the cluster name even if we're on the cluster manager
    if (($isCluster -eq $true) -and (($machine -eq ".") -or ($machine -eq "localhost")))
    {
    LogError "Invalid arguments.  Cluster queries cannot use `".`" for machine name."
    exit
    }

    # FIXME:  if isCluster, how do we make sure the input is in fact a cluster name?


    if ($isCluster -eq $true)
    {
    # specifying a cluster name, but requesting 
    #   VMs *not* in a cluster makes no sense
    if ($unclusteredOnly -eq $true)
    {
    LogError "Invalid arguments.  Cannot specify unclusteredOnly for a Cluster query."
    exit
    }
    }
    else
    {
    if ($unclusteredOnly -eq $true)
    {
    $cluster = $(Get-Cluster).Name
    if ([string]::IsNullOrEmpty($cluster))
    {
    $unclusteredOnly = $false
    $machine = gc env:computername
    }
    }
    if ((($machine -eq ".") -or ($machine -eq "localhost")))
    {
    $machine = gc env:computername
    }
    }

    $error.clear()

    # get list of cluster nodes
    $nodes = Get-ClusterNode -Cluster $machine

    # initialize array of vms to be outputted
    $vmlist_out = New-Object System.Collections.ArrayList
    $vmlist_out.clear()

    # get the vm(s)
    if (-not [string]::IsNullOrEmpty($vmname))
    {
        $vmlist_out = GetSingleVm

    #  Log an error if a specific VM was requested, but not found
    if ($vmlist_out.count -eq 0)
    {
    LogError ("Couldn't Find Requested VM: " + $vmname)
    exit
    }
    }
    else
    {
        GetVmList
    }

    # sort the list of vm's
    $vmlist_out = $vmlist_out | Sort-Object Name

    write-host ("<start>")
    if (($vmlist_out -ne $null) -and ($vmlist_out.Count -ne 0))
    {
    # first print count of VMs outputted
    write-host ("<VmCount>"+$vmlist_out.Count+"</VmCount>")

    # now loop through our list 
    foreach ($vm in $vmlist_out)
    {
    PrintVm($vm)
    }
    }
    else
    {
    write-host ("<VmCount>0</VmCount>")
    }
    write-host ("<stop>")

    --------------------------------------------------------------------------------------
    --------------------------------------------------------------------------------------

    Node 1 - VMS01  Node 2 - VMS02  Cluster - VMSCL

    Running the script shows only the maschines on the current node and shows a error for the maschines hostet on node two.
    --------------------------------------------------------------------------------------
    --------------------------------------------------------------------------------------

    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "APP01" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>C2648FA1-B83B-4CC5-BC0E-CCCC7B8A4EF7</Name>
    <Element Name>APP01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "ARC01" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>8F77588D-9B15-4E58-9294-0340289517B2</Name>
    <Element Name>ARC01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "BI01" -isCluster
    Get-VM : Ein Parameter ist ungültig. Ein virtueller Computer mit dem Namen "BI01" konnte von Hyper-V nicht gefunden
    werden.
    In C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2\GetVms.ps1:77 Zeichen:19
    +             $vm = Get-VM -ComputerName $node $vmname
    +                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidArgument: (BI01:String) [Get-VM], VirtualizationInvalidArgumentException
        + FullyQualifiedErrorId : InvalidParameter,Microsoft.HyperV.PowerShell.Commands.GetVMCommand

    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>C04FEFDA-5F30-4480-94C0-1C5D55050374</Name>
    <Element Name>BI01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS02</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CLOUD" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>6B8E7B4F-E61C-48D3-A6BA-7B33314B6EF9</Name>
    <Element Name>CLOUD</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CTI" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>4ACE33FB-1E93-4C05-80DC-055171178A60</Name>
    <Element Name>CTI</Element Name>
    <Enabled State>3</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CTI01" -isCluster
    Get-VM : Ein Parameter ist ungültig. Ein virtueller Computer mit dem Namen "CTI01" konnte von Hyper-V nicht gefunden
    werden.
    In C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2\GetVms.ps1:77 Zeichen:19
    +             $vm = Get-VM -ComputerName $node $vmname
    +                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidArgument: (CTI01:String) [Get-VM], VirtualizationInvalidArgumentException
        + FullyQualifiedErrorId : InvalidParameter,Microsoft.HyperV.PowerShell.Commands.GetVMCommand

    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>B44260B6-3156-4600-85E1-58168A3CC3D5</Name>
    <Element Name>CTI01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS02</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "CX" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>1200962C-E875-45F9-B4D9-E7BBA2BBBCFD</Name>
    <Element Name>CX</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>
    PS C:\Program Files (x86)\Quest\NetVault Backup\scripts\HyperV\v2> ./GetVms.ps1 "VMSCL" "DC01" -isCluster
    <start>
    <VmCount>1</VmCount>
    <vm>
    <Name>16A20330-7ABA-4694-A068-FCCA3D7E1A6F</Name>
    <Element Name>DC01</Element Name>
    <Enabled State>2</Enabled State>
    <Host Name>VMS01</Host Name>
    </vm>
    <stop>