Balance SOFS Cluster with SMA

I was at a customer and they needed to get their Scale Out File Servers Balanced. So I found this script: Link to blog post, after testing the script and seeing that it works. I thought, how I should run this on a schedule? I saw two options:

  1. Run it as a scheduled task in windows
  2. Run it with a SMA runbook.

I choose to run it with an SMA runbook due to that I think it’s more flexible than schedule task.

So the first thing I did was creating a new credential asset via the Azure Pack admin portal.

SMA_StorageRAA

The credentials asset is a PowerShell Credentials and the domain account needs to have administrative access to the cluster nodes to be able to run the powershell command “Move-SmbWitnessClient”

Then I created a new Runbook in the Azure Pack admin site named “Maintenance-SOFS-BalanceNodes” . Then I’ve added my PowerShell Workflow to the runbook:

The Script can be downloaded in the bottom of the post

###################################################################
#    Writen by Mattias Lehmus, TrueSec
#    Twitter: @onelehmus
#    Blog: onelehmus.com
#
#    Original Script: http://blogs.technet.com/b/josebda/archive/2013/04/17/file-server-tip-how-to-rebalance-a-scale-out-file-server-using-a-little-powershell.aspx
#
#    THIS CODE IS PROVIDED *AS IS*
###################################################################

workflow Maintenance-SOFS-BalanceNodes
{
param
(
[Parameter(Mandatory=$true)]
[string] $ClusterName,
[Parameter(Mandatory=$true)]
[string] $ClusterCredentials
)

# Connection to access Cluster Nodes.
#The ClusterCredentials should be a PowerShell Credential Asset in SMA with a user that has Admin rights on the cluster nodes
$CLCredentials = Get-AutomationPSCredential -Name $ClusterCredentials
$CLName = $ClusterName

inlinescript {
#Variables
$Clustername = $Using:CLName
$CimSession = New-CimSession -ComputerName $Clustername -Credential $Using:CLCredentials

        $clusterNodes = Invoke-Command -ComputerName $Clustername -ScriptBlock {Get-ClusterNode} -Authentication Default -Credential $Using:CLCredentials
Write-output $clusterNodes

        $witnessClientObject = @(Get-SmbWitnessClient -CimSession $CimSession | %{
$clientObj = @{};
$clientObj[‘WitnessClient’] = $_;
$clientObj[‘OpenFileCount’] = @(Get-SmbOpenFile -CimSession $CimSession -ClientUserName “*$($_.ClientName)*”).Count;
New-Object PSObject -Property $clientObj
} | sort-object OpenFileCount -Descending)

        if($witnessClientObject.count -gt 0)
{
Write-Output “Found $($witnessClientObject.Count) objects”
$witnessClientObject | ft {$_.witnessclient.ClientName}, {$_.OpenFileCount} -a
Write-Output “Getting node distribution”
$distributionOfFiles = @($witnessClientObject | Group-Object {$_.WitnessClient.FileServerNodeName})
$distributionObjects = @()

foreach($distribution in $distributionOfFiles)
{
$distributionObject = @{}
$distributionObject[‘FileServerNodeName’] = $distribution.Name
$distributionObject[‘OpenFileCount’] = ($distribution.Group | Measure-Object OpenFileCount -Sum).Sum
$distributionObject[‘Clients’] = $distribution.Group
$distributionObjects += New-Object PSObject -Property $distributionObject
}

            #add in any cluster nodes that have 0 witness connections

            foreach($unusedClusterNode in ($clusterNodes |? { $name = $_; -not($distributionOfFiles |?{ $_.Name -match $name}) }))
{
$distributionObject = @{}
$distributionObject[‘FileServerNodeName’] = $unusedClusterNode
$distributionObject[‘OpenFileCount’] = 0
$distributionObject[‘Clients’] = @()
$distributionObjects += New-Object PSObject -Property $distributionObject
}

            #sort by the number of open files per server node

$sortedDistribution = $distributionObjects | Sort-Object OpenFileCount -Descending
$sortedDistribution |%{ Write-Output “$($_.FileServerNodeName) – $($_.OpenFileCount)”}

#Balance where needed

            for($step = 0; $step -lt $sortedDistribution.Count/2; ++$step)
{
#Get the difference between the largest and smallest file counts for this step
#divide by two so we don’t flop a single connection back an forth on each run
$currentFileOpenVariance = [Math]::Ceiling(($sortedDistribution[$step].OpenFileCount – $sortedDistribution[-1 – $step].OpenFileCount)/2)
Write-Output “Variance for step $($step): $($currentFileOpenVariance)”
$moveTargets = @()
$moveOpenFiles = 0

foreach($client in $sortedDistribution[$step].Clients)
{
if($client.OpenFileCount -gt 0)
{
$varianceAfterMove = ($moveOpenFiles + $client.OpenFileCount)
Write-Output “Checking $($varianceAfterMove) to be less than or equal to $($currentFileOpenVariance) to be a move target”
if($varianceAfterMove -le $currentFileOpenVariance)
{
Write-Output “Client $($client.WitnessClient.ClientName) is a target for move”
$moveTargets += $client.WitnessClient
$moveOpenFiles += $client.OpenFileCount
}
}
}

            if($moveTargets.Count -gt 0)
{
foreach($moveTarget in $moveTargets)
{
Write-Output “Moving witness client $($moveTarget.ClientName) to SMB file server node $($sortedDistribution[-1 – $step].FileServerNodeName)”
Move-SmbWitnessClient -CimSession $CimSession -ClientName $moveTarget.ClientName -DestinationNode $sortedDistribution[-1 – $step].FileServerNodeName -Confirm:$false -ErrorAction Continue | Out-Null
}
}
else
{
Write-Output “No move targets available”
}
}
}
Write-Output “SMB Witness client connections should now be as balanced as possible”
}-PSComputerName $CLName -PSCredential $CLCredentials
}

To run the runbook you need to type in the name of the powershell credentials and the cluster name.

SMA_StartRBBalanceSOFS

After a succesful test run I’ve added a schedule to the runbook, in this case I’ve created a schedule to run the script daily at 23:30.

SMA_SCHEDULEBalanceSOFS

In the picture you can see how it looks after the schedule runs the runbook, there are also an output available to see what the runbook has done.

SMA_DashboadBalanceSOFS

SMA_OutputBalanceSOFS

I hope that can give some ideas of how you can use SMA instead of Scheduled Tasks to do maintenance.

The runbook can run on more than one cluster on a schedule. You just need to create a new schedule with another cluster name and other PS credentials if needed as parameters.

//Mattias

Link to Script: http://1drv.ms/1MUm1el


All scripts in the blog post is delivered AS IS, test it before using it in production!
This entry was posted in Powershell, SMA and tagged , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s