Get a server health status report for multiple servers
As administrators, we juggle multiple servers at a time. Hardly do we deal only with a handful servers. Imagine a situation where you use SCCM to patch your servers, and that the patching is completely automated using an automatic deployment rule and a maintenance window. Chances are, you are patching seventy servers at a time. Unattended.
This is a dream-come-true for many environments that have thousands of servers. However, it is also true that every good administrator wants to ensure their servers are up and running healthy, after the patching process. One way of ensuring this is to log into each of the servers and checking their health status. Another way of doing this is to wait for, say, SCOM to throw an alert. Neither is a good way, and as a good administrator, you would not rely on these methods. Nor should you.
This post discusses:
The requirement
A couple of years ago, when faced with a similar situation, I sat down to quickly write a function that was tailored to the environment that I was working on. This environment involved Citrix servers, hosting the frontend for the client’s ERP application. The ERP backend team had scheduled a maintenance activity for a certain day every month, after which the backend servers rebooted. They thought it would also be nice for the frontend servers to reboot at this time. They wanted a way to get a health status report from the frontend servers, along with some information about Citrix, to ensure the users could connect to the application without challenges.
The report
I decided to include the following data in the report:
- The server name
- Whether the RDP port is open on the server
- For how long the server has been up
- Whether the server is registered with the Citrix Delivery Controller
- Whether Maintenance Mode is turned on on the server
- A list of all automatic-but-not-running services
The service list would have to be separate because each server may have more than one automated service that isn’t running. I decided to make that a separate table.
I did not want this report to be naked text—that could look ugly. So I added some basic HTML styling to it.
The script
This script uses a slightly different approach. This starts with what is called the “main function”. The reason behind this is that the output of the report is non-standard non-object-like data. There is nothing return-ed in the regular sense. Which means, there’s some dirty work being done. If we make all of this into a single monolith, most of the script would not be reusable. In order to make the code reusable, you need to ensure each function does only one thing, and returns only one kind of object.
Therefore, I wrote three functions which each did only one thing, and each returned only one kind of object. The “dirty” work is all handled by main; work such as creating an email, adding styling to it, and things like that. So, this main function accepts all the necessary input, calls the other functions and collects the data, consolidates all the data, adds styling to the data to create a nice html, and then, sends the generated html report as an email to the administrators.
Here is the complete script I wrote (also available on GitHub).
function main {
begin {
$Wintel = 'citrixadmins@domain.com', 'windowsadmins@domain.com'
$Servers = 'CTXSVR000', 'CTXSVR001', 'CTXSVR002', 'CTXSVR003', 'CTXSVR004', 'CTXSVR005', 'CTXSVR006', 'CTXSVR007', 'CTXSVR008', 'CTXSVR009', 'CTXSVR010', 'CTXSVR011', 'CTXSVR012', 'CTXSVR013'
$Controller = 'CTXCTL.domain.com'
$ExcludedServices = 'clr_optimization_v4.0.30319_64', 'clr_optimization_v4.0.30319_32', 'sppsvc', 'stisvc'
$style = "<style>BODY{font-family:'Segoe UI';font-size:10pt;line-height: 120%}h1,h2{font-family:'Segoe UI Light';font-weight:normal;}TABLE{border:1px solid white;background:#f5f5f5;border-collapse:collapse;}TH{border:1px solid white;background:#f0f0f0;padding:5px 10px 5px 10px;font-family:'Segoe UI Light';font-size:13pt;font-weight:normal;}TD{border:1px solid white;padding:5px 10px 5px 10px;}</style>"
$SmtpServer = 'smtp.domain.com'
$From = 'CitrixMonitor@domain.com'
$Subject = 'Post-reboot service check on ERP Citrix Servers'
$ServerStatusReport = @()
}
process {
$ServiceStatus = Get-ServiceStatus -ComputerName $Servers -Exclude $ExcludedServices
if ($ServiceStatus) {
$ServiceStatusReport = $ServiceStatus | Select-Object ServerName, ServiceName | ConvertTo-Html -As Table -Fragment -PreContent '<h2>Service Status</h2><p>Here are the services that are set to start automatically on each of the ERP Citrix servers, but are not running post reboot.</p>' | Out-String
}
else {
$ServiceStatusReport = '<h2>Service Status</h2><p>All the critical services on all the ERP Citrix servers are running after the reboot.</p>'
}
foreach ($Server in $Servers) {
$WindowsStatus = Get-WindowsStatus -ComputerName $Server
$CitrixStatus = Get-CitrixStatus -ControllerFqdn $Controller -ComputerFqdn "$Server.domain.com"
$ServerStatusReportEntry = [ordered]@{
Server = $Server
RdpPortOpen = $WindowsStatus.RdpPortOpen
UpTimeMins = $WindowsStatus.UpTimeMins
RegistrationState = $CitrixStatus.RegistrationState
InMaintenanceMode = $CitrixStatus.MaintenanceMode
}
$ServerStatusReport += New-Object PSObject -Property $ServerStatusReportEntry
}
$ServerStatusReport = $ServerStatusReport | ConvertTo-Html -As Table -Fragment -PreContent '<h2>Server Status</h2><p>Also, here is a look at the other important parameters pertaining to the ERP Citrix servers.</p>' | Out-String
$Body = ConvertTo-Html -Head $Style -Body '<p>Hi Team,</p><p>The post-reboot service test was run on the ERP Citrix servers. The service status report follows.</p><h1>ERP Citrix Server Health Check Report</h1>', $ServiceStatusReport, $ServerStatusReport, '<p>Have a great day!</p><p>Regards,<br>Citrix Master Monitor</p>' | Out-String
Send-MailMessage -SmtpServer $SmtpServer -From $From -To $Wintel -Subject $Subject -Body $Body -BodyAsHtml
}
}
function Get-WindowsStatus {
param (
# Server names
[Parameter(Mandatory=$true, Position=0)]
[string]
$ComputerName,
# The RDP port number
[Parameter(Mandatory=$false, Position=1)]
[string]
$Port = '3389'
)
if (Test-Connection $ComputerName -Count 1 -Quiet) {
try {
$Null = New-Object System.Net.Sockets.TCPClient -ArgumentList $ComputerName, $Port -ErrorAction Stop
$PortStatus = 'Yes'
}
catch {
$PortStatus = 'No'
}
}
else {
$PortStatus = 'No'
}
try {
$Osdetails = Get-WmiObject win32_OperatingSystem -ComputerName $ComputerName -ErrorAction Stop
$UpTimeRaw = (Get-Date) - ($Osdetails.ConvertToDateTime($Osdetails.LastBootupTime))
$UpTime = [math]::Round($UpTimeRaw.TotalMinutes, 0)
}
catch {
$UpTime = 'Unable to fetch'
}
$Properties = [ordered]@{
Server = $ComputerName
RdpPortOpen = $PortStatus
UpTimeMins = $UpTime
}
New-Object PsObject -Property $Properties
}
function Get-CitrixStatus {
param (
# Server FQDN
[Parameter(Mandatory=$true, Position=0)]
[string]
$ComputerFqdn,
# Controller FQDN
[Parameter(Mandatory=$true, Position=1)]
[string]
$ControllerFqdn
)
begin {
try {
if (!(Get-PSSnapin Citrix* -ErrorAction SilentlyContinue)) {
Add-PSSnapin Citrix*
}
else {
Write-Verbose "Citrix snap-in is already loaded"
}
}
catch {
Write-Warning "Unable to load the Citrix snap-in"
break
}
}
process {
try {
$BrokerMachine = Get-BrokerMachine -AdminAddress $ControllerFqdn -DNSName $ComputerFqdn -Property InMaintenanceMode -ErrorAction Stop
$MaintMode = $BrokerMachine.InMaintenanceMode
$RegState = $BrokerMachine.RegistrationState
}
catch {
$MaintMode = 'Unable to fetch'
$RegState = 'Unable to fetch'
}
$Properties = [ordered]@{
Server = $ComputerFqdn
MaintenanceMode = $MaintMode
RegistrationState = $RegState
}
New-Object PsObject -Property $Properties
}
}
function Get-ServiceStatus {
param (
# Names of the servers
[Parameter(Mandatory=$true, Position=0)]
[string[]]
$ComputerName,
# Services to be excluded
[Parameter(Mandatory=$false, Position=1)]
[string[]]
$Exclude = $null
)
begin {
$ServiceTable = @()
}
process {
foreach ($Server in $ComputerName) {
$Services = (Get-WmiObject win32_service -Filter "StartMode = 'auto' AND state != 'Running'" -ComputerName $Server |
Where-Object Name -notin $Exclude).DisplayName
foreach ($Service in $Services) {
$Properties = [ordered]@{
ServerName = $Server
ServiceName = $Service
}
$ServiceTable += New-Object PSObject -Property $Properties
}
}
$ServiceTable
}
}
main # Call the main function
What the script does
We will talk about main in the end.
The first function you see is Get-WindowsStatus. This function is designed to look for the uptime of a server and whether its RDP port is open or not. Checking the RDP port is done by creating a System.Net.Sockets.TCPClient object, which is equivalent to telnet. This function works only on one server. This has been done to keep it simple. If you want this status to be shown for multiple servers, do it this way:
'SERVER01', 'SERVER02', 'SERVER03' | Foreach-Object { Get-WindowsStatus $PSItem }
If you have set a port other than 3389 as the RDP port, specify the port when calling the function. For example,
'SERVER01', 'SERVER02', 'SERVER03' | Foreach-Object { Get-WindowsStatus $PSItem -Port 45980 }
The next function, Get-CitrixStatus, is a simple one as well. This function accepts the fqdn of the server as well as that of the Delivery Controller. It loads the Citrix snap-in (only if the snap-in is not loaded onto the session), and then calls Get-BrokerMachine to get the necessary details for the Citrix server. Note that the property, InMaintenanceMode, has to be explicitly called for, like how you would call specific properties with Get-ADUser. We wrap this call within a try–catch block to handle errors better. You can customise the catch block, or add more error-specific catch blocks as per your requirement.
This function outputs an object that consists of the server name, whether the server is in maintenance mode, and whether the server is registered with the delivery controller.
Next, the Get-ServiceStatus function accepts multiple computer names as input, along with the optional Exclude parameter, which specifies the services that can be excluded. In my case, there were a few .NET services that we wanted to exclude.
This function makes a call to wmi on the server to fetch the service details. You can modify this to call Get-Service instead if you would like. This function checks for all the services that are set to automatically start, but aren’t running. The call has a Where-Object-based filter, that honours the exclusion list, and picks only the display names of the services. These services are then looped through to create the object we need.
At the end of the script, we make a call to the main function.
The main function is a little dirty. It has a good lot of definitions, including the email addresses to which the email report should be delivered. We then call Get-ServiceStatus to check if there is any server that matches our filter. If there’s none, the report would contain just a subheading, “Service Status”, and say that all the critical services are running. (It wouldn’t show an empty table.) Otherwise, the table would contain which service on which server matched our criteria.
Then, we loop through the entire server list. We call the two remaining functions for each of the servers, and get the required data. We then build an object using the data that we got from the two functions.
This data from all the three functions is converted to html tables, which are made into html fragments. (An html fragment does not contain its own head and body; it is just that table, essentially.) We add some pre-content, which is something that would be prepended to the table. All of this is wrapped into a single fragment. We pipe this to Out-String to force the object to be plain string—html tags and everything, in plain text.
Next, we build the html body. We add more text to the body, and then, add the styling to its head. This makes a complete html object. We pipe this to Out-String again, so we can use this with the Send-MailMessage cmdlet.
Finally, we call the Send-MailMessage cmdlet and tell it that the string we passed to it as Body is in fact, html. Therefore, the email that is sent by the cmdlet is a good-looking html report.
Summary
In this post, we looked at creating a health check report that is informative, as well as good-looking. (User experience matters.) This report contains details from Windows as well as Citrix standpoints; details such as the computer’s uptime, status of the RDP port, whether the server is registered with the Delivery Controller (XenApp 7.6), and whether the server is in maintenance mode. Also, the report contains a list of services that are supposed to automatically start, but aren’t running. This function also accepts exceptions: services that do not need to be monitored. The script creates an html report and sends it as an email to the specified recipients.
The script can be scheduled in Scheduled Tasks on Windows, to automatically run and send the report at a given time or interval.
I hope this script helps any other Citrix-Windows administrators out there, or helps you understand the way PowerShell objects work to give you the data you require.