vSphere Daily Environment Checking
vSphere is a renowned hypervisor platform and definitely this need not to be explained. VMware started some times ago (about 2 years) in building management stacks over the vSphere targeting to make sure the vSphere is being manageable and usable. Of course, the vRealize Operation Manager (vCenter Operation Manager) is a perfect tool for Capacity Planning, Holistic Monitoring and Reporting for your vSphere Environment. Following is the Home Page of the Operation Manager if you have never come across it.
This is a separate licensed product which definitely be worthy. Yet this blog post is not focusing in this product, we are doing something more dirty. This is driven by one of my customer who would like to have a daily health check of their vSphere Environment, so instead of a comprehensive Monitoring solution what they asked for is a configuration and status dump everyday. Thus, with the help of VMware PowerCLI and RvTools, I’ve implemented sort of Daily Report for their environment. I believe this maybe useful for many of you, so let me share the steps on implementing it.
Let’s have a look in the result first, i actually provide the customer two dumps for each of the vCenter Servers they have in their environment. While one is configuration based and another is status based dump and they are being provided by:
- RvTools – Configuration Dump
- vCheck Reports – PowerCLI based custom report for Point of Time Status Dump
We need one Windows Machine for running both Tools mentioned above. If you are using a Windows Based vCenter Server, you can actually use itself to configure this Health Check Mechanism. But if you have multiple vCenter Server Instances in the environment, a Dedicated Windows Server could be more useful. Followings are the two free tools we need:
RvTools from http://www.robware.net
PowerCLI from VMware WebSite
Under the hood, what I did is simple, I setup scripts for running RvTools Export and PowerCLI based vCheck Reports. And kicking off the scripts with Windows Scheduled Tasks.
After setting up the pre-requisites, we can start setting up the scripts and scheduled jobs as followings:
Configuration Dump Scripts
If you have use RvTools before, it’s a very simple but useful tools for dumping out the environment configurations. If you have used before, it can actually capture VM, Host, Virtual Switch, PortGroup, Datastore…etc objects. The configuration dumps are enough even in event if you need to rebuild anything from scratch again.
While the above screen captured the GUI output from the RvTools, but it can actually be exported into either CSV or Excel format. And the RvTool actually supports command line execution rather than just GUI driven, this is also the way I used to export the Configuration Dump Daily. And what you have to do is by copying the following script as a Batch file and Run this Batch file with the Windows Scheduled Tasks.
rem Include robware/rvtools in searchpath
set path=%path%;c:\program files (x86)\robware\rvtools
rem Set environment variables
set $VCServer=<The IP/Hostname of Your vCenter>
set $SMTPserver=<The IP/Hostname of Your Mail Server>
set $Mailsubject1=RvTools Daily Export
set $AttachmentFile=<File Prefix>_%$MYDATE%.xls
rem Start RVTools batch
rvtools.exe -u <vCenter Username> -p <vCenter Password> -s %$VCServer% -c ExportAll2xls -d %$AttachmentDir% -f %$AttachmentFile%
rem Send mail
rvtoolssendmail.exe /smtpserver %$SMTPserver% /smtpport %$SMTPport% /mailto %$Mailto% /mailfrom %$Mailfrom% /mailsubject %$Mailsubject% /attachment %$AttachmentDir%\%$AttachmentFile%
rem Move Files to Archive
move %$AttachmentDir%\%$AttachmentFile% %$AttachmentDir%\Archive
The script above will generate Excel based Export and also send it out as an attachment to the target recipient. On setting up Corresponding Windows Schedule Task, we can do a daily reporting. Followings are the sample setup.
Environment Status Dump Scripts
Besides configuration dump, more than often, many customer would also like to have a status dump on reporting stuffs like Snapshot Usage, Host Usage Status, NTP Status… etc. These information are not retrievable from the RvTools as mentioned above. And not to mention, we actually cannot have a very nice report formatted export from the RvTools. This is why besides the Configuration Dump above, I also created another Scheduled Task for running a PowerCLI based Script Report Daily. Following is a sample output of the vCheck Report we leveraged. It is developed and shared by VMware at this link.
Actually it is a bunch of Scripts developed in PowerCLI to export a lot informations including the followings:
- General Details
- Number of Hosts
- Number of VMs
- Number of Templates
- Number of Clusters
- Number of Datastores
- Number of Active VMs
- Number of Inactive VMs
- Number of DRS Migrations for the last days
- Snapshots over x Days old
- Datastores with less than x% free space
- VMs created over the last x days
- VMs removed over the last x days
- VMs with No Tools
- VMs with CD-Roms connected
- VMs with Floppy Drives Connected
- VMs with CPU ready over x%
- VMs with over x amount of vCPUs
- List of DRS Migrations
- Hosts in Maintenance Mode
- Hosts in disconnected state
- NTP Server check for a given NTP Name
- NTP Service check
- vmkernel warning messages ov the last x days
- VC Error Events over the last x days
- VC Windows Event Log Errors for the last x days with VMware in the details
- VC VMware Service details
- VMs stored on datastores attached to only one host
- VM active alerts
- Cluster Active Alerts
- If HA Cluster is set to use host datastore for swapfile, check the host has a swapfile location set
- Host active Alerts
- Dead SCSI Luns
- VMs with over x amount of vCPUs
- vSphere check: Slot Sizes
- vSphere check: Outdated VM Hardware (Less than V7)
- VMs in Inconsistent folders (the name of the folder is not the same as the name)
- VMs with high CPU usage
- Guest disk size check
- Host over committing memory check
- VM Swap and Ballooning
- ESXi hosts without Lockdown enabled
- ESXi hosts with unsupported mode enabled
- General Capacity information based on CPU/MEM usage of the VMs
- vSwitch free ports
- Disk over commit check
- Host configuration issues
- VCB Garbage (left snapshots)
- HA VM restarts and resets
- Inaccessible VMs
- Much, Much more…….
More than often, the above covers most usual status healthiness problem of an environment. But you can always add more Scripts on top of the vCheck Report to enrich the content of it. This is something I have done for my customer. The vCheck Report also supports sending out as attachment in Mail. But I edited it to export and store locally in Parallel.
Yet, instead of showing how to edit the PowerCLI scripts. I think it’s more important to show how the scheduled tasks are being created as we need some tweaks to Run PowerCLI script from Windows Scheduler. To achieve this, we would need to create a Batch File which is runnable by the Scheduled Tasks while the Batch File help kicking off the PowerCLI task. Following please see the one line batch file you need to prepare.
powershell.exe -ExecutionPolicy Unrestricted -NonInteractive -NoProfile -File C:\vCheck\vCheck-vSphere-master\vCheck.ps1
This Batch Script will Call the vCheck.ps1 which run and export the vCheck Report. And the Scheduled task in the Windows can be configured as following:
So, with these two tasks being setup, everyday you can have both your configuration dump and environment status dump on hand. Such that, it’s far easier for you to manage the environment when your vSphere keeps going. It also let you have a kind of daily backup of your environment in case incident outbreak and something has to be rebuild from scratched. Again, this is definitely not a comprehensive monitoring solution and reporting tools. But this gives some sort of ease in management if you don’t have a vRealize Operation Manager yet in the environment.