DRaaS dreams come true! vCloud Availability for vCloud Director (vCAV) – For Tenant

Following the previous lengthy setup of vCAV for the Service provider, and after we enabled a certain customer and one’s OVDC to have DRaaS to the cloud. We then need to start configuring the tenant side. And actually this is just thru’ one component, the vSphere replication appliance.

Tenant Setup

So, it’s comparatively easy for a tenant setup, as VMware doesn’t expect a Cloud User has to do a lot to consume the Cloud for DR. Actually, for production environment, you could only just need to setup the vSphere Replication at tenant side. But for test and development environment, we would need do a bit more. So followings are the steps to enable it.

  1. Prepare Your Environment to Install vSphere Replication – Check and Configure the vCenter FQDN Entry under advanced settings of the vCenter Server
  2. Deploy the vSphere Replication Virtual Appliance – I’m using 6.1.1. vSphere Replication for my testing, you can just deploy it like any ordinary vSphere Replication Appliance (as it is one actually).
  3. Register the vSphere Replication Appliance with vCenter Single Sign On
  4. Using a Self-Signed Certificate in a Development Environment –  then you would need to force trust the vCloud Director Cert from the ApplianceYou need to use the following command on the vSphere Appliance
    • # openssl s_client -connect cloud.vmwarehk.lab:443 -tls1 </dev/null 2>/dev/null | openssl x509 > /tmp/vcloud.pem
      # /usr/java/default/bin/keytool -noprompt -import -trustcacerts -alias vcloud -file /tmp/vcloud.pem -keystore /usr/java/default/lib/security/cacerts -storepass changeit

    • # service hms restart
      # service vmware-vcd restart

  5. Configure Cloud Provider – We need to add a Cloud Provider form the tenant side to replicate our VM into, this provides the exact same steps as if you are connecting to the vCloud Air for the DRaaS. Start doing this by click the highlighted icon which you may probably didn’t hit beforeThen you need to input the Address of your original VCD (NOT the Cloud Proxy) and input the organization and necessary credential Select the organization VDC from the list which has been enabled with the DRaaS  Confirm the setup and you will see an alert under the status So you need to click the alert entry to configure the networks for the Target Cloud Side, on setup the network, you have to problem Recovery Network and Test Network which would be used for an actual recovery and test drill correspondingly. Just like VMware Site Recovery Manager (SRM), you can map the recovery network to the network port group on premise And another network for DR Drill network Then you are good to click Finish to confirm the configuration The Alert will be gone after the configuration
  6. Configure Replication – So finally, you can configure the replication of your VM. This is just same as how you replicating a VM with ordinary vSphere Replication Operation. So you can start by right clicking a VM on premise Then Click “Replicate to a Cloud Provider”, amazing… this was only working with vCLoud Air Before Head… Boom! You can see your OVDC on your vCD environment, Click it And Select the Storage Policy to be copied towards Enable WAN Compression if you wanna save bandwidth Choose the RPO and enable Point in Time instances if you wanna recover the VM in different Snapshot of timeYo the replication starts. ***it’s normal to see unknown under VR server***

Great! Let’s perform the test! Yes, I think replication is not something you wanna test right? The test so called should be how we can perform DR Drill and Recovery, right?

DRaaS Testing

On completing both the service provider and tenant setup, we can finally perform the DRaaS test. So let’s start with looking at what your Cloud is like after the replication is completed. It’s as following:

You can see a replicated VM is being created on your Cloud. So let’s see what it will be when we are performing the testing recovery or actual recovery. And as mentioned, there is a new UI come with vCAV, the vCloud Availability for vCloud Director Portal. It’s designed to provide a dedicate portal for tenant to manipulate the DR actions. I will thus cover both the actions you can performed thru’ existing vSphere Web Client on premise and also by the vCAV UI at service provider side.

DR Drill (Test Recovery)

So let’s talk about vCAV UI first, this is by opening a browser and go to http://{vCAV-UI}:8443, login with the VCD credential and you will be directed to a summary page.

To trigger the test failover, go to the “Workspaces” tab and click the VM you would like to test failover with. And you could see the buttons on the right hand side control pane. Click the “Test” and “Start” afterwards to trigger the test failover.

The UI will reflect the status and result in failover

Of course, you can see the same result at the vSphere Web Client side.

So you can also trigger test recover from the vSphere Web Client directly too

But you would have to login on triggering the recovery, this is because you didn’t login the VCD on login the Web Client.

You got more option to choose from using the test recover in the Web Client

On confirmation the test recover will be executed

Again, the status and result of test failover will be visible from the vSphere Web Client

So how actually a test failover looks like? Just like the Site recovery Manager, you production workload will continue to run while the DR workload will be brought up but mapped to a testing network.

Two networks are visible on the VM at the Cloud, where one is for production in Actual recovery and a testing one for test recovery.

Test Recovery Clean Up

On completing the test failover, just like Site Recovery Manager again. We need to clean up the test recovered VMs, you can do that from both vCAV UI or vSphere Web Client again.

Do this by click the VM from the Workspaces tab, then select the “Cleanup” action from the right hand side pane.

Same action can be performed at the vSphere Web Client

Actual Recovery

On confirming the succeed in test recovery, we can confirm the actual recovery is good. Where we do not usually test an actual failover, we usually execute it in a planned migration or actually DR scenario. This is why VMware brings the vCAV UI actually, it makes a lot of sense for the later case, when you lost your vSphere Web Client on premise, you still can trigger the DR recovery.

Just like the test recovery action, you can trigger it by clicking the VM on the vCAV UI at the Workspaces tab and then trigger the Failover from the Right Hand Side Control Pane

You can Monitor the Recovery Status from the vCAV UI

Or you can also trigger the planned migration from the vSphere Web Client on premise, remember than you cannot trigger an actual DR here as the vSphere Web Client on premise is likely gone when DR scenario outbreak.

Again you can have more options in recovering a VM comparing with vCAV UI

From the vSphere Web Client, you can monitor the Status of Recovery too.


After recovery, when your site resumed, you could and you should reportect your VM in opposite direction to enable the DR protection even the VM is at the Cloud now.

This can be done at the vCAV UI with the Reverse button

Or the “Reverse Replication” button from the vSphere Web Client on premise

This simply help reversing the replication traffic to replicate the cloud VM back to on premise datacenter.


Finally, you can failback you VM back to the on premise datacenter after the reprotect has been completed. Of course, you should test recover again from the Cloud back to the datacenter first. But I would skip here.

Go to the vCAV UI, Workspaces, and click a VM under “Reversed” tab. Click the Failback button

In vSphere Web Client, instead you cannot see a “Reverse button”, but instead you need to go to the “Incoming Replication” which notes the Cloud to Datacenter replication. And Failback the VM by Start Recovery button

So on successful tailback, you should see all your VM back online in your datacenter and the tasks are Recovered Successfully

Great enough? That’s the DRaaS which is enabled by the vCAV and this is how simple you as an tenant can consume it from any of your vCAN service provider who has deployed it. I wish this is helpful for you!


DRaaS dreams come true! vCloud Availability for vCloud Director (vCAV) – For Service Provider

As introduced by the previous blog post, I’m going to brief the Service Provider side setup steps of vCloud Availability for vCloud Director (vCAV) here. The following summarise the components we would setup around the existing vCD Cells.

Service Provider Setup

From the service provider side, I’m assuming you have you vCloud Director deployed. In this blog, my vCloud Director is of 8.10 version, it’s a single cell deployment but leverage the one IP deployment method (sharing http and consoleproxy IP) which is a new feature in version 8.10. I’m following the official guide to:

  1. Prepare the environment
  2. Install the new vCAV components
  3. Configure the new vCAV components

As I’m just testing in the environment, some of the components I’m deploying for test and development purpose only, say I’m using Docker for my MQ and Cassandra. I’m deploying single instance of some components which you should actually consider NLB-ing multiple of them in your production environment. But anyway, just let see how simple stuffs work first. Again the steps I performed follows the official guideline which you can refer at HERE.

Preparing the environment

  1. Create vCloud Availability Installer Appliance – Trivial step, you would need to deploy the vCloud Availability Install Appliance from VMware.com. This component is the central control centre which you would use to install further the vCAV components. DO NOT delete this appliance even after your setup, as you would need this in case when you have to scale or reconfigure your DRaaS environment.
  2. Download vCloud Availability for vCloud Director Appliances – As mentioned, the installer Appliance in step one only help deploying the vCAV, but we still need to download the core vCAV appliances for the deployment. You can Download it from the VMware.com.We need to upload the downloaded vCAV Components binary into the vCAV Installer Appliance. Do this by using SFTP and take note that you would need to provide the path to these OVA/OVF files during the vCAV deployment. 
  3. Configuring vCloud Director for Installation – You would have to prepare your vCloud Director with the following items: (If you are interested in the detail steps for doing any of the following, you can refer to the Appendix of this blog series HERE. Or it could be tooooooo lengthy)
    • Use Wildcard certificate for the vCloud Director if you are not using it (I’m not…)
    • Migrate your Single Cell vCloud Director to a Multiple Cell Configuration (as for deploying the Cloud Proxy for vCAV).
    • Deploy and Configure MQ with SSL (This is not default for RabbitMQ).
    • Join the vCloud Director to the lookup services
  4. Prepare the vCloud Availability Installer Appliance for vCloud Availability for vCloud Director Installation – While there are two ways of vCAV installation, the “Full Commands Installation” and the “Simple Command Installation”, I am using the “Simple Command Installation” in this blog. And this is why I have to create the registry file under the location ~/.vcav of your vCAV Installer Appliance. The registry file defines the vCenter and vCloud Director endpoint information which is be used for component deployment, e.g. the which Datastore and Host should the Components be deployed on. So for my case, my registry file looks like the following:
  5. Enable Static IP Addresses Deployment – Besides following the official guide to create the IP pool. Which you can do it thru’ vSphere Client UI if you are more familiar with it, but the command provided in the official guide also works great. Do also assign static IPs for components and configure the AAAA record in the DNS. You would need to have at least 7 IP Addresses for the following components on the service provider:
    1. vCAV Installer x 1 IP (Hostname: vcav-installer)
    2. vCloud Proxy x 1 IP (Hostname: vmlvcd02)
    3. vSphere Replication Cloud Service (vRCS) Host x 1 IP (Hostname: vmlhcs01)
    4. vSphere Replication Manager (vRMS) Host x 1 IP (Hostname: vmlhms01)
    5. vSphere Replication Server (vRS) x 1 IP (Hostname: vmlhbr01)
    6. vCAV UI Appliance x 1 IP (Hostname: vmlvui01)
    7. (Optional) Docker Host x 1 IP (Hostname: vcav-docker01)
  6. Create a Trusted Connection with the vCloud Availability Installer Appliance – Ensure the certificates using in the environment are being (force) trusted by all the components.
  7. Create Cloud Proxy – This is actually another vCloud Director Cell yet with most of the functions being disabled. Cloud Proxy is required for handling the DRaaS tasks. This is why I’ve mentioned we have to migrate from a single cell deployment to a Multiple cell one. And the following red circled line is the corresponding setting in the global.properties file on the Cloud Proxy Cell
  8. Creating Containers (Optional) – If you have a MQ and a Cassandra in your environment already, you can skip this step. And if you are preparing for a production environment, skip this step too. This step is just for test and development purpose. We can leverage Container technology to deploy the MQ and Cassandra DB for the vCAV easily. (This is why VMware Loves Container). I use the Docker in my environment, you can following the official guide session HERE. It instructs you how to create the docker host and further the docker image and configuration for both MQ with SSL and Cassandra.We can deploy the docker host for running the MQ and Cassandra with the following Command
    • vcav docker create –vsphere=vmlvcs01 –ntp= –root-password-file=~/.ssh/.root –vm-name=vcav-docker01 –vm-address

  9. Deploy the MQ and Cassandra Images – After deploying the Docker Host, we can deploy the images on top and you can do this by doing the following at the vCAV Installer:
    • systemctl start docker

      docker pull cassandra:2.2
      docker pull rabbitmq:3.4

      vcav amqp create –docker-address= –container-name=vcav-amqp01 –amqp-port=5671 –amqp-user=vcd –amqp-password-file=~/.ssh/.amqp

      vcav cassandra create –docker-address= –container-name=vcav-cass01 –cassandra-port=9042

      vcav trust add –address= –port=5671 –accept-all

  10. Check vCloud Director Endpoints – After preparing the vCloud Director Environment, we definitely have to validate the preparation before we moving on to Install the Components. You could find my command a bit different from the one in the official document, perhaps you would need to add “-k” as mine too as my wildcard cert is not being secure enough. So in case you see the above message, you need to use a command like mine

Installing the new vCAV Component

Then we need to leverage the vCAC Installer Appliance to help deploying the vCAV Components actually here. And the step has to be preformed in CLI, so I would recommend you using SSH rather than the VM Console (you know you cannot copy and paste). So before running the actual installation commands, you have to prepare the password file which can be handy to ensure you didn’t type the password wrongly during deployments. Do this by running the following command at the vCAV Installer:

mkdir ~/.ssh
chmod 0700 ~/.ssh
echo ‘P@ssw0rd’ > ~/.ssh/.root
echo ‘P@ssw0rd’ > ~/.ssh/.amqp
echo ‘P@ssw0rd’ > ~/.ssh/.vsphere.sso

echo ‘P@ssw0rd’ > ~/.ssh/.truststore
find ~/.ssh -type f -name ‘.*’ -print0 | xargs -0 chmod 0600

And we need to follow the step below to setup all the vCAV components.

  1. Create vSphere Replication Manager (vRMS) Host – This may not be too unfamiliar for you, as we also need this component in vSphere Replication.The command to deploy vRMS from the vCAV Installer is as following, and you need to customise the IP, hostname and OVF URL for your deployment:
    • vcav hms create \–vsphere=vmlvcs01 –ntp= –root-password-file=~/.ssh/.root –vm-name=vmlhms01 –vm-address –ovf-url=/vCAV/vCloud_Availability_4vCD_OVF10.ovf

  2. Create vSphere Replication Cloud Service (vRCS) Host – As the name of it, this is the engine for enabling the DRaaS or Cloud Replication.The command to deploy vRCS from the vCAV Installer is as following, and you need to customise the IP, hostname and OVF URL for your deployment:
    • vcav hcs create –vsphere=vmlvcs01 –ntp= –root-password-file=~/.ssh/.root –vm-name=vmlhcs01 –vm-address –ovf-url=/vCAV/vCloud_Availability_4vCD_Cloud_Service_OVF10.ovf

  3. Create vSphere Replication Server (vRS) Host – The actual appliance for performing the replication tasks. If you are having a non-testing environment, you would probably need multiple instances of this to handling the actual traffic from customer site
    • vcav hbr create –vsphere=vmlvcs01 –ntp= –root-password-file=~/.ssh/.root –vm-name=vmlhrs01 –vm-address –ovf-url=/vCAV/vCloud_Availability_4vCD_AddOn_OVF10.ovf

  4. vCAC UI Portal Appliance Deployment – This is the appliance for running the separate UI dedicated for the DRaaS for customer consumptionThe command to deploy the vCAV UI from the vCAV Installer is as following, and you need to customise the IP, hostname and OVF URL for your deployment:
    • vcav vcd-ui create –vsphere=vmlvcs01 –ntp= –root-password-file=~/.ssh/.root –vm-name=vmlvui01 –vm-address –ovf-url=/vcloud-availability-for-vcd-ui-ova-1.0.1-b238-4945120.ova

  5. Validate Deployment – To ensure the initial deployment is good, we can validate the deployment with the following command. Again I’ve added “-k” for all the vCAV commands I used to connect to the vCloud Director
    • vcav vcd wait-for-api -k –vcd=vmlvcd01 –timeout=300
      vcav vcd is-federation-enabled -k –vcd=vmlvcd01

Configuring the new vCAC Components

  1. Configure vSphere Replication Manager (vRMS) – Registering the vRMS to the vCenter Server which is managing the Cloud Resource.
    • vcav hms configure -k —hms-address= –vsphere=vmlvcs01 –vcd=vmlvcd01
      vcav hms wait-for-extension -k —hms-address= –vsphere=vmlvcs01 –vcd=vmlvcd01
  2. Configure RabbitMQ Servers – Configure the vCloud Director Cells to use AMQP provided by the Rabbit MQ with SSL enabled. And you need to restart the VCD Services in your nodes after performing this, execute “service vmware-vcd restart” in your VCD nodes, NOT the vCAV Installer.
    • vcav vcd configure-amqp -k –vcd=vmlvcd01 –amqp-address= –amqp-port=5671 –amqp-user=vcd –amqp-password-file=~/.ssh/.amqp –amqp-vhost=/ –amqp-exchange=systemExchange

  3. Configure Cassandra Servers – Connect the vRCS to the Cassandra DB
    • vcav trust add –address= –port=9042 –accept-all
      vcav cassandra register -k –hcs-address= –cassandra-address= –cassandra-port=9042 –vcd=vmlvcd01
      vcav cassandra import-hcs-certificate –docker-address= –container-name=vcav-cass01 –hcs-address=

  4. Configure vSphere Replication Cloud Service (vRCS) – Register the vRCS to the vCenter Server which is managing the Cloud Resource (same as step 1) and Update the vCloud Director Role to include the vRCS privileges. ***Check the firewall if you are hitting issue in the second command***
    • vcav hcs configure -k –hcs-address= –amqp-password-file=~/.ssh/.amqp –cassandra-replication-factor=1 –vsphere-registry-list=vmlvcs01 –vcd=vmlvcd01
      vcav hcs wait-for-extension -k –hcs-address= –vsphere-registry-list=vc-01a –vcd=vmlvcd01
      vcav hcs add-rights-to-role -k –vcd=vmlvcd01 “–role=Organization Administrator”

  5. Congiure vSphere Replication Server (vRS) – Setup the vSphere Replication Server to register  to both vCenter and vCloud Director Resources.
    • vcav hbr configure -k –hbr-address= –vsphere=vmlvcs01 –vcd=vmlvcd01

  6. Configure vCloud Availability for vCloud Director Portal (vCAV UI) Host – Choose the deployment size and setup the UI appliance, using small will be good enough for a test and development purpose.
    • vcav vcd-ui configure –ui-address= –keep-self-signed-certificate –truststore-password-file=~/.ssh/.truststore –vcd=vmlvcd01

  7. Configure Service Provider vCloud Director Organisations – Finally, enable a tenant who is eligible to use the DRaaS. This is not by default enabled feature for all the organisations and all organization VDC. You have to enable it selectively:
    • vcav org-vdc enable-replication -k –vcd=vmlvcd01 –org=SE –vdc=se-ovdc-01

Great! The Service Provider part has been completed and we can proceed to the Tenant Setup. As a Service Provider, I strongly recommend you to have a look into this as this enable a lot of value added services you can provide to your customer on top of IaaS service. Do start the evaluation and i wish this is helpful for you!

DRaaS dreams come true! vCloud Availability for vCloud Director (vCAV) – Introduction


“The biggest thing since vCloud Director” – This is the most suitable phase I could use to describe the VMware vCloud Availability for vCloud Director (vCAV). I have been working in few vCloud Director projects in these years, including vCloud Director with Metro Cluster, vCloud Director Migrations and vCloud Director with Customized Portal On top. Those projects are great and I’ve benefit a lot in cloud building thru’ out. If you know me, you would know how much I love vCloud Director. It’s simple and it’s beautiful. Thus, I love to see how it evolves. I’m looking forward to the vCloud Director 8.20 so much which should be announce soon. But today, I would like to talk about another ultimate weapon which vCloud Director can be equipped with, the vCAV. This is so awesome a value-added functionality for a public cloud and it’s also at the same time so handy for customer to gain a DR solution.

Actually, vCloud Air has been empowering with this function for a while already and you can use that. And what vCloud Availability does, it simply let vCloud Air Network service providers to be equipped with the similar DR as a Service feature. Personally, I’ve been looking into this for so long. Primarily because there are no vCloud Air Coverage at Hong Kong, many of my customers are not allowed to put their VM to AWS or vCloud Air because of regulations for data location. This is why I believe there are chances for local service providers who can offer DRaaS here locally. And this is also why I would like to demonstrate how it actually looks like here, as I believe your local service provider (not just Hong Kong ones) would also be interested in this.

What does vCAV include?

So as mentioned in the previous blogs, vCloud Director is a very handy tool for Service Provider. It helps providing both the Cloud Engine and Portal Service for delivering IaaS service on top of VMware vSphere Environment. But what’s more and what’s next?

Actually VMware is trying to help and encourage Service provider to deploy and develop value added service on top. And vCAV is one which help providing DRaaS for vCloud Director Deployments. To deploy it, you would need the following items:

  • vCloud Availability Installer Appliance
  • Cloud Proxy
  • vCloud Director (vCD)
  • vSphere Replication Cloud Service (vRCS)
  • vSphere Replication Manager (vRMS)
  • vSphere Replication Server (vRS)
  • vCloud Availability for vCloud Director Portal (vCAV UI)
  • Cassandra
  • RabbitMQ

From the diagram below, you could found the basic architecture of the solution. Not all the items are being covered, but this diagram provides you the network requirements for deploying vCAV. And the beauty is, you actually only need to enable port 443 from the On Premise to the Provider Cloud.

While I would like to categorise the vCAV components as follows:

  1. Some Middleware – help integrating with existing infra (Rabbit MQ, Cassandra)
  2. Replication Engine – for copying VM data to the Cloud (vRMS, vRS)
  3. Core DRaaS Engine – for managing the VM Replications, Recovery, Drill Test (vRCS, Cloud Proxy)
  4. Separate UI – vCAC 1.0.1 provide another web UI for triggering Recovery, in 1.0 version, this can only be performed by API (vCAC UI)

Although it looks like there are a lot of components to be deployed, the vCloud Availability 1.0.1 provides you a really easy way to deploy all the components. No worry, you don’t have to deploy each of the component manually, you just need to leverage the vCloud Availability Installer Appliance. I would call it a Control Centre in deploying, scaling and reconfiguring the vCAV environment. Thus, it’s more than just an one off installer, I would recommend you NOT to delete it even after the deployment.

The official user guide actually includes step by step commands for your reference to deploy the whole vCAV infrastructure, but I would like to let you know what would be expected symptom and behaviour in each step and this is the reason why I’m writing this series of blogs. This is just a introduction post, and I will further create two blog posts, one for Service Provider and another for Tenant. Wish that would be helpful for you!






DRaaS dreams come true! vCloud Availability for vCloud Director (vCAV) – Appendix

As the Appendix of the captioned blog series, here I would provide the steps in “Preparing the vCloud Director” Step. As mentioned, there are few things we need to enable our existing vCloud Director Deployment before we can deploy the vCAV. To recap, they are:

  1. Use Wildcard certificate for the vCloud Director if you are not using it (I’m not…)
  2. Migrate your Single Cell vCloud Director to a Multiple Cell Configuration (as for deploying the Cloud Proxy for vCAV).
  3. Deploy and Configure MQ with SSL (This is not default for RabbitMQ).
  4. Join the vCloud Director to the lookup services

Generate Wildcard Certificate for vCloud Director Cells

I’m using Active Directory CA in my environment, so I use one of my domain joint machine to request for a wildcard certificate. This can be done at the MMC with the Certificate Snap-in import. Do request for a “Legacy Key” Template with PKCS#10 format.

Friendly name doesn’t have to be the wildcard, i use here just for easy in identification

Input the Subject Detail, CN = wildcard is a critical entry

Enable the Extensions as following

Make the Private Key Exportable

Then proceed to generate the certificate request

Copy the Certificate request content

And request the certificate from the AD

Generate as a Web Server Certificate

Download the Certificates from the AD

And Import it back to the machine we request the certificate

You can then see the wildcard certificate being available on the machine

We then can export it out to the vCloud Director Cells

Upload the Wildcard certificate onto the vCloud Director Cells and you can replace the existing certificates with it according to the VMware KB HERE.

Don’t forget to replicate the wildcard certificate at the vCloud Director Portal

Migrate from Single Cell to Multiple Cell vCloud Director Deployment

As there are numbers of blogs discussing about this. What I would like to recap here will be more high level steps:

  1. Create a NFS share for sharing between target vCD Cells
  2. Copy the files under /opt/vmware/vcloud-director/data/transfer of the existing vCD cell
  3. Stop the vCD service by “service vmware-vcd stop”
  4. Mount the NFS share to the vCD cell at the /opt/vmware/vcloud-director/data/transfer
  5. Start the vCD service by “service vmware-vcd start”
  6. Share the /opt/vmware/vcloud-director/etc/response.properites and certificate keystore among the hosts
  7. Install new vCD cells by mounting the same NFS share and using the response.properites and Certificate keystore

Deploy a Rabbit MQ server with SSL enabled (NOT Container)

I’ve come across a very good blog HERE for configuring the Rabbit MQ with SSL. I am not repeating  it.

Join the vCloud Director to vSphere Lookup services

This may not be difficult for you, as you can follow the standard procedure to add the federation setting at the vCloud Director Admin UI. But remember the following caveats, you would need to put “/cloud” after this URL in the vCloud Director setting. ***Even the hints under the text box didn’t said so*** I’m checking with support team on this cosmetic error.

Then you can just add the Lookup service URL under the Federation Tab

On succeed you would see this and you would have to login with SSO user. So do add SSO users as your system admin by granting the user right. Or if you want to login thru’ local user, do go to the URL at https://vCDFQDN/cloud/login.jsp.

So on completing all the above, your vCloud Director environment is being prepared well and you can continue the vCAV setup!!! Wish this is helpful for you!

Horizon 7 JIT Desktop and App Vol 2.12

From the latest VMware Blog HERE, VMware has just announced the Horizon View 7.1 where a new protocol BEAT is developed and Just in time (JIT) applications are being emphasised in the release. While the term JIT is referring to the instant clone technology which help provisioning VM in a very short time thru’ linked clone technology on both storage and memory aspect. So we can expect the JIT Application in horizon 7.1 would be an instant clone of RDSH hosts, yet as it has not been GA-ed yet, let’s just wait for it.

In this blog, instead, I would like to share the way to use JIT Desktop which is brought about from Horizon 7.0. Someone may said linked clone or JIT desktops are just something provisioning faster than full clone desktops. BUT, i would like to emphasis here again, this is absolutely NOT. In fully clone, we expect user data perhaps will be persisted on the desktop and such that one user may tie to one desktop. But actually Linked clone and JIT desktops target to make desktops a shared pool. We try to enable a pool of Desktops for users to consume such that in case one desktop broke down, they can use another with same user experience directly.

Why we need JIT desktop and App Volume???

To achieve this, we have to understand what make up the user’s “user experience” of a desktop, I think it can be layered as:

  1. Desktop OS layer
  2. Application layer
  3. User Environment layer

While in a physical desktop, we have all the three layers tying tight together. We generally don’t care about how to separate the layers. But in VDI, if we can decouple the layers, actually we can recompose an “user experience” base on a policy and making desktop management easy and possible.

While the JIT desktop from Horizon 7 enables we creating pool of consistent Desktop OS, the App Volume helps instead in providing decoupled applications on top of the Desktop. I didn’t cover it in this post, but VMware User Environment Management can help in performing the User Environment layer management perfectly further on top.

So how to build JIT desktops?

Actually, that’s simple. That’s even more simple than using linked clone in Horizon View. With linked clone deployment, you need to deploy the Horizon View Composer Server, right? But for JIT desktop, you don’t even have to use it at all. So, to begin with, you need to ensure the Horizon View Agent is being properly setup for JIT deployment, as differs from default option during the agent setup. You have to enabled the instant clone when you come to the following page

While the option for “View Composer” and “Instant Clone” are conflicting, we thus need to disable the “View Composer” and enable “Instant Clone” for enabling it, just like the following.

After prepared the image, you have to take a snapshot, just like how you are doing it for linked clone desktops. And then you can open the Horizon Admin page for setting up the instant clone Desktop Pools. But before that, go to the “View Configuration” > “Instant Clone Domain Admins” to add an administrator for performing the instant clone action. As we need to create Domain objects and join domain stuffs, do ensure the user right is proper.

Afterwards you can go the create a new Desktop Pool, there is not much special in provisioning an instant clone pool. Yet, user assignment has to be a floating one.

If not, you won’t be able to see the “Instant Clones” option

Select the snapshot just like how you do it for linked clone desktops

Wait for the completion of the cloning tasks, it is proper for you seeing cp-parent objects appear at least one per hosts and cp-replica one per datastore and cp-template VM there.

You can try the behaviour of the instant clone, if you try to power of the instant clone VM, it will be flagged out again

So after user assignment, you can test it out using either View Client or a browser if you enabled the HTML assess. DO try power off it from View Session too, you will see the magic of JIT desktop.

You don’t actually need to even recompose or refresh Desktop linked clone desktops, as the Desktop life can be as short as a power off action be triggered. And this is how a JIT desktop is being created.

So how to build App Volume?

So afterwards, we have to setup the App Volume Manager first, this is decoupled from Horizon View actually and you can use it with even Citrix VDI. The basic mechanism is we will install an agent into the VDI Desktops such that when a user login, the agent will help presenting the Applications including Desktop Icons and Binaries onto the Desktop. While we try to use VMDK based mounting and running natively on the desktop OS to achieve the application layer, you can expect the speed will be very fast and the performance of the application will not be limited by any sandbox environment.

Thus to achieve the above, we need to:

  1. Deploy the App Volumes Manager + Database
  2. Deploy the App volumes Agent
  3. Capture the Application as AppStacks for presenting

So, let’s get started!

Deploy the App Volumes Manager + Database

While the App Volumes Manager is a Windows based solution, the database required is thus a MSSQL. We need to create a database for the setup,

You can create a login dedicated for this database

So you just need to grant the dbo right of the DB for this new user, do grant the dbo right for the msdb database too.

Then you can start setting up the App Volume Manager on the Windows Server. Do this by download the IOS and mount it and running the setup binary, this is the same binary for setting up both Server and Client Agent

You can select the “Install App Volumes Manager” during the setup

Do select the External DB by “Connect to an existing SQL Server Database”

Input the DB connection string detail and proceed the setup

You could see the App vol icon on the desktop after the successful installation

After the setup, we need to perform the basic configuration of the App Vol Manager. Start by clicking the App vol icon on the desktop. Most of the admin tasks are being done on the web GUI

First you need to input a license or using the evaluation one

Then we have to connect the app volumes to the AD

Afterwards, we need to grant someone in the AD be the admin as the App Volume Admins

Add the vCenter Server under the “Machine Managers”, I choose “Mount On Host” to offload the tasks for vCenter Server which is by default the person to help mounting and unmounting the disks for the VDI Desktops.

You gonna accept the vCenter Certificates

And select the storage for the AppStack to be provisioned to

Start Upload the VMDK template as the seed of the AppStack and Writable Volume

Click the “Upload” to start importing the VMDK into the Datastore Selected

Click Okay to confirm and complete the initial setup

Following is the App Volumes main page after the initial configuration and we will come back here when we are creating the App Stacks for our applications again.

Deploy the App volumes Agent

So using the same App Volume installation binary, you can install the Agent on the desktop image. You just have to deploy it on the template we used in the section above in JIT Desktop and take a snapshot. Make sure the Desktop network can reach the app volume manager, do not use NAT in-between the networks if possible (as I have hit some issue before). And if you have Anti virus running in your desktop image, you would need to follow a VMware KB to amend a registry key about filter.


Input the App Volume Manager Information and “Disable Certificate Validation with App Volume Manager” if your app Vol certificates are not trusted.

Capture the Application as AppStacks for presenting

To capture an Application you would need a VDI Desktop with App Volume Agent but you would need to login with a domain user to ensure it’s connecting to the App Volume Manager. But to Click start the Application Capturing,

Go to the App volume management page and “Create AppStack” at the “Volumes” > “App Stacks”

Give the AppStack a name and choose which storage to be provisioned in

On clicking the Create the AppStack, it will create the VMDK for storing the AppStack for the target capturing applications

It will be in “Unprovisioned” mode, then we need to click the “plus” icon in the front to provision the App Stack which is actually kicking start with the new Application Capturing

But before “Provision” and capture the Application, as mentioned, do login one VDI Template or Desktop Image first in order to let App Volume to see this VDI for Application Capture

After login, you can see the VDI machine from the App Vol UI. And you can click the “Start Provisioning”.

You can see the status changed on the App Volumes UI, we then swap to the VDI desktop to install the corresponding Applications.

You can see the App Volume Agent prompted and following the message, do NOT click OK until you finished the application installation

Just setup the Application as usual

Click the “OK” to finish the setup and the application capture

Click “Yes” to complete the setup

Click OK to restart the VM to complete the Application Capture

You could see the following after the reboot and login. This is needed to complete the whole capture, thus, DO NOT skip this setup to religion the VDI image you use to capture the application.

Then you can see the status of the capture app is changed, and you can assign the app to different AD users or groups.

On Assignment, you can see the Application is being instantly provisioned on the Desktop

YES!!! You got JIT Desktop and Application Virtualisation done!!! In the upcoming blog we will study the User Environment Management considerations and see how to achieve this.



Lab upgrade for NSX 6.3 deployment for vSphere 6.5

Last week is an exciting week that the very first version of NSX has been released for supporting vSphere 6.5! I did deployed few 6.5 labs but as previously I cannot deploy NSX on top of it, honestly, I could not test thru’ it much. But finally, the NSX version 6.3 has been out which officially support the vSphere 6.5 (should be 6.5a actually). Besides supporting vSphere 6.5, there are a lot of enhancement and improvement in stability and performance aspect. You can refer to the release notes HERE. Actually, what make me more exciting is the support of vCloud Director 8.20 advanced networking services, this is why I am waiting for the release of vCloud Director 8.20 even more after the release of NSX 6.3.

But here, I would like to illustrate the upgrade steps and deployment procedure of NSX 6.3 instead. Actually, most of the steps are pretty much alike to the deployment of any NSX version, but here still let me capture the expected screen in deployment for your reference.

Begin the vSphere Upgrade

As the pre-requisite for NSX 6.3, you would need to have vSphere 6.5a, NOT 6.5. Although I think most of the production environment would not be at version 6.5 yet. This is why I have to first upgrade my existing 6.5 environment.

While my existing environment comprises of the followings:

  1. vCenter 6.5 Server Appliance
  2. Management Host ESXi 6.5 x 3
  3. Resource Host ESXi 6.5 x 2 (ROBO vSAN Enabled)
  4. vSAN Witness Appliance 6.5

The upgrade approach is comparatively trivial, from point 1 to point 4. Let’s get started! Do remember that as we are performing an upgrade instead of a green field deployment, you have to download the path from the patch download link HERE.

So you have to download the 6.5a patch for the ESXi hosts

And also the 6.5a patch for the vCenter Appliance

Upgrade the vCenter Server

So, you just have to upload the ISO and attach it to the vCenter Server Appliance VM after downloading the patch. Then you can upgrade the vCenter server by going to the management URL at https://<vCenter ip or FQDN>:5480.

Hit the Update and “Check CDROM”. This GUI based upgrade is so convenient to me as we have to do this thru’ CLI in the previous versions. Then you can proceed in hitting the “Install CDROM Updates”

On accepting the EULA, the upgrade will be proceeded.

Wait until the update is completed and you have to reboot the vCenter Server Appliance

And you can do it at the “Summary” tab by using the “Reboot” button

All done!!! Is it simple enough? The vCenter Upgrade has been improved by version to version, but I truly believe the 6.5 version is the best among all.

Upgrade the ESXi Hosts

So then we can upgrade the ESXi from 6.5 version to 6.5a version, this is simply achievable thru’ Updater Manager of course. But the beauty in 6.5 again, the Update Manager is there by default inside the vCenter Server Appliance.

So you just need to upload the patches downloaded above

Create the baseline to include the patch

Attach the base line to the Cluster

Then remediate the hosts, remember that even the VSAN Witness, you can just remediate it instead of the need to redeploy the Appliance again.

After the upgrade you are all done with the pre-requisites for deploying the NSX 6.3.

Deploy the NSX 6.3

After upgrading the vSphere, following would be an even easier task for deploying the NSX. You have to download the NSX OVA definitely. But the deployment is easy that you just import it same as any VM appliance.

Then, you can go to the http://<nsx ip or fqdn>/admin, to configure the NSX registration.

Register the NSX to vCenter 6.5a instance, so good… it works!!!

Yes, then you can see the “Networking & Security” icon in the web client

You can further proceed to the NSX detail Setup, first step deploy the NSX Controller

Deploy the NSX features and VXLAN

You can see I’m having a L2 over L3 vxlan deployment while cluster Mgmt is using 192.168.100.x/24 network and Res Cluster using 192.168.101.x/24

I skipped the steps in creating Segment ID and transport zone and logical switch as those are trivial but the end result is as follows, you can see my VXLAN is working great!


This is really the greatest news in the Chinese New Year such that I can shift my focus in vSphere 6.5 testing after the release of NSX 6.3. The upgrade steps for your existing 6.5GA environment are also trivial as shown above. I would like to invite you to test out the new NSX 6.3 too. Again let me update again when the vCloud Director 8.20 is out soon and I’m really looking forward seeing the new features and integration between the two products!


Again, as all the labs i deployed is nested on a vCloud Director based environment, I use my Domain controller as the router too. If you are using the same approach, do remember to configure the MTU on your router to enable 1600MTU for the VXLAN connection!

Protect you workloads with vSphere 6.5 VM Encryption

Security is a strong focus of the vSphere 6.5, you can have your vMotion traffic being encrypted which is very useful for you to migrate a VM cross site thru’ internet network and could be even more useful when later you would have to migrate it into the cloud. We also have the secure boot for UEFI which ensure the boot device is being trusted, as mentioned in the Auto Deploy Blog Series, this is the stuff that stop me booting my nested ESXi from PXE image. So in this blog, I would like to walk thru’ the setup steps and caveats you would have to be aware of. You can refer to the documentation HERE for the detail VM Encryption function supported by vSphere 6.5. But as I would like to let you visualise the setup, thus let me start by the setup procedure.

VM Encryption Setup

While the vCenter, ESXi will be responsible for the actual encrypting mechanism, we need to setup a KMS server for storing the keys which are used for encrypting and decrypting the VM files.

For testing purpose, I have (and would suggest you) following the blog post HERE by William Lam. In the post, we can use a docker container to hold the KMS server. Of course the key will be lost when the docker process is down, but this provides a really handy way for us to test the VM Encryption in this post. You can definitely use any docker host to flag out the container process, but here, I would like to use a Photon OS to do that for me. Followings are the steps I used to setup the KMS:

So first you need to prepare the docker host for running the KMS container, and you can download the Photon OS from the link HERE. I download the OVA with virtual hardware v11 version which fit in my vSphere 6.5 environment

So you can then deploy it as generic photon OS

As you cannot assign a static IP to photon OS thru’ the deployment wizard, you would have to login the VM Console to alter the configuration file under /etc/systemd/network for giving a static IP to the Photon OS.

So following the Post from William, you can run the command:

docker pull lamw/vmwkmip

To pull the image into the docker host, of course, your Photon OS thus has to have internet access for pulling the docker image down. After pulling the image down, you can run the following command to run the KMS docker image.

docker run –rm -it -p 5696:5696 law/vmwkmip

I have to state again, docker based KMS should not be used for production environment as it’s not stateful at all, the key will be lost when the docker process is quitted or down accidentally.

Anyway, as we are just testing the Caveats here, you can  continue the work and go back to the vSphere Web Client. You need to connect the vCenter to the KMS server from the “Configure” tab of the vCenter Object. And hit the “Add KMS Server” with the green plus icon.

You can see the following wizard which would let you entering the KMS server information. The mandatory item will be “Server Address” and “Server port”, while you can give the “Cluster name” and “Server alias” a name you want

Confirm the configuration by clicking “Yes”

The KMIP cert will be prompted to be trusted Manually

On successful configuration, you could see the KMS entry as following:

So after the basic configuration of the KMS server, we can start encrypting the VM

VM Encryption Test

So, far easier than what you think,  you actually just need to change the “VM Polices” of a VM, with the editing in VM Storage Policies when your VM is being powered OFF.

The wizard let you assign the VM Encryption Policy to the VM to encrypt the VM

You can find relevant task and events which are trying to “reconfigure VM” which is actually the encryption task

After the task is done, you can see more information from the VM summary.

  1. the VM logo is with a “Lock” beside it
  2. Encryption Entry under “VM Hardware”
  3. VM storage policy is compliant with Encryption Policy

Well all DONE! Is it simple enough?? And yes this is how you can encrypt your VM with VM encryption as the new feature in vSphere 6.5 environment.