Awaiting too long for this!!! vCenter HA – Part 3

This is the part 3 of the blog series for configuring the new vCenter HA protection for your vCenter Server 6.5. As the previous blog posts, I am performing a deployment with best availability level and with supported and recommended topology defined by VMware. We have already deployed the PSC01 and PSC02 as shown in the following logical diagram. In this blog we focus in the NLB deployment and configuration, accompanying with the setup of HA mode across PSC01 and PSC02. We would be ready for setup the vCenter and deployment vCenter HA in the next blog post.

slide1

So in this blog we are going to perform the following steps for enabling a High Availability PSC which is supported by a Load Balancer:

  1. Create a new machine SSL certificate. For more information, see:
    Configuring certificates for Platform Services Controller for High Availability in vSphere 6.5 (2147627)
  2. Configure the load balancer. For more information, see:
    Configuring Netscaler Load Balancer for use with vSphere Platform Services Controller (PSC) 6.5 (2147014)
  3. Verify the machine Certificate:
    vCenter Server Appliance – /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store MACHINE_SSL_CERT –text
  4. Verify the Load Balancer is presenting the same certificate:
    vCenter Server Appliance – openssl s_client -connect SSOLB.vmware.local:443
  5. Run the configuration scripts on the Platform Service Controllers. For more information, see
    Configuring PSC Appliance for High Availability in vSphere 6.5 (2147384)

Create a new machine SSL certificate

So according to the VMware KB, after deploying the two PSC servers. We have to configure the two nodes using the same SSL certs. Therefore, we have to generate new SSL certs and replace those on both PSC nodes. The following steps are refer to KB2147627:

Creating the certificate request

Using a text editor, create the psc_ha_csr_cfg.cfg file with these entries:

[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:psc-ha-a1.domain.com, DNS:psc-ha-a2.domain.com, DNS:psc-ha-vip.domain.com
[ req_distinguished_name ]
countryName = Country
stateOrProvinceName = State
localityName = City
0.organizationName = Company
organizationalUnitName = Department
commonName = psc-ha-vip.domain.com

Notes:
The subjectAltName values should contain all PSC FQDNs that will participate in this HA Site, including the Load Balanced FQDN.
The commonName value should be the Load Balanced FQDN.

For my case, I use psc.vmware.lab as the NLB FQDN. And psc01 and psc02 would be the two appliances I deployed in the previous blog post. And you don’t need to thing about where to generate the cert. You can directly generate it on the PSC nodes and just you need to do this step in one of the PSC of course.

cert-req

Run this command to create a psc-ha-vip.csr and a psc-ha-vip.key file.

openssl req -new -nodes -out /certs/psc-ha-vip.csr -newkey rsa:2048 -keyout /certs/psc-ha-vip key -config /certs/psc_ha_csr_cfg.cfg

Note: 2048 bit key length private key is created with rsa:2048. This value can be increased, 2048 is the minimum supported key length.

Again, I perform this step on the PSC server directly, again you just need to do this step once as the same cert files will be used by the other PSC server

gen-cert

So after that you have the necessary cert request files for proceeding the next for generating the cert out.

Generating a certificate from the VMCA

I leverage the VMCA directly instead of an external CA as i think for most of the case, the vCenter certificates using in environment are the one generated by the VMCA. So following the steps in the KB.

Run this command to create the certificate from the psc-ha-vip.csr and the the psc_ha_csr_cfg.cfg file outputting a psc-ha-vip.crt file.

openssl x509 -req -days 3650 -in /certs/psc-ha-vip.csr -out /certs/psc-ha-vip.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile /certs/psc_ha_csr_cfg.cfg

Run this command to copy the current VMCA root certificate and rename it to cachain.crt.

cp /var/lib/vmware/vmca/root.cer /certs/cachain.crt

cert-gen

Run this command to create Machine SSL Certificate that contains the newly created certificate and the VMCA root certificate named psc-ha-vip-chain.crt.

cat /certs/psc-ha-vip.crt >> /certs/psc-ha-vip-chain.crt
cat /certs/cachain.crt >> /certs/psc-ha-vip-chain.crt

cat

Preparing Certificates
Three certificates should have been created

  1. psc-ha-vip-chain.crt
  2. psc-ha-vip.key
  3. cachain.crt

Validate the certificate information
Run this command to open the certificate:

openssl x509 -in /certs/psc-ha-vip-chain.crt -noout -text

Ensure that the Subject CN value is the correct Load Balanced FQDN.
Ensure that the the DNS values contain all PSC FQDNs and Load Balancer FQDN.

So My DNS value shows all the three elements: FQDN of NLB, PSC01 and PSC02

san

And My CN is the NLB FQDN

cn
Replacing the Certificates on the Platform Services Controller
Launch the Certificate-Manager using this command:

/usr/lib/vmware-vmca/bin/certificate-manager

Select Option 1, then Option 2.
Provide the paths to the psc-ha-vip-chain.crt, psc-ha-vip.key and cachain.crt files created in the Preparing Certificates section.

For example:

Please provide valid custom certificate for Machine SSL.
File : /certs/psc-ha-vip-chain.crt
Please provide valid custom key for Machine SSL.
File : /certs/psc-ha-vip.key
Please provide the signing certificate of the Machine SSL certificate
File : /certs/cachain.crt
Important: Replace the Machine SSL Certificate of the additional PSC using the same certificate.

replace

On finishing, I personally copy the certs into the nodes to and perform the same cert replacement tasks. If you are trying to leverage scp, do remember you need to enable the bash shell in another PSC nodes: e.g.

enable-bash-shell

Configure the load balancer

So then we can configure the load balancer for the PSC nodes, I use Netscaler in this case. This is not because I have experience in Netscaler or a fans of Citrix. But because the deployment overhead is easier. We can have NSX or F5, but my lab has to be upgraded first before I got more resources for deploying those.

The load balancer requirement for PSC is actually easy to meet and thus I just download the free edition Netscaler and perform a one arm NLB (simplest possible configuration). The link is here.

screen-shot-2016-11-25-at-10-28-28-am

Of course I download the ESXi format one which level me deploy on the same ESXi hosts I setup for running the PSC and vCenter Nodes. There is a nice blog post here which can guide you thru’ the basic setup and deployment for the VPX Express.

So after the NLB is up, we are following the VMware KB2147014 to perform the simple setup. There are overlapping part in the KB due to documentation bug, but you can following the following steps (I added some more complementing wordings in bold):

To configure the Netscaler Load Balancer to provide the vSphere 6.5 Platform Services Controller (PSC) High Availability. You need to carry out 5 steps in your Netscaler:

  1. Add Platform Controller Servers Under the Server Tab
  2. Add Services
  3. Create Virtual Servers
  4. Create a Persistency Group
  5. Verify Servers, Services, Virtual Servers

So log into the Netscaler Web UI
Adding Platform Controller Servers

  • Navigate to Configuration > Traffic Management > Load Balancing > Servers.
  • Select Add.
  • Enter a Server Name for the First PSC Node.
  • Enter an IP Address for the First PSC Node.
  • Click Create.
  • Repeat these steps for the Additional PSC Node.

add-server

Adding Services

  • Navigate to Configuration > Traffic Management > Load Balancing > Services
    Select Add.
  • Enter a Service Name.
  • Select Existing Server.
  • Select the First PSC Node from the Server drop down menu.
  • Click Protocol and then select TCP.
  • Click Port and enter 443.
  • Click OK.
  • Repeat for these ports 389, 636, 2012, 2014 and 2020 for both PSCs.

services

Creating the Virtual Servers

  • Navigate to Configuration > Traffic Management > Load Balancing > Virtual Servers.
    Select Add.
  • Enter a Name.
  • Click Protocol and then select TCP.
  • Click IP Address and enter the Load Balanced IP Address.
  • Click Port and then 443.
  • Click OK.

Note: If you are asked to enable the ‘LB’ Feature, click Yes.

  • Under Services and Service Groups select No Load Balancing Virtual Server Service Binding.
    Click >.
  • Select the two services for port 443.
  • Click OK. The added Services should appear in the Select Service box.
  • Click Bind.
  • Repeat the preceding steps for ports 389, 636, 2012, 2014 and 2020.

Note: There should be 6 Virtual Servers after this process.

virtual-servers
Create a Persistency Group

  • Navigate to Configuration > Traffic Management > Load Balancing > Persistency Groups
    Click Add.
  • Click Group Name provide a name.
  • Click Persistence and then select SOURCEIP.
  • Click Time-out and enter 1440.
  • Click Virtual Server Name and then click the + button.
  • Click the > button to move all six PSC VIPs to the Configured pane.
  • Click Create.

persistency-groups

Verify Servers, Services, Virtual Servers

  • Navigate to Configuration > Traffic Management > Load Balancing > Servers.
  • Verify that both PSC Servers are online and enabled.
  • Navigate to Configuration > Traffic Management > Load Balancing > Services.
  • Verify that all Services are UP and that there are two Services for each Port
  • Navigate to Configuration > Traffic Management > Load Balancing > Virtual Servers.
  • Verify that all Virtual Servers are UP, that they map to the correct Load Balanced PSC HA IP Address and that there is a Virtual Server for each Port

Verify the machine Certificate

After the NLB is setup we can verify the Machine Certificate, this is by one single command. In the KB, you can see the “bin” is missing, hopefully it will got fixed soon.

 /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store MACHINE_SSL_CERT –text

Verify the CN, for mine it’s psc.vmware.lab

ver-cert-01

Verify the DNS again, for mine, it includes all the NLB and node FQDN

ver-cert-02

Verify the Load Balancer is presenting the same certificate

Check the Load balancer

vCenter Server Appliance – openssl s_client -connect SSOLB.vmware.local:443

For confirming the CN and DNS again

Run the configuration scripts on the Platform Service Controllers

Finally, as the last step in setting up your HA PSC with NLB. You need to run the script to configure the PSC nodes. We are following the VMware KB2147384.

Configuring PSC HA 6.5

 To configure the PSCs for load balancing, run updateSSOConfig.py and updateLsEndpoint.py scripts:
Notes:
  • The updateSSOConfig.py script updates information local to each PSC and must be ran on all PSCs in the HA instance.
  • The updateLsEndpoint.py script updates the ServiceRegistration Endpoints in VMDir and only needs to be ran on one of the PSCs in the HA instance.
Running the updateSSOConfig.py script
  1. Connect to the PSC appliance and log in with root credentials.
  2. Type shell to access the Bash shell.
  3. Navigate to /usr/lib/vmware-sso/bin with this command:cd /usr/lib/vmware-sso/bin
  4. Run this command:python updateSSOConfig.py –lb-fqdn=psc-ha-vip

    For example:

    python updateSSOConfig.py –lb-fqdn=loadbalancer.vmware.com

  5. Repeat these steps on remaining PSCs.

psc02 psc01

Running the updateLsEndpoint.py script
  1. Connect to the PSC appliance and log in with root credentials.
  2. Type shell to access the Bash shell.
  3. Navigate to /usr/lib/vmware-sso/bin with this command:cd /usr/lib/vmware-sso/bin
  4. Run this command:python UpdateLsEndpoint.py –lb-fqdn=psc-ha-vip.domain.com –user=administrative_user –password=password

    For example:

    python UpdateLsEndpoint.py –lb-fqdn=psc-ha-vip.domain.com –user=administrator@vsphere.local –password=VMware123$

    Note: Perform these step on a single PSC node only.

script

GREAT! Your PSC servers and NLB are working now! Finally, you can deploy the vCenter and further perform the vCenter HA configuration!!! I will document that in the next Blog.

 

Awaiting too long for this!!! vCenter HA – Part 2
Awaiting too long for this!!! vCenter HA – Part 4

Leave a Reply

Your email address will not be published / Required fields are marked *