Symptoms
- vCenter/PSC Services do not start due to expired certificate showing the following errors:
ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Thu Jan 02 09:22:13 EST 2020, endTime=Fri Jan 03 09:22:13 EST 2020] :: Signing certificate is not valid at Thu Jan 02 09:22:13 EST 2020, cert validity: TimePeriod [startTime=Wed Jan 06 20:44:39 EST 2010, endTime=Wed Jan 01 20:54:23 EST 2020]
Note: The endTime should be a date in the past if the certificate is expired.
- The following error is observed when logging into the Web Client:
- Accessing WebClient or UI Client will show below error message if vmware-vpxd service is not running due to expired certificate
- If WebClient is accessible, it might throw error message "User name and password are required" while trying to login
- Unable to replace any certificate on either PSC or VC because it fails to re-register services on lookup service
- Unable to add/modify/delete registrations from lookup service manually using lstool
- Cannot deploy a new PSC and do a cross domain repoint
- Cannot deploy a new PSC as a replication partner on the existing SSO domain
When the STS certificate expires, it does so without warning. On some systems, this expiry may occur as soon as two years from initial deployment.
Notes:
- Here are the scenarios where STS signing certificate is expected to have life time around 2 years.
- Not all 6.5 U2 or later but only 6.5 U2 or later on 6.5 release lines only.
- Fresh installation of PSC/vCenter Server 6.5 starting with U2 or later (6.5 lines only).
- Freshly installed PSC/vCenter Server 6.5 U2 or any later 6.5 releases and upgraded to a later version including 6.7 and 7.0.
- STS signing certificate has been replaced using certool post installation of PSC or vCenter Server.
- STS signing certificate has been replaced with custom certificate (Internal/External CA Signed).
Take offline snapshots concurrently for all vCenter Servers and Platform Service Controllers in the SSO domain before running the script.
This script should only be run once per SSO domain.
- Download the "fixsts.sh" script attached to this kb and upload to the impacted PSC or VC with embedded PSC in the /tmp folder, or copy its contents to a text file on the appliance using vi
- cd to /tmp folder
- Run chmod +x fixsts.sh to make the file executable
- Run ./fixsts.sh
- Restart services on all vCenters and/or PSCs in your SSO domain by using below commands:
service-control --stop --all
service-control --start --all
Note: Restart of services will fail if there are other expired certificates like Machine SSL or Solution User. Proceed with next step to identify and replace those expired certificates. - Check for expiration and replace any other expired certificates you might have, using certificate manager as shown in How to use vSphere Certificate Manager to Replace SSL Certificates or follow Option 8 as shown in How to regenerate vSphere 6.x certificates using self-signed VMCA if both Machine SSL and Solution User certificates are expired
- The following one-liner can determine other expired certificates for the vCenter Server Appliance:
If you replaced Machine SSL or VMCA Root certificates, you will need to re-register 2nd party solutions such as NSX, SRM, and vSphere Replication.
Note: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
The script will ask for the SSO administrator password and then proceed to regenerate and replace STS certificate.
This is an example of a successful output
NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================
Resetting STS certificate for vcsa1.gsslabs.org started on Fri May 22 14:39:40 UTC 2020
Detected DN: cn=vcsa1.gsslabs.org,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: vcsa1.gsslabs.org
Detected PSC: vcsa1.gsslabs.org
Detected SSO domain name: vsphere.local
Detected Machine ID: ce510c87-35e6-444e-82f0-60a7527608a3
Detected IP Address: 192.168.0.51
Domain CN: dc=vsphere,dc=local
==================================
==================================
Detected Root's certificate expiration date: 2030 May 16
Detected today's date: 2020 May 22
==================================
Exporting and generating STS certificate
Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success
Enter password for administrator@vsphere.local:
Amount of tenant credentials: 1
Exporting tenant and trustedcertchain 1 to /tmp/vmware-fixsts
Deleting tenant and trustedcertchain 1
Applying newly generated STS certificate to SSO domain
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
Replacement finished - Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================
==================================
Note: You may receive an error when you try to run the script:
bash: ./recreate_machine.sh: /bin/bash^M: bad interpreter: No such file or directory
This error is caused by DOS carriage returns added to the script when copying from a Windows based text editor. To resolve this problem, run the following command and rerun the script:
sed -i -e 's/\r$//' fixsts.sh
0 Comments