Whoh…! It has been a very long time since I wrote a blog on my site, last blogs were from Ignite, and the 2018 edition is about to start. So time to end the blog silence 🙂
Azure Stack Certificates
Certificates all have there lifetimes fortunately otherwise it will miss it’s goal entirely, so it’s inevitable that we have to rotate some certificates on Azure Stack. I had to rotate the public certificates recently. A public certificate on a multi node Azure Stack POC environment that is of the type Multi SAN Wildcard certificate. Best practice is to have a separate Wildcard certificate for the different roles of Azure Stack but since this is a POC al names are in one certificate. A lot about the Certificate requirements is describe here.
In this blog I am not explaining how to create the CSR or request the certificate, this is just about testing and rotating the public certificate. More details about generating a CSR can be found here
Prepare Certificate folder
Azure Stack expects a certain folder structure for all certificates and some properties on the .PFX file. The test tool will check for this. There is a powershell file on GitHub called CertDirectoryMaker.ps1 that you could use to create the folder structure. Then add your certificates to the right folder. In this case it was simple 1 certificate in all the folders.
Make sure you create a .pfx file with the current options enabled.
The CertDirectoryMaker tool also creates 2 folders for the Host Extension feature that is coming. The current Azure Readiness Checker tool does not expect them yet, so you need to remove these folders, otherwise the tool will complete it has 2 folders that does not suppose to be there. Below an example of a certificate that is not exported the correct way.
Now share this folder and make sure the ERCS server is able to reach it.
Test Certificate
With the folder in place and the certificate exported with the right properties the .pfx files can be copied to the folders created with the CertDirectoryMaker tool. Then run the following command. In my case we use AAD as identity, if you use ADFS you need to change the IdentitySystem parameter.
Install-Module -Name Microsoft.AzureStack.ReadinessChecker $pfxPassword = Read-Host -Prompt 'Enter PFX Password' -AsSecureString Start-AzsReadinessChecker -CertificatePath 'Path-to-certificate' ` -pfxPassword $pfxPassword -RegionName local -FQDN azurestack.local ` -IdentitySystem AAD
When everything is as it should you will receive several OK’s for all certificates and your are ready to go to the next step. In the example below I used a UNC path to test the certificate, because I used the same path in the certificate rotation later on. But you can also test the certificate locally and copy it to a share after the test.
Test Azure Stack
I recommend you also run a test Azure Stack test to validate if all the Azure Stack Role are healthy and there are no issue’s. When there are issues I recommend you resolve those first before running the certificate rotation. Running the Test-Azurestack CmdLet requires you to login to a PEPSession. Below an example to login and start the test
$cred = Get-Credential Enter-PSSession -ComputerName ERCS01-Server -ConfigurationName ` PrivilegedEndpoint -Credential $cred [ERCS-Server]: PS test-azurestack
After the test I should be all green “PASS” lines and your good to go!
Update!
When I created this blog I justed finished the cert rotation and in the mean while updated to 1808. But unfortunally the update failed and after a good week of throubleshooting with MS support it turns out there are some issue’s with the certificate/secrate rotation process. There are some processes that do not run which will bite you in the ass when you update to 1808. So be prepared with running this update first, before updating to 1808!
Rotate Secret
After al tests are green we can proceed with the certificate rotation. In this case I am replacing the External and Internal secrets. Rotating only internal secrets is described here. It’s recommended to start this process during off work hours. Normally al workloads will be online during the rotation process, but you might want to freeze changes to the environment, so no deployment or altering stuff from the portal, powershell or API’s.
It’s important to use the PEP sessions as a variable and not entering the session directly. Also beware, it takes a lot of time!! In my case almost 10 hours and I hear it from more cases! So if you start it, be very patient, also when the constantly refreshing output is showing the same status for hours, keep it running and leave it doing it’s thing!
$cred = Get-Credential
$session = New-PSSession -ComputerName ERCS-Server -ConfigurationName `
PrivilegedEndpoint -Credential $cred
$PEPsession = Get-PSSession -Name "$session"
$CertPassword = ConvertTo-SecureString "Put your PFX Password Here" `
-AsPlainText -Force
$CertShareCred = Get-Credential
$CertSharePath = "server-share-to-the-cert"
Invoke-Command -session $PEPsession -ScriptBlock {
Start-SecretRotation -PfxFilesPath $Using:CertSharePath -PathAccessCredential `
$Using:CertShareCred -CertificatePassword $Using:CertPassword
}
Remove-PSSession -Session $PEPSession
When the command is executed, it verifies the certificate first, it seems like on 1807 (the Azure Stack version I had at the moment) already has some Host Extension stuff in there it can’t find, but it can continue without issue’s.
When the secret rotation is completed a lot of logging is spit out of the powershell console, you could skim over it but it’s basically a large log of all that has been done. If it’s completed your have successfully rotated your secrets! You could do another test-azurestack to make sure everything is online.
If it went bad, don’t retry it but contact support to figure out what is wrong with it instead of retrying over and over.
If you have any questions or comments, leave them below or reach out by mail or twitter
Pascal Slijkerman