Migrating your applications to Azure using Virtual Machine Scale Sets, Packer and Virtual Machine extensions – Part 3
This is a continuation of the previous post about migrating your not ready cloud application to the Azure cloud. The last post discussed about creating a managed image to be able to be used by a virtual machine scale set for provisioning.
What will we do in this series
I decided to do a series of posts about this topic as it touches a variety of aspects. I will use a concrete example that may or may not have happened to you and I plan to cover
- Building a managed image from an Ubuntu image as base, and setting up a web server (Tomcat for instance) to host an application
- Creating a Virtual Machine Scale Set using ARM templates
- Adding a post provisioning extension that will be fetching files from a blob storage that is required for the application to run once a virtual machine in the set has started (this post)
Assigning the RBAC for the VMSS
Now that our image is built and the VMSS template is ready, we are ready to add the extension.
The tricky part here is that we want to use the managed identity of the VMSS to go and read data in a blob storage we have somewhere. To do that, I will modify the ARM template I used in part 2.
The first thing I need to do is grant the Storage Blob Data Reader role from the VMSS managed identity to the storage account blob container. The following ARM snippet allows me to do so. Add it to the existing ARM template.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
{ "type": "Microsoft.Storage/storageAccounts/blobServices/containers/providers/roleAssignments", "name": "[variables('vmssStorageRoleAssignmentName')]", "apiVersion": "2018-07-01", "location": "[variables('location')]", "dependsOn": [ "[concat('Microsoft.Compute/virtualMachineScaleSets/', variables('vmssName'))]" ], "properties": { "roleDefinitionId": "[variables('storageBlobDataReaderRole')]", "principalId": "[reference(concat('Microsoft.Compute/virtualMachineScaleSets/', parameters('vmssName')), '2019-07-01', 'Full').identity.principalId]" } } |
The vmssStorageRoleAssignmentName needs to be a unique guid. We will handle that in our variables declaration.
Variables and parameters
You will need to add the following variables in your main ARM template:
|
1 2 3 4 |
"storageBlobDataReaderRole": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/', '2a2b9908-6ea1-4ae2-8e65-a410df84e7d1')]", "uniqueRoleAssignmentId": "[guid(uniqueString(parameters('artifactsStorageName')))]", "artifactsStorageContainerName": "provisioning", "vmssStorageRoleAssignmentName": "[concat(parameters('artifactsStorageName'), '/default/', variables('artifactsStorageContainerName'), '/Microsoft.Authorization/', variables('uniqueRoleAssignmentId'))]" |
Adding the extension
If you add the extension directly into the ARM template, with dependencies (dependsOn), you may get a 403 from the extension when it’s trying to go and fetch the provisioning script. RBAC role assignments may take up to five minutes to propagate1. The role assignment has to be assigned and propagated before anything can happen. This is because the extension will use the managed identity to go and fetch the files listed in the fileUris array.
As such, I will move the extension to it’s own template. You need to wait about 5 minutes before provisioning it so that the RBAC propagation has time to occur.
If you are writing a PowerShell script to automate all of that, use the PowerShell command Start-Sleep -Seconds 300.
Create a new ARM template file. Add the following into the resources array:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
{ "type": "Microsoft.Compute/virtualMachineScaleSets/extensions", "name": "[concat(parameters('vmssName'),'/',variables('artifactsStorageContainerName'))]", "apiVersion": "2019-03-01", "location": "[variables('location')]", "properties": { "publisher": "Microsoft.Azure.Extensions", "type": "CustomScript", "typeHandlerVersion": "2.1", "autoUpgradeMinorVersion": true, "settings": { "skipDos2Unix": false }, "protectedSettings": { "commandToExecute": "[concat('sudo sh postprovisioning.sh ', '\"', parameters('artifactsLocation'),'\"')]", "fileUris": ["[parameters('vmssCustomScriptArtifactsLocation')]"], "managedIdentity": {} } } } |
Once the extension has successfully been added to the VMSS, it will run the command you specified in the commandToExecute under the protectedSettings property.
Tip: If you want to debug/troubleshoot, you can always go in the waagent folder where the file was downloaded (
/var/lib/waagent/custom-script/download/*) to see what is going on. The folder is located at. See the troubleshooting procedure for more information.
You will also need to supplement your template with the following parameters and variables:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
"parameters": { "vmssName": { "type": "string", "metadata": { "description": "The VMSS resource name" } }, "vmssCustomScriptArtifactsLocation": { "type": "string", "metadata": { "description": "The full location of the provisioning script. This must be in a storage account that can be accessed by the vmss." } }, "artifactsLocation": { "type": "string", "metadata": { "description": "The full location of the artifacts file. This must be in a storage account that can be accessed by the vmss." } } } variables: { "location": "[resourceGroup().location]", "artifactsStorageContainerName": "provisioning" } |
Necessity in your script
In order to download the file from the blob storage, we need to generate an access token and then query the storage account provider to go and grab the file. The following is an example of how to do this
|
1 2 3 4 5 |
artifactsFile=$1 filename=$(basename "$artifactsFile") access_token=$(curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F' -H Metadata:true | jq -r .access_token) wget --header="x-ms-version: 2019-02-02" --header="Authorization: Bearer $access_token" "$artifactsFile" -O /var/www/html/$filename |
Since we are passing our file to our script as a parameter, we can get it by using the $1 variable. To grab the access token, we query the identity metadata endpoint and ask it to generate a token for the storage provider. The response from the endpoint is a JSON object containing the JWT information. As we only need the access token, we use the jq utility to extract it. Once that is done, we use that access token to download the file and the save it to the appropriate location, in my case /var/www/html .
Tip: Install jq in the image you build with Packer so that it is available right way in your post provisioning script.
Verification
We can now see that our extension ran properly if we navigate to our web application. In my example, I downloaded a text file from the blob storage and added it to the serving directory.
To remember
If you delete everything related to the VMSS, your role assignment will not be deleted along with all the resources. For that you will need to delete it manually:
|
1 2 |
$roleAssignment = Get-AzRoleAssignment -RoleDefinitionName "Storage Blob Data Reader" -ResourceGroupName "<ResourceGroup of your storage account>" -ObjectType "ServicePrincipal" Remove-AzRoleAssignment -ObjectId $roleAssignment.ObjectId -RoleDefinitionName "Storage Blob Data Reader" -Scope $roleAssignment.Scope |
Updating your VMSS image and extension
If you want to update your VMSS image, as mentioned in the documentation, you can just run the following PowerShell cmdlet:
|
1 2 3 4 |
Update-AzVmss ` -ResourceGroupName "myResourceGroup" ` -VMScaleSetName "myScaleSet" ` -ImageReferenceId /subscriptions/{subscriptionID}/resourceGroups/{myResourceGroup}/providers/Microsoft.Compute/images/{myNewImage} |
If you want to update your extension for it to go and fetch a new file, you can just run the provisioning of the extension again and it will update its under template properties and settings. Just remember to change your artifactsLocation parameter.
Food for thought
If you don’t want to always update your extension to grab new files, you could use an Azure Function which would act to give you the latest URI to your file. In your extension, you would only pass in the Azure Function URI, which wouldn’t change, and the rest could be taken care of in your post provisioning script.
Conclusion
You can do the same with windows images and use DSC extensions. You will have to create and upload your DSC extension to a blob storage and the rest will work as explained in this series. Feel free to let me know your tips and tricks and questions. I’m always looking to improve.
