Understanding the security responsibilities of Azure VM Agents and Extensions #OMIGOD
UPDATED: Microsoft finally released a guidance for cloud extensions and who own the responsibility patching. Follow the guidance here in addition to the guidance that I highlighted below: https://msrc-blog.microsoft.com/2021/09/16/additional-guidance-regarding-omi-vulnerabilities-within-azure-vm-management-extensions/
Cloud security researchers @nirohfeld and @shirtamari uncovered yet another critical vulnerability that impacts Azure workloads. The vulnerability allows for Remote Code Execution (RCE) in situations where some network ports (1270, 5985, 5986) are exposed on vulnerable systems and Elevation of Privilege in scenarios where there is initial access to the vulnerable system.
This vulnerability has been tagged “#OMIGOD” as the affected component behind the vulnerability is the open sourced Open Management Infrastructure (OMI) which is used in some Microsoft products (and other 3rd-party products) including some Azure services. Microsoft has released a fix for the vulnerable component but there’s still a lot of confusion out there amidst the “hype” sorrounding the disclosure. I’ve read many comments of people not being sure of what gets automatically upgraded in relations to Azure agents and extensions or what they are to do manually.
My main goal here is not to repeat information that’s already out there. If you’re yet to, I’ll strongly recommend that you read the guidance provided by @nirohfeld here — https://www.wiz.io/blog/secret-agent-exposes-azure-customers-to-unauthorized-code-execution— I think it is awesome! Especially the sections on how to identify if your systems are vulnerable and some remediation guidance. I can’t stress this enough that this is not just an Azure issue, this is an issue that affects any application that uses a version of OMI that is less than the version that has the fix — 1.6.8.1. The best way to tackle this is to use a configuration/vulnerability/audit management service to identify the existence of the OMI package on your Linux systems where the version number is less than 1.6.8.1. But beyond “fire-fighting” vulnerabilities and reading about how bad the situation is, what else should we be learning about platform VM extensions and associated security responsibilities?
My goal with this write-up is to help you to clarify security responsibilities on upgrading platform agents and extensions deployed to your Azure VMs — the so called — “secret agents”. Hopefully this demystifies them!
Automatic VM agent and extensions
Every Linux VM in Azure has the Microsoft Azure Linux Agent (waagent) installed. The agent manages the interactions between the VM and the platform fabric controllers. The agent powers capabilities like image provisioning and network configuration — https://github.com/Azure/WALinuxAgent/blob/master/README.md.
This agent includes an “extension handler” that is responsible for provisioning additional platform agents (called “extensions”) on the VM depending on the functionalities enabled by cloud operators. The Azure platform supports many extensions that provides capabilities like backup, disk encryption, monitoring and security. The extensions are mainly published by Microsoft but there are a few situations where third party extensions are provided.
Most platform agents in Azure can be installed as a VM extension but a few of them supports both the VM extension installation and a manual agent installation option.
Cloud VM extension patching is different!
Your installation method of an Azure VM extension and the configuration that you have implemented during the installation determines how the upgrades works and who is responsible for them!
According to Microsoft’s documentation — If you have installed an Azure agent using the VM extension option, security updates or key bug fixes cannot be opted out of! — https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/features-linux#extension-updates. Also, updating a component manually may break functionality.
I have collected in the table below, a list of the most used Azure Linux VM extensions and if they use OMI or not.
GIST: https://gist.github.com/davidokeyode/ae7528229623cbb13398848ac2a941ce
As highlighted in the diagram, Microsoft seems to have patched new installations for all affected extensions with the exception of the “diagnostic agent”!
That being said, I only tested in the UK South region and extension upgrades are rolled out by region so it is possible for an extension update to be available in a region and not in another. You can use the Azure CLI command below to check the extension versions that are available in a region:
az vm extension image list --location uksouthThis is one of the lessons that Microsoft should take out of this. There needs to be clear transparency and visibility on the deployment of extension security updates. If I have existing extensions installed on VMs in a region, when will security updates be applied (if I’m not meant to apply manually)? Also, if my automation tools deploy a VM extension in a region, when will new extension with the applied security updates be made available so I don’t roll out vulnerable software in my organization? It will be good not to have to hunt these down ourselves!
- * UPDATE: Microsoft finally released a guidance for cloud extensions and who own the responsibility patching. Here is the link: https://msrc-blog.microsoft.com/2021/09/16/additional-guidance-regarding-omi-vulnerabilities-within-azure-vm-management-extensions/
In essence, here are the options: Wait for Microsoft to apply the security updates for existing extensions that are affected OR remove existing vulnerable extensions and reinstalling them if the new agent deployments have been patched as I highlighted in the above table OR remove existing VM extensions and blocking new installations till you’re comfortable that they are safe. BTW — When I tested removing an existing vulnerable VM extension (the diagnostics agent extension), it still left the vulnerable OMI version on the system and this had to be cleaned up manually! It will be great if the removal of an extension from the platform also cleaned up unused dependencies!
Automatic extension upgrade
So apart from the above information, it is good to be aware of configurations that we can enforce to also ensure the VM extensions that we’re installing are automatically patched.
For example, the above table shows that the Dependency agent can be installed via an Azure extension. If you installed the agent via an extension and set the enableAutomaticUpgrade property to true, the extension will be upgraded automatically whenever the extension publisher (Microsoft in this case) releases a new version for it. However, this must be specified explicitly as the default is to disable it as shown in the figure below:
enableAutomaticUpgrade disabled by default for supported extensionsWith that being said, not all extensions supports this property (only five extensions currently supports this — The dependency agent, the Azure monitor agent, the application health extension, the guest configuration extension and the key vault extension)— https://docs.microsoft.com/en-us/azure/virtual-machines/automatic-extension-upgrade#supported-extensions (This document is a bit out of date as AMA is not yet called out). You can validate this at scale using Azure policy by checking for the property to be set to TRUE.
Automatic extension Minor Version upgrade
What option do we have to auto-update extensions that do not support enableAutomaticUpgrade option? There is another property called autoUpgradeMinorVersion that we can configure to automatically apply minor version upgrades when available. This usually covers security and bug fixes. As far as I’m aware, this option is suppported for all Microsoft provided VM extensions.
autoUpgradeMinorVersion for Azure VM extensionsREFERENCE: https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/features-linux
My recommendations
- As shown in the above table, new deployments of the Linux diagnostics extension is yet to be patched. I’ll recommend disabling this in your environment till it is patched. You can do this using the Azure policy rules and parameters files shown below and specify the parameter —Microsoft.Azure.Diagnostics.LinuxDiagnostic :
2. Enable Automatic extension upgrade for all extensions that supports it to ensure that you automatically get the latest extension upgrade when the extension publisher releases a new version. Worth mentioning here that the upgrade process will follow an “availability first” approach which means that if a VM is not in a healthy state within 5 minutes of an upgrade completion, the extension version will automatically roll back to the previous version. You can read about that process here.
3. Set autoUpgradeMinorVersion to true for all VM extensions. This will ensure that you get the latest minor release bug fixes, it is highly recommended that you always select auto update in your extension deployments. Hotfix updates that carry security or key bug fixes cannot be opted out.
To get into more details on Azure Security, please check out my books on Implementing Azure Security Technologies — Defense and Penetration Testing Azure for Ethical Hackers (co-authored with @kfosaeen) — Offense
