Understanding Azure Logs from a security perspective — Part 1 — Activity Logs

Story Cover Image

Log collection and analysis is the foundation of security monitoring and digital forensics for the Azure platform. It is important to collect available logs in a proportionate manner and where possible, implement proactive analysis to detect unsafe activities and threats.

The Azure platform delivers multiple audit logs which enables us to track user, application and administration activity within our Azure environments. My purpose in this blog series is to cover the available audit logs, discuss the security insights that we can obtain from them and also to highlight existing blind spots that can save you a few headaches down the line. Here are SOME of the logs that will be covered:

• Activity Logs
• Network Flow Logs
• Azure AD (Tenant) Logs
• Resource Logs
• DNS Logs
• Application Logs

In this first blog in the series, the focus will be on the Azure Activity Log. I’ll answer the following key questions about this log:

• What does it log?
• How do we enable it?
• What is the security value? — If you don’t have the chance to read the entire blog, I’ll recommend reading this section and the “good practices” section.
• What is the event latency?
• Where are the azure activity logs stored and for how long?
• What are the analysis and long term retention options?
• Can attackers clean their tracks?
• What good practices will you recommend for handling Activity Logs?

What does it log?

Activity logs records events from multiple sources. The sources are highlighted below:

1. Administrative Events

• It logs all write operations (PUT, POST, DELETE) for Azure management API endpoints. This will be like event log in Windows and /var/log on Linux but for the Azure platform.
• It logs the start of the operation and subsequent success or failure of the operation
• It doesn’t include read operations (GET)
• It also logs Azure role assignments (the operation to assign resource permissions to identities)
• There is consistent logging irrespective of the management tool being used (Azure Portal, Azure CLI, Azure PowerShell, SDKs or Direct API calls) because of Azure singular management layer — Azure Resource Manager — ARM (Figure 1.1). This is because the logging happens at the ARM endpoint.
• The downside here is that there is no ID correlation so you might see entries about the “role xxxxxxxx-yyyy-zzzz-1111–22222222 is assigned to identity aaaaaaaa-bbbb-cccc-dddd-eeeeeeeee at scope 11111111–2222–3333–4444–555555555555” but then you have to go to figure out what those IDs mean.
• References

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#administrative-category

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/view-activity-logs

Figure 1.1 — Azure Resource Manager Events

2. Azure Service Health issues

• Service Health is a FREE monitoring service in Azure that reports on the following:
• health of Azure services — similar to the information that we get from the Azure status dashboard — https://status.azure.com but in a personalized way
• planned maintenance — upcoming platform maintenance work that can affect the availability of our services
• health advisories — upcoming changes in Azure services that we may need to respond to like the deprecation of Azure features or upgrade requirements;
• security advisories — security related notifications or violations that may affect the availability of our Azure services.
• If any of the above issues happen, an incident/notification is created in service health.
• If a service health incident/notification occurs, an event is automatically logged in activity logs by the Azure platform under this category. We do not need to enable or configure anything for this to happen.
• Notifications/Incidents are published to activity logs only if our subscription has a resource that would be impacted by the event
• References

https://docs.microsoft.com/en-us/azure/service-health/service-notifications

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#service-health-category

Figure 1.2 — Service Health Events

3. Resource Health events

• Resource health is a capability of the Azure platform that watches our Azure resources to check if they are running as expected
• If an issue is detected and a service is not running as expected, a notification will be logged in activity logs under this category.
• A resource is a specific instance of an Azure service, such as a virtual machine, web app, or SQL Database.
• An example of a Resource Health event is Virtual Machine health status changed to unavailable.
• But what can checks does Resource Health perform?
• It can check if a resource is available or unavailable
• If a resource is not available, an “unavailable” status notification will be logged in activity logs
• If Resource Health is unable to determine the health of the resource or if it hasn’t received information about the resource for more than 10 minutes, an “unknown” status notification will be logged in activity logs
• It can check if a resource is experiencing performance degredation
• If performance loss is detected for a resource even though it is still available for use, a “degraded” status notification will be logged in activity logs
• Does Resource Health perform all checks for all resource types in Azure?
• No. It currently performs checks for 51 resource types (out of over 180 resource types in Azure)
• Some resources only supports availability checks while some supports both availability and performance degredation checks
• The methods and signals used to monitor the health of resources also varies for different resource types
• References

https://docs.microsoft.com/en-us/azure/service-health/resource-health-overview

https://docs.microsoft.com/en-us/azure/service-health/resource-health-checks-resource-types

4. Azure Monitor alerts events

• This is more of a legacy thing now as classic alerts was retired in May 2021 for the Azure public cloud. It has now been replaced by a newer near real-time alerting model.
• If you have configured “Classic Azure Monitor alerts”, a notification will be logged in activity logs under this category when an alert is triggered.
• The new alert model does not log an event in activity logs when an alert is trigerred.
• References

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#alert-category

https://docs.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-classic.overview

5. Autoscaling events

• Autoscale is a capability of Azure that allows supported services to automatically add infrastructure capacity in response to events. There are currently about 12 resource types in Azure that support auto-scaling (see references).
• The start and status of all autoscaling events are logged in Activity logs under this category (Figure 1.3). This includes when the autoscaling event started and also its subsequent status of success or failure.
• References

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#autoscale-category

https://docs.microsoft.com/en-us/azure/azure-monitor/autoscale/autoscale-overview#supported-services-for-autoscale

Figure 1.3 — Azure Monitor Alerts

6. Security events

• Azure Security Center is a PAID Cloud Security Posture Management (CSPM) and Cloud Workload Protection (CWP) service in Azure.
• If an issue is detected by Security Center, a notification will be logged in activity logs under this category
• The events are only logged for security alerts detected by CWP (Azure Defender) and not for CSPM.
• References

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#security-category

Figure 1.4 — Security Center Security Alerts

7. Recommendation events

• Azure Advisor is a FREE best practice recommendation service in Azure
• It analyzes resource configuration and usage telemetry of Azure resources and provides recommendations on how to optimize cost-effectiveness, performance, reliability (high availability) and security
• If a recommendation is identified by Azure Advisor, a notification will be logged in activity logs under this category
• References

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#recommendation-category

8. Policy events

• Azure Policy is a FREE configuration assessment and enforcement service in Azure
• It analyzes the configuration of existing and new resources against assigned policy definitions
• If the resource configuration matches the defined definition, the corresponding action is applied
• The compliance status of resources are written to an activity log
• Non-compliant audit state is written with the “warning” level
• Non-compliant deny state is written with the “error” level
• References

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#policy-category

https://docs.microsoft.com/en-us/azure/governance/policy/how-to/get-compliance-data

How do we enable it?

• It is enabled by default. We don’t need to do anything to enable it.
• Some events may require other Azure Services to be configured for the logging to happen
• Policy events — Requires Azure Policy definitions or initiatives to be assigned
• Security events — Requires Security Center (Azure Defender Plan) to be enabled and configured
• Recommendation events — Requires Azure Advisor
• Autoscaling events — Requires autoscaling settings to be configured
• Azure Monitor alert events — Requires alerts to be configured in Azure Monitor

What security value can we obtain from this log?

1. Compliance insights: Information on service usage can be obtained from Activity logs and this can be used to identify usage that violates organization compliance policies. As Azure policy violation detections are logged in Activity logs, this can also be used to track resources that violates configured organization policies.
2. Proactive threat detection: Azure activity logs can exported to other services in Azure like Log analytics and Azure Sentinel OR to 3rd-party cloud security solutions like Prisma Cloud for proactive analysis. These services can be used to look for suspicious activities and patterns using various methods like correlation of events with threat intelligence sources, pre-built ML algorithms that is trained using the log data, etc.
3. Platform and Service level security advisories: As service health security advisory notifications are logged in activity logs, we can retrieve platform security advisories in order to implement recommended actions to mitigate vulnerabilities.
4. Forensics: Activity logs entries includes PUT, POST, DELETE management operations in Azure which is useful for forensic insights and to investigate breaches. For example, what did a bad actor do with a compromised credential? Remember that GET administrative operations are not logged so there is a chance of missing insights like where an attacker has used a compromised credential to read other sensitive information. Also, this does not log data plane events so for a more complete picture, you may have to correlate with resource log information that we will cover in this series.
5. Privileged entitlement assessment: Compare permissions granted with actual permissions used when correlated with active role assignments

Where the logs stored and for how long?

• The logs are stored on the Azure platform for 90 days
• This means that we can only view the activity logs only for the last three months
• https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log

But what if we want to send the logs to an external service or retain longer than 90 days?

• We can configure a diagnostic setting (Azure Portal → Azure Monitor → Activity Logs → Diagnostic Settings) to export Activity Logs to any of the following four options: An Azure Storage Account; An Event Hub namespace; A Log Analytics workspace; Supported native partner integrations like Datadog and Logz.io.
• Activity logs can also be collected directly from the platform via API (API-Based collection)
• These five export options have different use cases as I’ve highlighted below:
• Event Hub: Use case is mainly to do a push based stream of the logs to an external service for near real-time analysis. This is the method documented for tools like Splunk here:
• Log Analytics Workspace: Use cases are mainly to correlate the log data with other collected data in the workspace; to centralize log collections from multiple Azure subscriptions and tenants into a single location; to perform complex log analysis using queries based on KQL (Kusto Query language); to implement complex alerting logic based on log queries.
• Logs will be stored in a table called “AzureActivity”
• Unlike other logs sent to a Log Analytics workspace, data ingestion is not charged to Activity logs and data retention is not charged for the first 90 days
• Data can be stored for up to 730 days (2 years) in a log analytics workspace
• Storage Account: Azure blob storage is a cheap cloud storage service. The main use cases here is to either archive the logs or to temporarily store the logs for another service to collect from the blob container. If using for archiving purpose, logs can be stored indefinitely in a storage account with a retention setting configuration of “0”.
• API-Based collection: 3rd party services like Prisma Cloud implement this option to perform a pull-based collection of the logs for further analysis. The only supported operation is the LIST operation but filters and selctions can be applied to queries as described here — https://docs.microsoft.com/en-us/rest/api/monitor/activity-logs/list#examples
• There is a legacy collection method using “Log profiles” that you shouldn’t use annymore. You will still see references to this in Microsoft documentations and also in Azure policy

Figure 1.5 — Activity Logs Diagnostic Settings

What is the event latency?

• This varies by event category.
• My “unofficial tests” showed that Administrative events could take about 2 minutes to appear in the UI after the event has occurred so this is quick enough to be classified as “near real-time”.

Can attackers delete or overwrite log entries?

• No. Regardless of a user’s role, entries in the Activity log cannot be modified or deleted.
• This is great for forensics as both internal or external bad actors cannot delete or modify entries to cover their tracks.
• They are automatically deleted from the platform after 90 days
• https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log

Figure 1.6 — Activity Logs cannot be edited

What good practices will you recommend for handling Activity Logs?

I’ll cover this topic in more details later in this series but good logging practices should both be proportionate and RELIABLE. We want to ensure that we can easily enforce, scale and manage collection and analysis. Here are some good practices to follow.

1.Collect the logs centrally for easier analysis

• While we can configure this within individual subscriptions using diagnostic settings, my recommendation is to enforce centrally using Azure Policy. This way, we can apply policies at the root management group level to ensure that logs are sent to our central point of collection. The following Azure policies can be used for this:
• Built-In policies

Configure Azure Activity logs to stream to specified Log Analytics workspace — This built in policy uses a “DeployIfNotExists” effect to configure a diagnostic setting to stream Activity logs to a specified Log Analytics workspace

• Community contributed policies

Configure Azure Activity logs to stream to specified Storage Account — This built in policy uses a “DeployIfNotExists” effect to configure a diagnostic setting to send Activity logs to a specified storage account. This policy was contributed by Shachaf Goldstein who works at Microsoft. https://github.com/Azure/Community-Policy/tree/master/Policies/Monitoring/apply-diagnostic-setting-subscription-storage-account

• Configure Azure Activity logs to stream to specified Event Hub — This built in policy uses a “DeployIfNotExists” effect to configure a diagnostic setting to send Activity logs to a specified event hub. This policy was contributed by Tim Sullivan who works at Microsoft. https://github.com/Azure/Community-Policy/tree/master/Policies/Monitoring/deploy-diagnostic-setting-for-activity-log-event-hub

2. Configure alerts for sensitive administrative, security and policy operations

• We can use Azure monitor alerts to configure alerts for sensitive operations but I’ll recommend doing this from within a service that acts as a central point of visibility.
• That could be in a Log Analytics workspace using Azure Monitor Log Alerts (there is an additional cost) OR from a 3rd party cloud security solution like Prisma Cloud.
• For example, you may want to monitor the policy and administrative operations below to ensure that activities to assign or delete a policy are reviewed by the security team. These operations could be an indication of defense evasion by an attacker trying to bypass enforced policies or open backdoors. You can refer to Chapter 8 of my new book that I co-authored with Karl Fosaeen — Penetration Testing Azure for Ethical Hackers for more on opening backdoors.

# Policy Operations
"Microsoft.Authorization/policyAssignments/write",
"Microsoft.Authorization/policyAssignments/delete"
# Administrative Operations
"Microsoft.Sql/servers/firewallRules/write",
"Microsoft.Sql/servers/firewallRules/delete",
"Microsoft.Network/networkSecurityGroups/write",
"Microsoft.Network/networkSecurityGroups/delete",
"Microsoft.Network/networkSecurityGroups/securityRules/write",      "Microsoft.Network/networkSecurityGroups/securityRules/delete",

• For each operation that you’re alerting on, you should have documented processes that can be followed by SOC team members in response.

3. Implement services that proactively analyze activity logs for suspicious patterns and theats.

• These patterns could range from calls from IP addresses with bad reputation, consecutive failed operations across a range of services for the same identity (which could indicate a fingerprinting activity) and many others.
• Azure Security Center (Defender Plan) has a good list of detections that it offers for this — https://docs.microsoft.com/en-us/azure/security-center/alerts-reference#alerts-resourcemanager. This is charged at $4 for analyzing one million operations.
• 3rd-party security solutions like Prisma Cloud also analyze activity logs to identify anomalous compute provisioning activity (in the case of cryptomining hijacks) and unusual identity activities for both user and app identities (in the case of insider threats and compromised accounts based on the geolocation and the cloud resources accessed)
• For each detection type, you should have documented processes that can be followed by SOC team members in response.

In the next part of this series, I will answer the same questions for NSG flow logs in Azure. If there’s any question that you have about Azure logs, please feel free to ask as I cover thhis series. Also, checkk out my new books on Microsoft Azure Security Technologies and Penetration Testing Azure for Ethical Hackers.