Understanding Azure Logs from a security perspective — Part 2 — NSG Flow Logs

12 min readOct 9, 2021

Log collection and analysis is the foundation of security monitoring and digital forensics for the Azure platform. It is important to collect available logs in a proportionate manner and where possible, implement proactive analysis to detect unsafe activities and threats.

This blog is the second in a multi-part series to cover the available logs and telemetry of the Azure platform, discuss the security insights that we can obtain from them and also to highlight existing blind spots that can save you a few headaches down the line. In the last blog, I discussed activity logs. In this blog, I’ll cover NSG flow logs.

Azure Virtual Network Overview

The Azure cloud platform allows customers to create logically isolated private networks called “virtual networks” (vNets). These isolated networks can be segmented into subnets that we can deploy supported Azure IaaS and PaaS services into.

Azure Virtual Network Resources — Service Integration vs Private Endpoint

So what resources can we deploy into these networks? For most organizations, the main services that they will deploy into them are “virtual machines” (VM) and “virtual machine scale sets” (VMSS) but there are currently about 25 IaaS and PaaS services that can be deployed into an Azure virtual network subnet (including Azure SQL managed instances, container instances, cache for redis and Kubernetes service). Deploying a supported platform service into a virtual network subnet is referred to as “virtual network integration” (vNet integration).

**Virtual Network Intergration — IaaS and PaaS**

A full list of services that are supported for vNet integration can be found here — https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-for-azure-services#services-that-can-be-deployed-into-a-virtual-network.

Most PaaS services that supports this requires a dedicated subnet — this means that they cannot be combined with other services in the subnet that they are deployed into. Some PaaS services also require explicit permissions to auto-deploy service specific resources in the subnets. For example, when a SQL managed instance is deployed into a subnet, it automatically creates a network security group and a route table and applies them to that subnet. To perform these operations, the service requires the necessary permissions.

In addition to the 25 services that supports vNet integration, there are about 32 platform services (29 generally available and 3 in preview) that can be linked to a virtual network using the “private endpoint” feature. Enabling this feature for a platform service instance enables private access to that resource instance.

**Private endpoint — supported platform services**

The full list of services that supports private endpoint can be found here (29 services are generally available; 3 are in preview)— https://docs.microsoft.com/en-us/azure/private-link/availability. A few services support both vNet integration and private endpoint. Azure Cache for Redis falls into this category even though it is missed out on Microsoft’s documentation for services that are supported for private endpoint.

To filter network traffic to and from Azure resources within a virtual network subnet, we can implement “network security groups” (NSGs). Rules can then be configured in a NSG to allow or deny traffic by source/destination IP address, source/destination port, and protocol (known as the 5 tuple). We can associate a NSG at the subnet level or at the virtual machine NIC level (NIC level NSG is automatically applied for VM scale sets instances for consistency).

NSG Flow Logs

The Azure platform has a service called “Network Watcher” that can be used to collect network traffic logs from NSGs and store them in an Azure blob container. The feature in network watcher that we use for this is called “NSG flow logs”. Flow logs are collected outside the path of the network traffic so there is no latency impact.

Where are the logs stored and for how long?
NSG flow logs can be stored in an Azure blob storage container. We can specify the retention period at configuration time as shown in the diagram below. We can also collect in a Log analytics workspace (if Traffic Analytics is enabled) where the retention period of the workspace applies.
Which format is the log written?
Inbound and outbound flows are logged per NSG rule in JSON format. Each record includes the network interface (NIC) the flow applies to, 5-tuple information, the traffic decision

That being said, please be aware that NSG flow logs does not log every network flow for a subnet or VM NIC! There are certain considerations and limitations that you should be aware of and those are highlighted below:

NSG flow logs considerations and limitations

1. NSG flow logs is associated with a NSG resource. Network flows will only be logged where there is an associated NSG resource that has flow logs enabled. You can verify NSG flow log status with the commands below:

# List NSGs with flow logs enabled
az network nsg list --query "[?flowLogs!='null']" -o table
# List NSGs with flow logs not enabled
az network nsg list --query "[?flowLogs=='null']" -o table

2. Collected flow logs are stored in an Azure blob container in the same region as the virtual network. Manually rotating the access keys to the storage account will break NSG flow logging! The only way to fix this is to disable and re-enable NSG Flow Logs. You can only spot these errors (example in diagram below) if you have resource logs enabled for the storage account. Disabling key based access for the storage account will also break flow logging as the underlying machanism relies on this.

3. NSG flow logging is not supported for certain scenarios. Some of these “blind spots” are partly due to platform limitation and partly because some network flow implementation bypasses the NSGs applied at the subnet level! Those cases are highlighted below:

a. Application Gateway v2 SKU

Application gateways are platform resources that are deployed into a virtual network subnet with or without a public IP depending on resource configuration. It is commonly used as an ingress resource for web applications hosted on Azure services like AKS (Azure Kubernetes Service), App Service and VMs. There are currently two SKUs of this resource type — v1 and v2 with v2 supporting auto-scaling while v1 doesn’t.
NSG flow logging does not work properly (and is not currently supported) for NSGs that are associated with an application gateway v2 subnet. This limitation does not affect application gateway v1 SKUs — https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-faq#are-nsg-flow-logs-supported-on-nsgs-associated-to-application-gateway-v2-subnet.
We can still gain visibility of some network traffic information by collecting the access logs (v1 & v2) or WAF logs (v2). Both of these logs contain the source IP and port information for incoming requests — https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics#access-log. We will cover these logs later in this series.

b. Azure Container Instances (ACI)

Container groups can be deployed into a private virtual network subnet. NSGs can also be applied to the subnet but NSG flow logs are not supported — https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview#nsg-flow-logging-considerations
An alternative option to gain this visibility You could gain this visibility by implementing a network virtual appliance (NVA) with user-defined routes and then implementing logging from within your NVA.

c. Logic Apps

Logic App workflows that need access to private network resources can be configured to run in an Integrated Service Environment (ISE) deployed into a virtual network subnet.
Even though NSGs are allowed for the subnet, NSG flow logs are not currently supported — https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview#nsg-flow-logging-considerations

d. Azure Firewall, VPN Gateway, ExpressRoute Gateway

Associating a NSG to Azure Firewall, VPN Gateway or ExpressRoute Gateway subnets is disabled to prevent service interruption so NSG flow logs cannot be used to collect network flow logs — https://docs.microsoft.com/en-us/azure/firewall/firewall-faq#are-network-security-groups--nsgs--supported-on-the-azurefirewallsubnet.

**NSG association disabled for the firewall subnet and the gateway subnet**

For Azure Firewall, similar information to flow log records can be obtained in the “properties.msg” field of the firewall resource logs if enabled for collection (Application rule log, Network rule log; DNS proxy log) — https://docs.microsoft.com/en-us/azure/firewall/logs-and-metrics#diagnostic-logs

e. App service vNet integration

Traffic that’s used with virtual network integration doesn’t show up in Azure Network Watcher or NSG flow logs. — https://docs.microsoft.com/en-us/azure/app-service/overview-vnet-integration#how-regional-virtual-network-integration-works

4. NSG flow logs and private endpoint enabled services

NSGs and User Defined Routes (UDRs) were previously not supported for private endpoints which meant that we could not use NSG flow logs to collect network flows for private endpoint enabled resources — https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-overview#limitations
That being said, feature enhancements that extends NSG and UDR support for private endpoint enabled resources have now been announced in preview — https://azure.microsoft.com/en-us/updates/public-preview-of-private-link-network-security-group-support/
To test this feature, you can register the Microsoft.Network/AllowPrivateEndpointNSG feature and also configure the PrivateEndpointNetworkPolicies subnet property to be “enabled” (the property is automatically set to “disabled” when creating a private endpoint using the Azure portal).

# Register feature
az feature register --namespace Microsoft.Network --name AllowPrivateEndpointNSG# View status
az feature show --namespace Microsoft.Network --name AllowPrivateEndpointNSG --query properties.state# Propagate feature registration
az provider register -n Microsoft.Network# Configure subnet property
az network vnet subnet update --disable-private-endpoint-network-policies false --name <subnet-name> --resource-group <resource-group-name> --vnet-name <vnet-name># Verify
az network vnet show -n <vnet-name> -g <resource-group-name> --query 'subnets[*].{Name:name, NSG:networkSecurityGroup, PrivateEndpointPolicy:privateEndpointNetworkPolicies, PrivateLinkPolicy:privateLinkServiceNetworkPolicies}' -o table

Where are the logs stored and for how long?

NSG flow logs can be stored in an Azure blob storage container and in a Log analytics workspace (if Traffic Analytics is enabled).

If you are using a firewall-enabled storage account, you need to configure the “exception option” for “trusted Microsoft services” to allow the logs to be stored in the container (Azure Portal → Storage Account → Security + networking → Networking → Firewalls and virtual networks → Exceptions → Allow Azure services on the trusted services list to access this storage account)

**Enable exception for firewall enabled storage account**

When we enable NSG flow logs collectiion to a blob container, we need to also configure the retention period. If we set the value to zero, the data will be retained forever.

If Traffic Analytics is enabled, the logs will also be stored in a Log analytics workspace. Azure Traffic analytics is an Azure native service to process flow logs, extracts insights and visualize flow logs.

But what if we want to send the logs to an external service for analysis?

SIEMs and cloud native security solutions like Prisma Cloud can collect NSG flow logs from the blob container for analysis. This usually requires either key-based access (using the access key or SAS token) OR identity-based access (using a service principal) with permissions to read data in the storage account (usually with the built-in READER + DATA ACCESS role).

If “Traffic Analytics” is enabled, the logs can be queried and analyzed using KQL in log analytics and pulled via API.

What is the logging latency?

My “unofficial tests” showed it may take up to 5 minutes for log entries to appear in the blob storage after the network activity has happened.

Can log entries be modified to evade detection?

Depends on the log destination. Yes for blob storage. No for Log Analytics (If “Traffic Analytics” is enabled).

Anyone with modify access to the blob container can delete or modify the log entries. This is one of the reasons why we want to collect this centrally in a separate subscription with locked down access (see the good practices section below).

What security value can we get from this log?

Compliance insights: Flow logs show actual network traffic. This can be used to identify network access patterns that violates organization policies. If Traffic analytics is configured and the logs are collected in a Log Analytics workspace, they can be queried using KQL. Cloud native security solutions like Prisma Cloud also supports querying the logs using a query language called RQL — Resource Query Language.
Proactive threat detection: Flow logs can be exported to SIEMs, network analysis solutions like IDS OR to cloud native security solutions like Prisma Cloud for proactive analysis. These services will analyze the logs to identify suspicious activities and patterns. They use various methods like correlation with threat intelligence sources (like IP reputation databases), analysis pre-built ML algorithms that is trained using the log data to detect unexpected and irregular traffic patterns.
Forensics: Flow log records can be useful when investigating operations performed by a bad actor in the event of a breach. Just keep the blind spots that we discussed earlier in mind.

NSG flow logs collection good practices

1. When possible, collect the logs centrally in a separate subscription

If you have multiple Azure subscriptions and you need a way to manage the collection and analysis of NSG flow logs, my recommendation is to enforce this using Azure policy where possible but keep the logs in a separate “security subscription” with locked down access. The remaining points highlights the steps you need to do to achieve this.

2. Ensure that every production subnet has an associated NSG

You can access compliance using this built-in Azure policy — “Subnets should be associated with a Network Security Group”. Remember that gateway subnets and Azure firewall subnets are not allowed to have NSGs associated so remember to exclude them.
Also familiarize yourself with the port requirements for platform services that can be deployed in a delegated subnet before applying a NSG to the subnet. If you deny ports required by the service, it will break its functionality.
To prevent subnets from being created without an associated NSG, you can use this community policy contributed by “mrajess” — https://github.com/Azure/Community-Policy/blob/master/Policies/Network/enforce-subnets-must-have-nsg-and-nsg-must-have-same-suffix-as-subnet
You can also assess this using 3rd-party cloud security solutions like Prisma Cloud.

3. Create a network watcher in each Azure region where you have virtual network resources

You can enforce this using this built-in Azure policy — “Deploy network watcher when virtual networks are created”. This policy uses the “DeployIfNotExists” effect to create a network watcher resource in regions with virtual networks. It is best to apply it at a higher scope in your Azure resource hierarchy.

4. Create one standard tier storage account in each Azure region where you have virtual network resources.

This is because the storage account used to store NSG flow logs must be in the same region as the NSG.

Storage Account Type: General purpose v2 Storage accounts (GPv2). Configuring retention where logs are automatically cleaned up after the retention period is available only if we use GPv2 storage accounts.
Performance Tier: Only standard performance tier storage accounts is supported. Premium tier is not supported.
Default Access Tier: If your purpose is to archive the logs for future analysis in the case of a need for forensic analysis, you could set the default access tier to “cool” to save costs (Storage Account → Settings → Configuration → Blob access tier). Otherwise, set the default access tier to cool and then use the lifecycle management capabilities of the storage account to transition older logs to cool and archive tiers.
Redundancy: You can consider using GRS if you consider network flog log data to be critical to your organization in a DR scenario

5. Configure an “Activity log” alert to notify if the storage account access keys are rotated or if key-based access is disabled.

This is because this action breaks flow logging and you will need to disable and re-enable it. Alternatively, you can automate this with a Logic App workflow, Automation runbook or Function.

6. Enforce NSG flow log collection using Azure policy

We can audit flow logs status for our NSGs using this built-in Azure policy — “Flow logs should be enabled for every network security group”
To enforce the collection, we can use this community policy contributed by “mrajess” who works at Microsoft — https://github.com/Azure/Community-Policy/tree/master/Policies/Network/deploy-nsg-flow-logs

To get into more details on Azure Security, please check out my books below on Implementing Azure Security Technologies — Defense and Penetration Testing Azure for Ethical Hackers (co-authored with @kfosaeen) — Offense