Azure Security Tools 01: Maester

Continuous Security Validation of Entra ID, Exchange Online and SharePoint Online

1 — Overview

• https://maester.dev/
• Summary:
• Many cloud breaches are due to preventable misconfigurations. It can be a difficult problem to solve as the constant evolution means that the goalpost keeps shifting. Insecure configuration of identity platforms and cloud services like Entra ID and Exchange Online can lead to vulnnerabilities that can be exploited. Continuous security validation is one aspect of managing this challenge.
• Maester is a security test automation framework that we can use to validate and report on configuration of our Entra ID and Exchange Online services. It uses Pester behind the scene.
• It comes with a collection of ready to use tests: Tests that validate Entra ID configuration (Maester, EIDCA, CISA); Tests that validate Exchange Online configurations (CISA); Tests that validate Azure configurations (CISA); New tests are added by the Maester team/community. Tests things like conditional access policies — if they are created or not, effective or not.
• Because the tests can be done using the command line, it presents an opportunity to bring DevOps practices to cloud service config validation. It can be integrated with Azure Automation ; DevOps platforms like Azure DevOps and GitHub ; Azure Container App Jobs.
• Creators: An Open Source project
• Use Cases:
• Automate continuous tests of Entra ID and M365 cloud services
• Conditional policy assigned to a group of guest users
• Someone deleted the group

2 — Install pre-requisite PowerShell modules

1. Install and validate Pester

• Maester uses Pester behind the scene.

Install-Module Pester -SkipPublisherCheck -Scope CurrentUser -Force

Get-Module -ListAvailable -Name Pester

• Maester needs to be able to connect with Entra ID, Exchange Online and Azure to validate their configurations so we also need to install the nnecessary PowerShell modules for these services.

2. Install and validate the Azure PowerShell module

Install-Module Az -Scope CurrentUser -Force

Get-Module -ListAvailable -Name Az*

3. Install and validate the ExchangeOnline PowerShell module

Install-Module ExchangeOnlineManagement -Scope CurrentUser -Force -AllowClobber

Get-Module -ListAvailable -Name ExchangeOnlineManagement

4. Install and validate the Microsoft Graph PowerShell module

Install-Module Microsoft.Graph -Scope CurrentUser -Repository PSGallery -Force -AllowClobber

Get-Module -ListAvailable -Name Microsoft.Graph

3 — Install the Maester module and review its tests

1. Install the Maester PowerShell module

Install-Module Maester -Scope CurrentUser -Force

2. Create directory to store the tests

New-Item -ItemType Directory -Name "maester-tests"

Set-Location -Path "maester-tests"

3. Review the default tests

• https://maester.dev/docs/tests/

Maester tests

• Entra ID tests only
• https://maester.dev/docs/tests/maester/
• maester-tests\Maester folder

Entra ID Security Config Analyzer (EIDSCA) tests

• Entra ID tests only
• https://maester.dev/docs/tests/eidsca/
• EIDSCA folder

CISA tests (Includes Entra and Exchange Online tests)

• Entra ID, Exchange Online, SharePoint Online and Azure tests
• Entra — https://maester.dev/docs/tests/cisa/entra
• Exchange Online — https://maester.dev/docs/tests/cisa/exo
• cisa folder
• SCuBA (Secure Cloud Business Applications) is a project run by the US governmment agency — CISA (Cybersecurity and Infrastructure Security Agency)
• The project provides guidance and capabilities to US federal agencies’ on how to best secure their cloud business application environments and protect sensitive information stored in them.
• Maester supports tests that we can use to validate that a M365 tenant’s configuration aligns with the policies defined in the SCuBA security configuration baseline documents
• https://github.com/cisagov/ScubaGear/blob/main/baselines/README.md
• Entra ID security configuration baseline
• https://github.com/cisagov/ScubaGear/blob/main/PowerShell/ScubaGear/baselines/aad.md

CIS tests (Exchange Online tests)

• Exchange Online tests
• Entra — https://maester.dev/docs/tests/cisa/entra
• Exchange Online — https://maester.dev/docs/tests/cisa/exo

4 — Install the Maester tests

Install-MaesterTests
explorer .

## Maester tests
maester-tests\Maester folder

## Entra ID Security Config Analyzer
EIDSCA folder

## CISA (Includes Entra and Exchange Online tests)
cisa folder

5 — Connect to your cloud environments

1. Sign into your Microsoft 365 tenant and run the tests.

• https://maester.dev/docs/commands/Connect-Maester
• Maester needs to be able to connect with Entra ID, Exchange Online and Azure to validate their configurations

Connect-Maester
Connect-Maester -Service All
Connect-Maester -Service Azure,Graph
Connect-Maester -SendMail
Connect-Maester -SendTeamsMessage
Connect-Maester -Privileged

6 — Run the tests

• https://maester.dev/docs/commands/Invoke-Maester/

## Reports are generated in the default ./test-results folder
## Run all tests in the current folder
Invoke-Maester
Invoke-Maester -Verbosity Normal
Invoke-Maester -Verbosity Detailed
Invoke-Maester -Verbosity Diagnostic

## Run tests in a specific folder
Invoke-Maester ./maester-tests
Invoke-Maester -Path ./maester-tests/EIDSCA

## Runs the tests with specific tags
Invoke-Maester -Tag 'CA'
**Tags: CA, App, CISA, EIDSCA, Entra, MS.AAD, MS.EXO, Maester, Privileged, Security

## Specify different output folder
Invoke-Maester -OutputFolder './my-test-results'
Invoke-Maester -OutputHtmlFile './test-results/TestResults.html'

## Runs tests and send a report of the results to mail recipient.
Invoke-Maester -MailRecipient <email_address>

## Runs tests and post a summary of the results to a Teams channel.
Invoke-Maester -TeamId '<team_id>' -TeamChannelId '<team_channel_id>'

## Run the tests using a saved configuration
$configuration = New-PesterConfiguration
$configuration.Run.Path = './maester-tests'
$configuration.Filter.Tag = 'CA'
$configuration.Filter.ExcludeTag = 'App'
Invoke-Maester -PesterConfiguration $configuration

7 — Maester — Integrate with GitHub (CICD)

7a. Import repository

• https://github.com/new/import
• The URL for your source repository: https://github.com/maester365/maester-tests
• Your new repository details:
• Owner: Your GitHub account OR organization name
• Repository name: maester-tests
• Private: Selected
• Begin import

7b. Set up the GitHub Actions workflow

• There are many ways to authenticate with Microsoft Entra from GitHub Actions. The recommended approach is to use workload identity federation which is a more secure option.

Create an Entra Application and grant permissions to Microsoft Graph

# Create the Azure AD app and capture appId and appObjId
appId=$(az ad app create --display-name "maester-github-repo" --query appId --output tsv)

appObjId=$(az ad app create --display-name "maester-github-repo" --query id --output tsv)

# Make a note of the IDs
echo $appId
echo appObjId

Grant permissions to Microsoft Graph

# Required permission GUIDs
permission_guids=(
"7ab1d382-f21e-4acd-a863-ba3e13f7da61"  # Directory.Read.All
"ae73097b-cb2a-4447-b064-5d80f6093921"  # DirectoryRecommendations.Read.All
"6e472fd1-ad78-48da-a0f0-97ab2c6b769e"  # IdentityRiskEvent.Read.All
"246dd0d5-5bd0-4def-940b-0421030a5b68"  # Policy.Read.All
"37730810-e9ba-4e46-b07e-8ca78d182097"  # Policy.Read.ConditionalAccess
"4cdc2547-9148-4295-8d11-be0db1391d6b"  # PrivilegedAccess.Read.AzureAD
"230c1aed-a721-4c5d-9cb4-a90514e508ef"  # Reports.Read.All
"ff278e11-4a33-4d0c-83d2-d01dc58929a5"  # RoleEligibilitySchedule.Read.Directory
"c7fbd983-d9aa-4fa7-84b8-17382c103bc4"  # RoleManagement.Read.All
"38d9df27-64da-44fd-b7c5-a6fbac20248f"  # UserAuthenticationMethod.Read.All
"fee28b28-e1f3-4841-818e-2704dc62245f"  # RoleEligibilitySchedule.ReadWrite.Directory
)

# Add each permission to the app
for guid in "${permission_guids[@]}"; do
az ad app permission add --id $appId --api 00000003-0000-0000-c000-000000000000 --api-permissions $guid=Role
done

az login

# Admin consent for the app (in case of roles requiring admin consent)
az ad app permission admin-consent --id $appId

Grant permissions to Exchange Online

• Exchange.ManageAsApp: Necessary to support tests that validate Exchange Online configurations, such as the CISA tests.

az ad app permission add --id $appId --api 00000002-0000-0ff1-ce00-000000000000 --api-permissions dc50a0fb-09a3-484d-be87-e023b12c6440=Role
az ad app permission admin-consent --id $appId

Grant permissions to Azure

subId=$(az account show --query id --output tsv)

# Assign 'Reader' role to the service principal at subscription scope
az role assignment create --assignee $appId --role "Reader" --scope "/subscriptions/$subId"

# Assign 'Reader' role to the service principal for Azure Active Directory Identity Management
az provider register -n microsoft.aadiam
az role assignment create --assignee $appId --role "Reader" --scope "/providers/Microsoft.aadiam"

Add federated credentials

• Azure Portal → Entra ID → Manage → App registrations → All Applications → Select Application → Manage → Certificates & secrets → Federated credentials → Add credential → GitHub Actions deploying Azure resources
• Organization: Your GitHub account OR organization name
• Repository: maester-tests
• Entity type: Branch
• GitHub branch name: main
• Credential details
• Name: maester-github-repo
• Add

Create GitHub secrets

• GitHub Repo → Settings → Security → Secrets and variables → Actions → Repository secrets → New repository secret
• Name: AZURE_TENANT_ID
• Value: Enter your Azure Tenant ID
• Name: AZURE_CLIENT_ID
• Value: Enter the App ID of your app that you made a note of earlier

Enable GitHub Actions

• GitHub Repo → Settings → Code and automation → Actions → General → Actions permissions
• Select “Allow all actions and reusable workflows”
• Save

Create GitHub Action worklow for Maester

• GitHub Repo → Actions → Skip this and set up a workflow yourself
• Uses this marketplace action — https://github.com/marketplace/actions/maester-action

name: Maester Daily Tests

on:
  push:
    branches: ["main"]
  # Run once a day at midnight
  schedule:
    - cron: "0 0 * * *"
  # Allows to run this workflow manually from the Actions tab
  workflow_dispatch:

permissions:
      id-token: write
      contents: read
      checks: write

jobs:
  run-maester-tests:
    name: Run Maester Tests
    runs-on: ubuntu-latest
    steps:
    - name: Run Maester action
      uses: maester365/maester@main
      with:
        client_id: ${{ secrets.AZURE_CLIENT_ID }}
        tenant_id: ${{ secrets.AZURE_TENANT_ID }}
        include_public_tests: true # Optional: Set to false if you are keeping to a certain version of tests or have your own tests
        step_summary: true         # Optional: Set to false if you don't want a summary added to your GitHub Action run
        artifact_upload: true      # Optional: Set to false if you don't want summaries uploaded to GitHub Artifacts
        # Other inputs are available and can be reviewed in the action.yml in the Maester repository

• Commit changes.. (top right) → Commit changes

Review the workflow

• Actions → maester-daily-tests to view the status of the pipeline
• Review “Summary” (top left)

Update the tests with latest tests from the Maester team/community

• Create a branch from the main branch
• Clone to your PC
• Update the Maester PowerShell module to the latest version and load it.
• Change to the maester-tests\tests directory.
• Run Update-MaesterTests.