An Introduction to Microsoft Azure SRE Agent

Managing cloud environments often involves jumping between alerts, logs, dashboards, and code repositories when something breaks.  Azure SRE Agent aims to simplify operational troubleshooting.  The tool:

1. Investigate incidents
2. Automates routine workflows
3. Helps resolve issues faster

What is the Azure SRE Agent?

The SRE Agent combines 1) Brain - Large Language Models (LLMs, either OpenAI or Anthropic models) with 2) Azure-specific product knowledge - tools and knowledge, and 3) Instructions - custom configuration.  Allowing it to reason over operation support data.

The agent is deployed directly inside an Azure Resource Group, which sits within an Azure Subscription.  I have only used the Reader Scope for my assigned identity permissions; it can operate in Reader Scope (for inspection and telemetry analysis) or Privileged Scope (to execute changes and automated mitigations).

The Azure SRE Agent operates across three main operational patterns

Automate Incidents: When an alert fires, the agent queries monitoring platforms, correlates signals across systems, determines probable root causes, and suggests or executes fixes

Automate Scheduled Workflows: It can perform scheduled background tasks, such as routine health checks and compliance sweeps

Investigate and Advise (Virtual Team Member): Engineers can ask natural-language questions (e.g., "what changed in the last hour?") in chat interfaces like Microsoft Teams to get grounded operational insights.

Running Incident Response Issues

Instead of forcing engineers to manually correlate disparate data, the agent follows a streamlined incident workflow: when an alert fires (in Azure Monitor, in my case), the SRE agent picks it up.  It queries logs using correlation IDs to connect the data so that the agent can do the Root Cause Analysis & Ticketing: The agent identifies the probable cause (such as a bad commit or memory trend) and can prefill an incident ticket in ServiceNow or PagerDuty with diagnostic findings

The support engineer, in this case me, reviews and approves the remediation.

Connects natively with Azure Monitor, Application Insights, Log Analytics, PagerDuty, ServiceNow, GitHub, and Azure DevOps

Tip: External tools can also connect via the Model Context Protocol (MCP), basically making any API accessible.

Customisation: Capabilities can be extended using Skills (Azure CLI scripts and marketplace runbooks), Python tools, Custom sub-agents, and Agent hooks

Billing: Operational reasoning is metered using Agentic Units (AUs) rather than standard raw tokens, with standard monthly allocations; I still have no idea what this will cost or how it is calculated.

Summary

The Microsoft Azure SRE Agent provides a structured, governed way to use AI for incident management. By pulling together context from observability tools and code repositories into a single thread, it reduces tool switching and speeds resolution.

Dynatrace Product Play

 Dynatrace is pretty similar to Azure Monitor.

• Dynatrace (really good if you use multi cloud) Dynatrace - Saas offering is on AWS.  Can be on-prem.  
• Making workloads observable is using Logs, Traces, Events, and metrics into Dynatrace.  From these ingested events we analyse and can automate behavior. 
• OneAgent is deployed on the Compute i.e. VM, Kubernetes.  Can import logs from other SIEMs or Azure Monitor, so you can eventually get Azure service logs such as App Service or Service Bus.  
• Does Full stack and includes code-level and applications and infrastructure monitoring, also can show User monitoring.  
• Dynatrace offers scalable API's that are sitting on Kubernetes.  
• "Davis" is the AI engine used to help figure out the problems.  
• Alerting is solid.  
• Dynatrace can log against 1) network/Infra 2) SDK 3) DEM (User monitoring,..)  logs, traces, metrics are ingested either using OneAgent or OpenTelemetry.
• Management Zones - user only see's information they have access to and need.
• Define a Site Reliability Guardian (SRG) to each program/project, this allows you to identify thru RAG boards the current and recent state of the various pieces.  There are Guardian templates to use as a starting point.
• W3C Trace Context is used - it allows for end-to-end tracing.  OpenTelemetry or Dynatrace keep the trace and provide in headers (traceparent.
• Create documentation and tutorials for Dynatrace.  Dynatrace has a playground tenant for playing on.

High-level Architecture hosted on AWS.

High-level architecture for capturing logs et al. and then using the data.

Product Screen Shots:

Azure & Dynatrace

• Abnormality detection using AI. shall greatly improve observability and security. 
• End-to-end visibility is what makes it so amazing.
• Enterprises often use Dynatrace as there central SIEM solution, shipping from Azure in Dynamics takes planning but works well, categorise and ensure the right into is pushed into Dynatrace.  
• Dynatrace is the leader in Gartner and Forrester in it's space.
• Grail - lake house, schema-less, allows for easy fast query.  Massive scale.  Bring all data together and query at hyperscale.  Grail is in 15 regions either on AWS, Azure, or GCP for customers to use.  UK looks like AWS only. 
• Grail: Record level protection, masking data, support access controls (elevate privileges).  

Dynatrace architecture for Grail from Barcelona conference 5 Oct 2023.

Collect all events in Grail, automate the process of identify suspicious activity relating to security.  Faster reaction time.

Azure offers Dynatrace as a SaaS service

Updated 16 Feb 20224