Power Platform Logging, Monitoring, and Alerting

This post relates to a previous strategy blog post, read that first https://www.pbeck.co.uk/2023/02/setting-up-azure-application-insights.html

Overview:  Microsoft uses Azure Application Insights to natively monitor Power Apps using an instrumentation key at the app level.  To log for model driven apps and Dataverse this is a Power Platform config at the environment level e.g. UAT, Prod.

When setting up Application Insights, use the Log Analytics workspace approach and not the "Classic" option as this is being deprecated.

Power Apps (Canvas Apps): Always add the instrumentation key to all canvas apps, it is set in the "App" level within each canvas app.  Deploy solutions brings challenges with changing keys for app insights logging (unmanaged layers).

"Enable correlation tracing" feature imo. should always be turned on, it is still an experimental feature but with it off, the logging is based on a sessionid

"Pass errors to Azure Application Insights" is also an experimental feature.  Consider turning it on.

Canvas Apps have "Monitor", Model driven apps also have this ability to monitor, and Power automate has it's own monitoring

Log to App Insights (put app insights on Azure Log analytics), simple example with customDimensions/record.

Trace("My PB app ... TaxAPI.NinoSearch Error - Search - btnABC",
        TraceSeverity.Error, // use the appropriate tracing level
        {
            myappName: $"PB App: {gblTheme.AppName}",
            myappError: FirstError.Message,  // optional
            myappEnvironment: gblEnv,
            myappErrorCode: 10010,
            myappCorrelationId: GUID() // unique correlationId
        }
    );

Query the logs using kusto:

traces
| extend errCode = tostring(customDimensions["myappErrorCode"]), err = tostring(customDimensions["myappError"])
| where errCode == "100100"

Coming June 2023

Push cloud flow execution data into Application Insights | Microsoft Learn

Allows logging to tie Flows back to the calling Canvas app.  You can now do this manually but it has to be applied at all calls to or after the flow.

Below is a basic checklist of decisions to ensure you have suitable logging

Logging Checklist:

1. Setup Azure Log Analytics (1 per DTAP env e.g. uat, prd)
2. Get the workspace key needed for logging to Log analytics "Agents" > "Log Analytics agent instructions", copy the Workspace Id and the Secondary Key
3. Create an Azure Application Insights per DTAP
4. Each Canvas app needs an instrumentation key (check, have you aligned DTAP log instances with the Canvas App DTAP)
5. Power Automate has great monitoring, but it is a good idea to setup logging for Dataverse (which shall cover model apps), done thru Power Platform Admin Studio > Environment
6. Enable Logging Preview Feature for Canvas apps & check the power automate push cloud execution feature state.
7. Do you have logging patterns in you Canvas app for errors, do you add tracing, and is it applied consistently?
8. Do you have a Pattern for Power Automate runs from Canvas apps?  I like to log if the workflow errors after the call.
9. Do you have a Pattern for Custom Connectors?
10. Do you correlation trace Custom API (internal and 3rd party)? 
11. Do you have a Try, Catch, Finally scope/pattern for Workflows.  How do you write to the logs, most common is to use an Azure Function with the C# SDK.  I like to use the Azure Log Analytics Connector in my catch scope to push error info into the workspace log using a custom table.
12. Ensure all Azure Services have instrumentation keys. Common examples are Azure Functions, Azure Service Bus, API Manager, the list goes on...
13. Do you implement custom APIM monitoring configuration?
14. Do you use the SDK in your code (Functions etc.)?
15. Setup Availability tests - super useful for 3rd party API's.

Once you have the logs captured and traceable (Monitor & Alerting Checklist):

1. Create Queries to help find data
2. Create monitoring dashboard using the data
3. Use OOTB monitoring for Azure and the platform
4. Consider linking/embedding to other monitors i.e. Power Automate, DevOps, Postman Monitor
5. Setup alerting within the Azure Log Workspace using groups, don't over email.  For information alerts, send to Slack or Teams (very simple to setup a webhook or incoming email on a channel to monitor)
6. Power Automate has connectors for adaptive cards channel messaging, consider using directly in Flows or from alerts, push the data into a flow that will log the alert using an adaptive card right into the monitoring channel.

Distribute tracing using Azure Application Insights across Azure SaaS product

Overview:  Building SaaS products using multiple underlying Azure PaaS, and IaaS services with multiple Microservices supporting and calling each other is great.  The issue is we need to be able to trace, debug, and observe the logic flow through multiple Microservice calls.  Distribute Tracing on Azure supports technical players such as devOps teams, developers, support, technical leads and/or architects to find and trace the entire execution to figure our what is/has happened.  Application Insights provides rich functionality on Azure PaaS services.

Distribute Tracing:  A good option for providing consistent traceable logging is to use Application Insights with the Distributed Tracing to trace the flow of each transaction.  The original request generates an Id which is set as the operation_parentId.  Now we can easily follow the execution of a specific operation.

It is fairly easy to tie multiple operations together.  For example, an operation that fires timer jobs, each timer job would be seen as unique operations with their own full trace.  By referencing the original operation when the timer jobs are setup, long running distributed jobs can be tied together.

Thoughts:  Distributed tracing generally catches exceptions from the underlying infrastructure services such as SQL, Azure Functions, App Service on Windows but in code you can add additional tracing information.  The tracing info can be exception based but most of this is picked up anyway or trace base (when an even happens you want to record).

It is a good idea to instantiate the Telemetry client once per service e.g. webAPI and merely call using the same instantiate telemetry object instance throughout each application.

On exception in both client and Server side code write to App Insights telemetry.  Below is the C# server side code snippet:

try

{

    ...

    telemetry.TrackTrace(message, SeverityLevel.Warning, properties);

}

catch (Exception ex)

{

    telemetry.TrackException(ex);

}

More Info:

Distributed Tracing in Azure Application Insights - Azure Monitor | Microsoft Docs

Application Insights API for custom events and metrics - Azure Monitor | Microsoft Docs

Finding Correlation Errors on a SP2013 farm

Background: SP 2013 has rich and expansive logging/tracing capabilities.  Logging is done via the Unified Log Service (ULS).  This will add logs to the tracelogs (often refered to as the ULS logs or ULS trace logs or ULS, it doesn't matter except you need to understand the ULS service is not only the trace log) and the Windows Event Viewer.  Anything logged in the Event Viewer log will also be in the ULS trace logs.

It is worth check how your logging is setup on your farm.  I change my default location for my ULS trace logs.  Change the logging so it matches your farms requirements.

On a small farm, it's normally pretty easy to take a Correlation Id / the unique GUID generate for the SharePoint request, open the trace log using notepad and find the error.  The default is to create a trace log every 30 minutes, these log files have a lot of data in them on busy production farms, and as you may have a large farm you also have multiple logs to check. I use Microsofts'd unsupported ULSViewer to look at all my logs regardless of farm size.  You can trace the logs in a live format and then filter out what you need.  Another option is to open existing errors to get historical issues.  If you know the datetime and server where the error occured, you open the correct log file (it is labled with a datetimestamp) and then either filter for the correlationId or look around the time the error occured.

Lastly, timer jobs ship entries from the ULS logs into the SharePoint Logging Database (SP_UsageandHealth).  You can directly query the SP_UsageandHealth database using T-SQL.

Tracing Correlation Errors on a SharePoint 2013 Farm.
User passes you a correlation Id and the date/time when the error occured, find the apprioate ULS trace log.  Open the log using ULSViewer and filter for the CorrelationId.  If you can reproduce the bug, you have the developer dashboard that can be turned on (performane penalty) selectively, their is a new SP2013 tab "ULS" this will show you the ULS trace snippet relating to this request.

On a big farm you may want to1st find out which server in the farm had the error:

Merge-SPLogFile -Path ".\error.log" -Correlation "5ca5555c-8555-4555-555b-f555af4d5555"

Tip: Be aware this is a heavy process, so restrict which logs you will merge.
 

Use ULSViewer to find the correlationId and review the logs.

Use IT tools or Fiddler to examine the http response from SharePoint to get the correlation Id, this is the SPRequestGUID (assuming it is not show on the error message).

 

More Info:

Tobias Zimmergren has a great post on working with Correlation Id's.

http://www.sharepointblog.co.uk/2012/09/logging-capabilities-of-sharepoint/

http://habaneroconsulting.com/insights/An-Even-Better-Way-to-Get-the-Real-SharePoint-Error

List of the ULS viewers

http://www.sharepointblog.co.uk/2012/09/logging-capabilities-of-sharepoint/

Sandbox solution ULS logging

Overview: Step-by-step instructions to create a logging module for Sandbox solutions using a full trus proxy that logs to the ULS.

Steps:
1.> Ensure Sandbox solutions service is running:

1.1.> If the "Microsoft SharePoint Foundation Sandboxed Code Service" is stopped start it either thru CA (shown above or PS). 

PS> Start-Service -Name SPUserCodeV4

1.2.> Ensure the Windows service is running.

2.> Create project to test the full trust ULS logging proxy.

2.1> Create a Visual Studio project that deploys as a sandbox solution (test rig)

2.2.> Create a visual web part for sandboxed solutions (VSIX/Extension maneger: Visual Studion 2010 SharePoint Power Tools)

Note:  The submit button does not perform any action as the Full Trust Proxy must be created and deployed so we can use it to log code to the ULS.

3.> Create a ful trust proxy that code in the sandbox solution can call to add the error message to the ULS logs.

3.1.> Create an empty SharePoint project using Visual Studio 2010that deploys to the farm.
3.2.> CKSDev (VSIX extension has a SPI template to create a full trust proxy).  I used the SPI to create my ful trust proxy.
3.3.> Create the code to log to the ULS

3.4.> Edit the full trust proxy to use the ULS logging code created above.

4.> Add the proxy code call to the test rig/proxy caller i.e. sandboxed solution code.

Tip: Resart the Sandbox solution work process after each deployment

Tip: Use Powershell to check a correlation Id quickly:
PS>Get-SPLogEvent | ?{$_.Correlation -eq ""} | ft Category, Message -Autosize

References:
http://msdn.microsoft.com/en-us/library/gg622617.aspx
http://www.zimmergren.net/tag/sps
http://www.wictorwilen.se/Post/Working-with-SharePoint-2010-Correlation-ID-in-PowerShell-and-code.aspx

Logging custom error for SharePoint 2010 custom code

Problem: SP2010 has good logging in the ULS logs. Unlike MOSS you can write to the ULS logs in SharePoint 2010, allowing for a single consistant place to log your errors. The event viewer logs errors to a lesser extent than ULS where administrators tend to look first. However, on all projects the topic of how and where to log custom coding errors comes up. What level should the event be caught at and where should they be placed. I have seen multiple was of doing this and it really comes down to what the previous projects used and how do you want to monitor errors.
Tips:

1. Don't write code to function using error catching i.e. I have seen developers catch a specific error and from this position they know the code is to follow specific logic -it's very inefficient. Write code to deal with all situations.
2. Catch errors as specifically as possible, then decide if the error should be bubbled up or can it be dealt with via the logs. But catch the appropriate errors so they are logged.
3. You can write to Unified Logging Service(ULS) logs but these are often not checked or hard to find issues you have thrown up in your code, so consider using a logging block such as Microsoft's Enterprise Logging blocks or Log4net. Ted Pattison suggests writing to ULS and he know his stuff so if you don't have another specified logging policy write to the ULS. And if you do have another logging method consider writing to the ULS anyway.
4. There are a lot of logging applications for .NET, and most companies tend to have logging code ready for implementation on your SP 2010 project.
5. I have seen a MS gold partner use tracing on all projects. So when an error occurs they turn on tracing and try replicate the issue in production environments. Far better to catch errors so you can get your issues resolved quickly and don't need to change config setting or leave tracing enabled on the live production boxes.
6. A web part error can cause an entire page to throw an application error. However don't throw try catch blocks around all code, rather try catch the errors at more appropriate junctions such as at the service layer. Once again it really depends on the WP.

Summary: Avoid using errors for logic (pretty obvious). Log the errors to a distinct area (database, file system(xml files are pretty useful)) to identifying where and what the issue is. Catch specific errors -try not to catch general exceptions and if you do need to catch general exceptions make sure you have looked for more specific exceptions such nullrefobjectexceptions.

More Info:
MSDN SharePoint Logger - http://msdn.microsoft.com/en-us/library/ff798361.aspx
Writing to the ULS
Update: 5 Dec 2010 - Writing to ULS using SP2010 by Waldek Mastykarz
Update: 18 Jab 2011 - MSDN article on logging and debugging

The SharePoint Logger
using Microsoft.Practices.ServiceLocation;ILogger logger = SharePointServiceLocator.GetCurrent().GetInstance();
using Microsoft.Practices.SharePoint.Common.ServiceLocation;
using Microsoft.Practices.SharePoint.Common.Logging;
IServiceLocator serviceLocator = SharePointServiceLocator.GetCurrent();
ILogger logger = serviceLocator.GetInstance();
logger.TraceToDeveloper("Unexpected condition");

Update 14 Dec 2010 - ULS Viewer - Tool to view ULS log and filter data
Update 23 Dec 2010 - Tracing using CorrelationId