Showing posts with label trace logs. Show all posts
Showing posts with label trace logs. Show all posts

Friday 19 May 2023

Logging and Monitoring Advanced Canvas Apps

Overview: Most Canvas apps reach back to 3rd parties or internal storage e.g. SQL, Dataverse, blobs to persist data.  In an ideal world we want to be able to identify error and performance concerns and be able to trace quickly.

Scenario: A canvas app performs a search against a custom connector, that goes to an external API, for example search's for Company Numbers from an Open API.

  • The annotated diagram, in orange records the button pressed to run the search.  This only shows if the "Upcoming feature", "Enable Azure Application Insights correlation tracing" is enabled.
  • A custom connector, in blue, makes  a call to an REST/Open API.
  • The last call is to an external API, highlighted in purple, (in this case it was actually a mocked APIM endpoint, but the icon would change it it was totally external such as call API search to the IRS, HMRC, Inland Revenue.

Tip: Always create App Insight Instances using a workspace and not the classic-mode app insights as it is being deprecated by Microsoft in early 2024.

App Insights Understanding:

App Insights setup using the Workspace-based setup (Log Analytics) can be queried in two ways:

  • Query thru App Insights
  • Query thru the Log Analytics workspace (the Kusto/KQL is slightly different, but it's rather minor)

Tip: If you upgrade from classic to the workspace base app Insights, the log history is still "query-able" as App Insights combines the logs from AppInsights Classic (stored in app insights directly) and the logs stored in Log Analytics.

Tip: Power Automate has a connector to Log Analytics so it's good to use this for flows so you can trace canvas apps using flow journeys.  Most people tend to build a custom connector to a function that uses the AppInsights SDK.  I've used both and they are both valid approaches and shown in the annotated diagram below.

The two options for logging Flows into App Insights.

Note: If you create a new App Insights workspace-based instance remember to update the loggers in all you Azure Services to the new instance (app key).  For example functions, APIM and Service Bus are common components.

Note: You can log to multiple workspace/app insights in a tenant and the correlations will be retrieved so you see the full history, assuming you have permissions to all log sources.

Learning: the instrumentation key for app insights has 3 parts to it: 
1) instrumentation key (basically a unique identified to find and allow logs to be saved into AppInsights, 
2) ingestion endpoint (URL for the log), and 
3) monitoring metric endpoint (URL for metrics/performance counters/live metrics/failed requests/). 

Here is an example and you can see the 3 parts:
InstrumentationKey=2675bxxx-xxxb-xxxx-bf5558009ccf;IngestionEndpoint=https://uksouth-1.in.applicationinsights.azure.com/;LiveEndpoint=https://uksouth.livediagnostics.monitor.azure.com/

Tuesday 14 March 2023

Power Platform Logging, Monitoring, and Alerting

This post relates to a previous strategy blog post, read that first https://www.pbeck.co.uk/2023/02/setting-up-azure-application-insights.html

Overview:  Microsoft uses Azure Application Insights to natively monitor Power Apps using an instrumentation key at the app level.  To log for model driven apps and Dataverse this is a Power Platform config at the environment level e.g. UAT, Prod.

When setting up Application Insights, use the Log Analytics workspace approach and not the "Classic" option as this is being deprecated.

Power Apps (Canvas Apps): Always add the instrumentation key to all canvas apps, it is set in the "App" level within each canvas app.  Deploy solutions brings challenges with changing keys for app insights logging (unmanaged layers).

"Enable correlation tracing" feature imo. should always be turned on, it is still an experimental feature but with it off, the logging is based on a sessionid

"Pass errors to Azure Application Insights" is also an experimental feature.  Consider turning it on.

Canvas Apps have "Monitor", Model driven apps also have this ability to monitor, and Power automate has it's own monitoring

Log to App Insights (put app insights on Azure Log analytics), simple example with customDimensions/record.

Trace("My PB app ... TaxAPI.NinoSearch Error - Search - btnABC",
        TraceSeverity.Error, // use the appropriate tracing level
        {
            myappName: $"PB App: {gblTheme.AppName}",
            myappError: FirstError.Message,  // optional
            myappEnvironment: gblEnv,
            myappErrorCode: 10010,
            myappCorrelationId: GUID() // unique correlationId
        }
    );
Query the logs using kusto:
traces
| extend errCode = tostring(customDimensions["myappErrorCode"]), err = tostring(customDimensions["myappError"])
| where errCode == "100100"

Coming June 2023

Push cloud flow execution data into Application Insights | Microsoft Learn

Allows logging to tie Flows back to the calling Canvas app.  You can now do this manually but it has to be applied at all calls to or after the flow.

Below is a basic checklist of decisions to ensure you have suitable logging

Logging Checklist:

  1. Setup Azure Log Analytics (1 per DTAP env e.g. uat, prd)
  2. Get the workspace key needed for logging to Log analytics "Agents" > "Log Analytics agent instructions", copy the Workspace Id and the Secondary Key
  3. Create an Azure Application Insights per DTAP
  4. Each Canvas app needs an instrumentation key (check, have you aligned DTAP log instances with the Canvas App DTAP)
  5. Power Automate has great monitoring, but it is a good idea to setup logging for Dataverse (which shall cover model apps), done thru Power Platform Admin Studio > Environment
  6. Enable Logging Preview Feature for Canvas apps & check the power automate push cloud execution feature state.
  7. Do you have logging patterns in you Canvas app for errors, do you add tracing, and is it applied consistently?
  8. Do you have a Pattern for Power Automate runs from Canvas apps?  I like to log if the workflow errors after the call.
  9. Do you have a Pattern for Custom Connectors?
  10. Do you correlation trace Custom API (internal and 3rd party)? 
  11. Do you have a Try, Catch, Finally scope/pattern for Workflows.  How do you write to the logs, most common is to use an Azure Function with the C# SDK.  I like to use the Azure Log Analytics Connector in my catch scope to push error info into the workspace log using a custom table.
  12. Ensure all Azure Services have instrumentation keys. Common examples are Azure Functions, Azure Service Bus, API Manager, the list goes on...
  13. Do you implement custom APIM monitoring configuration?
  14. Do you use the SDK in your code (Functions etc.)?
  15. Setup Availability tests - super useful for 3rd party API's.

Once you have the logs captured and traceable (Monitor & Alerting Checklist):

  1. Create Queries to help find data
  2. Create monitoring dashboard using the data
  3. Use OOTB monitoring for Azure and the platform
  4. Consider linking/embedding to other monitors i.e. Power Automate, DevOps, Postman Monitor
  5. Setup alerting within the Azure Log Workspace using groups, don't over email.  For information alerts, send to Slack or Teams (very simple to setup a webhook or incoming email on a channel to monitor)
  6. Power Automate has connectors for adaptive cards channel messaging, consider using directly in Flows or from alerts, push the data into a flow that will log the alert using an adaptive card right into the monitoring channel.

Saturday 14 January 2023

APIM Logging

Overview: Azures API Management is a big service, it is worth understanding the logging capability so you can effectively analyse traffic.

Thoughts:

  • Multiple App Insights can be setup with default logs going to a specific App Insights.
  • Each API can be overridden to log to any of the API's added to API.
  • The old "Classic" App Insights, stored data internally, whereas the new "workspace-based" app insights", I think of it it as "V2 App Insights connected to a Log Analytics", the new data is stored in the workspace.
  • If you upgrade App Insights, the results blend from two storage locations, the old data stored internally with App Insights and the new data stored within Log Analytics - if you query Log analytics, you only see the new log analytics data.
  • Security for App Insights should be done at the Resource Group (RG) level, ther are AppInsight roles for use at RG level, if the workspace is on a different resource group to the app insights connected instance, ensure you sort out the permssions in both RGs.
  • Open Telemetary project is making strides forward, and for API's it will be great.

Problem: I recently migrated a customer Dev, Test, Appearance, Pre-prod and Production (Not yet) to use the AppInsights instance running on Log Analytics (sometimes refereed to as V2).  Logging wasn't work correctly.


Initial Hypothesis: I have complicated resource groups differing crossing DTAP boundaries.  By default, APIM has a logging catch all setup per APIM instance and then specific API's settings are changed to log to specific App Insights.

Steps:

My AppInsights instance was to rename the old classic type AppInsights e.g. "appinsights-dev" becames "appinsights-dev-delete".

Create a new AppInsights instance using the V2 Log Analytics option  and name it the original name "".  The client opted for the name to be the same.  It would be simpler to give it a name like "appinsights-dev02".  The clients also wanted to use a shared Log Analytics instance per env e.g. "loganalytics-dev-shared".






Friday 10 July 2020

Power Apps Tracing to App Insights Not Working in edit mode

Overview: Power Apps integrates to directly log to App insights.  This post looks at the issues around Tracing in App insights.

Setup & Verify:
App Insights instance Instrument Key Required, Config you Power App to Trace to App Insights, and create a button to test.
The Monitor Tool is fantastic for tracing all outbound traffic, you no longer need to go to the service and check if Power Apps is reaching.  For example I use to have to look at the APIM Azure App logs for custom Connectors.
I can see all my interactions in Power Apps and travelling out.  This is a massive win for Power Apps so giving Tracing from the front end.

Greg Lindhorst, wrote a post on using the Power Apps Monitor Tool for debugging and performance improvement.
Note: Traces written to App Insights don't work in preview.  Verify, see below.

Problem: It normally takes a few minutes (like 2 minutes) to show up in App Insights, it is not showing up after 15 minutes.  Let's see, I've drop the Power Apps team a message and i think it's a bug that has crept in recently.  Today is 10 July 2020.  I've also notice that none of my Session page tracing is showing up in App Insights.

Update: 11 July 2020, the Power Apps behaviour has been on my mind.  I'm in edit mode, and when I publish the app and use it, App Insights logs perfectly.  When in edit mode running the app, even direct traces, no logging into Power Apps.  I did not realise this, but it kind of makes sense.

Resolution: Publish & Run the app, and the Tracing and Power App session tracing shows up in App Insights.

Only when I Run the Published App to my Page Views and Custom Traces get logged in App Insights.

Friday 17 September 2010

Unified Logging Service (ULS)

Overview: ULS is made up of 3 components: Trace logs, Reporting database & the Windows Event viewer. 

Trace Logs:
Trace Logs by default are located in the 14 hive/SharePoint Root, by default they are created every 30 minutes and can be viewed in notepad. 
ULSviewer is available on codeplex to improve usability as the logs are easier to follow than if you use notepad.
Trace logs can grow extremely quickly.  Only log the appropriate messages for troubleshooting without making the logging to big.  Verbose logging cause huge amounts of logging but it is extremely detailed.  The amount of logging or what is logged can be set using the CA> Monitoring > Diagnotic Logging "Configure Diagnostic Logging".
SharePoint 2010 uses NTFS compression for log files to reduce storage on the WFE (logging) servers.
Consider moving the trace logs to a new drive so that the logs don't fill up the default drive (usually the c drive).  This change applies to all servers in the farm.
Logs by default are kept for 14 days, you can also hold them using sizes rather than duration.
Event Log Flood Protection in CA should always be turned on to stop repeated errors being recorded multiple time in you logs.
Correlation ID's as the name suggest helps us track errors from the UI through to the Trace Logs.  The end user when the receive an error are shown a unique correlation ID that can be used to track down the problem.  Correlation Id's as also shown on the developer dashboard and can be used to tract from the UI into the trace logs without errors.

Update 7 Oct 2010 - Useful Monitoring artlicle:
http://www.sharepointproconnections.com/content.aspx?topic=How-SharePoint-2010-s-Monitoring-Features-Can-Help-Admins&catpath=sharepoint
Update 14 Dec 2010 - ULS Viewer
Tool to view ULS log and filter data


Various SharePoint Logs Explained:
http://stackoverflow.com/questions/525197/where-is-the-default-log-location-for-sharepoint-moss