App Insights for Power Platform - Part 11 - Custom Connector

Overview:  Power Automate can set retry policies on custom connectors, Canvas apps using a Custom Connector, does not have any retry configuration.  FYI is the Custom Connector gets a 5.x.x error it shall retry 4 times.  Proven using 500 and 502 errors.  408 (timeout) and 429 (to busy) errors appear to throw 4 times (retry driven by canvas app; using the Custom Connector trigger shall only try once).  

My example:  My Canvas app uses a custom connector, that calls my Azure Function, in turn this calls my APIM, and APIM calls the 3rd party.

My Azure function returns a 500 or 502 or 408 (response timeout) or 429 if the response has not been received in 10 seconds, and I push the http code back to the Custom connector.  I can see the response from the 3rd party is taking +-35 seconds.  I can see from network traces 4 invoked calls that all fail with 408 http response codes.

Result:  The custom connector retries 4 times resulting in my Power App being locked for 40+ seconds.

Possible remediation: 

1. Return a 418 HTTP response code (I am a teapot), my app calls the function using the custom connector once.  So using 418 (also tested with 400 - but 400 is not the right response) errors behaves differently from the 5xx, 408, 429 errors from the API.  

Note: Originally I was caught out as I trigger using the custom connector test rig, this only tries once.  But when called from Power Apps, it shall try four times.  Returning 408 is not a fix.  Returning 418 ensures I only try once, get a better user experience and now I have 418 logs that I add error details to.

2. 3rd party API should not take 35 seconds, improve it.  

3. I could set my timeout on the specific function to 40 seconds, however if the call starts going to 41 seconds, my canvas app will be locked for over 160 seconds.  

4. Go to all API's and if they are fast set the timeout as short as possible so the app does not get locked out while waiting for the 4 responses.

Summary: Examine the 3rd party API's, get them stable and performant and per the agreed SLA's.  If you only want to try once ensure that time outs on the 3rd arty are set to the SLA or if you intercept the request, you can choose the timeout, by examining the API's you can see the optimum time to avoid timeouts and using the 418 response code, the call only happens once.

Series

App Insights for Power Platform - Part 1 - Series Overview 

App Insights for Power Platform - Part 2 - App Insights and Azure Log Analytics 

App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key)

App Insights for Power Platform - Part 4 - Model App Logging

App Insights for Power Platform - Part 5 - Logging for APIM 

App Insights for Power Platform - Part 6 - Power Automate Logging

App Insights for Power Platform - Part 7 - Monitoring Azure Dashboards 

App Insights for Power Platform - Part 8 - Verify logging is going to the correct Log analytics

App Insights for Power Platform - Part 9 - Power Automate Licencing

App Insights for Power Platform - Part 10 - Custom Connector enable logging

App Insights for Power Platform - Part 11 - Custom Connector Behaviour from Canvas Apps Concern (this post)

App Insights for Power Platform - Part 10 - Custom Connector Logging Thoughts

Overview: One of our developers was asking about a log he was struggling to trace, and it took me awhile and a lot of help from the community to truly understand the issue.  My scenario is shown below:

Scenario: I have a Dataverse change triggering a flow, the flow calls a Custom Connector, that in turn calls an Azure Function (that I control).  So the flow fails, and I have used a pattern in the flow to catch the error and log it into Log Analytics.  All good, then I don't see the event where the action calls the function, my function has logging enabled.  I can see I am getting a 401 unauthorized error.  

Initial Hypothesis: The Power Platform use APIM internally to implement Custom Connectors, and there is no access for clients/tenants to see the internal logging/traffic.  Microsoft have provide the ability to use iLogger on the custom connector to log the traffic.  

We have flows that intermittently get a 401, when the flow is manually rerun, the flow works and I can see the traffic coming into the Function.

The failure rate is extremely low and retries almost always fix the issue, and a third try always ensures the transaction goes thru.

Resolution:  Add logging to the custom connector so we can speak with MS support about the issue.  Add alerting to notify support, they can contact the user or chose to rerun the flow. 

Alternative: If I  enable the code, I can override the behaviour and inject C# code to work with the backend, or handle logic such as replacing text,... 

1. In step "4. Code" tab of the custom connector, add the code below:

You can do any C# logic, I'm sending the original request thru and if it doesn't return me a 200, I'm logging it as critical. 

2. Update the connector, go to the next step "5. Test" > "Update Connector" (Tip: follow the steps)

3. Run the "Test operation", open the Response and validate the response body is correct, then open the "Code logs" tab.  If it is blank, re-run the "Update Connector" (irritating but true). 

304 return from the API, which is cached and not a problem, but 400, 500 would be an issue, could also look out for 429s.

Full C# Code:

public class Script : ScriptBase

{

    public override async Task<HttpResponseMessage> ExecuteAsync()

    {       

        this.Context.Request.Method = HttpMethod.Get;

        HttpResponseMessage response = await this.Context.SendAsync(this.Context.Request, this.CancellationToken).ConfigureAwait(continueOnCapturedContext: false);

        Context.Logger.LogTrace("Custom Connector ListBooks called "); 

        if (response.StatusCode == HttpStatusCode.OK)

        {    Context.Logger.LogTrace("Success");      }   

        {    Context.Logger.LogCritical("Critical | " + response);     }

        return response;

    }

}

More Info:

https://learn.microsoft.com/en-us/connectors/custom-connectors/write-code (NB)

https://never-stop-learning.de/logging-in-custom-connector-code/ (NB) The 2nd part of this post on the Alternative , is a rehash of this amazing post - I amended the logic and now I'm wondering is I could write to App Insights using the SDK?

Series

App Insights for Power Platform - Part 1 - Series Overview 

App Insights for Power Platform - Part 2 - App Insights and Azure Log Analytics 

App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key)

App Insights for Power Platform - Part 4 - Model App Logging

App Insights for Power Platform - Part 5 - Logging for APIM 

App Insights for Power Platform - Part 6 - Power Automate Logging

App Insights for Power Platform - Part 7 - Monitoring Azure Dashboards 

App Insights for Power Platform - Part 8 - Verify logging is going to the correct Log analytics

App Insights for Power Platform - Part 9 - Power Automate Licencing

App Insights for Power Platform - Part 10 - Custom Connector enable logging (this post)

App Insights for Power Platform - Part 11 - Custom Connector Behaviour from Canvas Apps Concern

Common Scrum Team improvements

 WIP

Problem: Bugs must not change scope - common to see a bug and it morphs into improvements, other changes.  

Fix: Raise new bug. Don't keep amending and adding new features.

Problem: Team testers/automation testers do not understand the requirement.  Cause by the User Story (US) or bug being ambiguous and Acceptance Criteria (AC)

Fix: Ensure User Stories are in a standard format i.e. "As a <>, I want to <>, so that <>.".  Ensure Gherkin is used for AC (Scenario, Given, When, Then, and preferably more than 1 AC per user story.  Could have: Annotate diagrams to clearly convey artefact information.  Also a narrated recording for bugs.  This could be the person recording the bug, the person that fixed the bug as proof to increase understanding. 

App Insights for Power Platform - Part 9 - Power Automate Licencing

Overview:  Licencing is extremely complicated, but there are threshold limits that are being reduced at the moment, August 2023.  

O365 users get get the lowest priority profile, can only run the standard connectors, and have a "request" limit of 6,000 requests per day.

What is a Request?

Each flow consists of a combination of triggers, actions, and responses when cloud flow is run, the instance walks thru the actions such as Create a SharePoint list item, setting variables, 

What counts as a Power Platform Request

"Here are some guidelines to estimate the request usage of a flow.

• One or more actions run as part of a flow run. A simple flow with one trigger and one action results in two "actions" each time the flow runs, consuming 2 requests.

• Power Automate Flows, by default, run in the context of the Flow Owner.  The "actions" are worked out against the Flow Owner.

• Every trigger/action in the flow generates Power Platform requests. All kinds of actions like connector actions, HTTP actions, built-in actions (from initializing variables, creating scopes to a simple compose action) generate Power Platform requests. For example, a flow that connects SharePoint, Exchange, Twitter, and Dataverse, all those actions are counted towards Power Platform request limits.

• Both succeeded and failed actions count towards these limits. Skipped actions aren't counted towards these limits.

• Each action generates one request. If the action is in an apply to each loop, it generates more Power Platform requests as the loop executes.

• An action can have multiple expressions but it's counted as one API request.

• Retries and additional requests from pagination count as action executions as well."

Here are my thoughts which seem to differ from the MS notes provide above: Not all Actions count as a request, If i look at the Power Automate Analytics it gives me a break down on the API calls to understand the "Request" counting.  Basically any action that does an API call when run adds to the request count.  

Guide for planning for limitations:

• O365 users get 6k request per days
• Dynamics and most per user plans get 40k requests per day.
• As a rough guide, I count simple workflows as 3 requests average, medium as 7 requests, large can be over 100 so it is better to build the workflow and from the analytics you can get the number of requests per day.
• For each flow multiply by the estimated number of calls
• Understand who the quest is attributed to (either the user or the owner of the flow, the requests are counted against the flow owner unless the flow use a pay per flow model.)

Example: to calculate billable actions/billable requests

I have a single Flow running against my O365, the flows has a Power Apps trigger, then creates a new list item and lastly responds to Power Apps.

1 Cloud flows that has 3 billable actions run 5 times will result in 15 billable actions.

I have 6k per 24 hrs on an O365 licence, most of the other licences such as Power Automate premium, an account has 40k requests per 24 hours.

I could run the flow 1,200 times in 24 hrs under an O365 licence.

Series

App Insights for Power Platform - Part 1 - Series Overview 

App Insights for Power Platform - Part 2 - App Insights and Azure Log Analytics 

App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key)

App Insights for Power Platform - Part 4 - Model App Logging

App Insights for Power Platform - Part 5 - Logging for APIM 

App Insights for Power Platform - Part 6 - Power Automate Logging

App Insights for Power Platform - Part 7 - Monitoring Azure Dashboards 

App Insights for Power Platform - Part 8 - Verify logging is going to the correct Log analytics

App Insights for Power Platform - Part 9 - Power Automate Licencing (this post)

App Insights for Power Platform - Part 10 - Custom Connector enable logging

App Insights for Power Platform - Part 11 - Custom Connector Behaviour from Canvas Apps Concern

App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key)

App Insights for Power Platform - Part 1 - Series Overview 

App Insights for Power Platform - Part 2 - App Insights and Azure Log Analytics 

App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key) (this post)

App Insights for Power Platform - Part 4 - Model App Logging

App Insights for Power Platform - Part 5 - Logging for APIM 

App Insights for Power Platform - Part 6 - Power Automate Logging

App Insights for Power Platform - Part 7 - Monitoring Azure Dashboards 

App Insights for Power Platform - Part 8 - Verify logging is going to the correct Log analytics

App Insights for Power Platform - Part 9 - Power Automate Licencing

App Insights for Power Platform - Part 10 - Custom Connector enable logging

App Insights for Power Platform - Part 11 - Custom Connector Behaviour from Canvas Apps Concern

Overview: Logging & monitoring for Canvas apps is done in two parts: App Insights, and using the Canvas app Monitor.  This post focuses on logging via App Insights.

Note: Once a solution that contains a Instrumentation key, they app logging key cannot be alter unless you make the environment have unmanaged layers.  You can use PowerCli and compose a new managed solution for each DTAP environment but it's a new compile for each environment.

Example:

In the annotated diagram below including a log snippet.  

1. Canvas App has an instrumentation key, the log captures the front end action

2. Calls to Dataverse & Power automate Flows are logged (relies on step 1)

3. Custom connector is calling an Azure Function (Function is logging to Log Analytics or app Insights),

4. the function logs into APIM and sends APIM a request (APIM logging is setup on the end points), and

5. APIM sends an outbound API request and captures the response (relies on step 4)

Note in this example I have Correlation tracking enabled on the Canvas App to get the full timeline, as shown below, it has been an experimental feature for a few years now.

When I turn off the Correlation, it is not as easy to trace items from start to finish.  All I get by default is the steps 3&4 data in my transaction search timeline.

All 5 pieces are still captured but the timeline has to be pieced together for tracing.

I would also enable the preview feature for logging as well as the experimental if the clients governance allows experimental features to be turned on.

Summary: Always add as many logging features as possible in Canvas Apps, think about where your logs go and also setup logging on Azure services to transaction can be traced.

App Insights for Power Platform - Part 1 - Series Overview 

App Insights for Power Platform - Part 2 - App Insights and Azure Log Analytics 

App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key) (this post)

App Insights for Power Platform - Part 4 - Model App Logging

App Insights for Power Platform - Part 5 - Logging for APIM 

App Insights for Power Platform - Part 6 - Power Automate Logging

App Insights for Power Platform - Part 7 - Monitoring Azure Dashboards 

App Insights for Power Platform - Part 8 - Verify logging is going to the correct Log analytics

App Insights for Power Platform - Part 9 - Power Automate Licencing

App Insights for Power Platform - Part 10 - Custom Connector enable logging (this post)

App Insights for Power Platform - Part 11 - Custom Connector Behaviour from Canvas Apps Concern

Microsoft Purview DLP Policies

Overview: Microsoft Purview allows a company to create end device DLP policies and push out to all client devices.

Create policies, and takes up to 2 hours to update end client devices and show on the "Device Monitoring" dashboard.  Client Analyser tool - check endpoint client devices are getting the latest policies, and can speed it up.  Device obviously has to be online to get the latest update.

WIP

Visual Studio App Center

Visual Studio App Center is an Azure/Microsoft Service for packaging and deploying mobile apps. "Visual Studio App Center lets you automate and manage the lifecycle of your iOS, Android, Windows, and macOS apps. Ship apps more frequently, at higher-quality, and with greater confidence." Microsoft Learning

Useful for Canvas apps that will be deployed via the stores or InTune to mobile user devices.

To login to the App Center go to https://appcentee.ms/apps (I logged in with an Azure Tenant admin account)

First create an Organisation, in the organisation you can create your mobile Apps for the Organisation.

Note: Max 2 new orgs per day.  I delete all my Orgs and was messing around to find i could not add anymore organisations.

Problem: I cannot create new organisation in the App Center and i get the error "You can not create more organizations at this time. Please try again later."

Hypothesis: I thought I had broken my tenant and tried repairing it with the AppCentre Cli, no joy.  Went to ms support and they immediately told me that a max of 2 new orgsanizations can be created per day.

"The limit for creating organizations is indeed 2 organization's per day, to prevent abuse by malicious actors. Now you have to wait 24 hours to create other 2 more organizations."

The error message could provide a clearer expalination.

Resolution: I'll wait until tonight as my 24 hour 2 org limit should be available in a few hours.