Overview: My system notified me that my production errors were going crazy. Quickly I knew all the tenants at my largest customer were down.
Problem: Apps would load and stay in a loading state. I could see the Canvas apps but that is where is would continue try to load.
Initial Hypothesis: Originally, I thought is was 1 environment and I know it was loading a SharePoint list so i thought it may be permissions, but it was on all my environments and my continuous test was picking them all up.
I checked the Microsoft Services and all the services are working: https://azure.status.microsoft/en-gb/status
I went to QA and Dev and they were also failing with the same issue. I ran the Canvas app in debug mode and could see the error was relating to Connectors to the European APIM for Dataverse.
Next I went to other environments on my client in other regions, they too were also failing. I was a bit surprised as I wasn't getting any feedback from other clients, or feeds, so i logged onto my own company Power Platform tenant and in environments, they were working. So this was only to this specific client. And now I knew the extent, and i could not run flows only on the client environments.
Here are the CI test report and results for a subset of the apps on the business units production environment.
Ran a test to check a single Production department Environment with 24 Canvas apps:
![]() | ||||
Simple test: runs on 3 worker processes on a single browser engine chromium.
All 8 tests were not finding the Title of the page. I log from Playwright and from Canvas apps using App Insight traces, the error was supper easy to pickup even without the Power Platform trace in the Dev environment. Resolution: Raise a ticket, tell MS that we had the issue and provide the info (not raised by me, but by a support engineer). 30 minutes later, the company has been advised to close the browsers and try again. Did this manually and issue resolved. I still had the old token for SPO in Playwright for Chromium, so I ran the test on for all 3 browser engines. Chromium fails with the old token, Firefox and webkit pass as they grabbed new login tokens.
If I find out what cause the issue, I'll post what MS did and found out. Useful Dashboards I used: Portal.azure.com OOTB App Insights Url to retrieve an OperationIDs' history https://portal.azure.com/#blade/HubsExtension/BladeRedirect/bladeName/Microsoft_Azure_LogicAppsRunBlade/ runId/<OPERATION_ID>/ logicAppName/<LOGIC_APP_NAME>/ resourceGroupId/%2Fsubscriptions%2F<SUBSCRIPTION_ID> %2FresourceGroups%2F<RESOURCE_GROUP_NAME>%2Fproviders%2FMicrosoft.Logic%2Fworkflows%2F<LOGIC_APP_NAME> Series App Insights for Power Platform - Part 1 - Series Overview App Insights for Power Platform - Part 2 - App Insights and Azure Log Analytics App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key) App Insights for Power Platform - Part 4 - Model App Logging App Insights for Power Platform - Part 5 - Logging for APIM App Insights for Power Platform - Part 6 - Power Automate Logging App Insights for Power Platform - Part 7 - Monitoring Azure Dashboards App Insights for Power Platform - Part 8 - Verify logging is going to the correct Log analytics App Insights for Power Platform - Part 9 - Power Automate Licencing App Insights for Power Platform - Part 10 - Custom Connector enable logging App Insights for Power Platform - Part 11 - Custom Connector Behaviour from Canvas Apps Concern App Insights for Power Platform - Part 12 - A fix story (this post) |



0 comments:
Post a Comment