Radimaging Ltd - Paul Beck's Technical Working Notes for Microsoft Technology: App Insights for Power Platform - Part 12

Overview: My system notified me that my production errors were going crazy. Quickly I knew all the tenants at my largest customer were down.

Problem: Apps would load and stay in a loading state. I could see the Canvas apps but that is where is would continue try to load.

Initial Hypothesis: Originally, I thought is was 1 environment and I know it was loading a SharePoint list so i thought it may be permissions, but it was on all my environments and my continuous test was picking them all up.

I checked the Microsoft Services and all the services are working: https://azure.status.microsoft/en-gb/status

I went to QA and Dev and they were also failing with the same issue. I ran the Canvas app in debug mode and could see the error was relating to Connectors to the European APIM for Dataverse.

Next I went to other environments on my client in other regions, they too were also failing. I was a bit surprised as I wasn't getting any feedback from other clients, or feeds, so i logged onto my own company Power Platform tenant and in environments, they were working. So this was only to this specific client. And now I knew the extent, and i could not run flows only on the client environments.

Here are the CI test report and results for a subset of the apps on the business units production environment.

Ran a test to check a single Production department Environment with 24 Canvas apps:

Simple test: runs on 3 worker processes on a single browser engine chromium.

Reports logs show all 8 of these sites are not working

All 8 tests were not finding the Title of the page. I log from Playwright and from Canvas apps using App Insight traces, the error was supper easy to pickup even without the Power Platform trace in the Dev environment.

Resolution: Raise a ticket, tell MS that we had the issue and provide the info (not raised by me, but by a support engineer). 30 minutes later, the company has been advised to close the browsers and try again. Did this manually and issue resolved.

I still had the old token for SPO in Playwright for Chromium, so I ran the test on for all 3 browser engines. Chromium fails with the old token, Firefox and webkit pass as they grabbed new login tokens.