Playwright Post 6 - Automating Canvas App MFA login for Playwright unattended for Canvas apps

Overview:  Modern security makes automating logins requiring MFA rather difficult.  This post looks at possible approaches to automate the login.

Option 1. Turn off MFA—not really, but you can set a conditional rule in EntraId to not perform MFA. This is not an option in many enterprises.

Option 2. Time-based One-Time Password (TOTP)—Microsoft Authenticator makes this pretty difficult. At least I can't do it, as the APIS are relatively limited. This is kind of expected, as it's a security measure.

Option 3. Programmatically acquire an access token without browser automation, use MSAL with a client secret or certificate (for confidential clients). 

Option 4.  Use Playwright to record the login and intercept the access token once logged in.  Then you can store it and use it.  There are a few easy options to get the session:

4.1. Retrieve the access token from the response once logged in

4.2. Retrieve from your local storage:

  const token = await page.evaluate(() => {

    return window.localStorage.getItem('adal.idtoken') || window.sessionStorage.getItem('adal.idtoken');

  });

4.3. Retrieve the token using Playwrite at the command run level

Note: This adds the token to my repository. Don't save the token to your repository if you don't realise that the Access/Bearer token will expire depending on what your EntraId sets. The default is 1 hour.

Option 4.3.1. Like option 4.3, use the refresh token to silently generate a new Access token. You store the refresh token during the recorded login (by default, it lasts for 90 days) to generate a new access token when you need it.

Option 4.3.2.  Take it further back to generate the refresh token using the access code you get at the original login, renew the "refresh token", and generate a new access token to run your tests.

If you decide to store your access token, refresh token or code, don't store them in your code repo.  You know why if you got this far.

Thought: as a refresh token works for 90 days on a sliding scale, I've never used the option 4.3.2, as by storing the refresh token, all I need to do is to extend the refresh token by using it to get an access token and the refresh token has 90 days from that point. 

This is the plan I'm thinking of using:

1. Store the refresh token in Azure Key Vault (AKV).
2. Your app retrieves the refresh token from AKV.
3. Uses the refresh token to get a new access token.
4. Store the new access token to run my ALM/CI/CD playwright tests

Playwright Series

Playwright Post 1 - Overview of E2E testing using Playwright

Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled

Playwright Post 3 - Add App Insights logging inside your Playwright tests 

Playwright Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor

Playwright Post 5 - Understanding how Playwright works

Playwright Post 6 - Unattended testing when secured with MFA (this post)

Playwright series - Post 2 - Refactored TS code for Consciously verify Apps in Production

Overview: Create a function with Playwright tests that loops through all my production apps, logs in, and validates the Page title load on each app's home page.

Steps:

1. Create the spec.ts code that reads app.json to loop thru and validate sites

2. Record and Store the session state for all future test runs (Used for MFA in the tests runs)

3. Create an apps.json file containing URLs to open and validate

4. After running the test, you'll see that the 3 tests were completed successfully. In my case, there were 2 Power Apps with MFA enabled and an anonymous public website that had been checked.

Optional: 

Create short cuts to run your tests using PowerShell

PS C:\repos\PW> npx playwright test -g "Prod-CanvasApps" --project=chromium --headed

Next Steps:

Run continuously using the Azure Playwright Service.

=========================

Playwright Series

Playwright Post 1 - Overview of E2E testing using Playwright

Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled (this post)

Playwright Post 3 - Add App Insights logging inside your Playwright tests 

Playwright Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor

Playwright Post 5 - Understanding how playwright works

Playwright Post 6 - Unattended testing when secured with MFA 

Other Posts

Upgrading two C# Blazor web applications and verifying using Playwright - super fast 

Mendix - Part 2 - Diving deeper (E2E automation testing of Mendix using Playwright)

Low-code testing with playwright walkthru

Continiously Monitor Apps using Playwright with TS

Testing Canvasa apps with Playwright using C# (rather use TS, it's better)

Low Code testing with Playwright - 1. Intro Exercise (15 min)

Exercise 1.  Install and setup you first Playwright Test

1. Verify VS Code is installed 

2. Ensure the Microsoft Playwright Extension is installed (use the default language TypeScript)

 

3. Create a folder using Windows Explorer as shown here: C:\Users\paulb\source\repos\Playwright\Mendix

4. Open the folder in VS Code and ensure the project is initialised C:\Users\paulb\source\repos\Playwright\Mendix> npm init playwright@latest --yes -- --quiet --browser=chromium --browser=webkit

5. Verify your screen looks similar to this...

Exercise 2.  Record and run your first Playwright test

1. On the "Testing" area, select "Record new"

2. The recorder opens, type in a url in my case I used "https://www.paulbeck.co.uk"

3. Click and assert text exists on the page

4. Stop the recorder and close the browser.

5. Run the test, as shown below, validate the result

Playwright Series

Playwright Post 1 - Overview of E2E testing using Playwright 

Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled 

Playwright Post 3 - Add App Insights logging inside your Playwright tests 

Playwright Series - Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor (this post)

Playwright Post 5 - Understanding how playwright works

Mendix - Part 2 - Diving deeper (E2E automation testing of Mendix using Playwright)

Mendix Series

1.  Overview of Mendix 

2. Mendix - Part 2 - Diving deeper (this post)

3. Extensibility for Mendix Studio 

4. Building a Mendix Widget for the Mendix Market Place (Convert text to Audio using Azure AI)

5. Mendix Tips & Thoughts

6. Competitors

AI with Mendix (current version Mendix 10.5.x):

1. Logic bot - recommends what you are likely to do, like a copilot as you go along building the app
2. Performance bot - shows redundancies, recommends performance improvements 
3. Chatbot in beta

Playwright is a good UI testing tool for Mendix:

For more advanced applications, Playwright is a good testing framework that can help developers know their code is running end-to-end, useful for monitoring applications and behaviour, and also can be used as part of the CI process to validate Mendix end user accessibility as shown in this mp4 (7 minutes - good video).

mp4 walking thru testing a Mendix site, including WCAG accessibility

WCAG accessibility testing in Playwright starts at min 5:25. Build in WCAG testing to your playwright tests> npm install @axe-core/playwright

Thoughts:

• Deployment can take time when the project grows.
• Domain modelling: can use existing models and add to Mendix.
• Playwright is perfect for Mendix testing, here is a basic dev example.
• The Mendix Community is a good place to post questions.

Mendix Tip: I needed to change from US format to UK date time format:

The community has the answer: Mendix Forum - Question Details

Mendix Series

1.  Overview of Mendix 

2. Mendix - Part 2 - Diving deeper (this post)

3. Extensibility for Mendix Studio 

4. Building a Mendix Widget for the Mendix marketplace (Convert text to Audio using Azure AI)

5. Mendix Tips & Thoughts

6. Competitors

Playwright Series

1. Playwright Post 1 - Overview of E2E testing using Playwright 

2. Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled 

3. Playwright Post 3 - Add App Insights logging inside your Playwright tests 

4. Playwright Series - Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor

5. Playwright Post 5 - Understanding how playwright works

Playwright series - Post 1 - Overview of E2E testing using VS Code for Low Code

Setup: I have installed Node 20.100.0 and the VS code extension for Playwright.  The installation and getting started guides are straightforward and of a high quality.  https://playwright.dev/docs/intro  I am running on Windows 10 Surface 4 with 16GB.  I use TypeScript (ts) as the default, and the recording mechanism works well with ts.  Previously, I used C# as my language of choice, but it is easier to maintain, and there is no need for complex logic/functions in end-to-end (e2e) UI testing.  New features always come out in TS/JS first.

Thoughts:  Postman is easy to use, fast, configurable and flexible.  UI e2e testing allows me to know my app/sites are working as expected.  Manual testing is time consuming, and amending automated tests can be hard.

Setup Reminder:

1. Install the Playwright extension using VSCode (once at initial setup)

2. Open a new folder in VSCode, and open the "Command Pallette" (once for each new project)

>Install Playwright

These are the defaults and will use TypeScript as the base language, stick to this it is the simpliest.  VSCode builds the default file scaffolding as shown above

3. Create your first New Playwright UI Test:

3.1. Record new

3.2. Enter a URL in the recorder browser, and click around (optional add Asserts) 

3.3. Save the Test

3.4. Execute the test

The Green tick can be used to quickly run the test locally.  In the "Test Results" terminal, you can see the same test was run 3 times, my configuration is set to test Chrome, Firefox and webkit.

Why Playwright?

• Easy to understand/follow,
• Easy to record,
• Open source, 
• No paid licencing, 
• Faster than Selenium,
• Various coding languages supported (bindings for C#, Python, Java, JS, TypeScript),
• UI verification using screenshots and AI to minimize flakiness/static DOM reliance,
• The ability to debug and trace is strong,
• Can do API testing,

Possible Playwright UI testing Layers: 

1. Full regressions go into detail and run in Chrome, Firefox, WebKit and specified devices 
2. Check-in tests are comprehensive on a single browser for code check-ins
3. Continuous testing: Record logging in, reading from a database, and calling an API. You can write to logs, e.g., Dynatrace, Azure Monitor, and SolarWinds using APIs. Doing this every 5 minutes will tell you at a high level if the service and its dependencies are running and if there is a performance change.
4. A developer can write detailed local tests when working in an area and reuse them if he comes back and changes any code.

Testing Challenges:

Unit testing is a challenge in low code - while they are fast and ideal for C# or code, not easy to implement for Low code.  Their is a new beta feature for component testing in Playwright, i don't think it adds value.  API Testing - I use Postman for API testing including controlling my CI.  Low code automation testing is hard in the Power Platform, E2E playwright testing in context works pretty well.  API's/ network traffic needs to be mocked.

Challenger products:

• Selenium - QA's highly skilled here
• Cypress - Devs tended to use this over Selenium
• Specific products like Power Platform Test Studio and ...
• I also like BrowserStacks low code testing, especially if no CI/CD and can manage from here and use different low code technology.  

Summary: Generally, I'd go for Playwright over all the others. 

=========================

Playwright Series

Playwright Post 1 - Overview of E2E testing using Playwright (this post)

Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled 

Playwright Post 3 - Add App Insights logging inside your Playwright tests 

Playwright Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor

Playwright Post 5 - Understanding how Playwright works

Playwright Post 6 - Unattended testing when secured with MFA 

Other Posts

Upgrading two C# Blazor web applications and verifying using Playwright - super fast 

Mendix - Part 2 - Diving deeper (E2E automation testing of Mendix using Playwright)

Low-code testing with playwright walkthru

Continiously Monitor Apps using Playwright with TS

Testing Canvasa apps with Playwright using C# (rather use TS, it's better)

Review of Browser Stacks Low Code Automated test tool

Overview: Low code testing relies heavily on complete UI end-to-end testing.  It needs to be fast, flexible, quick to correct, scale-able, highly configurable and BrowserStack's low code test tool is in beta and definitely on the right path - for me, it needs a few features.  I ran my testing against customised apps created on three platforms:

1. Mendix low code,
2. Microsoft's Blazer hosted on Azure Web App, and
3. Canvas app within Power Platform. 

Tip: I've looked and use BrowserStack for many years and it has moved from being an device emulators infrastructure testing provider to a full ALM testing platform.  The low code Browser stack has a recorder to capture steps.

Where does Low Code fit into Browser Stack:

Image1. Low code automation works well as part of the full BrowserStack Platform or just using the product by itself.

Pros of the Low Code BrowserStack Product:

• The local recording feature is easy to set up and use
• Seamless integration with the cloud version running on BrowserStack's infrastructure
• Logical layout of UI, little to no training required
• UI validation using the DOM or, more importantly, screenshots using BrowserStack's AI verification (required further review) has the potential to self-heal as in the screen changes, but the validation can be smart enough to understand it is just an updated screen (example a single colour in the page and the position of the name is moved).
• SDK is available to work with the full BrowserStack platform.
• Not Low code specific but BrowserStack generally has the new phones included in their offering within days of being release.

Cons:

• Provide a webhook or allow for a REST client call as a step (I'd want to log directly from the test run into Azure Monitor)
• More run options, I'm sure it's already on the road map, but the ability to run every hour for continuous monitoring.
• Refresh tokens on a schedule (allows you to not use MFA such as SMS codes or Authenticator).
• Make it clear if the run is from the local or the browser, and keep the historical logs for both together.
• Export results - I could not find this, but it would help compare step performance.
• I use DevOps, I'm unlikely to take the whole BrowserStack platform unless i need the emulators which is what BrowserStack was originally famous for.

Summary: This is an excellent tool for testing; the low code product was still in beta when I reviewed it.  It is a nice tool and has the potential to disrupt the market.  I feel Playwright is a better point solution and integrates to CI/ALM platforms.

References: https://www.browserstack.com/low-code-automation/features?utm_source=google&utm_medium=cpc&utm_platform=paidads&utm_content=668760067900&utm_campaign=Search-Brand-EMEA-Navigational&utm_campaigncode=Core+9045914&utm_term=e+browserstack

Other: 

Image 2. Emulate a Samsung Galaxy phone on Android using the Chrome browser.

Thought: I like BrowserStack's reporting, clean and simple on tests and easy to get the history.

Testing Power Apps using Microsoft Playwright with C#

Overview: I am using a C# Console app to run my power Apps to verify that it is working correctly.

Approach:  I'm not using NodeJs but I'll use C# and add it into my devOps pipelines.  I assume you have VS Code installed (you need the C# extensions, I have kept my notes at the bottom of the post).  I'm using .NET 6 core SDK and the Microsoft.Playwright package.  I'm not using any of the .NET testing frameworks to keep it simple.

Tips: Playwright by default use headless browser testing, I want the test to use my local JWT that I have stored against my Chrome browser.  If I tell Playwright to use show the browser i.e., headless=false, it logs me in using my cached credentials.  You should parameterise the login and store the username/pswd somewhere secure like a key vault. 

Code Example: 

Replace the C# code in the console as I have done above, and hit F5 (Run>Debug).  The test logs me into the Canvas app and returns the Title of the page. 

Note:  I like the Canvas app testing feature but this is better as I can easily add testing to my CI/CD pipelines, tooling and recording is better and there are more ways I can use the Playwright approach.  

Notes for installing Playwright on VSCode to use C#

1. Install VS Code and add the following two extensions

2. Open the VS Code terminal, create a C# Console app using .NET6, add the reference to the Playwright package, install the headless browsers against you profile (only needed once per machine). 

3. Write and run the tests.  Tip: It's easier to record the code and then add it to the C# console.

Note: It's worthwhile using NUnit as it's much easier to test than doing it the way I've done it here.

Playwright using the NUnit framework:

Playwright Series

Playwright Post 1 - Overview of E2E testing using Playwright 

Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled 

Playwright Post 3 - Add App Insights logging inside your Playwright tests 

Playwright Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor 

Playwright Post 5 - Understanding how playwright works

Other Posts

Upgrading two C# Blazor web applications and verifying using Playwright - super fast 

Mendix - Part 2 - Diving deeper (E2E automation testing of Mendix using Playwright)

Low-code testing with playwright walkthru

Continiously Monitor Apps using Playwright with TS

Testing Canvasa apps with Playwright using C# (rather use TS, it's better)