Playwright series - Post 2 - Refactored TS code for Consciously verify Apps in Production

Overview: Create a function with Playwright tests that loops through all my production apps, logs in, and validates the Page title load on each app's home page.

Steps:

1. Create the spec.ts code that reads app.json to loop thru and validate sites

2. Record and Store the session state for all future test runs (Used for MFA in the tests runs)

3. Create an apps.json file containing URLs to open and validate

4. After running the test, you'll see that the 3 tests were completed successfully. In my case, there were 2 Power Apps with MFA enabled and an anonymous public website that had been checked.

Optional: 

Create short cuts to run your tests using PowerShell

PS C:\repos\PW> npx playwright test -g "Prod-CanvasApps" --project=chromium --headed

Next Steps:

Run continuously using the Azure Playwright Service.

=========================

Playwright Series

Playwright Post 1 - Overview of E2E testing using Playwright

Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled (this post)

Playwright Post 3 - Add App Insights logging inside your Playwright tests 

Playwright Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor

Low Code testing with Playwright - 1. Intro Exercise (15 min)

Exercise 1.  Install and setup you first Playwright Test

1. Verify VS Code is installed 

2. Ensure the Microsoft Playwright Extension is installed (use the default language TypeScript)

 

3. Create a folder using Windows Explorer as shown here: C:\Users\paulb\source\repos\Playwright\Mendix

4. Open the folder in VS Code and ensure the project is initialised C:\Users\paulb\source\repos\Playwright\Mendix> npm init playwright@latest --yes -- --quiet --browser=chromium --browser=webkit

5. Verify your screen looks similar to this...

Exercise 2.  Record and run your first Playwright test

1. On the "Testing" area, select "Record new"

2. The recorder opens, type in a url in my case I used "https://www.paulbeck.co.uk"

3. Click and assert text exists on the page

4. Stop the recorder and close the browser.

5. Run the test, as shown below, validate the result

Mendix - Part 2 - Diving deeper (E2E automation testing of Mendix using Playwright)

Mendix Series

1.  Overview of Mendix 

2. Mendix - Part 2 - Diving deeper (this post)

3. Extensibility for Mendix Studio 

4. Building a Mendix Widget for the Mendix Market Place (Convert text to Audio using Azure AI)

5. Mendix Tips & Thoughts

6. Competitors

AI with Mendix (current version Mendix 10.5.x):

1. Logic bot - recommends what you are likely to do, like a copilot as you go along building the app
2. Performance bot - shows redundancies, recommends performance improvements 
3. Chatbot in beta

Playwright is a good UI testing tool for Mendix:

For more advanced applications, Playwright is a good testing framework that can help developers know their code is running end-to-end, useful for monitoring applications and behaviour, and also can be used as part of the CI process to validate Mendix end user accessibility as shown in this mp4 (7 minutes - good video).

Thoughts:

• Deployment can take time when the project grows.
• Domain modelling: can use existing models and add to Mendix.
• Playwright is perfect for Mendix testing, here is a basic dev example.
• The Mendix Community is a good place to post questions.

I needed to change from US format to UK date time format:

The community has the answer: Mendix Forum - Question Details

Mendix Series

1.  Overview of Mendix 

2. Mendix - Part 2 - Diving deeper (this post)

3. Extensibility for Mendix Studio 

4. Building a Mendix Widget for the Mendix marketplace (Convert text to Audio using Azure AI)

5. Mendix Tips & Thoughts

6. Competitors

Playwright series - Post 1 - Overview of E2E testing using VS Code for Low Code

Setup: I have installed Node 20.100.0 and the VS code extension for Playwright.  The installation and getting started guides are straightforward and of a high quality.  https://playwright.dev/docs/intro  I am running on Windows 10 Surface 4 with 16GB.  I use TypeScript (ts) as the default, and the recording mechanism works well with ts.  Previously, I used C# as my language of choice, but it is easier to maintain, and there is no need for complex logic/functions in end-to-end (e2e) UI testing.  New features always come out in TS/JS first.

Thoughts:  Postman is easy to use, fast, configurable and flexible.  UI e2e testing allows me to know my app/sites are working as expected.  Manual testing is time consuming, and amending automated tests can be hard.

Setup Reminder:

1. Install the Playwright extension using VSCode (once at initial setup)

2. Open a new folder in VSCode, and open the "Command Pallette" (once for each new project)

>Install Playwright

These are the defaults and will use TypeScript as the base language, stick to this it is the simpliest.  VSCode builds the default file scaffolding as shown above

3. Create your first New Playwright UI Test:

3.1. Record new

3.2. Enter a URL in the recorder browser, and click around (optional add Asserts) 

3.3. Save the Test

3.4. Execute the test

The Green tick can be used to quickly run the test locally.  In the "Test Results" terminal, you can see the same test was run 3 times, my configuration is set to test Chrome, Firefox and webkit.

Why Playwright?

• Easy to understand/follow,
• Easy to record,
• Open source, 
• No paid licencing, 
• Faster than Selenium,
• Various coding languages supported (bindings for C#, Python, Java, JS, TypeScript),
• UI verification using screenshots and AI to minimize flakiness/static DOM reliance,
• The ability to debug and trace is strong,
• Can do API testing,

Possible Playwright UI testing Layers: 

1. Full regressions go into detail and run in Chrome, Firefox, WebKit and specified devices 
2. Check-in tests are comprehensive on a single browser for code check-ins
3. Continuous testing: Record logging in, reading from a database, and calling an API. You can write to logs, e.g., Dynatrace, Azure Monitor, and SolarWinds using APIs. Doing this every 5 minutes will tell you at a high level if the service and its dependencies are running and if there is a performance change.
4. A developer can write detailed local tests when working in an area and reuse them if he comes back and changes any code.

Testing Challenges:

Unit testing is a challenge in low code - while they are fast and ideal for C# or code, not easy to implement for Low code.  Their is a new beta feature for component testing in Playwright, i don't think it adds value.  API Testing - I use Postman for API testing including controlling my CI.  Low code automation testing is hard in the Power Platform, E2E playwright testing in context works pretty well.  API's/ network traffic needs to be mocked.

Challenger products:

• Selenium - QA's highly skilled here
• Cypress - Devs tended to use this over Selenium
• Specific products like Power Platform Test Studio and ...
• I also like BrowserStacks low code testing, especially if no CI/CD and can manage from here and use different low code technology.  

Summary: Generally, I'd go for Playwright over all the others. 

=========================

Playwright Series

Playwright Post 1 - Overview of E2E testing using Playwright (this post)

Playwright Post 2 - Continuously Test/Monitor Canvas apps and website with MFA enabled 

Playwright Post 3 - Add App Insights logging inside your Playwright tests 

Playwright Post 4 - 6 Min walkthru of Playwright testing with Azure Monitor

Review of Browser Stacks Low Code Automated test tool

Overview: Low code testing relies heavily on complete UI end-to-end testing.  It needs to be fast, flexible, quick to correct, scale-able, highly configurable and BrowserStack's low code test tool is in beta and definitely on the right path - for me, it needs a few features.  I ran my testing against customised apps created on three platforms:

1. Mendix low code,
2. Microsoft's Blazer hosted on Azure Web App, and
3. Canvas app within Power Platform. 

Tip: I've looked and use BrowserStack for many years and it has moved from being an device emulators infrastructure testing provider to a full ALM testing platform.  The low code Browser stack has a recorder to capture steps.

Where does Low Code fit into Browser Stack:

Image1. Low code automation works well as part of the full BrowserStack Platform or just using the product by itself.

Pros of the Low Code BrowserStack Product:

• The local recording feature is easy to set up and use
• Seamless integration with the cloud version running on BrowserStack's infrastructure
• Logical layout of UI, little to no training required
• UI validation using the DOM or, more importantly, screenshots using BrowserStack's AI verification (required further review) has the potential to self-heal as in the screen changes, but the validation can be smart enough to understand it is just an updated screen (example a single colour in the page and the position of the name is moved).
• SDK is available to work with the full BrowserStack platform.
• Not Low code specific but BrowserStack generally has the new phones included in their offering within days of being release.

Cons:

• Provide a webhook or allow for a REST client call as a step (I'd want to log directly from the test run into Azure Monitor)
• More run options, I'm sure it's already on the road map, but the ability to run every hour for continuous monitoring.
• Refresh tokens on a schedule (allows you to not use MFA such as SMS codes or Authenticator).
• Make it clear if the run is from the local or the browser, and keep the historical logs for both together.
• Export results - I could not find this, but it would help compare step performance.
• I use DevOps, I'm unlikely to take the whole BrowserStack platform unless i need the emulators which is what BrowserStack was originally famous for.

Summary: This is an excellent tool for testing; the low code product was still in beta when I reviewed it.  It is a nice tool and has the potential to disrupt the market.  I feel Playwright is a better point solution and integrates to CI/ALM platforms.

References: https://www.browserstack.com/low-code-automation/features?utm_source=google&utm_medium=cpc&utm_platform=paidads&utm_content=668760067900&utm_campaign=Search-Brand-EMEA-Navigational&utm_campaigncode=Core+9045914&utm_term=e+browserstack

Other: 

Image 2. Emulate a Samsung Galaxy phone on Android using the Chrome browser.

Thought: I like BrowserStack's reporting, clean and simple on tests and easy to get the history.

Testing Power Apps using Microsoft Playwright with C#

Overview: I am using a C# Console app to run my power Apps to verify that it is working correctly.

Approach:  I'm not using NodeJs but I'll use C# and add it into my devOps pipelines.  I assume you have VS Code installed (you need the C# extensions, I have kept my notes at the bottom of the post).  I'm using .NET 6 core SDK and the Microsoft.Playwright package.  I'm not using any of the .NET testing frameworks to keep it simple.

Tips: Playwright by default use headless browser testing, I want the test to use my local JWT that I have stored against my Chrome browser.  If I tell Playwright to use show the browser i.e., headless=false, it logs me in using my cached credentials.  You should parameterise the login and store the username/pswd somewhere secure like a key vault. 

Code Example: 

Replace the C# code in the console as I have done above, and hit F5 (Run>Debug).  The test logs me into the Canvas app and returns the Title of the page. 

Note:  I like the Canvas app testing feature but this is better as I can easily add testing to my CI/CD pipelines, tooling and recording is better and there are more ways I can use the Playwright approach.  

Notes for installing Playwright on VSCode to use C#

1. Install VS Code and add the following two extensions

2. Open the VS Code terminal, create a C# Console app using .NET6, add the reference to the Playwright package, install the headless browsers against you profile (only needed once per machine). 

3. Write and run the tests.  Tip: It's easier to record the code and then add it to the C# console.

Note: It's worthwhile using NUnit as it's much easier to test than doing it the way I've done it here.

Playwright using the NUnit framework: