Mendix - Part 2 - Diving deeper (E2E automation testing of Mendix using Playwright)

Mendix Series

1.  Overview of Mendix 

2. Mendix - Part 2 - Diving deeper (This post)

AI with Mendix (current version Mendix 10.5.x):

1. Logic bot - recommends what you are likely to do, like a copilot as you go along building the app
2. Performance bot - shows redundancies, recommends performance improvements 
3. Chatbot in beta

Playwright is a good UI testing tool for Mendix:

For more advanced applications, Playwright is a good testing framework that can help developers know their code is running end-to-end, useful for monitoring applications and behaviour, and also can be used as part of the CI process to validate Mendix end user accessibility as shown in this mp4 (7 minutes - good video).

Thoughts:

• Deployment can take time when the project grows.
• Domain modelling: can use existing models and add to Mendix.
• Playwright is perfect for Mendix testing, here is a basic dev example.
• The Mendix Community is a good place to post questions.

I needed to change from US format to UK date time format:

Community has the answer: Mendix Forum - Question Details

Playwright series - Post 1 - Overview of E2E testing using VS Code for Low Code

Setup: I have installed Node 20.100.0 and the VS code extension for Playwright.  The installation and getting started guides are clear and of a high quality.  https://playwright.dev/docs/intro  I am running on Windows 10 Surface 4 with 16GB.  I am using TypeScript (ts) as it is the default and the recording mechanism works well with ts.  Previously I have use C# as it's my language of choice but I feel ts is easier to maintain and there is no need for complex logic/functions in end-2end (e2e) UI testing.  New features always come out in TS/JS first.

Thoughts:  Postman is easy to use, fast, configurable and flexible.  UI e2e testing allows me to know my app/sites are working as expected.  Manual testing is time consuming, and amending automated tests can be hard.

Setup Reminder:

1. Install the Playwright extension using VSCode (once at initial setup)

2. Open a new folder in VSCode, and open the "Command Pallette" (once for each new project)

>Install Playwright

These are the defaults and will use TypeScript as the base language, stick to this it is the simpliest.  VSCode builds the default file scaffolding as shown above

3. Create your first New Playwright UI Test:

3.1. Record new

3.2. Enter a URL in the recorder browser, and click around (optional add Asserts) 

3.3. Save the Test

3.4. Execute the test

The Green tick can be used to quickly run the test locally.  In the "Test Results" terminal, you can see the same test was run 3 times, my configuration is set to test Chrome, Firefox and webkit.

Why Playwright?

• Easy to understand/follow,
• Easy to record,
• Open source, 
• No paid licencing, 
• Faster than Selenium,
• Various coding languages supported (bindings for C#, Python, Java, JS, TypeScript),
• UI verification using screenshot and AI to minimize flakiness/static DOM reliance,
• Ability to debug and trace is strong,
• Can do API testing,

Possible Playwright UI testing Layers: 

1. Full regressions goes into detail and runs in Chrome, Firefox, webkit/devices 
2. Check in tests are comprehensive on a single browser for code check ins
3. Continuous testing - record logging in and reading from a db and calling an API.  Can write to logs i.e. Dynatrace, Azure Monitor, SolarWinds using API's,  Doing this every 5 minutes will tell you at a high level if the service and it's dependencies are running and if there is a performance change.
4. Developer can write detailed local tests when working in an area, and reuse them if he comes back and changes any code.

Testing Challenges:

Unit testing is a challenge in low code - while they are fast and ideal for C# or code, not easy to implement for Low code.  Their is a new beta feature for component testing in Playwright, i don't think it adds value.  API Testing - I use Postman for API testing including controlling my CI.  Low code automation testing is hard in the Power Platform, E2E playwright testing in context works pretty well.  API's/ network traffic needs to be mocked.

Challenger products:

• Selenium - QA's highly skilled here
• Cypress - Devs tended to use this over Selenium
• Specific products like in Power Platform Test Studio and ...
• I also like BrowserStacks low code testing, especially if no CI/CD and can manage from here and use different low code technology.  

Summary: Generally I'd go for Playwright over all the others. 

Review of Browser Stacks Low Code Automated test tool

Overview: Low code testing relies heavily on complete UI end-to-end testing.  It needs to be fast, flexible, quick to correct, scale-able, highly configurable and BrowserStack's low code test tool is in beta and definitely on the right path - for me, it needs a few features.  I ran my testing against customised apps created on three platforms:

1. Mendix low code,
2. Microsoft's Blazer hosted on Azure Web App, and
3. Canvas app within Power Platform. 

Tip: I've looked and use BrowserStack for many years and it has moved from being an device emulators infrastructure testing provider to a full ALM testing platform.  The low code Browser stack has a recorder to capture steps.

Where does Low Code fit into Browser Stack:

Image1. Low code automation works well as part of the full BrowserStack Platform or just using the product by itself.

Pros of the Low Code BrowserStack Product:

• The local recording feature is easy to set up and use
• Seamless integration with the cloud version running on BrowserStack's infrastructure
• Logical layout of UI, little to no training required
• UI validation using the DOM or, more importantly, screenshots using BrowserStack's AI verification (required further review) has the potential to self-heal as in the screen changes, but the validation can be smart enough to understand it is just an updated screen (example a single colour in the page and the position of the name is moved).
• SDK is available to work with the full BrowserStack platform.
• Not Low code specific but BrowserStack generally has the new phones included in their offering within days of being release.

Cons:

• Provide a webhook or allow for a REST client call as a step (I'd want to log directly from the test run into Azure Monitor)
• More run options, I'm sure it's already on the road map, but the ability to run every hour for continuous monitoring.
• Refresh tokens on a schedule (allows you to not use MFA such as SMS codes or Authenticator).
• Make it clear if the run is from the local or the browser, and keep the historical logs for both together.
• Export results - I could not find this, but it would help compare step performance.
• I use DevOps, I'm unlikely to take the whole BrowserStack platform unless i need the emulators which is what BrowserStack was originally famous for.

Summary: This is an excellent tool for testing; the low code product was still in beta when I reviewed it.  It is a nice tool and has the potential to disrupt the market.  I feel Playwright is a better point solution and integrates to CI/ALM platforms.

References: https://www.browserstack.com/low-code-automation/features?utm_source=google&utm_medium=cpc&utm_platform=paidads&utm_content=668760067900&utm_campaign=Search-Brand-EMEA-Navigational&utm_campaigncode=Core+9045914&utm_term=e+browserstack

Other: 

Image 2. Emulate a Samsung Galaxy phone on Android using the Chrome browser.

Thought: I like BrowserStack's reporting, clean and simple on tests and easy to get the history.

Testing Power Apps using Microsoft Playwright with C#

Overview: I am using a C# Console app to run my power Apps to verify that it is working correctly.

Approach:  I'm not using NodeJs but I'll use C# and add it into my devOps pipelines.  I assume you have VS Code installed (you need the C# extensions, I have kept my notes at the bottom of the post).  I'm using .NET 6 core SDK and the Microsoft.Playwright package.  I'm not using any of the .NET testing frameworks to keep it simple.

Tips: Playwright by default use headless browser testing, I want the test to use my local JWT that I have stored against my Chrome browser.  If I tell Playwright to use show the browser i.e., headless=false, it logs me in using my cached credentials.  You should parameterise the login and store the username/pswd somewhere secure like a key vault. 

Code Example: 

Replace the C# code in the console as I have done above, and hit F5 (Run>Debug).  The test logs me into the Canvas app and returns the Title of the page. 

Note:  I like the Canvas app testing feature but this is better as I can easily add testing to my CI/CD pipelines, tooling and recording is better and there are more ways I can use the Playwright approach.  

Notes for installing Playwright on VSCode to use C#

1. Install VS Code and add the following two extensions

2. Open the VS Code terminal, create a C# Console app using .NET6, add the reference to the Playwright package, install the headless browsers against you profile (only needed once per machine). 

3. Write and run the tests.  Tip: It's easier to record the code and then add it to the C# console.

Note: It's worthwhile using NUnit as it's much easier to test than doing it the way I've done it here.

Playwright using the NUnit framework: