Testing Power Apps using Microsoft Playwright with C#

Overview: I am using a C# Console app to run my power Apps to verify that it is working correctly.

Approach:  I'm not using NodeJs but I'll use C# and add it into my devOps pipelines.  I assume you have VS Code installed (you need the C# extensions, I have kept my notes at the bottom of the post).  I'm using .NET 6 core SDK and the Microsoft.Playwright package.  I'm not using any of the .NET testing frameworks to keep it simple.

Tips: Playwright by default use headless browser testing, I want the test to use my local JWT that I have stored against my Chrome browser.  If I tell Playwright to use show the browser i.e., headless=false, it logs me in using my cached credentials.  You should parameterise the login and store the username/pswd somewhere secure like a key vault. 

Code Example: 

Replace the C# code in the console as I have done above, and hit F5 (Run>Debug).  The test logs me into the Canvas app and returns the Title of the page. 

Note:  I like the Canvas app testing feature but this is better as I can easily add testing to my CI/CD pipelines, tooling and recording is better and there are more ways I can use the Playwright approach.  

Notes for installing Playwright on VSCode to use C#

1. Install VS Code and add the following two extensions

2. Open the VS Code terminal, create a C# Console app using .NET6, add the reference to the Playwright package, install the headless browsers against you profile (only needed once per machine). 

3. Write and run the tests.  Tip: It's easier to record the code and then add it to the C# console.

Note: It's worthwhile using NUnit as it's much easier to test than doing it the way I've done it here.

Playwright using the NUnit framework:

SaaS Azure Testing Thoughts

 Tooling:

1. API Automation - Postman, Newman
2. UI Automation - Selenium
3. IDE - Visual Studio 2019
4. Test Organization - Azure DevOps Test Plan
5. CI/CD - Azure DevOps

Code reviews:

Code review is used as a verification technique to ensure that each unit is coded as per standards and expected business logic and inline with coding standards and best practices.  Automate code review built into Azure Pipelines should include:  

• WhiteSource Bolt - Scan packages for vulnerabilities.
• SonarQube - Static Analysis, 
• Blackduck - Open-Source Scanning (OSS) tool.  Used to look for license risks and unused references.
• Checkmarx - Static Application Security Testing (SAST) tool benefits include: Detect security vulnerabilities, Improve developer practices, and reports on code ownership.  Static code anaylsis.  VeraCode is a competitor product.
• BugSuite

Code should pass OWASP (Open Web Application Security Project) shows the most common code vulnerabilities.  OWASP ASVS (Application Security Verification Standard) - framework for controls when building applications to cover functional and NFR's for web applications.

Unit testing:

Unit tests are written to ensure every unit of code is working as expected, and to prevent a defect from going to the next level on all C# code.  Xunit and Moq are the tools to be used for unit testing using the standard Arrange > Act > Assert pattern.

As long as Unit test coverage is high and of a good standard, I don't mind if the tests are written before the code (TDD) or as most developers tend to do the tests after the code is written.

API testing:

All API must use Postman collections and Environments for local testing.  The tests need to cover all API's dealing with authentication, authorisation, checking status codes, body responses, headers, data persistence, and post test clean-up.  Use Newman to integrate postman tests into Azure pipelines:

https://www.npmjs.com/package/newman-reporter-htmlextra

Selenium testing:

Code for UI must be automated where possible.

SonarQube: "automatic code review tool to detect bugs, vulnerabilities, and code smells in your code" SonarQube documentation

Code Smells:  Bloaters, OO abusers, ....

Checkmax detects potential security issues

Disposable email addresses: You often need to test login/account creation and it's useful to have temporary disposable email addresses:

• YOPmail - Disposable Email Address
• Home - Mailinator

Non Functional Testing for SharePoint

Overview:  Functional Requirements are the business requirements that the business define for the application being built.  Non-functional testing is concerned with performance, reliability, scalability, recovery, load,  security and usability testing.  For SharePoint it is a good idea to test this at a platform level and then verify the individual application non functional testing is appropriate.

SharePoint Non Functional Testing:
All of these test should be performed against your various SharePoint platforms and will dictate the SLA's offered to the business using SharePoint as a service.  Baseline testing is a good idea as the differences can be used to determine the efficiency of the individual application being created.

Proxies:
Fiddler is my favourite (other tools for capturing web traffic Charles, BurpSuite, WireShark and you can use the developer tools shipped with the browsers). tcpdump, goPacket are awesome for network monitoring.

Use Fiddler to:

• Observe traffic (http/https requests, headers,..)
• Replay sessions, 
• Evaluate performance,
• Set break point

A common misconception is the function of performance vs load testing.  

Performance is primarily concerns with looking at typical user usage scenarios and see how long each page takes to load.  So a recorded script with wait time between recorded calls is useful  It's also worth looking at the same page with minimal data and also with large amounts of data.  

Load Test involves recording the standard users interaction (ensure some users query heavy and some light amounts of data), there are no wait times.  The norm is to multiple this concurrent number of users by 100 to know the amount of users the farm business application can support.  e.g. a concurrent user load of 100 users where the performance is acceptable means the farm should handle 10,000 users (100 users times 100).  You run the 100 users in a steady state for a few hours.

Stress testing is similar to load testing but you keep stepping up the number of concurrent users until the SharePoint farm starts running out of resources and throttling requests.  Once the system degrades and performance is not acceptable is pretty much how many users the application on the SharePoint farm are supported.  So if the performance throttles at 170 users, the farm can handle 17, 000 normal usage users.

References:
http://www.guru99.com/non-functional-testing.html

DevOps Tooling

DevOps Tooling Notes

DevOps Tooling is broken down into the following areas, note the tools often overlap in function.  The list is not exhaustive but these are the more common tools I have come across.

1. Version Control: TFS, Git, SVN, ...
2. Bug Tracking: ServiceNow, Jira, ZenDesk
3. Continuous Testing: Selenium, Jasmin or Mocha or Unit.js (JavaScript testing), NUnit, Web Tests (Visual Studio), SpecFlow
4. Continuous Integration (CI): TeamCity, Jenkins, Azure DevOps (bigger) 
5. Configuration Management and Deployment:  Puppet, Chef, ANSIBLE, SALT  (all installed on Linux, obviously work on Windows environments)
6. Containers: Docker, Kubernetes, Microsoft Containers. I think the Azure AKS is pretty much containers for Azure now.
7. Other:  PowerShell, VMWare, HyperV

RESTful API Tooling

1. Swagger - awesome.  Swagger is a set of tools that help document, build and test your API  (Your API conforms to the OpenAPI specification or Swagger specification).  Great way to get a contract for users of the API early on.  Updated 2019/11/25:  Link to Swagger post
2. Swagger UI, Swagger Integrator,...
3. Apiary - UI to create an API and publish with mocks.  I prefer Swagger or on simple projects APIM.
4. API Management (APIM) - flexible Azure service for bring together multiple API securely.  Same as MuleSoft.  Can import OpenAPI's v2 or v3 to create a hosted API.  Can mock and built in test tool.
5. RAML is an alternative to Swagger and Apiary (never used)
6. Blueprint - API documentation tool.  Pretty simple and nice results.
7. Postman - send http requests to the API.  Postman is a REST client useful to check your API.  This is my main tool for testing, exploring REST based API's.  
8. SoapUI - if working with SOAP/XML.
9. Slate - API documentation - I always use OAS/OpenAPI/Swagger.
10. Fiddler - I'm old school and still love Fiddler and it's capabilities.  Fiddler is a great HTTP debugger.  
11. BURP - an HTTP debugger to review traffic.  I've used BURP for security testing and it is great for API debugging.  
12. Charles is another HTTP debugger (never used).
13. cURL - Cmd line to test API's using HTTP, separate exe to run on Windows, Windows 10 has cURL built in.
14. Visual Studio
15. Wireshark - Over the years I have needed packet sniffing to fix issues and always go to Wireshark, I used the tool in the 90's but it had a different name.  Extremely useful for issues relating to firewalls, especially when an environment reacts differently to another working DTAP environment.
16. Tcpdump is another packet sniffer

Testing:
http://www.incyclesoftware.com/2014/02/executing-selenium-ui-tests-release-management/

More Info:
http://blog.sharepointsite.co.uk/2014/02/devops-and-sharepoint.html
http://www.networkworld.com/article/2172097/virtualization/puppet-vs--chef-vs--ansible-vs--salt.html
http://blog.sharepointsite.co.uk/2013/11/iac-presentation-for-sharepoint.html

Unit Testing for SharePoint 2010 - Part 2

Part 1 of Unit Testing for SharePoint 2010

Overview:  Part 1 discusses setting up nUnit and TDD.NET on you dev machine and writing a simple unit test.  This post looks at using TypeMock Isolator to unit test SharePoint dependent code.

//Arrange
//Act
//Assert

Background:  I haven't used TypeMock Isolator in the last 2 years but the latest version still is a good candidate for testing SharePoint 2010 based applications.   The TypeMock Isolator allows you to fake calls to SharePoint so that you can run unit testing against the SharePoint object model with actually having SharePoint installed or going to a specific instance of SharePoint 2010.  The full version (i.e. not the SharePoint only version of TypeMoch Isolator) also allows you to do create isolation mocks of you code (see Rhino Mocks is a good Isolator Framework).  So pretty useful for testing SharePoint or Entity Framework.  You simply mock out the expected behaviour and don't need to test the Entity Framework by connecting to it.

Tip:  You can use this exact same technique to test MOSS.
Tip: TypeMock needs a separate build server licence from the developer licences.  You can use the full product that includes the ability to do SharePoint testing or just the SharePoint TypeMock isolator.  As of 6 March 2011 the SharePoint edition costs 249 euros per year.  Licencing is expensive especially for large dev teams with multiple build servers but the returns are good if your organisation buys into TDD.
Tip: The SharePoint edition is fine for testing SharePoint calls but you can't mock your own .NET methods so you can't simply use a factory pattern and then mock your entire .NET method call as was shown to me by Ketul Patel (@Pateketu).

Objectives: Create a unit test that uses TypeMock Isolator that mocks the call to SharePoint.
TypeMock to Isolate a method call:
Add 4 reference for nUnit and TypeMock frameworks.
Add a reference to the library you are testing in my case "MagicBall.dll".
Create a new unit test, I isolate a call to the method "GetSPQuestions" and return a list of strings (this has allowed me to override the normal call that would be made to SharePoint).

My business logic for the "Magic 8 Ball" is shown below.  The key take away is I fake the method call GetSPQuestions() using TypeMock so it doesn't try access SharePoint.  This allows me to test my business logic using SharePoint's object model.

TypeMock to Isolate a SharePoint call:
The previous technique works if the return data is not SharePoint objects, so TypeMock allows you to mock up SharePoint objects.  Whereas this example shows how to unit test by faking SharePoint objects.

• As you can see the code fakes the SPSite Object, then we fake the "Contracts" SPList object (fakeContractList). 
• At this point all properties inside the objects are null.  So our list would simple return a list with no objects.
• The next step is to create 3 fake list items that are added to the "fakeContractList".

The code below shows the business logic/code of my "Magic 8 Ball":

The method QuestionSPList() has child methods that use SharePoint objects.  This example directly fakes the SharePoint object model to allow for unit testing.
Summary:  I prefer to build my calls to the SharePoint API or any external API's in a low level private service method.  Then all I need to do it stub out the entire method as shown in the 1st example.  Faking the SharePoint object model using TypeMock is a good option for testing SharePoint reliant code. 

More Info:
Paul Hunt has done 3 great blogs on Andrew Woodward's SPRetreat so this will give you another viewpoint on unit testing for SharePoint 2010.

Andrew Woodward has a codeplex project (SharePoint Magic 8 Ball) that will give you tons of example code on unit testing.

Unit Testing for SharePoint - Part 1

Overview:  I am not an expert in Test Driven Development (TDD) but the SharePoint Retreat in London has made me think about using this approach for SharePoint 2010 development.  There are various tools to help you perform TDD however, these 2 posts show the approach I am most comfortable with thanks to Andrew Woodward. 

This whole article is based on Andrew's session and I really would recommend to any serious development team to get Andrew in to do some upfront consulting and potentially training.  Upfront costs and his experience would definately be quickly recovered.  If you have already decided to use a TDD get Andrew in to get you dev team up to a good standard with TDD for SharePoint, anyway enough of my sycophantic
dialogue.

Tools:

• Visual Studio (VS) 2010 any edition;
• NUnit - preferable to MSTest for SharePoint dev
• TestDriven.Net (optional)
• ReSharper (optional)

Objectives:
Part 1 focuses on using setting up a basic "Magic 8 Ball" to unit test the business logic using nUnit. 

Tip: using coding Katas (really good approach to training, this is the 1st time I've used them and they actually really work) to get a full unit testing system working. 
Tip: Write the unit test and then write the code to fix the test.

At the SharePoint retreat, each session started with a:

1. Review of the last sessions code (if appicable),
2. Followed by an overview of what will be the next bit of coding logic (instructor lead teaching/demo exercise),
3. Break into a new pair programming team to code (old code is throw away and a fresh set of code is started that includes all the exercise from all the previous sessions), and
4. Loop - Back to step 1 (Review each pairs code).

In this post I am describing the 1st session, what you are trying to do is setup unit testing and write a couple of tests for you "Magic 8 Ball". 

Setup

1. Open VS2010, create a new solution file (MagicPost1.Sln).

2. Create 2 new C# class projects, the 1st project contains the business logic code (BOL project) and the 2nd project contains the unit tests (BOLTest project).

3. Change the default class on the BOL project to Ball and the default class on the BOLTest project to BallTest.

4. Add 3 references to the BolTest project. 

All 4 steps are shown below:

dll reference locations:

\Program Files (x86)\NUnit 2.5.9\bin\net-2.0\lib\nunit.core.dll

\Program Files (x86)\NUnit 2.5.9\bin\net-2.0\framework\nunit.framework.dll

5.  Add the basic testing structure to you BallTest.cs code as shown below:

6. Add this basic code snippet to your Visual Studio code snippets (Resharper has better code snippet support).  7. Write you 1st unit test (the simpler the better) as show below:

8. Build your project to ensure it compiles. 9. Comptete the test code as shown below and run the test.

Tip: Strictly following TDD principals your test should fail and then you write code to fix it in the BOL code.

More Info:
Andrew Woodward's codeplex project on the magic 8 Ball which provides the code you can use to build the magic 8 Ball with TDD.
Here is a slide deck to a presentation on the Magic 8 Ball TDD project.

Next Post:
Part 2 of the series looks at testing SharePoint using Type Moch Isolator.

SharePoint Retreat Saturday

SharePoint Retreat London 2011, Saturday, March 05, 2011 London held by Andrew Woodward (@AndrewWoody) & Wes Hackett.  A brilliant day, if you get a chance to go to 1 of these you should.  Met some good SP people and got great tips & pointers.  The bonus was I won a Type Mock Isolator licence.

Andrew gave a clear example that we worked on throughout the day, enjoyed the unit testing with nUnit.  I'm not totally sold on the TypeMoch as I feel like I'm testing the SharePoint API but it does increase code coverage.    

More Info:
http://spretreatlondon11.eventbrite.com/
Paul Hunt (@cimares) has done 3 great posts on the SPRetreat event.  Paul has transcribed the sessions extremely well and with good examples delivered in a step by step approach.
Update 10/08/2011: Unit testing Guidance