Sunday 8 August 2021

Tools for Architects

Overview:  As an architect we use multiple tools, it's a good idea to standardise tool usage especially in larger businesses. It's pretty common to se architects using different tool for drawing architecture diagrams (viso, draw.io,...)

Common Drawing tool I've seen architects use:

  • Visio
  • Draw.io
  • Balsamiq (my odd tool of choice that offers low fidelity, alternatives are Figma which is high fidelity for UI and Ux design, issue is end users thing its the website and get confused, Axure is also a nice Ux prototyping tool).
  • Lucidchart (competitor products: Visio, Miro, Balsamiq, ) Lucidscall can pull diagrams from architecture.  The is also a cool python git project that allows you to specify a diagram using Python and it draws the diagram and you can also then use the Python code to provision the infrastructure.
  • Miro (great for sharing).  Similar to Lucidcharts, with good integration, I tend not to introduce Miros as I use Teams and it's whiteboarding.  FigJam is pretty good for brainstorming from the makers of Figma.
  • PowerPoint (sic. but it happens and some architects are good with it)

Tools for Retrieving Azure Architecture and creating documentation:

Enterprise Architecture Tooling:

  • Sparx
  • Archimate

Dev Tooling I use a lot:

  • Visual Studio
  • SQL Server Management Studio 
  • Postman/Swagger

WIP


Sunday 1 August 2021

JMeter - The basics

JMeter is an easy to use open source load testing tool by simulating network requests.  JMeter is good for figuring out how well the server side responses are working under different test conditions.  JMeter is built with Java and can run on Linux, Mac or Windows using a Java Virtual Machine (JVM). 


JMeter is Single Agent:

  • JMeter runs from the machine it is installed on so it does not have multiple agents.  Saying that it can simulate hundreds of users on fairly low spec machines.  
  • To avoid network latency, test on the same subnet or data center.  A simple VM in Azure (with 2 vCPU's and 8 GB RAM) can mimic over a thousand requests per second.
  • You can run tests off multiple machines to generate extreme loads (first I would use 8 cores and 64GB ram until the network traffic is maxed).
  • Install the Windows JDK 11 before installing installing JMeter.
Updated 2 May 2023: The current version of JMeter is Apache JMeter 5.5

Download and record web tests using the JMeter GUI tool.
Azure Load Testing needs the recorded tests generated by the JMeter GUI.
Create a new Azure Load Test Resource and use the recorded JMX/Test script file.


JMeter GUI
Open /bin/jmeter.bat


Wednesday 28 July 2021

Azure DevOps User Stories Tips

Quick Point on User Stories and Acceptance Criteria.

1.      User Stories description must follow the format:

As the <role> , I want to <feature> so that <benefit>

Note: Always follow the exact same format and bold up the standard/fixed parts for user stories.  Pls keep consistency across your teams user stories.  Under the user story in the description, feel free to add more description, annotated images (very useful) and links to Figma, Axure, UI mocks or Miro.  User Stories should also follow the INVEST (Independent, Negotiable, Valuable, Estimatable, Small, and Testable) breakdown.

2.      Acceptance Criteria (Use Gherkin Language) under the user story (ensure it goes into the User Story section and not comments or the description)

Scenario:
  Given
  When
  Then

Example

Scenario: Employee requests leave
  Given an employee has sufficient leave available in the year
  When the employee schedules leave (holiday)
  Then the employee is informed his request is valid and his manager is informed of the request.

Note: Always follow the exact same format and bold up the standard/fixed parts for user stories.  Pls keep consistency across your acceptance criteria.  I bold and use the four parts as shown above in the example.  You can use "and" to extend the story, just try keep them within the idea of INVEST.

3.      Other:  Order is Tasks belong to User Stories, User Stories belong to a Feature.  Features belong to Epics  These items must be related within Azure DevOps.   Epics > Features > User Stories (Acceptance criteria) > Tasks.

Scrum - Part 1
Scrum - Part 2 
Scrum - Part 3 - Scaling Scrum
Scrum - Part 4 - Kanban and Scrum
Scrum - Part 5 - Certification PSM 1

Azure DevOps - Introduction

Tuesday 20 July 2021

Open Banking & Crypto currency - Capital Gains Tax

In the 2019/2020 & 2020/2021 UK tax years, I bought and sold crypto currencies; I happened to make a little money out of it thru no skill of my own.  "A rising tide lifts all boats".  It has been bothering me so I looked up how taxing crypto works and it falls under Capital Gains. 

UK individuals get £12,300 I am well below the threshold for owing HMRC additional tax.  I still need to report my gains on my SA100/Self Assessment with HMRC using SA108 for reporting Capital Gains.  There is also a great initiative Payment Services Directive 2 (PSD2)/Open Banking offered by companies like TrueLayer that provide API's to get current account statements/transaction using OAuth2 permissions.  PDS2 is pretty useful for anti-money laundering (AML), specifically Know Your Customer (KYC) that falls under AML.

It is worth using this capital gains allowance if you can get solid returns off applicable assets such as shares or cryptocurrency.

"When you dispose of cryptoasset exchange tokens (known as cryptocurrency), you may need to pay Capital Gains Tax.

You only have to pay Capital Gains Tax on your overall gains above your tax-free allowance. The Capital Gains tax-free allowance is: £12,300".

TrueLayer and Token is a service that connects to all UK commercial banks instead of you having to individually connect to the banks Open Banking API's.

References

Payment Services Directive 2 and Open Banking | UK Finance

https://www.gov.uk/capital-gains-tax/allowances

Elastic Database Client Library for client database segregation on Azure PaaS for SaaS

Overview:  Provide a logically separated database instance for each client on my SaaS solution.  Using the Elastic Database client library from Microsoft on Azure PaaS services provides logical security separation of data, performance is on a per customer, and easy scalability.  Use Azure SQL Elastic Pools (HA redundant secondary database, built in DR).  Also add temporal tables for a full history of all transactions.

PoC:

  1. Provision 3 databases - A Shard Map Manager (Catalogue) database and 2 client databases (tenants/shards).
  2. Add shard related metadata to the Catelogue database for each of these databases.
  3. Create below Three service principals in Azure AD: 
    • Management Service Principal: for creating shard metadata structure.  A database contained user in Shard Map Manager db and each tenant db.
    • Access Service Principal: to load shard mapping at application side.  A database contained user in Shard Map Manager db.
    • Connection Service Principal: to connect tenant database.  Database contained user in each tenant db.


                        Management Service Principal: for creating shard metadata structure

CREATE USER [shard-map-admin-sp] FROM EXTERNAL PROVIDER

EXEC sp_addrolemember N'db_ddladmin', N'shard-map-admin-sp'

EXEC sp_addrolemember N'db_datareader', N'shard-map-admin-sp'

EXEC sp_addrolemember N'db_datawriter', N'shard-map-admin-sp'

GRANT EXECUTE TO [shard-map-admin-sp]

 

Access Service Principal: to load shard mapping at application side

CREATE USER [shard-map-access-sp] FROM EXTERNAL PROVIDER

EXEC sp_addrolemember N'db_datareader', N'shard-map-access-sp'

GRANT EXECUTE TO [shard-map-access-sp]

                                                         

Connection Service Principal: to connect client/tenant database

CREATE USER [tenant-connection-sp] FROM EXTERNAL PROVIDER

EXEC sp_addrolemember N'db_datareader', N'tenant-connection-sp'

EXEC sp_addrolemember N'db_datawriter', N'tenant-connection-sp'

EXEC sp_addrolemember N'db_ddladmin', N'tenant-connection-sp'

GRANT EXECUTE TO [tenant-connection-sp]


References:

https://docs.microsoft.com/en-us/azure/azure-sql/database/elastic-database-client-library


Saturday 10 July 2021

Modeling and working with data on the Dataverse within the Power Platform

The Common Data Model - Is an industry agreed approach to storing commonly used data.  Use it to store applications across your company/organisation.  Results in a single source of truth instead of multiple copies with data contained in different schema's.

Two types of relationships namely: 1:N and N:N (Dataverse hides the associate entity/intersect entity that is created in the background).  

Use "Option Sets" for small static data and use "Lookups" for larger or changing data.  In Multiselect Option sets for N:N relationships, rather use "Lookups".

SSMS can be used to view data using T-SQL, better to use "SQL 4 CDS" as it provides a full ability to work with data within the XrmToolBox.

XrmToolBox - is a 3rd party download that has a ton of contributed tools and there are good options for modelling data.  This is a collection of tools that are unbelievably useful and get continually updated and new tools added.

Excel and browser plugins - Can be used to import/export data and there are some nice Edge/Browser plugins to help such as "Level up for Dynamics 365/Power Apps".  Level up for Dynamics 365/Power Apps is a fantastic tool that I encourage any developer to add to Edge or Chrome as an extension.

Level up for Dynamics 365/Power Apps Browser extension
Dynamics 365 Power Pane Browser extension.

Dynamics 365 Power Pane is also a useful extension shown below:
Power Pane options

The built in browser Dataverse management tool is super easy to use to model your Common Data Model further.

Wednesday 7 July 2021

Microsoft Dataverse (CDS) - Overview

Overview:  Dataverse is CDS, there is a long story on the naming but ultimately Dataverse is a data store with a advanced security model, Open API's, workflows, pipeline injection...  It is awesome.

It is high performance, and would take considerable effort and components to deliver similar functionality or even semi close functionality.  It does have limitations mainly around performance but don't let that fool you, Dataverse is fast and powerful but for massive industrialized storage it's not the right option.  The costs are also a key consideration.

The biggest mistake I see is people making the same mistakes as they do with relational databases namely: 

Poor Dataverse implementation down to 1) poor entity relationship design, 2) either too many table containing duplicate data or to few table being expanded for a dev teams capability but ignoring existing systems, 3) poor security 4) too many cooks.

Basically, like any Database service, you need to have owners and try keep the structure logical and expand it appropriately.  The idea behind the data model used by the dataverse is to have centralized secure shareable data like customers or account information.  It's simple, treat dataverse as you would your most precious core database, have an owner that needs to understand and approve changes.

Note:  Microsoft have had some trouble naming Dataverse, it was previously known as the Common Data Service (CDS).

Dataverse logo

Overview: Dataverse helps improve processes.  And Dataverse helps reduce time to build IT capability, remove shadow IT, improve security and governance.  Data is the common data store we need to use to be effective.  As part of the Power Platform, it allows us to build custom software fairly quickly.

Updated 07-July-2022

Dataverse provides relation data storage (actually runs on Azure SQL (Azure Elastic Pools), Cosmos DB, and Blobs), lots of tools e.g. modelling tools.  I think of it as a SQL database with lots of extra features.  Most importantly business rules and workflow.

  • Dataverse relies on AAD for security
  • Easy data modelling and supports many-to-many relationships NB!
  • Easy to import data using PowerQuery compatible data sources
  • Role-based data (previously called row) and column (previously called fields) level security.  See Dataverse security in a nutshell at the bottom of this post.
  • Provides a secure REST API over the Common Data Model, it's awesome
  • Easy to generate UI using PowerApps model driven app
  • Ability to inject business rules when data comes in or out of the Dataverse (can also use .NET core code)
  • Can also stored files (ultimately in blob storage)
  • Search that indexes entities and files
  • CDS used tables, Dataverse calls them Entities.  Some of the UI still refers to table.  Just assume Entity and Table are interchangeable terms.
Dataverse basically allows you to have a PaaS data hosting service that mimics what we have done for many years with databases and Open API, has advance features and tooling and it is all securely managed for you.  

The cons are basically: is that it is expensive.  So you need to know your size and keep buying add-ons to the plans.  Scaling Dataverse is expensive.

Common Data Model: Collection of tables (previously called entities and most CRM people still call them entities) e.g account, email, customers for a business to reuse.  Comes from CRM originally, the starting point consists of hundreds of entities pre-created.  Common standard for holding data.

Each Power Platform Environment has a single Dataverse associate to it.  It's a good idea to have more than one environment but at it's simplest, use a trial to learn and progress to production.

Once I have a new environment, I can use Power Apps to access my environments Dataverse and model out a new table to store info, I am storing people tax returns.
Go into the Dataverse and model directly

Model the table in you Dataverse instance

Dataverse Security in a Nutshell:
  1. A user is linked from AAD to the User entity in the Dataverse.
  2. User Entity record is aligned to the AAD User.
  3. AAD Users can be part of AAD security groups.
  4. Dataverse Teams (Dataverse Group Teams) can have Users and or Security Groups assigned.
  5. Dataverse Group Teams are aligned to Business Units.
  6. Business Units have roles (rights).
"Security is additive" in Dataverse (generally the whole MS and security world these days).  i.e. no remove actions.  If you have permission in any of the groups you can access the data/behavior.

Business Units used to restrict access to data.  Can be hierarchical i.e. Enterprise > Audit > EMIA > UK (Don't use it like this, keep it simple)
Security Roles define a users permissions across the Dataverse entities i.e role can read only from Accounts entity 
Teams consist of users and security groups.  That get assigned roles.  There are two types of Teams in Dataverse: Owner teams & Access Teams
Field-level security, only allows specified users to see the field data

https://learn.microsoft.com/en-us/power-platform/admin/wp-security-cds (Good clear post on Dataverse security, core concepts are Business, Units, Teams, Roles, Users & OAuth/AAD)