Thursday 30 December 2021

Azure DevOps Series - Overview

Azure DevOps is a SaaS platform that provides tools for deploying software using DevOps techniques generally within Agile software delivery projects.  DevOps is useful for gathering requirements, building the solution, performing daily “integration builds”, and having deliverable end-of-sprint demos.  The key to building software today versus several years ago is that we should automate as much as possible.  Azure DevOps provides excellent tooling to automate and implement automation results in better quality, reduced timelines.  Building software is easy as long as you have great people and precise requirements.  Agile practices and DevOps processes and tooling can help get you to the desired state.

  1. Azure Boards are great for planning and getting the requirements cleanly broken down.
  2. Visual Studio coupled with Azure boards items with Azure Repos (source control options are GitHub & TFVS) is ideal for the development using most languages such as C#, NodesJS, Angular, React, TypeScript.
  3. Azure Pipelines are good at deploying solutions by setting up the infrastructure (I prefer to use PaaS and get out of the Infrastructure world, using ARM templates) and deploying code with the appropriate DTAP environment configuration.  Azure Test Plans are used to verify builds. 
  4. Monitor and Alert – Azure Monitor/App Insights is ideal for monitoring the infrastructure and operating code to detect issues early.
  5. Azure Artifacts help create and share Nuget code packages.

More Info:

Sunday 7 November 2021

Figuring out SaaS licencing and SLA's

Overview: Buyers be they B2B or B2C will want to understand you licensing, probably cost, and level of service.  Keep it simple, keep it understandable, make sure you cover what availability, performance, actions users can use you service for are all clearly outlined in you Service Level Agreement (SLA).

Licensing pricing options:  Pay per use one-off, yearly, pay per user monthly or annual, pay per consumption e.g. stripe.  

SLA:

  • Availability 99.9 or better is good, it really depends on what you are offering but there are often penalties for missing availability SLA.  If I build a pretty standard SaaS application that utilises App Services, APIM (standard, premium geo loaded has a higher SLA) and Azure SQL, I can't make a 99.9% SLA excluding AAD and any patching, or application caused downtime.  At a SaaS product level providing an actual 99.999% (5 nines) SLA is not as easy as the marketing and legal stakeholders might assume.

It becomes easier to offer 99.9+% SLA's if you as a company assume the risk.  i.e it's unlikely all the downtime will occur and affect you sequentially so offering money back is absolutely possible.  Also most SaaS companies require clients to claim, it is not monitored and automatically applied to your bill.
  • Support - phone, bot, email and max time to respond and time to resolve.  
  • B2B Monitoring - Good idea to monitor you SaaS provider and not just take their word for it.  Technically, monitor availability of individual services (web sites or API's), it also good to know when items outside of your control (with the SaaS vendor) are unavailable in internal support.  Examples: page load times, login times, you are looking for availability and speed.   How much of the service is down, how much does this affect end customers.   You may want to use a 3rd party tool or write your own as a last resort to monitoring.  When relying on 3rd parties to provide services ensure you do a hazard risk assessment.  Plan for when things happen, how will you respond, adjust.

SLA vs SLO vs SLI:
  • SLA (Servcie Level Agreement - contractual agreement SaaS company makes with the customer.
  • SLO  (Service Level Objective) - Goal availability (and acceptable performance) of the microservice or application.  Measurement goal.
  • SLI (Service Level Indicator) - checks if SLO is achieved.  Actual Measurement.

As part of High Availability and scalability, it is a good idea to know how many instances and how autogrowth is setup, here is an example for Azure App Services.


Scale Out (CPU or Memory) - Matrix Threshold (Avg): 70, Duration:  5 Min, Cool down Time:  5 Min, Increase Count: 1
Scale In   (CPU or Memory) - Matrix Threshold (Avg): 40, Duration: 30 Min, Cool down Time: 10 Min, Decrease Count: 1

Power Apps - MicrosoftTeams.CreateATeam() connector method not working

Problem:  I am provisioning a new team inside my tenant using Power Apps.  The code throws an error and I can't get the TeamId, however, the Team is being created.


Hypothesis:  The call creates the team and using the Monitoring Tool inside Power Apps I can see the call is working.  The issue appears to be when Power Apps reads the response.

Possible Resolutions:  

  1. Use Power Automate and fire from the Power App
  2. Create a custom Postman Collection using the Teams Graph API



Sunday 3 October 2021

Asking for Gender Pronouns and Title in forms

 Here are a few pointers when asking for Gender, Sex, Pronouns and Users Title

SaaS Onboarding & Payment Collection

Overview:  Selling a SaaS generally can be split into B2C and B2B.  Both SaaS models require the ability to onboard a customer and collect the payments for the service.  And to do this a website /Content Management System is need to allow the customer to trial, buy, purchase add-on and collect recurring revenue.

B2C SaaS (small to large):  You need "brochureware", web pages that show the service and allow the user to purchase.   As I generally sell SaaS software, and B2C is often 1 off or pay as you need, I'd recommend Shopify, there are add-ons for selling digital goods.  You can always use marketplaces like amazon or eBay also.  WooCommerce (integrates with WordPress), BigCommenrce, Magento, Wix, SquareSpace can be used for selling physical goods but need some thought and add-ons for digital goods.  All the options are not great at Monthly Recurring Revenue (MRR) e.g. Netflix, or ARR (annual, e.g. Grammarly, Blinkist) billing, or "pay as you consume" also referred to as "metered billing" or "pay-as-you-go pricing" (Uber, AWS, Twilio, Stripe, Azure, GCP) revenue models.  Pay-as-you-go pricing has variable costs but allows your to reduce the cost to your customer by only charging them for what they use.

Update 2022/06/24: Webflow is a great tool for semi web literate developers to build websites.  UI drag and drop experience.  In the Wix space but you drag vs choosing a template.  I've used it to generate clean semantically correct HTML/CSS to implement in a custom developed SPA solution.  Always consider Webflow, it has checkout and can easily integrate with Shopify and Zapier for simple workflows.  Has free SSL and you can host on their platform.  Priced per website but reasonable with various options.

There are plenty of add-ons but in the Shopify world it's bring in an add-on and a few moving pieces.  For a medium sized SaaS selling MMR solutions, the overhead of setting up and managing the processes is fairly steep.  As the business gets bigger, it's worth the integration or using a dedicated solution like ChargeBee.

B2B SaaS: Could use any revenue model but it is best if your product lends itself to subscription-based selling.  e.g. Office 365, Workday, Legal practice management software to manage clients and work for law firm.  For small SaaS startups selling digital services use a solution like ChargeBee (low end) or Paddle (top end).  You can use anything in-between and addons to get a solution but for the price, setup, expansion as a general rule ChargeBee is good:

  1. Recurly
  2. Chargify
  3. Zuora
  4. Chargify
  5. Stripe
  6. Billby (Good for startups)
  7. ChargeBee (Good for startups)
  8. Bill.com for Accounts payable and receivable.
In the UK, we need direct debits often setup, you can look at something like Bottomline PTX.  I also like recharge.

Privacy Management: OneTrust provides a good configurable SaaS service for cookies and privacy.  If not built into your CMS/platform, then OneTrust is an option.
Pay-as-you-go/metered billing:  You pay based on consumption of services you owed x for the last 30 days e.g. AWS, Azure.  I try to stay away from this model as it's normally difficult to understand and customers in SaaS as a general rule don't like complex unknown pricing models.

Traditional:  Basically, pay as you buy a license.  So you get a perpetual license.  Akin to physical shopping but for digital goods.  Shopify is perfect for this model with a digital download add-on.  E.g. bjjfanatics.com

SME SaaS Business Checklist:

  1. Where you you selling, physical vs virtual
  2. Subscription or 1 off payments (account maintenance), trials, upgrade, upsell and cross selling.  Autorenewals.
  3. Jurisdictions (Tax, VAT, shipping, currency)
  4. Cost (fees, what's included, percentage of sales, growth)
  5. Support (Tech touch, Low touch e.g. email vs 24 hrs phone support, is this subscription based)
  6. Retention (Churn, New, Length of time for customers, support churn warnings, unsubscribing).
  7. Does my billing/subscription allow me to sell on web, native mobile, marketplaces.
Methods to set the SaaS price:
  1. Value based pricing - set the price based on the value the customer saves/gets
  2. Cost plus pricing - your cost plus markup
  3. Competitor pricing - what our our competitors charging
  4. Art of adjustment pricing - set a price see how much demand, change price how does it affect demand, MRR and total expected revenue.
Metrics to capture in SaaS Sales:
  1. MRR
  2. ARR
  3. CLV (Customer lifetime value)
  4. Churn rate
  5. Cost of Customer Acquisition

Marketinghttps://sproutsocial.com/

Support:  It's important to minimize human support effort, automating as much as possible is key.  Bots, knowledge articles that are easy to find are awesome.  Coveo does a nice job of setup for community channels.

User tracking:  Google analytics is pretty good for getting stats.  I do like a new tool to me Pendo which is expensive but extremely powerful.  Pendo's main 2 features for me are user interaction/how they use the site, and providing help tips/Guides are html injected into applications.



Wednesday 22 September 2021

Azure Synapse Intro

I have not done any serious big data work in many years and I'm using this post to understand the Azure options and update my skills.   

Azure Synapse Analytics - built for limitless scale for unstructured and structure data for big data  - supports petabyte analytics.  Ingest and prepare data for BI.  Instance can be a dedicated or serverless PaaS service.  Data lake Gen 2 assigned to each synapse instance.  GA Dec 2020.


Use to be Data Lake (massive, semi structured data) and Data warehouse, Azure Synapse Analytics can be referred to as Data Lakehouse.  Basically all the sperate pieces are under 1 umbrella i.e. ADF, SSIS, Dataware house,..
  • SQL Serverless Pools is pay be use/pay per query.  Can use SSMS to manage data.  Good for small or new data ware housing.  T-SQL works perfectly so simple for smaller projects or PoCs. 
  • Go from 3NF in relational data to using Fact and dimension tables to put data into a star model for MPP.
  • Data Warehouse Units (DW) - Same as DTU, made up of CPU, Mem, IO.  Simple to upgrade.  Lowest is DW100 can be huge, and simple to scale up and down.  More DW's means more VM's/work process work on the data.

Azure Synapse Studio  - SaaS development experience provide code free and code first (C#, Scala, Python, SQL, Java), notebooks are used for working with data.




Wednesday 8 September 2021

Observability in Azure SaaS Solutions

Problem:  Software has many places where errors and tracing is logged to.   Support get an incident, they need to investigate figure out how widespread the issue is and then try patch together various logs to figure out the problem.

Thoughts:

Observability is not a new concept, we need to be able to: 1) view and connect logs & 2) tracing and view metrics & notifications.

Implementing Observability must cover:

CI/CD allows devOps teams to find issues early using Unit testing.   Automated testing on UI.  API automation testing is also great.

Azure offers continious monitoring by performing various API calls to ensure your servie is running and any failures are picked up hopefully before any customers are aware.  You can also be notified of performance slow down, check performance speed between releases.  Which is great for identifying bottlenecks and with the Azure PaaS world, it is easy to increase the processing causing the bottleneck.  

Performance metrics built into the CI/CD and developers work allows us to identify issues early and costs miles less to correct early. 

Security and LINTing in CI/CD also allows us to pickup issues early and correct at a way lower cost.

Instrument you hardware and software, well on Azure you can use App Insights and you have a fantastic instrumenting platform that captures events.  A big reason to use Azure Services for as many of the function pieces in your solution.

Work In Progress ...

TBC Azure App Insights detail, ParentOperationId, Linking operations with a ServiceBus or work process call.



Sunday 5 September 2021

R&D UK tax Credits for Software development

Overview: R&D incentive claims with HMRC are designed to attract and encourage investment in R&D in the UK.  By providing for a taxable credit at 13% on qualifying expenditure, namely: Staff costs, consultants, Software licenses, hosting.  Claiming R&D costs reduces the companies tax liability and can be attractive for starting a new business or building new SaaS products.

The idea is:

PoC, Concept Design, software design and initial testing are likely to be deemed as probably qualifying for R&D claims.  The product/service must seek to advance (new or appreciable improvement of a process, product or increase technical capability) technology.

Specification/Requirements, scale up to commercial release, training may be qualifying activities. 

Thoughts:  R&D relief claims are complex.  There are different rates for SME vs large corps and rates if the company is in profit.  It is worth doing, and I believe it is pretty common to pay experts based on a percentage of R&D claim.  The big players have divisions to do R&D claims, i.e. KPMG.  There are also tons of specialized smaller firms that will do R&D claims.  For a SaaS startup, I'd contact a specialist provider to make the R&D claims and agree to a fixed percentage of the savings.

Wednesday 18 August 2021

Creating a Power Platform Custom Connector from a Postman Collection

Problem:  I like to discover an API and play with it using Postman.  I save the collection and theoretically I should be able to upload the postman collection into the custom connector to generate the Power Platform connector.  Until a few days ago, the custom connectors only supported Postman v1.0 collections.  And Postman has not save using the v1 format for a long time.  

I was happy to see the Custom connectors support v2.0 and 2.1 of postman collections.  Only problem is while it says it does if is still broken.  So I have to go the old school way.

I wanted to generate custom connectors for HMRC's REST API so that I can connect with the Power Platform.  Option 2 below gives me a way to build my collections using HMRC's API's and generate the appropriate version 2 Swagger definition.

Resolution

Option 1: 

  1. Export the postman collection in v2.1
  2. Upload the postman file to https://apitransform.com/ and generate the OpenAPI specification (version 3.0.0) gets generated.
  3. Upload the OpenAPI 3.0.0 specification file to Transformer | APIMatic and generate the JSON Open API 2.0 version.
  4. In Power Apps add a new custom connector based off the Open API file (version 2.0) to generate the connector.

The process is a pain, but this approach does not need any software installed.  I do feel Power Platform are about to improve by accepting the Postman v2 collections and it would be great if it could read Open API file in YAML and JSON in both V2.0 & 3.0 but any of these permutations would be a game changer.

Warning: Large collections end up with information being lost in the translations.

Option 2: 

  1. Export the postman collection in v2.1
  2. Upload the Postman 2.1 Collection file into Transformer | APIMatic and generate the JSON Open API 2.0 version.
  3. In Power Apps add a new custom connector based off the Open API file (version 2.0) to generate the connector.

Monday 16 August 2021

Distribute tracing using Azure Application Insights across Azure SaaS product

Overview:  Building SaaS products using multiple underlying Azure PaaS, and IaaS services with multiple Microservices supporting and calling each other is great.  The issue is we need to be able to trace, debug, and observe the logic flow through multiple Microservice calls.  Distribute Tracing on Azure supports technical players such as devOps teams, developers, support, technical leads and/or architects to find and trace the entire execution to figure our what is/has happened.  Application Insights provides rich functionality on Azure PaaS services.

Distribute Tracing:  A good option for providing consistent traceable logging is to use Application Insights with the Distributed Tracing to trace the flow of each transaction.  The original request generates an Id which is set as the operation_parentId.  Now we can easily follow the execution of a specific operation.

It is fairly easy to tie multiple operations together.  For example, an operation that fires timer jobs, each timer job would be seen as unique operations with their own full trace.  By referencing the original operation when the timer jobs are setup, long running distributed jobs can be tied together.

Thoughts:  Distributed tracing generally catches exceptions from the underlying infrastructure services such as SQL, Azure Functions, App Service on Windows but in code you can add additional tracing information.  The tracing info can be exception based but most of this is picked up anyway or trace base (when an even happens you want to record).

It is a good idea to instantiate the Telemetry client once per service e.g. webAPI and merely call using the same instantiate telemetry object instance throughout each application.

On exception in both client and Server side code write to App Insights telemetry.  Below is the C# server side code snippet:

try

{

    ...

    telemetry.TrackTrace(message, SeverityLevel.Warning, properties);

}

catch (Exception ex)

{

    telemetry.TrackException(ex);

}

More Info:

Distributed Tracing in Azure Application Insights - Azure Monitor | Microsoft Docs

Application Insights API for custom events and metrics - Azure Monitor | Microsoft Docs

Sunday 8 August 2021

Tools for Architects

Overview:  As an architect we use multiple tools, it's a good idea to standardise tool usage especially in larger businesses. It's pretty common to se architects using different tool for drawing architecture diagrams (viso, draw.io,...)

Common Drawing tool I've seen architects use:

  • Visio
  • Draw.io
  • Balsamiq (my odd tool of choice that offers low fidelity, alternatives are Figma which is high fidelity for UI and Ux design, issue is end users thing its the website and get confused, Axure is also a nice Ux prototyping tool).
  • Lucidchart (competitor products: Visio, Miro, Balsamiq, ) Lucidscall can pull diagrams from architecture.  The is also a cool python git project that allows you to specify a diagram using Python and it draws the diagram and you can also then use the Python code to provision the infrastructure.
  • Miro (great for sharing).  Similar to Lucidcharts, with good integration, I tend not to introduce Miros as I use Teams and it's whiteboarding.  FigJam is pretty good for brainstorming from the makers of Figma.
  • PowerPoint (sic. but it happens and some architects are good with it)

Tools for Retrieving Azure Architecture and creating documentation:

Enterprise Architecture Tooling:

  • Sparx
  • Archimate

Dev Tooling I use a lot:

  • Visual Studio
  • SQL Server Management Studio 
  • Postman/Swagger

WIP


Sunday 1 August 2021

JMeter - The basics

JMeter is an easy to use open source load testing tool by simulating network requests.  JMeter is good for figuring out how well the server side responses are working under different test conditions.  JMeter is built with Java and can run on Linux, Mac or Windows using a Java Virtual Machine (JVM). 


JMeter is Single Agent:

  • JMeter runs from the machine it is installed on so it does not have multiple agents.  Saying that it can simulate hundreds of users on fairly low spec machines.  
  • To avoid network latency, test on the same subnet or data center.  A simple VM in Azure (with 2 vCPU's and 8 GB RAM) can mimic over a thousand requests per second.
  • You can run tests off multiple machines to generate extreme loads (first I would use 8 cores and 64GB ram until the network traffic is maxed).
  • Install the Windows JDK 11 before installing installing JMeter.
Updated 2 May 2023: The current version of JMeter is Apache JMeter 5.5

Download and record web tests using the JMeter GUI tool.
Azure Load Testing needs the recorded tests generated by the JMeter GUI.
Create a new Azure Load Test Resource and use the recorded JMX/Test script file.


JMeter GUI
Open /bin/jmeter.bat


Wednesday 28 July 2021

Azure DevOps User Stories Tips

Quick Point on User Stories and Acceptance Criteria.

1.      User Stories description must follow the format:

As the <role> , I want to <feature> so that <benefit>

Note: Always follow the exact same format and bold up the standard/fixed parts for user stories.  Pls keep consistency across your teams user stories.  Under the user story in the description, feel free to add more description, annotated images (very useful) and links to Figma, Axure, UI mocks or Miro.  User Stories should also follow the INVEST (Independent, Negotiable, Valuable, Estimatable, Small, and Testable) breakdown.

2.      Acceptance Criteria (Use Gherkin Language) under the user story (ensure it goes into the User Story section and not comments or the description)

Scenario:
  Given
  When
  Then

Example

Scenario: Employee requests leave
  Given an employee has sufficient leave available in the year
  When the employee schedules leave (holiday)
  Then the employee is informed his request is valid and his manager is informed of the request.

Note: Always follow the exact same format and bold up the standard/fixed parts for user stories.  Pls keep consistency across your acceptance criteria.  I bold and use the four parts as shown above in the example.  You can use "and" to extend the story, just try keep them within the idea of INVEST.

3.      Other:  Order is Tasks belong to User Stories, User Stories belong to a Feature.  Features belong to Epics  These items must be related within Azure DevOps.   Epics > Features > User Stories (Acceptance criteria) > Tasks.

Scrum - Part 1
Scrum - Part 2 
Scrum - Part 3 - Scaling Scrum
Scrum - Part 4 - Kanban and Scrum
Scrum - Part 5 - Certification PSM 1

Azure DevOps - Introduction

Tuesday 20 July 2021

Open Banking & Crypto currency - Capital Gains Tax

In the 2019/2020 & 2020/2021 UK tax years, I bought and sold crypto currencies; I happened to make a little money out of it thru no skill of my own.  "A rising tide lifts all boats".  It has been bothering me so I looked up how taxing crypto works and it falls under Capital Gains. 

UK individuals get £12,300 I am well below the threshold for owing HMRC additional tax.  I still need to report my gains on my SA100/Self Assessment with HMRC using SA108 for reporting Capital Gains.  There is also a great initiative Payment Services Directive 2 (PSD2)/Open Banking offered by companies like TrueLayer that provide API's to get current account statements/transaction using OAuth2 permissions.  PDS2 is pretty useful for anti-money laundering (AML), specifically Know Your Customer (KYC) that falls under AML.

It is worth using this capital gains allowance if you can get solid returns off applicable assets such as shares or cryptocurrency.

"When you dispose of cryptoasset exchange tokens (known as cryptocurrency), you may need to pay Capital Gains Tax.

You only have to pay Capital Gains Tax on your overall gains above your tax-free allowance. The Capital Gains tax-free allowance is: £12,300".

TrueLayer and Token is a service that connects to all UK commercial banks instead of you having to individually connect to the banks Open Banking API's.

References

Payment Services Directive 2 and Open Banking | UK Finance

https://www.gov.uk/capital-gains-tax/allowances

Elastic Database Client Library for client database segregation on Azure PaaS for SaaS

Overview:  Provide a logically separated database instance for each client on my SaaS solution.  Using the Elastic Database client library from Microsoft on Azure PaaS services provides logical security separation of data, performance is on a per customer, and easy scalability.  Use Azure SQL Elastic Pools (HA redundant secondary database, built in DR).  Also add temporal tables for a full history of all transactions.

PoC:

  1. Provision 3 databases - A Shard Map Manager (Catalogue) database and 2 client databases (tenants/shards).
  2. Add shard related metadata to the Catelogue database for each of these databases.
  3. Create below Three service principals in Azure AD: 
    • Management Service Principal: for creating shard metadata structure.  A database contained user in Shard Map Manager db and each tenant db.
    • Access Service Principal: to load shard mapping at application side.  A database contained user in Shard Map Manager db.
    • Connection Service Principal: to connect tenant database.  Database contained user in each tenant db.


                        Management Service Principal: for creating shard metadata structure

CREATE USER [shard-map-admin-sp] FROM EXTERNAL PROVIDER

EXEC sp_addrolemember N'db_ddladmin', N'shard-map-admin-sp'

EXEC sp_addrolemember N'db_datareader', N'shard-map-admin-sp'

EXEC sp_addrolemember N'db_datawriter', N'shard-map-admin-sp'

GRANT EXECUTE TO [shard-map-admin-sp]

 

Access Service Principal: to load shard mapping at application side

CREATE USER [shard-map-access-sp] FROM EXTERNAL PROVIDER

EXEC sp_addrolemember N'db_datareader', N'shard-map-access-sp'

GRANT EXECUTE TO [shard-map-access-sp]

                                                         

Connection Service Principal: to connect client/tenant database

CREATE USER [tenant-connection-sp] FROM EXTERNAL PROVIDER

EXEC sp_addrolemember N'db_datareader', N'tenant-connection-sp'

EXEC sp_addrolemember N'db_datawriter', N'tenant-connection-sp'

EXEC sp_addrolemember N'db_ddladmin', N'tenant-connection-sp'

GRANT EXECUTE TO [tenant-connection-sp]


References:

https://docs.microsoft.com/en-us/azure/azure-sql/database/elastic-database-client-library


Saturday 10 July 2021

Modeling and working with data on the Dataverse within the Power Platform

The Common Data Model - Is an industry agreed approach to storing commonly used data.  Use it to store applications across your company/organisation.  Results in a single source of truth instead of multiple copies with data contained in different schema's.

Two types of relationships namely: 1:N and N:N (Dataverse hides the associate entity/intersect entity that is created in the background).  

Use "Option Sets" for small static data and use "Lookups" for larger or changing data.  In Multiselect Option sets for N:N relationships, rather use "Lookups".

SSMS can be used to view data using T-SQL, better to use "SQL 4 CDS" as it provides a full ability to work with data within the XrmToolBox.

XrmToolBox - is a 3rd party download that has a ton of contributed tools and there are good options for modelling data.  This is a collection of tools that are unbelievably useful and get continually updated and new tools added.

Excel and browser plugins - Can be used to import/export data and there are some nice Edge/Browser plugins to help such as "Level up for Dynamics 365/Power Apps".  Level up for Dynamics 365/Power Apps is a fantastic tool that I encourage any developer to add to Edge or Chrome as an extension.

Level up for Dynamics 365/Power Apps Browser extension
Dynamics 365 Power Pane Browser extension.

Dynamics 365 Power Pane is also a useful extension shown below:
Power Pane options

The built in browser Dataverse management tool is super easy to use to model your Common Data Model further.

Wednesday 7 July 2021

Microsoft Dataverse (CDS) - Overview

Overview:  Dataverse is CDS, there is a long story on the naming but ultimately Dataverse is a data store with a advanced security model, Open API's, workflows, pipeline injection...  It is awesome.

It is high performance, and would take considerable effort and components to deliver similar functionality or even semi close functionality.  It does have limitations mainly around performance but don't let that fool you, Dataverse is fast and powerful but for massive industrialized storage it's not the right option.  The costs are also a key consideration.

The biggest mistake I see is people making the same mistakes as they do with relational databases namely: 

Poor Dataverse implementation down to 1) poor entity relationship design, 2) either too many table containing duplicate data or to few table being expanded for a dev teams capability but ignoring existing systems, 3) poor security 4) too many cooks.

Basically, like any Database service, you need to have owners and try keep the structure logical and expand it appropriately.  The idea behind the data model used by the dataverse is to have centralized secure shareable data like customers or account information.  It's simple, treat dataverse as you would your most precious core database, have an owner that needs to understand and approve changes.

Note:  Microsoft have had some trouble naming Dataverse, it was previously known as the Common Data Service (CDS).

Dataverse logo

Overview: Dataverse helps improve processes.  And Dataverse helps reduce time to build IT capability, remove shadow IT, improve security and governance.  Data is the common data store we need to use to be effective.  As part of the Power Platform, it allows us to build custom software fairly quickly.

Updated 07-July-2022

Dataverse provides relation data storage (actually runs on Azure SQL (Azure Elastic Pools), Cosmos DB, and Blobs), lots of tools e.g. modelling tools.  I think of it as a SQL database with lots of extra features.  Most importantly business rules and workflow.

  • Dataverse relies on AAD for security
  • Easy data modelling and supports many-to-many relationships NB!
  • Easy to import data using PowerQuery compatible data sources
  • Role-based data (previously called row) and column (previously called fields) level security.  See Dataverse security in a nutshell at the bottom of this post.
  • Provides a secure REST API over the Common Data Model, it's awesome
  • Easy to generate UI using PowerApps model driven app
  • Ability to inject business rules when data comes in or out of the Dataverse (can also use .NET core code)
  • Can also stored files (ultimately in blob storage)
  • Search that indexes entities and files
  • CDS used tables, Dataverse calls them Entities.  Some of the UI still refers to table.  Just assume Entity and Table are interchangeable terms.
Dataverse basically allows you to have a PaaS data hosting service that mimics what we have done for many years with databases and Open API, has advance features and tooling and it is all securely managed for you.  

The cons are basically: is that it is expensive.  So you need to know your size and keep buying add-ons to the plans.  Scaling Dataverse is expensive.

Common Data Model: Collection of tables (previously called entities and most CRM people still call them entities) e.g account, email, customers for a business to reuse.  Comes from CRM originally, the starting point consists of hundreds of entities pre-created.  Common standard for holding data.

Each Power Platform Environment has a single Dataverse associate to it.  It's a good idea to have more than one environment but at it's simplest, use a trial to learn and progress to production.

Once I have a new environment, I can use Power Apps to access my environments Dataverse and model out a new table to store info, I am storing people tax returns.
Go into the Dataverse and model directly

Model the table in you Dataverse instance

Dataverse Security in a Nutshell:
  1. A user is linked from AAD to the User entity in the Dataverse.
  2. User Entity record is aligned to the AAD User.
  3. AAD Users can be part of AAD security groups.
  4. Dataverse Teams (Dataverse Group Teams) can have Users and or Security Groups assigned.
  5. Dataverse Group Teams are aligned to Business Units.
  6. Business Units have roles (rights).
"Security is additive" in Dataverse (generally the whole MS and security world these days).  i.e. no remove actions.  If you have permission in any of the groups you can access the data/behavior.

Business Units used to restrict access to data.  Can be hierarchical i.e. Enterprise > Audit > EMIA > UK (Don't use it like this, keep it simple)
Security Roles define a users permissions across the Dataverse entities i.e role can read only from Accounts entity 
Teams consist of users and security groups.  That get assigned roles.  There are two types of Teams in Dataverse: Owner teams & Access Teams
Field-level security, only allows specified users to see the field data

https://learn.microsoft.com/en-us/power-platform/admin/wp-security-cds (Good clear post on Dataverse security, core concepts are Business, Units, Teams, Roles, Users & OAuth/AAD)

Sunday 4 July 2021

UK Tax - MTDfITSA

Update 21 December 2022:  MTD for ITSA delayed 2 years, will now start 5 April 2026, and applies to a way smaller base.   Theses are the key changes: 

Starting 6 April 2026 shall apply to roughly 740k users, people that need to register have a combined Self employment and property income of over £50k.  From April 2027, the threshold drops from £50k to £30K and will increase the user base to roughly 1.6 millions people.

  1. There is no plan for Partnerships to fall under MTD for ITSA.
  2. It's all subject to change, and it's likely the threshold will also be applied to lower threshold self employed people.
  3. HMRC are looking at the API's, no road map but will put it together at some point.  The Sandbox will be reviewed for API testing (let see what comes out).
  4. The new penalty points system only starts in 2026.
  5. MTDfCT does not have any dates i.e. the 2026 start has been scrapped and no new date set.  HMRC plan to do MTDfCT at some point.
  6. There is no change to the Basis Period Reform (BPR).

Making Tax Digital (MTD) for Income Tax Self Assessment (ITSA) is the replacement process to HMRC's Self Assessments (SA) that starts 6 April 2024 6 April 2025.  Of the 12 million UK people registered for Self-Assessment, roughly 40% are mandated to switch over to MTDfITSA on the 6th of April 2024 (circa 4.2 million individuals).  A combined gross income of over £10k from people with either a self employed business or a property business will need to register.

Are you or you self-employed client in?

Do you have over £10k income from self-employed businesses, then yes, from April 6 20224 you'll need to be using digital records to do quarterly submissions.  You first MTD for ITSA submissions are due submission for the quarter 6 July 2024 and 5 Aug 2024.  

What software options are the options?

There are approved vendors on HMRC's website for ITSA submissions, everyone has to submit via approved software vendors.  Anecdotally, Free Agent, Sage, Xero and QuickBooks are going to offer Quarterly self assessments submissions tied to their bookkeeping products.  Free agent is good for small customers, keep it simple.   I also quiet like what I have seen from Sage for accountants coupled with bookkeeping.  Personally, I've use QB for many years and played with Xero, both are excellent products and are strongly focuses on have MTDfITSA quarterly submissions.

There are software vendors that shall also look at bridging software solutions which will be more useful to self-employed businesses that support spreadsheets and are likely to be used by individuals to keep their costs down.  Accounts are likely to look at TR, Sage for Accountants, Capium, BTC, and WK for full practice management tooling including MTDfITSA, these vendors providing full suite solutions are likely to offer bridging and retrieve from source (booking and open banking) options for holding the digitally linked data.  Coconut looks interesting for individuals.

How do I enroll a client on MTDfITSA pilot?

Call HMRC and get client registered with HMRC for the pilot.  Software can be used to assign client to their accountants software. 

Is there a new penalty system coming in?

There is a new penalty and fining systems, the amounts of returns and work to complete an ITSA is significantly more than the single once a year SA.

Summary of comparison between ITSA & SA:

SA

ITSA

1 SA return done each year per person.

4 quarterly returns via approved software per business, required a digital record link to the underlying transactions per self-employed business

1 End of Period Statement (EOPS) per business.

Crystallization/Finalization using HMRC approved software per person

Paper submission due 31 Oct for previous year.

Online submissions due 31 Jan – 9 months after financial/tax year end.

Approved Digital Software to submit (no paper returns).

1 month after each quarter, submit the quarterly return, these can be revised later.

Each EOPS is due on 31 Jan the following year.  

The Crystallisation/Finalisation/Final Declaration is also due 31 Jan the following year.

MTDfITSA has been pushed back twice (although it is in pilot) and it's scoped changed.  ITSA follows on from Making Tax Digital for VAT (MTDfV) that is already live.  

HMRC use paper for self-assessments, then allowed XML submissions, and is starting MTDfIT to enable individuals to do their own Self-assessments.  Many people doing Self Assessments will be affected, and the associate costs & time shall be higher.  Just under 40% of people file SA themselves. If they need to be on MTDfITSA, it is highly likely that these smaller businesses will be severely impacted.  I believe Accountants that are doing small business self-assessment in the £400-£1,400 range will probably need to double or more these fees.  It all comes down to will the client do the quarters and the account just does the final submission, in which case there should not be much of a cost change.  Realistically, some clients will need booking to record the transactions and others will merely need guidance and year end adjustments from the accountant.  SA submissions start at about £120 (ex Vat) for simple Self-Assessments, these individuals, if they fall under ITSA shall probably be charged at least triple the amount compared to a TA doing all the work under self assessment or need to do considerably more work themselves.

MTDfVat took several years to implement (April 2022, all businesses, including those registered for MTDfVat but under the VAT threshold, must use MTDfV), and MTDfITSA is way more complicated than MTDfVat.

MTD for Corporation Tax (MTDfCT) is scheduled to start April 2026.  A pilot is scheduled to begin in April 2024 (unlikely).  Update earmarked for 2026, likely to be later.

MTD for ITSA (referred to as ITSA or MTDfITSA) 

  • Who is in - Individuals and Partnerships.  Require either trading income from self-employed businesses and/or property businesses (if you have either of these business types), if their combined revenue/income total is over £10K.  (To determine, need to look two years' back at the individual's self-assessment income).   If not trading in the two years previous means you are outside MTDfITSA.  I.e. new self-employed businesses would get two years grace before they need to file using MTDfITSA.  Unlikely to be enforced by penalties.  The mandatory start for quarters is 6 April 2024. Must have combined property and self-employed (unincorporated) business revenue over £10k combined. 
  • Partnerships are in if all partners are individuals (starts 6 April 2025).  I think there is a disconnect between the draft regulation and what HMRC are providing.  Partnerships declare income using SA800 for a partnership.  Then each individual partner has to do a self-assessment.  There is no way partnership income counts as self-employed income so partnerships are not a criteria for MTDfITSA.  Need to confirm what the "General partnerships with income over £10K that have individual partners need to join MTDfITSA" refers to, I suspect it is incorrect. 
  • Trusts/estates, LLP and Ltd are out at the start.  
  • 1.3 million landlords in the UK as of 2021.
  • Roughly 40% of Self Assessments are filed by individuals for themselves.  This is going to be tricky with MTDfIT.  If these individuals wish to file, they will need to do the quarters, have digital records, and use free software (HMRC are not providing MTDfIT free software).  They will also need to do MTDfIT EPOS and crystallisation.
  • Individuals doing Self Assessment can have ten or many self-employment businesses, 0 to 3 (4 - Foreign property is not clear) property businesses under MTDfITSA (Ord UK property, FHL UK, FHL EEA, Foreign property).
  • One can defer MTDfIT by changing year ends.  Individuals/practices also can apply for exclusion based on age, religion...
  • The HMRC ITSA service is only able to accept end clients who have accounting period aligned to the tax year (I believe this means 31 March or 5 April).  Basis period reform is coming and will sort out this issue.
  • Digital links/keeping digital records.  Can't re-key/copy and paste.   There is no requirement to use bank feeds/PDS2 data.  Some booking software firms are likely to file quarterly MTDfIT returns for each self-employed (and property) business.  Spreadsheets are an acceptable form of record keeping.   Excel and bridging software is sufficient for the source for filing.  If recording sales can use daily sales totals for the digital source but ideally link to the raw input system.  
  • PDS2/bank feed - unlikely to work, useful for bringing in data.  The issue is that the bank account would need to represent a single business; also, the mapping would be very rough into the MTDfIT categories.  Things like purchases from Tesco can't understand where to place the transaction.  As MTDfITSA quarterly is a rough estimate.  It is possible as an option.
  • Quarterly MTDfITSA is done per business and is due 1 month after quarter period ends.  Penalty points for late filing, missing 4 quarters in 24 months, is £200 penalty.  Record of last 24 months retained.  Property business quarters and year end run in the same cycles as personal tax namely start 6 April and end the following year 5 April.  As MTDfIT starts on 6 April 2024 (was 6 April 2023 but postponed again notice provide on 22 Sept 2021), the 4 quarterly submission for the 2024-2025 tax year and filling due dates are:

Qrt start date

Qtr end date

Qtr submission due date

6 Apr 2024

5 July 2024

5 Aug 2024

6 July 2024

5 Oct 2024

5 Nov 2024

6 Oct 2024

5 Jan 2025

5 Feb 2025

6 Jan 2025

5 Apr 2025

5 May 2025

  • Self Employed businesses with a year-end of 5 April or 31 Mar, are treaded as using the same period dates as property businesses'.  The majority of Self-employed businesses are already aligned with personal tax year ends and are "in".  If you tax year is outside of this i.e. 31 Dec each year you can't use MTDfITSA.  
  • Each property business for the individual needs to submit an End of Period Statement (EOPS) each year and also a declaration for each business attached to an individual.  
  • Lastly, the Final "Crystallisation" (basically the final Self-assessment) needs to be submitted covering the self-employed business, the property businesses and the personal tax affairs of the individual.
  • Late payment has interest penalties, basically no penalty for 15 days late, then 2% for 16-31 days and then 4% is paid after this. 4% is on the outstanding balance from day past due.  Payment is due following year 31 Jan.
  • MTDfITSA adjustments can be done after the quarter and resubmit or at the EOPS.
  • Quarterly submissions are estimates and can be adjusted per quarter or finalised at year-end for each business.  Revenue could ask for proof of digital linking, so you can't just throw in a rough estimate.
  • After every quarterly obligation is submitted, the response gets a CalculationID, that can be used to view the end clients "year to date tax estimate" for the business.
There are basically Two possible flows for completing ITSA Tax returns:
  1. Full Process/Accountant/Practice/TA has booking data and does the quarters for each trade/business, the practice then does the end of year pieces also, the issue is it will be expensive.
  2. Client Individual Driven Process is where the end user shall submit their ITSA quarterly returns, and then use a practice to submit the business/trades end of years and then the individuals end of year/Crystallization. 

High-Level Process for TA (full process):

  1. Clients sign up to MTDfITSA on HMRC using their Government gateway credentials, assuming they meet the criteria (Agent if authorised to act on the clients behalf can also sign the client up to MTDfITSA).
  2. Clients must maintain digital records/digital linking, and it must be digitally linkable (spreadsheets are acceptable, min req allows for daily totals in a spreadsheet).  Need to record the transaction date, category and amount.
  3. Perform quarterly submissions per registered unincorporated business.  It can be submitted up to 1 month after the period ends.
  4. EOPS submission per business.  Due 31 Jan following year.  HMRC will return the tax calc for the unincorporated business.  Also, a declaration for each business must be completed by the TA or individual.
  5. Final Crystalisation & Declaration (HMRC does the calculation for the individual) are due 31 Jan following year.
  6. The payment date remains the same for Tax liability due 31 Jan following year.

Note:  All opinions are my own, and I am not a personal tax expert.

Numbers:
  • 32 Million Income tax payers, 4.1 million people are higher rate tax payers.
  • 450k people on additional rate (45%).
  • Median income per UK individual is £25k (males £27,400 females £22,200)
  • +-35k Tax & Accounting Practices in the UK (rough estimate).
  • 12 million people do self-assessments in the UK.
  • MTD ITSA will impact around 4.3-4.7m people (Self employment & landlords).
  • MTDfITSA affects 1.3m individual landlord.
  • HMRC recon they will gather an extra £9 billion per year.
  • HMRC estimate 1 million MTDfITSA customers will quality for the free software from vendors (less than £85k turnover and using cash basis accounting).
  • of the 12.2 million SA's due for the year ending 2020/2021, 10.2 million were filed on time by 31 Jan 2022.  It gets pretty crazy in filing day see HMRC's info.
  • HMRC recon that +-8 million of the 12.2 million 2020/21 self assessment (SA) tax returns have been filed as of 24 January 2022 (1 week to 31 Jan deadline, with Feb no penalty extension in 2020/21 year).  Interestingly the year before 9 million we filed at the same point the year before.
  • The pilot ITSA HMRC sign up is extremely low.  With HMRC estimating 4.3 million ITSA user sign-ups for the year starting 6 April 2024.
My Thoughts 10 Sept 2021 on ITSA:
HMRC have a lot of open questions both on the mechanics of ITSA and the API for MTDfITSA.  The regulation should be finalised +-end of Sept/Oct.  There are very few people that are eligible for the Pilot 6 April 2022 and very few accountants wanting to put their client on MTD.  My personal opinion is that HMRC will end up delaying the start for ITSA as they did for MTDfVat until is if more flushed out.  The API sandbox is not fit for pre-production however, all the vendors are doing best endeavors.  So not impossible to keep the existing timelines but i do feel there will be huge changes to ITSA coming along shortly from Revenue.

Self Assessment filling options:
  1. Most people use the current XML online filing done 31 Jan after the personal tax year.
  2. Some people still use paper based self-assessment due Oct after tax year,
  3. MTD for ITSA will be due 1 month after each businesses quarter, EOPS and finalization/crystallization process due 31 Jan the year after the personal tax year.