Wednesday, 22 September 2021

Azure Synapse Intro

I have not done any serious big data work in many years and I'm using this post to understand the Azure options and update my skills.   

Azure Synapse Analytics - built for limitless scale for unstructured and structure data for big data  - supports petabyte analytics.  Ingest and prepare data for BI.  Instance can be a dedicated or serverless PaaS service.  Data lake Gen 2 assigned to each synapse instance.  GA Dec 2020.

Use to be Data Lake (massive, semi structured data) and Data warehouse, Azure Synapse Analytics can be referred to as Data Lakehouse.  Basically all the sperate pieces are under 1 umbrella i.e. ADF, SSIS, Dataware house,..
  • SQL Serverless Pools is pay be use/pay per query.  Can use SSMS to manage data.  Good for small or new data ware housing.  T-SQL works perfectly so simple for smaller projects or PoCs. 
  • Go from 3NF in relational data to using Fact and dimension tables to put data into a star model for MPP.
  • Data Warehouse Units (DW) - Same as DTU, made up of CPU, Mem, IO.  Simple to upgrade.  Lowest is DW100 can be huge, and simple to scale up and down.  More DW's means more VM's/work process work on the data.

Azure Synapse Studio  - SaaS development experience provide code free and code first (C#, Scala, Python, SQL, Java), notebooks are used for working with data.

Wednesday, 8 September 2021

Observability in Azure SaaS Solutions

Problem:  Software has many places where errors and tracing is logged to.   Support get an incident, they need to investigate figure out how widespread the issue is and then try patch together various logs to figure out the problem.


Observability is not a new concept, we need to be able to: 1) view and connect logs & 2) tracing and view metrics & notifications.

Implementing Observability must cover:

CI/CD allows devOps teams to find issues early using Unit testing.   Automated testing on UI.  API automation testing is also great.

Azure offers continious monitoring by performing various API calls to ensure your servie is running and any failures are picked up hopefully before any customers are aware.  You can also be notified of performance slow down, check performance speed between releases.  Which is great for identifying bottlenecks and with the Azure PaaS world, it is easy to increase the processing causing the bottleneck.  

Performance metrics built into the CI/CD and developers work allows us to identify issues early and costs miles less to correct early. 

Security and LINTing in CI/CD also allows us to pickup issues early and correct at a way lower cost.

Instrument you hardware and software, well on Azure you can use App Insights and you have a fantastic instrumenting platform that captures events.  A big reason to use Azure Services for as many of the function pieces in your solution.

Work In Progress ...

TBC Azure App Insights detail, ParentOperationId, Linking operations with a ServiceBus or work process call.

Sunday, 5 September 2021

R&D UK tax Credits for Software development

Overview: R&D incentive claims with HMRC are designed to attract and encourage investment in R&D in the UK.  By providing for a taxable credit at 13% on qualifying expenditure, namely: Staff costs, consultants, Software licenses, hosting.  Claiming R&D costs reduces the companies tax liability and can be attractive for starting a new business or building new SaaS products.

The idea is:

PoC, Concept Design, software design and initial testing are likely to be deemed as probably qualifying for R&D claims.  The product/service must seek to advance (new or appreciable improvement of a process, product or increase technical capability) technology.

Specification/Requirements, scale up to commercial release, training may be qualifying activities. 

Thoughts:  R&D relief claims are complex.  There are different rates for SME vs large corps and rates if the company is in profit.  It is worth doing, and I believe it is pretty common to pay experts based on a percentage of R&D claim.  The big players have divisions to do R&D claims, i.e. KPMG.  There are also tons of specialized smaller firms that will do R&D claims.  For a SaaS startup, I'd contact a specialist provider to make the R&D claims and agree to a fixed percentage of the savings.