Building better Software Thoughts

Overview:  I see a lot of development teams, and they always seem to have areas they are good at and capabilities teams need improvement on.  Key is culture and building a happy team where team members trust and help one another.

Building a culture where teams enjoy code reviews is also key for successful Software projects.   To improve software, reviewing various areas not only code reviews are essential.  For me, clear requirements are the number 1 factor in improving teams performance.  

Companies are getting better at building software; I aim to work on these topics to improve the delivery of software within scrum teams:

1. Code Reviews & Peer Reviews (Daily reviews are awesome, should be pretty short and enjoyable not someone trying to show off or hours long)
2. Collaboration (Standups, Slack/Teams, Code tools have collaboration built in)
3. Documentation & Requirements Reviews
4. Better tooling including better CI/CD tooling including static analysis tools
5. Unit Testing, automate coding standards, Integration testing, UI Testing, and API testing 
6. Requirements (Use Stories are clear and Acceptance Criteria)
7. Cadence is improving thanks predominately to Agile practices; I like short release cycles (2-3 weeks depending on the team and industry).  Changing requirements, indecision kills software projects.  Agile helps, but decisive knowledgeable product owners increase the likelihood of the project succeeding.

Benefits of Code, Documentation and Requirement Reviews:

1. Improved software quality & product delivery
2. Share domain knowledge
3. Training team members (useful for onboarding new team members)
4. Reduce support and fix costs
5. Lower cost & faster development

Options Layering API's on Data Sources - Micrososervices kind of

Hasura takes data sources such as SQL, Postgress & MySQL and converts it into GraphQL API's.  SQL Server is in preview.  Service is available on Azure and hooks into AAD and AAD B2C.  Hasuru looks extremely interesting and useful.  Potentially a great time saver.

CDS/DataFelx/Oakdale - Allows for Entity creation and provides REST API's.

SharePoint lists provide HTTP API's for CRUD operations.

REST API's vs GraphQL

OpenAPI specification (previously known as the Swagger specification) is my default for an API, this allows for a known RESTful API that anyone with access can use.   Open API has set contracts that returned defined objects which is great, you can work with the API like a database with simple CRUD operations as defined by the specification.  The issue is that the returned objects are fixed in structure so you may need 2 or more queries to get the data you are looking for.  Alternatively, GrapghQL allows the developer to ask for the data exactly as the want it.

Open API example:

/api/user/{2} returns the user object  // Get the user object for user 2

/api/users/{2}/orders/10  // Returns the last 10 orders for the user

GraphQL example:

Post a single HTTP request.

query {

 User(id: "") {

    name

    email

    orders(last: 10 {

      orderid

      totalamount

      datemodified

    }

 }

} 

Database Option Notes:

PostgreSQL - Hybrid relation database (Communitity edition free, standard and enterprise higher loads) sits on Linex but can also use Windows OS.  Replicas are used for additional read-only nodes (I think up to 5 geo relicas allowed) 

Can be hosted on Azure or AWS IaaS.  PostgreSQL is also offered as a fully managed service (PaaS) on Azure (either single server or hyperscale).

MariaDB community fork off MySQL - Open source relation database used in the LAMP app stack.  Azure MySQL manage service can have up to 5 replicas for read heavy workloads.

Working with CDS data structures for non CRM types

Overview:  I am working on a Power Platform solution and I need to use CDS.  Basically, I need to be able to see and edit values within CDS.

Option 1: Microsoft SQL Server Management Studio (SSMS) version 18.6 allows connectivity and read-only access.  Here are the instructions.

Option 2:  XrmToolBox has fantastic tools for Dynamics and Power Apps.  There are a lot of individual tools from various contributors.

Here I am using "Entity Relationship Diagram Creator" to look at the relationships between the CDS entities.

Reducing Power Apps Dynamic calls and where to store Power Apps data.

Overview:  Power Apps is driven by data and generally that data comes from Connectors.  So the great news is there are a lot of different connectors and if in trouble I always find the custom connector can be relied upon.  When working with Power Apps, it is not as simple as just having a data source and consuming it, one needs to consider all the data sources, do we need live data, performance.  Basically, going and dynamically pulling lots of dynamic sets of data repeatedly leads to poor performance.

CDS performance has relatively few layers when retiring or updating data so it generally is fast to work with.  Custom connectors have more layers so can be slower and you are dependent on the underlying REST API so watch for performance.  The on-prem azure data gateway is pretty slow but amazing if you need it.

Identify data sources:  CDS, Azure SQL, SharePoint SQL on-prem., CosmosDB, RSS, Open API's...

Understand the security, and the amount of data being pulled.  For example, if we need all the airport codes in the world for a drop down so the user can choose their closest airport e.g. JFK is for New York John F Kennedy airport.  There are roughly 4000 commercial airports in the world. 

Options: Call an Open API service.  Power App by default returns sets of 500, Power Apps max return count is 2000.  You still need to perform 2 calls with paging to get the full data set.  You could use a type ahead if the API supports it, but their will be a lag after each keystroke when Power Apps runs out to the service.  And there will be a lot of calls.  More suitable would be to do 2 calls with 2,000 record for each call and bind the control to the returned data.

A further improvement would be to store the airport lookup on data load or on first request, then subsequent requests would use the table/collection.  In effect, you are locally caching all the airport codes using 2 calls for each Power Apps user session.

For the airport example, one could also store the data using a Excel import, but beware the data is imported into Power Apps and store locally.  Big issue is the data is static in Power Apps, to update it, you need to re-import the Excel table.  So brilliant for sat days of the week or Months of the year as these never change.  Fairly static data like airport codes work well, but require a publisher level overwrite to update the list.  Also, storing extreme amounts of static data leads to bloat of the app and that data still needs to be loaded.  So I would not consider it for 100k+ items as a general rule.  

FYI Microsoft recommends to use less than 30 connections per application.

More Info:

Todd Baginski has a great video on using Excel import and creating language variations/multi-lingual Power Apps using Excel imports.

Alagunila Meganathan on C# Corner has a good post on Excel Imports for Power Apps.

AWS vs Azure services offering comparison for Solution Architects

Overview: Microsoft provides a useful list that allows me to know AWS services aligned to Azure Services.  This is pretty useful if you know 1 platform considerably better than another to quickly figure out your options on either AWS or Azure.

My Service comparison notes:

Amazon CloudWatch - same as Azure Monitor.

Amazon Relational Database Service (RDS) – SQL Server, Oracle, MySQL, PostGress and Aurora (Amazon’s proprietary database).  

Azure SQL lines up with Amazon's RDS SQL Server Service.  Although Aurora is probably also worth the comparison as it's AWS's native DB option.

Amazon DynomoDB – Same as CosmosDB – NoSQL database.

AWS API Gateway - Azure API Management

Amazon Redshift is the data warehouse.  Can be encrypted and isolate.  Support Petabytes of data.

Amazon ElastiCache run Redis cache and MemCached (simple cache).

AWS Lamda – Azure Functions. i.e. Serverless.

AWS Elastic Beanstalk – Platform for deploying and scaling web apps & Services.  Same as Azure App services.

Amazon SNS – Pub/Sub model – Azure Event Grid.

Amazon SQS – Message queue.  Same as Azure Storage Queues and Azure Service Bus.

Amazon Step Functions – Workflow. Same as logic apps

AWS Snowball – Same as Azure Box.  Physically copy and transport to data centre for upload.

Virtual Private Cloud (VPC) – Azure virtual network 

Amazon AppStream - Azure VDI (Virtual desktop) I think.

Amazon QuickSight - Power BI (Tableau Business Intelligence).

AWS CloudFormations - ARM and Bicep

Tip: I am glad that I did the AWS Certified Cloud Practitioner exam as it helped my understand of the AWS offering which has been very useful in large integration projects.  I have worked with AWS IaaS (EC2, API gateway and S3 historically).  Like Azure, there are a lot of Services and features.  Basically, there are equivilant services for Azure & AWS.  It may be a 2-to-1 service offering or it is not something offered by the cloud provider.

AWS vs Azure vs GCP Comparison

Overview:  I predominately use Azure & Microsoft for all my cloud services.  

Source: https://www.cloudhealthtech.com/blog/aws-vs-azure-vs-google

I have installed multiple SharePoint farms and setups on AWS EC2 instances and I'm currently preparing for the Cloud Practitioner AWS exam.  I have used Google for authentication, SaaS nut not as a IaaS offering.  I'm also a huge fan of Heroku which is great for PaaS and I used this to host my game built for Facebook games.  I've also seen IBM's cloud offering a few years ago.  For me it is too niche and not as feature rich.  So basically I understand Azure's offering well so I found this comparison pretty useful.

My Thoughts:  The contenders:  I really like Heroku for it's simplicity.  I feel for a small Indie developer or company, Heroku has a good free and cheap simple billing options.  GCP, I really can't comment from a good position of knowledge but from what I've used, I like GCP.  GCP is the third biggest Cloud provider.  As a large organisation, I'd only consider the big three: Microsoft Azure, AWS, and GCP to be our cloud partner.  Multi-cloud partner is a demand from some organisations, it's truely extra expensive.  Azure uses ARM templates and has many options for provisioing the IAAS, PaaS offerings.  If you are thinking multi-cloud consider Terraform by Hashicorp for IaC.  There is also the concept of Click-Ops (sic) which allows you to click thru the UI of the management of the Cloud services to get the the desired architecture, this is fine for simple small architecture but you can't do this at any scale or agility and it's super error prone.  Click-ops is more a joke term for the laziest way to build infrastructure and we need to make it sound modern.  IBM's offering, well if you are a partner, you cloud go with this option but it is aimed more a large business partners.  IBM's cloud is IaaS focused, with some PaaS offerings but once again I'm not an expert.

AWS, has always been really easy to use.  It is big and complex like Azure with many offerings.  Basically, I'd choose AWS if the organisation was already using it and the people in the org know have experience with AWS.  AWS originally was aimed at the B2C/startup market but was first to market at scale.

Azure, so in my world Azure and O365 feel like the dominant player but the diagram below provides a great insight into the relative size of the Cloud infrastructure market.  Azure SaaS offering O365/M365 is also huge and hosted on Azure.   Azure security is well thought out and their thinking on BYOK and geo-location appear to be important.  Microsoft offer Arm templates and DSC for configuring environments, they are also adding Bicep which is an abstract layer that will run ARM templates into Azure.

Source: https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/

There is good resource CloudWars.co that goes into looking at the various cloud providers.  My current take away is Amazon is the biggest player in the IaaS field.  Azure has IaaS, a large PaaS offering and a massive SaaS (including Dynamics and O365) offering (Amazon has no equivalent).  I am focused on PaaS solutions for my customers so as to remove the infrastructure and process overheads of IaaS.

Off the top of my head reasons for moving and objections I hear for the cloud regardless of platform:

Why Cloud:

1. Save Money
2. More Secure
3. Fast Delivery/More Agile/Easy to scale/Increase business resilience
4. Eco-friendly

Challenges:

1. Lack budget
2. Spiraling costs
3. CAPEX model vs OPEX is business common norm that some business find difficult to switch
4. Resources/Skills
5. Believe security is an issue/Don't trust the Cloud
6. Migrate legacy apps (for me don't move to the cloud unless you get significant advantage)

Onboarding staff and New Starters for SaaS

Overview:  With large SaaS projects, various stakeholders tend to start at different times.  For me it takes a while to get all team members up to speed with: domain knowledge, technical knowledge, our design and big picture, company specifics. 

Thoughts:

In person and Teams/Skype meetings are great for building relationships and getting the most information to the new starter.

Record presentations, then shorten and annotate the presentation to make focused high impact onboarding material that can be reused across various roles.

For each starter, ensure they have an introduction/onboarding plan:  Book meetings, get the people the background, have resources e.g. websites, mp4. glossary of terms ready:

Resources:

List out useful resource specific to the role.  I tend to keep these in SharePoint but as long as content if searchable and consistent and of value, I am in the right place.