Common Software Architectural Patterns

The goal of Solution Architecture is to:

Have a common understanding of how a technical solution shall be reached, the diagrams are useful too have a communicable roadmap and ensure it is complete.  Later the diagrams are used to ensure all relevant parties have a clear unambiguous shared understanding of the IT solution.

The main tools to communicate the architectural solution design are diagrams & documents that utilise common previously used and understood patterns to ensure a safe, scalable, stable, performant, and maintainable solution.  "4+1 view, which includes the scenario, logical, physical, process, and development views of the architecture", source.

Below are patterns and thoughts that I have come across and used to solve building high quality solutions.

3/N Tier Architecture/Layered:
1) Presentation/UI layer
2) Business Logic
3) Data Layer/Data source
Here are a couple of possible example over the years you could of used
ASP > C++ Com > SQL Server 2000
ASP.NET (Web Forms) > C# Web Service (XML/SOAP) > SQL Server 2008
ASP.NET C# > C# Business Object Layer > SQL Server 2008
KO > MVC > SQL 2012
Angular 3 > C# Web API (swagger contract) > SQL 2016
REACT.JS > Node.JS > Amazon Redshift
UI > Azure Functions/Serverless > SQL Azure
Flutter > C# Web API .NET Core 3 (swagger/OpenAPI) published on Azure App Service > SQL Azure/Cosmos

API's:  Over the years we have seen many different API's at a high level:  

• Proprietary formatted API's > 
• XML with SOAP coming out of XML based API's > 
• REST/JSON (other popular formats are: RAML, GraphQL > 
• Event Driven API's may be the next big jump.

Thoughts:  As time has progressed, scaling each of these layers has become easier.  For instance Azure SQL has replication and high Availability and scalability automatically built in.  No need to think about load balancing in depth.  Plug and play and ask for more if you need it.
Microsoft SQL Server use to be a single server, then came replication, clustering, Always-on-availability, scaling greatly improved performance.
Middle Tier or Business layer use to be a singleton pattern - go thru a single server for business logic, slowly load balancing improved and caching become better.  Nowadays merely ramp on on you cloud provider.

Sharded Architecture: Application is broken into many distinct units/shards.  Each shard lives in total isolation from the other shards.  Think SOA or Microservice architectures often use this approach.  "SOA is focused on application service reusability while Microservices are more focused on decoupling".  

Source: https://kkimsangheon.github.io

The problem with tight coupling multiple services are:

1. Complexity - Difficult to change code and know the effects.  Also services need to be deployed together to test changes.
2. Resilience - Service goes down, the whole suite goes down.
3. Scalability - Can be an issue as the slowest component becomes the bottleneck.  

For instance build a complete application to handle ordering and a separate system that handles inventory.  So both could be in different data stores so let's say orders are on CosmosDB and Inventory is on Azure SQL.  Some of inventory data is static in nature so I decide to use App Caching (Redis).  Both the data sources site on independent server-less infrastructure.  So if you see inventory has an issue, merely scale it.  The front end store would seamless connect to both the separate.  "Sharding" databases/horizontal partitioning is a similar concept but only at the database level.  Sharding can be highly scale-able, allow for leveraging and reusing existing services, can be flexible as it grows.  Watch out for 2 Phase Commit (2PC/Sagas/Distribute transactions)

Thoughts Pros: 

1. Developers & Teams can work independently
2. Great to reuse existing services instead of creating yourself. e.g. App Insights on Azure.
3. Great for high availability and targeted scalability.
4. Zero trust security.  Least privilege access.  Defense in depth.
5. Focused costs by scaling the individual microservices.

Cons:

1. Services need to be Independence or deployment becomes a challenge.
2. Increased latency - you may need to go to various systems in sequential order.
3. Need keys to manage e.g. clientId for this decouple architecture type, this architecture can also become complex especially if you need to expand a shard to do something it doesn't do today.
4. Data aggregation and ETL can become complex and have time delays.
5. Referential integrity, guarenteed comit is an issue, can use SAGA or 2PC to improve but not ACID.
6. Need rules and strict governance and communication between teams.
7. Monitoring and troubleshooting can be tough.  Build a great traceable service (App insights for instance needs to be pollyfilled forlong running operations, SPA's need unique correlationIds)

Event-driven architecture: Client sends a request that includes a response for the server to contact when the event happens.  So if asking a server to do a complex calculation, the client could keep polling a long running operation until the server has the answer or use an Event-drive architecture to can you pls calculate and when you are done, send the response to me at...  Types of Event-driven architectures are: WebHooks, WekSockets, ESB (pub-sub), Server Sent Events (SSE).

Only run when an event happens.  They are loosely coupled.  In Azure it generally covers: Functions, Logic Apps, Event Grid (event broker) and APIM.  Easy to connect using Power Platform Connectors.

• Client/Service sends a broadcast event
• Consumers listens for events to see if they want to use the event

Hexagonal Architecture - Related/founder to Microservices,
Command Query Responsibility Segregation (CQRS) - pattern/method for querying and inserting data are different./seperated.  This is a performance and scaling pattern.
Domain Driven Design (DDD) - Design software inline with business requirements.  The structure and language of the code must match the business domain.  DDD Diagrams help create a share understanding of the problem space/domain to aid with conversation and further understanding within the team.   "Bounded Context is a central pattern in Domain-Driven Design. It is the focus of DDD's strategic design section which is all about dealing with large models and teams. DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships."  Martin Fowler.

RACI Diagram - visual diagram showing the functional role of each person on a team or service.  Useful for seeing who is responsible for what part of a service or their role within a team.

Event Sourcing Pattern - Used for event based architecture

AMQP is a standard used for passing business messages between systems.  AMQP is the default protocol used in Azure Service Bus.  AmazonMQ, RabbitMQ also support AMQP and is the main standard for the messaging protocol for event messaging.

Competing Consumer Pattern – Multiple consumers are ready to process messages off the queue.

Priority Queue pattern -Messages have a priority and are ordered for processing based on priority.

Queue-based load leveling.

Saga design pattern is a way to manage data consistency across microservices in distributed transaction scenarios.  Similar use case to 2PC but different. 

2PC (Two phase Commit): Simple pattern to ensure multiple distributed web services are all updated or no transaction is done across the distributed services.  

Throttling pattern

Retry pattern - useful for ensuring transient failures are corrected,

The Twelve-Factor App methodology is a methodology for building software-as-a-service (SaaS) applications.

API Gateway Pattern

CircuitBreaker Pattern

Key Design Decision (KDD) Document helps outline why decisions where made.  This is also often called Architectural Decisions Document or Template.

RAID Log - 

Streaming/MessageBus: Kafka, IoT,
Azure Messaging Service is made of of 6 products:
1. Service Bus - Normal ESB.  Messages are put into the queue and 1 or more apps can dirrectly connect or subscripbe to topics.
2. Relay Service - Useful for SOA when you have infra on prem.  Exposes cloud based endpoints to your on-prem. data sources.
3. Event Grid - HTTP event routing for real time notifications.
4. Event Hub - IoT ingestion, highly scalable.
5. Storage Queues - point-to-point messaging, very cheap and simple but very little functionality.
6. Notification Hub - 

Azure Durable Functions - Azure Functions are easy to create logic but are not good at long running or varying length duration functions.  To get around the timeout limits there are a couple of patterns for Functions making them better at handling long running operations.  The most common patterns are: Asyn HTTP API's (Trigger a a function using HTTP, set off other functions and the client waits for an answer by polling a separate function for the result), Function Chaining (Execute functions sequentially once the last function completes), and Fan out/Fan-in (first function call multiple functions that run in parallel) 

Lambda: great for large data architectures.  Has a batch vs streaming concept.  Each transaction pushed into a queue/stream (Kafka/Azure Queues/Azure Event Grid) and large data can be stored for later batch processing.

"Onion Architecture is based on the inversion of control principle. Onion Architecture is comprised of multiple concentric layers interfacing each other towards the core that represents the domain. The architecture does not depend on the data layer as in classic multi-tier architectures, but on the actual domain models." Codeguru.com

Distributed Application Runtime, Dapr: Video - Event drive portatable runtime for building distribute applications on the cloud.  Open source project tries to support any language/framework, consistent portable API's, extensible components that is platform agnostic.  HTTP API.    Secure Service to service calls, state management, publish and subscribe (For Azure ESB, Azure Queue), resource binding (Azure Functions), observability (Azure Signal-R service), Secret stores (Azure Key Vault) components to provide specific functionality.  Building blocks are made up of components.  

Cell Architecture: collection of components that are connectable and observable.  Cell Gateway is basically the service exposed.  Similar to APIM/API Gateway.  1 or more cell components make up Cell Gateway for ingress data.  Egress is done using Sidecar (App Insights is a great example of a sidecar service pattern), ...  Basically API first architecture.

SAST/DAST: are application security testing methodologies used to find vulnerabilities in applications.  Another threat modeling approach is STRIDE.

DACI (Decision Making Framework): stands for "driver, approver, contributor, informed", used to make effective and efficient group/team decisions.

OpenAPI vs GraphQL

OpenAPI specification (previously known as the Swagger specification) is my default for an API, this allows for a known RESTful API that anyone with access can use.   Open API has set contracts that returned defined objects which is great, you can work with the API like a database with simple CRUD operations as defined by the specification.  The issue is that the returned objects are fixed in structure so you may need 2 or more queries to get the data you are looking for.  Alternatively, GraphQL allows the developer to ask for the data exactly as the want it.

Open API example:

/api/user/{2} returns the user object  // Get the user object for user 2

/api/users/{2}/orders/10  // Returns the last 10 orders for the user

GraphQL example:

Post a single HTTP request.

query {

 User(id: "") {

    name

    email

    orders(last: 10 {

      orderid

      totalamount

      datemodified

    }

 }

} 

You can see that for complex changing systems, GraphQL is potentially a better choice.  I also like the idea of using HASURA for ORM using GraphQL against PostgreSQL (hopefully SQL Server and others).

More Info:

Microsoft Architecture Patterns

AAD Conditional Access

What is Conditional Access on AAD: Microsoft AAD with conditional access allows for users or groups to verify themselves more securely as after the login attempt an additional check is required to identify if the account may be compromised/at risk or is good.  Microsoft use algorithms and a ton of collated information to determine the risk on the attempted login.  A simple example would be a users location is unusual or logging in from different places in the world in too short a period.

• First factor Authentication happens before conditional access. 
• Setting up conditional AAD access 
• Conditional Access is part of Azure MFA
• Configure conditions for access
• Easy to bypass MFA if a used is a ADFS federated user or coming from a specific IP range (head office location) or region.  Can also allow a one time bypass if a user loses there phone.
• Required Azure AD Premium licences

Azure DevOps/TFS Basics

Overview:  There is a lot you can do with Azure DevOps to monitor your projects.  A couple of simple charts can be used to motivate (or demotivate) your team.  Start simple and build...

Knowledge Transfer/Support Handover

Problem:  Projects that I tend to work on are complete by Scrum teams filled with specialist and specialist contractors who move on after project completion.  Support is generally handled by dedicate people/teams offshore.

Hypothesis: Having high quality support people working alongside you throughout the project is not very common due to costs.  I believe there are key points to cover to ensure that the operational support is effective.  Too many companies merely focus on checklists and the ops team don't get a fundamental understanding of the system.

Resolution:
1. People/Support: Understand the domain - Hard
2. People/Support: Understand the architecture - Easy
3. People/Support: Understand who is responsible for level 1- level 3 support and what that entails.  Easy if done correctly.
4. People/attitude: Hire patient collaborative, eager people in support (most key point) that want to learn and take ownership. Easy if done correctly.
5. Knowledge base - have a wiki or equivalent.  The same issues always present, so document and have an answer that can help your uses.  I also like to record mp4's for different levels of support.  Record the sessions as it is too easy for level 3 people to say they never got a handover or covered something.  This allows people to look back, easily train additional users.  Easy if done correctly.
6. Ensure you have automated tests, they are a great source of how your system works.  And if a fix has to be released, it also easy to validate that the original logic still works.  Hard but it returns great benefits if used.

My Solution Documentation Thoughts

It all depends on the project but this post outlines what I have found to be the best practices for documentation on projects. 

Documentation should not be an after thought but done effectively throughout the development of any project.  It helps clarify thoughts, communicate and should save time.  Documentation is generally poor as it is dumped on people that tend to write it from the wrong point of view.  For example, developers know the products or components but write the code from their point of view not necessarily effective to the enterprises understanding.

Documentation Should Cover

• Overview & Start-up Documentation - Get the team with a common understanding.  I always like to have a Project Initiation Document (PID) that is kept short and up to date throughout a programme.
• Architectural Design Decisions (ADD) - Get the technical people on the team with a common understanding.  Software Design Document (SDD)/architecture design document  - Description /overview.  High Level Design (HLD) & Low Level Design (LLD).  Architectural design decisions are stored in a Architectural Design Repository (can be a simple as a file server, I prefer SharePoint and a Wiki index).  

Possible Solution Architecture Information Architecture for holding SA docs 

• Ensure documentation for: Solution Architecture, Dev processes, Support (wiki's), end user documentation, technical specifications (API's integration points), inline code must be simple and contain appropriate comments, and changelogs
• Requirements - User Stories/Use Cases.  Get good clear requirements from the business.  This gives the team and architects, developers a clear idea/vision of what is to be built and often helps the product owner/stakeholders have a full clear agreed picture.  User stories are a great way to break apart large piece of functionality.  It's always a good idea to have functional (FR's also often referred to as Business Requirements (BR's)) and no-functional requirements (NFR's).  For me the best way to capture requirements is to use User Stories.  FURPS is a way of categorizing requirements, useful to ensure adequate non-functional requirement areas have been covered.  I also like to use the old fashion MoSCoW (Must, Should, Could, Would) for prioritizing.  The most common mistakes I see in projects are requirements are:  1)  "Analysis paralysis" (very common in SDLC but more an issue with usage of SDLC than the methodology. 2) Gall's Law - stakeholders trying to put to much into a system from the start.  KISS/MVP - always opt for Keep it simple and only aim to deliver the minimum viable product.  Acceptance Criteria is a good way to validate when a User Story has been achieved.  Ideally a User Story should have less than 5 or 6 User stories.  If it has more, it is likely that the User story is too big and should be broken up into multiple user stories.  Weighted Shortest Job First (WSJF) is an Agile prioritizing system where you identify the highest priority items to do first.  Weighted matrix is another I have seen.  I also like an informal spend valuation that replies on effort/cost being already assigned.  Propriety Poker is also pretty common with multiple key stakeholders.  Stack ranking is also an easy option.
• Code Documentation - Code comments & API Documentation/Swagger.  API's are often an architectural constraint in that you as a business may decide to everything needs to be implemented using REST API's.  APIM on Azure is a great tool for documentation and cross cutting concerns.  The developer portal documentation allows 3rd parties or other systems to securely access and documented API.
• Performance And Testing
• User/System Documentation - User Guides and knowledge bases. Reduce escalation or time to get end users working.  Support documentation, I use Wiki's, they are easy to use, update, once a problem is solved, it is easy to add a new wiki and all future support is much easier.  Wiki's are quick and easy and should be kept current, don't hold old decisions.  Wiki's are searchable and tag-gable.

Tip: I record a lot of decision and support using Snagit.  It's fast, brilliant for knowledge bases and end user training.  Considerably less effort than written documentation.
Note:  A lot of specific documentation is needed for legal and complaint/regulation, this can be pretty heavy but still best to understand the requirements and do it from day 1.
Thought: Technical Writer (can be a dev, BA, technical architect or a dedicate technical writer) - I believe the BA should also be the test lead on non-scaled Agile products.  They understand the requirement, therefore are best to understand the testing and write clear concise documentation in the form of test cases or acceptance criteria and user stories.
Tip: Use Grammarly and do documentation professionally.  Ensure your documentation is easy to follow, do not have spelling mistakes or grammar issues.  Lastly, consistent layout between different documentation writers must be consisted be this in code comments for full end user documentation.
Thought:  Write in present tense in an active voice, if forces people to look at the now and future.
Note: Companies have guidance and documents, ensure you know the format of documents and comply with company guidelines, this may be as simple as fonts and colours in your documentation to specific document formats such as TOGAF documentation standards.  Make it easy for your project with a little planning.
Thought:  Code comments - Naming should do most of the documentation, but complex logic or implementation decisions should be commented using the KISS principles.  Don't document exactly what the code says e.g. If (status=21)  // Apply logic if status is 21 // Rather us // Update the Customer Web Service if the users email address has change
Comments should not be used to delete code in case the developer needs it.  You have source control, delete the code.

Agile Documentation: Does not mean no or low documentation.  Agile documentation should be clean, concise and save time overall for the team members.  Essential documentation, don't over document or items that are obvious.  Prioritize documentation like we do in backlog evaluation.

Slack/Teams/Email:
I was a Slack evangelist, it is awesome for Agile projects especially for projects with people in different locations.  Well now I am a Teams guy.  It's awesome, simple and let's you remove so many dependencies.  If you haven't used it before and you have office 365, it's a "no brainer".  In 2 weeks everyone will love using teams.  I have had many dysfunctional teams that needed coaching, teams that document everything and in stand-ups you hear "I sent you that in an email".  The first thing I tell these teams is "email is not a defence", go tell or speak to the person.  These teams are To and CC nearly all there email.  I immediately enforce the rule To: means i want a reply CC means it's important to you.  If someone then sends and email that is CC'ed, I ask them why and they generally learn to use email conservatively.  I stopped a team several years back using email for 2 sprints to get them communication and trusting each other again.

Micro services Pattern without Microsoft Gateway for on-prem. data

PB APIM Series:

APIM Notes

APIM Testing with Postman

APIM Mocking

APIM Mocking part 2

APIM Mocking part 2

Secure APIM with AAD B2C

Handling Security Incidents

Security Incident: An incident that potentially has compromised a companies systems or data.

Goal:  Focus on restoring confidentiality of systems/data and prevent further attack.  Contain the incident and eradicate the issue.  Full resolution target timeline is met for incidents.  These incidents can take up to 100 days but depends on the complexity.  

Examples:  Virus, Trojan Horse, Stolen data, increased unauthorized permissions, compromised server, copying data, DoS, unauthorized system access, ....

Need to record each event and work through the life-cycle (ISO 27035).  Can be dedicated software or modules such as ServiceNow's Security Incident Response (SIR).

1. Plan & Prepare
2. Detection
3. Assessment and Decision - Get logs, review/analyse, document the findings, notify leadership teams.  Impact/Priority e.g. Critical vs Low business impact.
4. Response - limit damage plan, decide on approach, notify if needed and remediate.
5. Lessons Learnt - ensure the threat is removed and potential lessons can help improve the attach surface for similar issues.

https://en.wikipedia.org/wiki/Computer_security_incident_management

Note: Be careful not to delete forensic evidence.

Tip: Organisations must have a Security Incident Plan.  Plan, be ready, know what to do in advance improves the handling of Security incident.