Friday 10 July 2020

Power Apps Tracing to App Insights Not Working in edit mode

Overview: Power Apps integrates to directly log to App insights.  This post looks at the issues around Tracing in App insights.

Setup & Verify:
App Insights instance Instrument Key Required, Config you Power App to Trace to App Insights, and create a button to test.
The Monitor Tool is fantastic for tracing all outbound traffic, you no longer need to go to the service and check if Power Apps is reaching.  For example I use to have to look at the APIM Azure App logs for custom Connectors.
I can see all my interactions in Power Apps and travelling out.  This is a massive win for Power Apps so giving Tracing from the front end.

Greg Lindhorst, wrote a post on using the Power Apps Monitor Tool for debugging and performance improvement.
Note: Traces written to App Insights don't work in preview.  Verify, see below.

Problem: It normally takes a few minutes (like 2 minutes) to show up in App Insights, it is not showing up after 15 minutes.  Let's see, I've drop the Power Apps team a message and i think it's a bug that has crept in recently.  Today is 10 July 2020.  I've also notice that none of my Session page tracing is showing up in App Insights.

Update: 11 July 2020, the Power Apps behaviour has been on my mind.  I'm in edit mode, and when I publish the app and use it, App Insights logs perfectly.  When in edit mode running the app, even direct traces, no logging into Power Apps.  I did not realise this, but it kind of makes sense.

Resolution: Publish & Run the app, and the Tracing and Power App session tracing shows up in App Insights.

Only when I Run the Published App to my Page Views and Custom Traces get logged in App Insights.

Tuesday 30 June 2020

Multi-Geo for MS Teams

O365 offers multi-geo tenants to meet data residency rules for 13 countries and regions (as of 30 June 2020):
  1. Australia
  2. Asia Pacific 
  3. Canada
  4. European Union
  5. France
  6. India
  7. Japan
  8. Korea
  9. United Kingdom
  10. United States
  11. United Arab Emirates
  12. South Africa
  13. Switzerland.
Teams data resides in SharePoint Online, OneDrive for Business and Exchange Online.  With Multi-Geo enabled, a company can specify where data will reside.  There are 2 parts to multi-geo:
  • User specific data.  This data is stored in various satellite Geo for each user e.g. email, OneDrive
  • Company/Project/division specific data e.g. file shares, Document libraries
For more info on Multi-Geo on O365

Microsoft "Multi-Geo is currently available to Enterprise Agreement customers with a minimum of 250 Microsoft 365 Services subscriptions."

The UK South Azure Region has 3 data centres/zones, and it's geo-paired paired with UK West there is over 150 mile between the regions. 

MS Teams Background Info:
https://www.pbeck.co.uk/2019/12/microsoft-teams-governance.html
https://www.pbeck.co.uk/2020/05/microsoft-teams-overview.html

Note: Microsoft since 2022 I think have Microsoft Priva to help manage country privacy & compliance laws 

Wednesday 24 June 2020

Postman API Builder Intro

Overview: Tools for building and mocking API's.  Swagger has good tooling and my original preferred choice.  APIM - Great tooling, part of Azure and easy to replace mocks as you go along with the live implementation.  Postman is offering a great set of functionality to rival Swagger and APIM.  This post looks at Postman's new functionality around building API's.

Postman API Builder:
Not only a test rig, it now offers the ability to build API's and mock:
  • Mock - so you can test supports key and OAuth authentication
  • Assert Tests - You can specify asserts in postman
  • Test suite - generate collections/Collection Runner - Allows a set of related tests to run sequentially.
]
  • Document the API
  • Monitor
  • Version control for changes e.g. GITHub
  • API Versions supported
  • Note: Free plan has all of this, limited on the number of API's but all the features are on the free plan.  The main notation formats are support including:  Open API specification (OAS) & GraphQL
Summary:
I like Swagger tooling, I have done a few projects find APIM fantastic for building API's quickly.  Postman historically was merely my test rig but looking at the functionality, Postman API Builder is a great option for designing and building API's.  Postman is a good tool for building into CI/CD pipelines to validate API's.

Few more assert examples:

Postman offers a service to monitor API's using your postman collections, these can be triggered using Curl so can build into DevOps, Power Automated scheduled flows,....
sentry.io looks good as an alternative option

Thursday 28 May 2020

Microsoft Teams Power Apps Integration

Overview: Teams are amazing, I was a complete Slack fan, but I'm 100% now a teams supporter.  It's part of O365, replaces Skype (which was great but only a chat app like zoom), you get your email, and can add all your apps and websites to your Team.

Adding your custom Power Apps to Teams:

Adding A Power App to MS Teams:




Notes:
  1. MS Teams uses the Chrome engine (Chromium) as it's browser.
  2. A feature I don't like about Teams is that when i switch focus to say a chat window and come back to my Power app within MS Teams, I loose my place in my power app and the app is loaded from scratch.
  3. I believe the problem of apps maintaining session state will be solved shortly with pop out Windows in Teams around July/Aug 2020.
High-Level Flow to Build a native MS Team application:

Some Teams Dev Options: Power Apps, Blazor SPFx, React, Flutter

Friday 22 May 2020

Micrososervice and SOA introduction

What are Microservices: A type of SOA, basically distributed computing.  Microservices are a subset of SOA but with loosely couple architecture.  Independent deploy-able services, tries to reduce/remove other services dependency on a specific service.  I can change a service and it does not affect other service/break existing services.  Allows us to have a small scope of release, therefore easy to test and increase release frequency.  Services will depend on other services.  SOA is older but both are well tried and tested.  Microservices is generally preferred but it's much easier to move to SOA for a monolith.  

Monolith Systems:  Monolith are the way most systems were built, large full deployments.  All code is packaged into a large module of code.  They are simple and often are sometimes fairly appropriate such as on-prem. software.  Beware the Distributed Monolith:  Separate services but high dependent, end up having to co-ordinate deployment between code bases.

More on SOA: Independently deploy able,  DDD for scoping works well for defining Microservices.  Need to get the boundaries right is key.  Using Bounded Context is a good way to have multiple related models and show their overlapping points.

Each service operates in isolation which are processes.  Allow us to scale independently.  Containers are good for hosting these services.  Communication generally done over request/Response (most commonly HTTPS) e.g. REST API or Event-Driven e.g. Azure Service Bus.  Event is broadcast from the service, listening services is responsible for listening to the event.

Encapsulation of data, the data is hidden inside the Microservice boundary. Allows for independent deployment.

Main Benefits of SOA:

  • "Microservice buy you options";
  • As smaller independent units are deployed, which leads itself to faster deployments to production using CI and CD.  More teams/devs  can work independently;
  • Defense in Depth Security;
  • Scalability from performance - Scale up, identify bottlenecks (change tech or scale service hosting bottleneck);
  • Failure Independence - tolerate partial failures, service redundancy;
  • Cost - focus cost as needed, scale up and down.  Deliver service quickly, can replace with better technology e.g. service use an expensive workflow software to do some logic, as business grows, build a C# service that does the work faster, better, and lower cost and no need to BPM.

Note: SOA does not guarantee these benefits, you need to adjust to get them

Tip:  Still use similar technology stacks and patterns, don't allow each service to be built without considering the consumers.  Try keep on a similar stack e.g. don't use C#, NodeJS, Python, Go for different web services.


Friday 8 May 2020

cURL for Windows 10 & Azure Cognitive Service Primer

In this example I am using Azure Cognitive service to provide a jpeg using curl on my Windows 10 Surface laptop.

Updated: 2021-10-28: The latest on Cognitive search, Semantic Search looks interesting

Sunday 3 May 2020

Common Software Architectural Patterns

The goal of Solution Architecture is to:
Have a common understanding of how a technical solution shall be reached, the diagrams are useful too have a communicable roadmap and ensure it is complete.  Later the diagrams are used to ensure all relevant parties have a clear unambiguous shared understanding of the IT solution.

The main tools to communicate the architectural solution design are diagrams & documents that utilise common previously used and understood patterns to ensure a safe, scalable, stable, performant, and maintainable solution.  "4+1 view, which includes the scenario, logical, physical, process, and development views of the architecture", source.

Below are patterns and thoughts that I have come across and used to solve building high quality solutions.

3/N Tier Architecture/Layered:
1) Presentation/UI layer
2) Business Logic
3) Data Layer/Data source
Here are a couple of possible example over the years you could of used
ASP > C++ Com > SQL Server 2000
ASP.NET (Web Forms) > C# Web Service (XML/SOAP) > SQL Server 2008
ASP.NET C# > C# Business Object Layer > SQL Server 2008
KO > MVC > SQL 2012
Angular 3 > C# Web API (swagger contract) > SQL 2016
REACT.JS > Node.JS > Amazon Redshift
UI > Azure Functions/Serverless > SQL Azure
Flutter > C# Web API .NET Core 3 (swagger/OpenAPI) published on Azure App Service > SQL Azure/Cosmos

API's:  Over the years we have seen many different API's at a high level:  
  • Proprietary formatted API's > 
  • XML with SOAP coming out of XML based API's > 
  • REST/JSON (other popular formats are: RAML, GraphQL > 
  • Event Driven API's may be the next big jump.

Thoughts:  As time has progressed, scaling each of these layers has become easier.  For instance Azure SQL has replication and high Availability and scalability automatically built in.  No need to think about load balancing in depth.  Plug and play and ask for more if you need it.
Microsoft SQL Server use to be a single server, then came replication, clustering, Always-on-availability, scaling greatly improved performance.
Middle Tier or Business layer use to be a singleton pattern - go thru a single server for business logic, slowly load balancing improved and caching become better.  Nowadays merely ramp on on you cloud provider.

Sharded Architecture: Application is broken into many distinct units/shards.  Each shard lives in total isolation from the other shards.  Think SOA or Microservice architectures often use this approach.  "SOA is focused on application service reusability while Microservices are more focused on decoupling".  
Source: https://kkimsangheon.github.io

The problem with tight coupling multiple services are:
  1. Complexity - Difficult to change code and know the effects.  Also services need to be deployed together to test changes.
  2. Resilience - Service goes down, the whole suite goes down.
  3. Scalability - Can be an issue as the slowest component becomes the bottleneck.  
For instance build a complete application to handle ordering and a separate system that handles inventory.  So both could be in different data stores so let's say orders are on CosmosDB and Inventory is on Azure SQL.  Some of inventory data is static in nature so I decide to use App Caching (Redis).  Both the data sources site on independent server-less infrastructure.  So if you see inventory has an issue, merely scale it.  The front end store would seamless connect to both the separate.  "Sharding" databases/horizontal partitioning is a similar concept but only at the database level.  Sharding can be highly scale-able, allow for leveraging and reusing existing services, can be flexible as it grows.  Watch out for 2 Phase Commit (2PC/Sagas/Distribute transactions)

Thoughts Pros: 
  1. Developers & Teams can work independently
  2. Great to reuse existing services instead of creating yourself. e.g. App Insights on Azure.
  3. Great for high availability and targeted scalability.
  4. Zero trust security.  Least privilege access.  Defense in depth.
  5. Focused costs by scaling the individual microservices.
Cons:
  1. Services need to be Independence or deployment becomes a challenge.
  2. Increased latency - you may need to go to various systems in sequential order.
  3. Need keys to manage e.g. clientId for this decouple architecture type, this architecture can also become complex especially if you need to expand a shard to do something it doesn't do today.
  4. Data aggregation and ETL can become complex and have time delays.
  5. Referential integrity, guarenteed comit is an issue, can use SAGA or 2PC to improve but not ACID.
  6. Need rules and strict governance and communication between teams.
  7. Monitoring and troubleshooting can be tough.  Build a great traceable service (App insights for instance needs to be pollyfilled forlong running operations, SPA's need unique correlationIds)
Event-driven architecture: Client sends a request that includes a response for the server to contact when the event happens.  So if asking a server to do a complex calculation, the client could keep polling a long running operation until the server has the answer or use an Event-drive architecture to can you pls calculate and when you are done, send the response to me at...  Types of Event-driven architectures are: WebHooks, WekSockets, ESB (pub-sub), Server Sent Events (SSE).
Only run when an event happens.  They are loosely coupled.  In Azure it generally covers: Functions, Logic Apps, Event Grid (event broker) and APIM.  Easy to connect using Power Platform Connectors.
  • Client/Service sends a broadcast event
  • Consumers listens for events to see if they want to use the event
Hexagonal Architecture - Related/founder to Microservices,
Command Query Responsibility Segregation (CQRS) - pattern/method for querying and inserting data are different./seperated.  This is a performance and scaling pattern.
Domain Driven Design (DDD) - Design software inline with business requirements.  The structure and language of the code must match the business domain.  DDD Diagrams help create a share understanding of the problem space/domain to aid with conversation and further understanding within the team.   "Bounded Context is a central pattern in Domain-Driven Design. It is the focus of DDD's strategic design section which is all about dealing with large models and teams. DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships."  Martin Fowler.
RACI Diagram - visual diagram showing the functional role of each person on a team or service.  Useful for seeing who is responsible for what part of a service or their role within a team.
Event Sourcing Pattern - Used for event based architecture
AMQP is a standard used for passing business messages between systems.  AMQP is the default protocol used in Azure Service Bus.  AmazonMQ, RabbitMQ also support AMQP and is the main standard for the messaging protocol for event messaging.
Competing Consumer Pattern – Multiple consumers are ready to process messages off the queue.
Priority Queue pattern -Messages have a priority and are ordered for processing based on priority.
Queue-based load leveling.
Saga design pattern is a way to manage data consistency across microservices in distributed transaction scenarios.  Similar use case to 2PC but different. 
2PC (Two phase Commit): Simple pattern to ensure multiple distributed web services are all updated or no transaction is done across the distributed services.  
Throttling pattern
Retry pattern - useful for ensuring transient failures are corrected,
The Twelve-Factor App methodology is a methodology for building software-as-a-service (SaaS) applications.

Key Design Decision (KDD) Document helps outline why decisions where made.  This is also often called Architectural Decisions Document or Template.
RAID Log - 

Streaming/MessageBus: Kafka, IoT,
Azure Messaging Service is made of of 6 products:
1. Service Bus - Normal ESB.  Messages are put into the queue and 1 or more apps can dirrectly connect or subscripbe to topics.
2. Relay Service - Useful for SOA when you have infra on prem.  Exposes cloud based endpoints to your on-prem. data sources.
3. Event Grid - HTTP event routing for real time notifications.
4. Event Hub - IoT ingestion, highly scalable.
5. Storage Queues - point-to-point messaging, very cheap and simple but very little functionality.
6. Notification Hub - 

Azure Durable Functions - Azure Functions are easy to create logic but are not good at long running or varying length duration functions.  To get around the timeout limits there are a couple of patterns for Functions making them better at handling long running operations.  The most common patterns are: Asyn HTTP API's (Trigger a a function using HTTP, set off other functions and the client waits for an answer by polling a separate function for the result), Function Chaining (Execute functions sequentially once the last function completes), and Fan out/Fan-in (first function call multiple functions that run in parallel) 

Lambda: great for large data architectures.  Has a batch vs streaming concept.  Each transaction pushed into a queue/stream (Kafka/Azure Queues/Azure Event Grid) and large data can be stored for later batch processing.

"Onion Architecture is based on the inversion of control principle. Onion Architecture is comprised of multiple concentric layers interfacing each other towards the core that represents the domain. The architecture does not depend on the data layer as in classic multi-tier architectures, but on the actual domain models." Codeguru.com

Distributed Application Runtime, Dapr: Video - Event drive portatable runtime for building distribute applications on the cloud.  Open source project tries to support any language/framework, consistent portable API's, extensible components that is platform agnostic.  HTTP API.    Secure Service to service calls, state management, publish and subscribe (For Azure ESB, Azure Queue), resource binding (Azure Functions), observability (Azure Signal-R service), Secret stores (Azure Key Vault) components to provide specific functionality.  Building blocks are made up of components.  

Cell Architecture: collection of components that are connectable and observable.  Cell Gateway is basically the service exposed.  Similar to APIM/API Gateway.  1 or more cell components make up Cell Gateway for ingress data.  Egress is done using Sidecar (App Insights is a great example of a sidecar service pattern), ...  Basically API first architecture.

SAST/DAST: are application security testing methodologies used to find vulnerabilities in applications.  Another threat modeling approach is STRIDE.

DACI (Decision Making Framework): stands for "driver, approver, contributor, informed", used to make effective and efficient group/team decisions.

OpenAPI vs GraphQL
OpenAPI specification (previously known as the Swagger specification) is my default for an API, this allows for a known RESTful API that anyone with access can use.   Open API has set contracts that returned defined objects which is great, you can work with the API like a database with simple CRUD operations as defined by the specification.  The issue is that the returned objects are fixed in structure so you may need 2 or more queries to get the data you are looking for.  Alternatively, GraphQL allows the developer to ask for the data exactly as the want it.
Open API example:
/api/user/{2} returns the user object  // Get the user object for user 2
/api/users/{2}/orders/10  // Returns the last 10 orders for the user
GraphQL example:
Post a single HTTP request.
query {
 User(id: "") {
    name
    email
    orders(last: 10 {
      orderid
      totalamount
      datemodified
    }
 }
You can see that for complex changing systems, GraphQL is potentially a better choice.  I also like the idea of using HASURA for ORM using GraphQL against PostgreSQL (hopefully SQL Server and others).

More Info: