App Insights - Website and API Monitoring

Overview:  App Insights has functionality to run scheduled web requests and log the output in App Insights.  There are multiple advantages to this including end to end active monitoring of web sites and API's, and keeping the application warm.

Below I show a simple request to my blog (public website) and the results, Azure refers to this test type as a URL Ping test which is basically a URL HTTP GET request.  

Wait a few minutes and Refresh to see the results:

Very easy way to include a constant check that your API or Website is running.  There is also the options to create "Multi-step web test" using Visual Studio.  You can record the authentication and assert for known response content to build advanced constant monitoring.

Tip: The URL does need to be publicly available.

The content I used to test out the functionality comes from the Microsoft Docs site.

Also see Live Metric Stream that is part of App Insights.

Monitoring using Azure Monitor Dashboards:

• The image above shows a dashboard that can be used to monitor a SaaS applications PaaS Infrastructure.
• It's a good idea to create multiple dashboards and they can show the overview and allow the user to drill into specific areas.
• Internal boards watching key API's, HTTP uptime ping type requests is also a good idea.

Updated: 11 Jan 2023 - Checkly looks like a great monitoring service that works with Playwright for continuous monitoring.  I like to keep all my SaaS on Azure, but Checkly is independent an as it is monitoring.

More Info: 

App Insights MultiStep Tests  Replacement Option for MultiStep Test based on Azure Functions

App Insights - Basic Introduction

Overview:  Azure App Insights is a great platform for collecting logs and monitoring cloud based applications on Azure.  All Azure Services can push logging information into App Insight instances.  This can be errors, usages, performances logging that in turn is easy to query.  There are SDKs for developers that can be used to add custom logging to applications.  I am a big fan of AppDynamics for logging and monitoring but for SaaS and on a new project I'd go with App Insights.

Retention:  App Insights can keep 730 days worth of logs.  For long term storage, "Continuous Export" can be used to push data into storage accounts as soon as it arrives in AppInsights.  Retaining the App Insight logs for 90 days has no additional cost, so the default to store logs should be set to 90 days at least in most situations.

What is logged and what can be logged:  

• All Azure Services can be configured to send service logs to a specific App Insight instance.
• Instrument packages can be added to services to capture logs such as IIS, or background services.  You can pull in telemetry from infrastructure into App insights e.g. Docker logs, system events.
• Custom code can also call the App Insight instance to add logging and hook into exceptions handling.  There are .NET, Node.JS, Python and other SDK's that should e used to add logging, exception capturing, performance and usage statistics.

App Insights has a REST API to query the logs.  The "API Explorer" tool is awesome for querying App Insights online.  

The data below comes from Microsoft Docs.

"What kinds of data are collected?

The main categories are:

• Web server telemetry - HTTP requests. Uri, time taken to process the request, response code, client IP address. Session id.
• Web pages - Page, user and session counts. Page load times. Exceptions. Ajax calls.
• Performance counters - Memory, CPU, IO, Network occupancy.
• Client and server context - OS, locale, device type, browser, screen resolution.
• Exceptions and crashes - stack dumps, build id, CPU type.
• Dependencies - calls to external services such as REST, SQL, AJAX. URI or connection string, duration, success, command.
• Availability tests - duration of test and steps, responses.
• Trace logs and custom telemetry - anything you code into your logs or telemetry."

App Insights creates a hierarchy of requests built up from the operationId, and operation_parentId.

Application Insights is part of Azure Monitor and makes it easy to trace user interaction.  Independent infrastructure for recording issues and tracing.   App Insights in 3 parts. 

• Collect: Track infra/PaaS via instrumentation (throughput, speed, response times, failure rates, exceptions etc.), and via SDK (e.g. JavaScript SDK, C#) to add custom logging and tracing.  Blue boxes
• Store: Stores the data.  Purple Box
• Insights: Alerts, PowerBI, live metrics, REST API.  Green Box

Extending App Insights:

For long running operations like using queues or ESB you will need to tie the operations together, and it's really easy to connect this in a hierarchy using distributed tracing.  

SPA's:  There is a JavaScript SDK but logging on SPA's needs configuration and understanding as not every operation is logged uniquely for tracing.

Smart Detection: automatically tries to quickly warn you of problems/abnormalities and there root cause.

Snapshot Debugger/profiler: VS remote debugging can be hooked to an issue.  Shows execution traces from your live app.

Transaction Search:  Easy way to query and find data or unique info.

Azure Dashboards: Custom KPI dashboard using App Insights &  More Info 

Secure APIM using AAD B2C

Overview:  I have never connected AAD B2C to APIM myself, other on my project teams havde done it so I went thru it and it was super easy.

Followed the instructions: 

https://docs.microsoft.com/en-us/azure/active-directory-b2c/secure-api-management?tabs=applications-legacy

Postman to validate:

PB APIM Series:

APIM Notes

APIM Testing with Postman

APIM Mocking

APIM Mocking part 2

Secure APIM with AAD B2C

Developer Portal - Example documenting your API within YAML

Building better Software Thoughts

Overview:  I see a lot of development teams, and they always seem to have areas they are good at and capabilities teams need improvement on.  Key is culture and building a happy team where team members trust and help one another.

Building a culture where teams enjoy code reviews is also key for successful Software projects.   To improve software, reviewing various areas not only code reviews are essential.  For me, clear requirements are the number 1 factor in improving teams performance.  

Companies are getting better at building software; I aim to work on these topics to improve the delivery of software within scrum teams:

1. Code Reviews & Peer Reviews (Daily reviews are awesome, should be pretty short and enjoyable not someone trying to show off or hours long)
2. Collaboration (Standups, Slack/Teams, Code tools have collaboration built in)
3. Documentation & Requirements Reviews
4. Better tooling including better CI/CD tooling including static analysis tools
5. Unit Testing, automate coding standards, Integration testing, UI Testing, and API testing 
6. Requirements (Use Stories are clear and Acceptance Criteria)
7. Cadence is improving thanks predominately to Agile practices; I like short release cycles (2-3 weeks depending on the team and industry).  Changing requirements, indecision kills software projects.  Agile helps, but decisive knowledgeable product owners increase the likelihood of the project succeeding.

Benefits of Code, Documentation and Requirement Reviews:

1. Improved software quality & product delivery
2. Share domain knowledge
3. Training team members (useful for onboarding new team members)
4. Reduce support and fix costs
5. Lower cost & faster development

Options Layering API's on Data Sources - Micrososervices kind of

Hasura takes data sources such as SQL, Postgress & MySQL and converts it into GraphQL API's.  SQL Server is in preview.  Service is available on Azure and hooks into AAD and AAD B2C.  Hasuru looks extremely interesting and useful.  Potentially a great time saver.

CDS/DataFelx/Oakdale - Allows for Entity creation and provides REST API's.

SharePoint lists provide HTTP API's for CRUD operations.

REST API's vs GraphQL

OpenAPI specification (previously known as the Swagger specification) is my default for an API, this allows for a known RESTful API that anyone with access can use.   Open API has set contracts that returned defined objects which is great, you can work with the API like a database with simple CRUD operations as defined by the specification.  The issue is that the returned objects are fixed in structure so you may need 2 or more queries to get the data you are looking for.  Alternatively, GrapghQL allows the developer to ask for the data exactly as the want it.

Open API example:

/api/user/{2} returns the user object  // Get the user object for user 2

/api/users/{2}/orders/10  // Returns the last 10 orders for the user

GraphQL example:

Post a single HTTP request.

query {

 User(id: "") {

    name

    email

    orders(last: 10 {

      orderid

      totalamount

      datemodified

    }

 }

} 

Database Option Notes:

PostgreSQL - Hybrid relation database (Communitity edition free, standard and enterprise higher loads) sits on Linex but can also use Windows OS.  Replicas are used for additional read-only nodes (I think up to 5 geo relicas allowed) 

Can be hosted on Azure or AWS IaaS.  PostgreSQL is also offered as a fully managed service (PaaS) on Azure (either single server or hyperscale).

MariaDB community fork off MySQL - Open source relation database used in the LAMP app stack.  Azure MySQL manage service can have up to 5 replicas for read heavy workloads.

Working with CDS data structures for non CRM types

Overview:  I am working on a Power Platform solution and I need to use CDS.  Basically, I need to be able to see and edit values within CDS.

Option 1: Microsoft SQL Server Management Studio (SSMS) version 18.6 allows connectivity and read-only access.  Here are the instructions.

Option 2:  XrmToolBox has fantastic tools for Dynamics and Power Apps.  There are a lot of individual tools from various contributors.

Here I am using "Entity Relationship Diagram Creator" to look at the relationships between the CDS entities.

Reducing Power Apps Dynamic calls and where to store Power Apps data.

Overview:  Power Apps is driven by data and generally that data comes from Connectors.  So the great news is there are a lot of different connectors and if in trouble I always find the custom connector can be relied upon.  When working with Power Apps, it is not as simple as just having a data source and consuming it, one needs to consider all the data sources, do we need live data, performance.  Basically, going and dynamically pulling lots of dynamic sets of data repeatedly leads to poor performance.

CDS performance has relatively few layers when retiring or updating data so it generally is fast to work with.  Custom connectors have more layers so can be slower and you are dependent on the underlying REST API so watch for performance.  The on-prem azure data gateway is pretty slow but amazing if you need it.

Identify data sources:  CDS, Azure SQL, SharePoint SQL on-prem., CosmosDB, RSS, Open API's...

Understand the security, and the amount of data being pulled.  For example, if we need all the airport codes in the world for a drop down so the user can choose their closest airport e.g. JFK is for New York John F Kennedy airport.  There are roughly 4000 commercial airports in the world. 

Options: Call an Open API service.  Power App by default returns sets of 500, Power Apps max return count is 2000.  You still need to perform 2 calls with paging to get the full data set.  You could use a type ahead if the API supports it, but their will be a lag after each keystroke when Power Apps runs out to the service.  And there will be a lot of calls.  More suitable would be to do 2 calls with 2,000 record for each call and bind the control to the returned data.

A further improvement would be to store the airport lookup on data load or on first request, then subsequent requests would use the table/collection.  In effect, you are locally caching all the airport codes using 2 calls for each Power Apps user session.

For the airport example, one could also store the data using a Excel import, but beware the data is imported into Power Apps and store locally.  Big issue is the data is static in Power Apps, to update it, you need to re-import the Excel table.  So brilliant for sat days of the week or Months of the year as these never change.  Fairly static data like airport codes work well, but require a publisher level overwrite to update the list.  Also, storing extreme amounts of static data leads to bloat of the app and that data still needs to be loaded.  So I would not consider it for 100k+ items as a general rule.  

FYI Microsoft recommends to use less than 30 connections per application.

More Info:

Todd Baginski has a great video on using Excel import and creating language variations/multi-lingual Power Apps using Excel imports.

Alagunila Meganathan on C# Corner has a good post on Excel Imports for Power Apps.