Observability in Azure SaaS Solutions

Problem:  Software has many places where errors and tracing is logged to.   Support get an incident, they need to investigate figure out how widespread the issue is and then try patch together various logs to figure out the problem.

Thoughts:

Observability is not a new concept, we need to be able to: 1) view and connect logs & 2) tracing and view metrics & notifications.

Implementing Observability must cover:

CI/CD allows devOps teams to find issues early using Unit testing.   Automated testing on UI.  API automation testing is also great.

Azure offers continious monitoring by performing various API calls to ensure your servie is running and any failures are picked up hopefully before any customers are aware.  You can also be notified of performance slow down, check performance speed between releases.  Which is great for identifying bottlenecks and with the Azure PaaS world, it is easy to increase the processing causing the bottleneck.  

Performance metrics built into the CI/CD and developers work allows us to identify issues early and costs miles less to correct early. 

Security and LINTing in CI/CD also allows us to pickup issues early and correct at a way lower cost.

Instrument you hardware and software, well on Azure you can use App Insights and you have a fantastic instrumenting platform that captures events.  A big reason to use Azure Services for as many of the function pieces in your solution.

Work In Progress ...

TBC Azure App Insights detail, ParentOperationId, Linking operations with a ServiceBus or work process call.

Catch Error in Power Apps and App Insight Logging

Error Handling:
App Insights logging: https://sharepains.com/2019/01/24/powerapps-experimenting-with-error-handling/  Replaced as Microsoft have built in telemetry as of 3 Feb 2020.
https://powerapps.microsoft.com/en-us/blog/log-telemetry-for-your-apps-using-azure-application-insights/

Example Error capturing and tracing to Azure AppInsights:
IfError( // Perform API Call , // Fallback so log here! ,

    Trace("Pauls Unique PowerApp",TraceSeverity.Error, {UserName:User().Email,         Role:gblRole, ErrorMsg:ErrorInfo.Message, ErrorControl:ErrorInfo.Control,         ErrorProperty:ErrorInfo.Property});     Notify("Err message ..." & ErrorInfo.Message); // Display the error on the UI

More detail..

Possible Canvas Apps Error Handling Pattern:

1. Ensure AppInsights key is added to each canvas app
2. Use IfError() to check calls and logic
3. Use the Trace method to write info to App Insights
4. Do I want to enable the Experimental error handling features (great to trace by correlationId)
5. Consider all Power Automate that use Power Apps (ensure you use the V2 Connector)
6. Never use IfError to handle business logic

To Review your App Insights Logging:
Open you Azure Portal > Open your App Insights blade >
Click the "Search" navigation option > Free text entry e.g. "Loyalty PowerApp"

App Insights, finding Traces generated in Power Apps

Monitoring Tool within Power Apps
The Monitor tool in Power Apps is great for debugging and tracing.

Start a monitor on the open Power App.

Monitor Tool - Showing a GET via a custom Connector and the returned response

Function/Code Logging:

Server-side code should log to App Insights or you logging framework.

It is ideal with the Trace within Power Apps explained above to be used in conjunction with 3rd party API calls.

Overview: C# code needs to have logging. If an error occurs an appropriate response must be bubbled up for the next lay

Possible C# Error Handling Pattern:

1. All catch write exception to Log analytics or App insights 
2. Calls to data sources, Azure Services and third party API's and complex logic ideally should be wrapped in a try catch and log the error to App insights using the C# App Insights SDK 
3. The catch blocks ideally return the failed information so the caller code can deal with the logic using the output.  If you don't deal with the returned message, simply log the exception and rethrowing the error (this needs to be a conscious decision on each catch) 
4. Catch specific errors: log, if you don't pass info to caller rethrow the error if applicable (bubble), respond accordingly i.e. catch the specific error and lastly use a catch all. - Heavy, but only add to existing code where this happens often or we are having problems, i.e. be specific
5. Don't use Try, Catch to deal with business logic

Thought: Bubble up means: Code must log exceptions and returns appropriate reply to the caller, if you don't send the appropriate reply rethrow the exception after logging it so the caller has to deal with it.

Power Apps Quick Tips

Some of my Power Apps notes:

Licencing:
Determine your Power Apps licence using Power Apps Web user interface:
Power Apps > Settings > Plans

Note:  Microsoft changed how Power Apps licencing works circa Nov 2019.  The Power Apps licence with O365 E3/E5 does not have the connectors so is useful only for simple apps, and using flow, the apps can get more advanced.

Testing:
I saw a preview of the Power Apps Test Framework as the Microsoft Ignite Tour in mid Jan 2020.  The framework allows for recording UI tests and inserting asserts.  The tests are stored alongside the project inside Power Apps.  PowerApps Testing is in beta, it will be release by region so presumably it's already available in some US regions on the public beta.  The CI/CD pipeline integration was not shown but I'm sure some smart fella will look at this shortly.  I'd use Selenium for automated web testing at this stage, but that may change especially for mobile app development.

Updated 28 Jan 2020: Test Studio for Power Apps released today.  Here is the announcement.  Here is another recording outlining testing of Power Apps.

In the Power App > File > Settings > Advanced Settings > Scroll down to "Formula-level Error Management" and turn on the feature.
Open the "Advanced tools"

Accessing Test Studio in Power Apps

Power Apps Development Standards:
Error Handling - Should an unexpected event occur, log so the issue can be traced, as of 3 Feb 2020, PowerApps has App Insight integration.
Naming standards - Label visual controls and variables
Approved Colours - Digital branding standards
Reuse - Control reuse with shared branding
Roles - AAD connector allows me to figure out the user group the current user belongs too so they get appropriate menus, screens, and actions.
Menus -  @Laura-GB https://www.youtube.com/watch?v=5G-gVWRItmc  Build a menu component, input allows one to combine with AAD roles to create dynamic menus for users.
Copy and pasting OOB controls - such as a textbox, if you use a formula, the values are persist and passed around.  So if you have a color set by a variable, copy and paste results in the new text box referencing the colour.  Taking this further, if you put in the position X and Y a formula e.g. X=66*1 and Y=100*1, the control will inherit the parent controls property rather than reset the property.

Steps to setup and Use a Custom Connector:

Get the http response from a custom connector and add to a collection

Common Functions:
DateTimeValue() - Convert string into a date
Text() - Format a string
Value()- Format a string into a numeric
Text(DateTimeValue(TextInput1.Text), "dd-mmm-yyyy")
An approach I like to advanced Power App Input Validation.

Lookup() - Finds the first record in a table, closely related to Filter() and Search()

I.e. Fx> Set(Lookup(<DataverseTblName>,<UniqueRowId>=GUID("1234-1234-111...") 

Patch() - Update or add new, best way to save as it handle collections

Patch() function used to update an existing "Appeal" record.  Also note that a json string is saved- This is how to escape quotes.

Error Handling and Logging:
PowerApps has pretty basic error handling IfError(Try, if fail fall back)
Error Handling for Power Apps
The is a built in mechanism to log from Power Apps into App Insights.

Components:
Build a common UI element such as a menu or header.  The component has input and output parameters so you need to pass in the data to be used and return the output.  This allow the component to be re-used.  Obviously it makes sense that you can't use global variable within components, you need to pass the global variables in using input parameters.
Great example from @Laura-GB https://www.youtube.com/watch?v=5G-gVWRItmc

SPMetal is throwing a ThreadAbortException

Problem: I am inserting new items into a list using Linq-to-sharepoint (SPMetal), my code throws a ThreadAbortException exception. 

Initial Hypothesis: When I run the code as a Site Collection Owner the new list item is inserted.  So this is a permissions issues.  I increase the permissions on the user that can't insert the list item.  It works so this is clearly a permissions issue.  After the user tries to perform the insert list item, they are directed to the "Error: Access Denied" message on the web page.  Clearly the error message is misleading however, SharePoint seems to understand the ThreadAbortException is a permissions issue.  The issue is the user does not have sufficient permissions to perform the action.  By elevating the user permissions the error goes away.  All well except the error logging is throwing an odd error, after some digging this issue has been blogged before however not for LINQ to SharePoint.

Resolution: Assign the correct permissions to the user performing an action.

Logging custom error for SharePoint 2010 custom code

Problem: SP2010 has good logging in the ULS logs. Unlike MOSS you can write to the ULS logs in SharePoint 2010, allowing for a single consistant place to log your errors. The event viewer logs errors to a lesser extent than ULS where administrators tend to look first. However, on all projects the topic of how and where to log custom coding errors comes up. What level should the event be caught at and where should they be placed. I have seen multiple was of doing this and it really comes down to what the previous projects used and how do you want to monitor errors.
Tips:

1. Don't write code to function using error catching i.e. I have seen developers catch a specific error and from this position they know the code is to follow specific logic -it's very inefficient. Write code to deal with all situations.
2. Catch errors as specifically as possible, then decide if the error should be bubbled up or can it be dealt with via the logs. But catch the appropriate errors so they are logged.
3. You can write to Unified Logging Service(ULS) logs but these are often not checked or hard to find issues you have thrown up in your code, so consider using a logging block such as Microsoft's Enterprise Logging blocks or Log4net. Ted Pattison suggests writing to ULS and he know his stuff so if you don't have another specified logging policy write to the ULS. And if you do have another logging method consider writing to the ULS anyway.
4. There are a lot of logging applications for .NET, and most companies tend to have logging code ready for implementation on your SP 2010 project.
5. I have seen a MS gold partner use tracing on all projects. So when an error occurs they turn on tracing and try replicate the issue in production environments. Far better to catch errors so you can get your issues resolved quickly and don't need to change config setting or leave tracing enabled on the live production boxes.
6. A web part error can cause an entire page to throw an application error. However don't throw try catch blocks around all code, rather try catch the errors at more appropriate junctions such as at the service layer. Once again it really depends on the WP.

Summary: Avoid using errors for logic (pretty obvious). Log the errors to a distinct area (database, file system(xml files are pretty useful)) to identifying where and what the issue is. Catch specific errors -try not to catch general exceptions and if you do need to catch general exceptions make sure you have looked for more specific exceptions such nullrefobjectexceptions.

More Info:
MSDN SharePoint Logger - http://msdn.microsoft.com/en-us/library/ff798361.aspx
Writing to the ULS
Update: 5 Dec 2010 - Writing to ULS using SP2010 by Waldek Mastykarz
Update: 18 Jab 2011 - MSDN article on logging and debugging

The SharePoint Logger
using Microsoft.Practices.ServiceLocation;ILogger logger = SharePointServiceLocator.GetCurrent().GetInstance();
using Microsoft.Practices.SharePoint.Common.ServiceLocation;
using Microsoft.Practices.SharePoint.Common.Logging;
IServiceLocator serviceLocator = SharePointServiceLocator.GetCurrent();
ILogger logger = serviceLocator.GetInstance();
logger.TraceToDeveloper("Unexpected condition");

Update 14 Dec 2010 - ULS Viewer - Tool to view ULS log and filter data
Update 23 Dec 2010 - Tracing using CorrelationId