App Insights for Power Platform - Part 1 - Series Overview

Overview: Microsoft have great capabilities for logging and monitoring.  In this series of posts I will be examining the various parts of logging that may be useful in building solutions that are well monitored, provide alerting, easy tracing, and identifies issues or potential issues as soon as possible.

I am looking at App Insights for Power Platform monitoring.  So this includes: 

• Power Apps (Canvas, and model apps),
• Power Automate,
• APIM, 
• Azure Functions, 
• Azure Service Bus, and
• App Insights.

I shall be setting up a demo environment and these are the logical components being covered.

All the components making up the solution shall log into Log Analytics (left-hand side of the diagram).

For Continuous Integration, my clients will be Postman monitor (it's awesome and so easy to use all those postman collections), DevOps is great and I'll use it to run smoke tests after new releases.  I also use flows, to report on flows (sounds nuts but i love it).  These are at the bottom of the diagram. 

Lastly on the right of the diagram, I look at extracting logs for reporting (Power BI), and Monitoring using Azure DevOps (p.s. think about Grafana instead of DevOps Dashboards, it so nice).

Couple of extras are: Availability Logging, alerting, automating Canvas app testing, Playwright.  

From the diagram, you can see the data is now held in Log analytics and it can be queried via Log Analytics or App Insights using Kusto.  Note: the syntax is slightly different.

Series

App Insights for Power Platform - Part 1 - Series Overview (this post)

App Insights for Power Platform - Part 2 - App Insights and Azure Log Analytics 

App Insights for Power Platform - Part 3 - Canvas App Logging (Instrumentation key)

App Insights for Power Platform - Part 4 - Model App Logging

App Insights for Power Platform - Part 5 - Logging for APIM (this post)

App Insights for Power Platform - Part 6 - Power Automate Logging

App Insights for Power Platform - Part 7 - Monitoring Azure Dashboards 

App Insights for Power Platform - Part 8 - Verify logging is going to the correct Log analytics

App Insights for Power Platform - Part 9 - Power automate licencing

App Insights for Power Platform - Part 10 - Custom Connector enable logging

App Insights for Power Platform - Part 11 - Custom Connector Behaviour from Canvas Apps Concern

Tip: The Power Platform Admin Centre has a good overview of the Power Platform, but to make logging and monitoring better push data into Azure Log analytics and monitor and alert centrally.

Also see: View and download Dataverse analytics - Power Platform | Microsoft Learn

Uploading files to a Dataverse table using the "Add picture" control in a Canvas App

Overview: I need to be able to upload pictures or an file for that matter and persist the file in a Dataverse entity.  This took me a little longer than it should have.

Notes: I added a "Add picture" control to the screen, I also added a label and a icon ro Clear the uploaded image.

I also added a save button, here I persist to a Dataverse table named "Evidence", The Evidence table has a column called "File" of type "File". The part that took me awhile in the Patch statement was getting the file into the correct format (as it is a record not a picture).  This works for any file upload but has the drawback of needing to change the lookup to "All file types".  

Audit log retention in Dataverse

Overview: Audit data log retention is now fairly easy to implement in Dataverse, you can set whatever is audited and set the for how long duration easily.

Thoughts: As a simple version, I'd audit all changes into the Dataverse and set the retention to 7 years.  Now this could end up costing you a considerable amount of money so consider, do I need to audit everything, do I need to retain this long, can a use a long term storage retention approach.  There are a variety reasons for customising Datavervse data retention including: to comply with laws and potentially the need for litigation, to comply with industry standards/certification, and to keep a full history to understand why we have the current data position.

  

Ultimately, I need to identify/understand how to store audit history, clean up when no longer needed, ensure it is no affecting you live system performance, and can be retrieved by authorised people in the timeline required for each project or at an enterprise level.

If a system changes a lot and uses blobs, the audit history will be large and Dataverse is not necessarily the best place to store long term audit history.

Technical: Dataverse stores data in an Audit entity (table), the infrastructure has been changed in late 2022 to handle the audit data separately to allow for better non-functional requirements to available.

More Info:

https://powerapps.microsoft.com/en-us/blog/announcing-new-dataverse-auditing-features/

Reporting from Dataverse HLD options

Overview:  Recently I was looking at reporting from the Dataverse/CDS and I drew up these options that give the business various options but the overhead grew as the solution improves.  Client is on the MS stack so I have looked at the reporting options, There are a ton of variations but this is a good start for my options.

Reporting options from the Dataverse

Synapse, can be replace by any Data warehouse or Data lake solution and as Dataverse is not massive could also just use regular SQL Server for reporting.

Power BI can be replaced with other tools such as Tablau but for embedding and the MS stack, Power BI makes the most sense.

Model Apps - Controlling the UI with JavaScript

Problem:  I have a form of view in a Model Driven App (Power Apps) that needs to show 1 of 2 tabs depending on a control in the form having a value.

js> If the "Status"="Active show the "Agent" tab else show the "General" tab

Hypothesis: inject JS into the specific form that will check the status display control, and if active, get the Agent tab via the DOM, and hide it.

Solution:

1. Ensure you have two tabs on your form, my form is called "Information", see below

2. Create a JavaScript file to manipulate the DOM, my example is below:

Create a .js file to upload

Reference Blog for most common JS CRM work

3. Upload the file and publish.

4. Bind the JS function on the form or page being adjusted.

5. Result when the Status is active to hide the "Agents" tab.

Generating a Word Document from a Row change in Dataverse without SharePoint

Problem:  Dataverse/Dynamics has a great way to generate Word or Excel documents in Model apps using the UI.  I want to generate a generate letter from a CDS entity/Table when the status changes.  And I'm not allowed to use OneDrive/SharePoint (governance restriction).

High-Level Steps:

1. Create Dataverse Table - ensure "Notes" & "Business process flows" are enabled

2. Generate the Word Template & add it to the "Template View"

Generate Word Template with dynamic column vals

Add Template

3. Create a Process to generate the output/docx from the Word template i.e. generate when status of the row changes

Activate the Workflow

4. Use Power Automate to call the Dynamics Process

Dynamics 365 CE Intro

Overview: Dynamics 365 CE (Customer Engagement) is a subset of the Power Platform and Dataverse.  It provides commonly used entities such as Accounts and Contacts.  There are five apps that can add specific CRM functionality.  It is Microsoft's CRM solution.

There are 4 core subject (model) apps in Dynamics 365 CE

1. Sales (MB-210) - client management including leads and turning:
1.  leads (Qualify step) into an 
2.  opportunity (Develop step), goes to the 
3.  Proposal step consists of generating quote, revising quote, marking acceptance, and invoice, 
4. Order created/lost (Closed step).
2. Marketing (MB-220) - Lead generation & qualification, turn prospects in customers.  Build campaigns to market your business.  Email, social media, content creation to convert leads.  Ensure you target the right audience using the correct channels and breaking leads into segments.  I have only used this lightly and it is powerful.  Events are useful for planning such as speakers and managing leads/event analytics. Referred to as Dynamics 365 Marketing.  Customer Voice integrates with Marketing, e.g., embed survey sent our via email.
3. Customer Service (MB-230)/Customer support - Dynamics 365 Customer Service - Low tough Customer Experience(Cx) and it also provides for high tough as needs demand, provides omnichannel options, chatbot, email, SMS, service requests, wiki's, phone in.  Case management, raise record, route to where can be solved, and get to answer.
4. Field Service (MB-240) - Dynamics 365 Field Service.  Send people out to customer sites, schedule, ensure technician has work order and materials to complete the job.  

Common Data Model - Agreed between various companies including Microsoft so there is a set schema that we use as a base for our businesses.  Example entities:

• Account - Company or individual (I think of this as a legal entity).  Accounts have a primary contact, can have multiple contacts per account.  Accounts can have parent accounts.
• Contact - is a person, can belong to multiple account.

LinkedIn Sales Navigator - Sync LinkedIn, so a contact changes, can be pulled into dynamics.  Messages from Linked in are tract in dynamics.

Dynamics 365 Sales Insights - understand your relationships, and evaluate activities.

Dynamics 365 Customer Service / Customer Service Hub - 

    1) Raise a case via most suitable formats (align with SLA), 

    2) route case (align with SLA) e.g.,  

    3) Enter research such as calling customer, spoke to colleague against, knowledge base against the case.  Lastly, can record fix e.g., called customer and walked them thru steps, and problem resolved

    4) Close Case (includes reason)

Customer Service Notes

• Allows to integrate with other tools, ....
• Knowledge management - write articles in knowledge base (KB), submit for review, all staff can search to find fixes/info.  Can also import from SharePoint, EpiServer, Wordpress and others into KB.
• Omnichannel - communication with customer on multiple channels e.g., Teams, SMS, virtual agent, social media. Hook into external channels also.
• Analytics - measure customer service activity.  Show kpi's to show emerging topics, agent performance, how quickly cases are being resolved.
• Also has remote device management.

Dynamics 365 Field Service - Deliver on site service & manage workforce example is plumbing business or mining machinery maintenance.  Importantly deals with scheduling & work orders.  Deploy, fix issues/perform work from work order, the audit completion.

Note: MB-910 exam covers Dynamics 365 CE solutions.  It is a simple exam that does not require annual renewal and gives the hold CRM Dynamics Fundamentals Status.  The MB-210 specialises in the Dynamics 365 Sales and provides a Associate status which needs to be renewed annually with Microsoft. 

There are specific solutions focusing on ERP (Think SAP (Enterprise), Oracle/JD Edwards (enterprise), Baan now Infor, Sage(small-medium), Dynamics 365 (small to medium), Workday (seems to be everywhere lately) ) focus on areas like Finance, Operations, Supply chain management, Machine maintenance.  MB-920 covers the ERP fundamentals certification.

Generating a pdf from a word binary - Power Platform

Overview: Move a word document into a pdf stream in a Power Automate flow.

Solution: I am triggering a flow when a word document is created in Dataverse.  I get the word document in a stream and use OneDrive for Business (OD4B) to persist the docx to OneDrive.  I use the Power Automate Word for Business connector to convert the docx in OD4B into a pdf binary stream in my flow.  

Tip: The Location (OneDrive site collection) gets converted to a guid, so if you need a separate site collection for OneDrive or SharePoint, you can use the MS Graph and I believe this URL also works: 

https://radimaging.sharepoint.com/personal/paulbeck/_api/v2.0/drives.

Modeling and working with data on the Dataverse within the Power Platform

The Common Data Model - Is an industry agreed approach to storing commonly used data.  Use it to store applications across your company/organisation.  Results in a single source of truth instead of multiple copies with data contained in different schema's.

Two types of relationships namely: 1:N and N:N (Dataverse hides the associate entity/intersect entity that is created in the background).  

Use "Option Sets" for small static data and use "Lookups" for larger or changing data.  In Multiselect Option sets for N:N relationships, rather use "Lookups".

SSMS can be used to view data using T-SQL, better to use "SQL 4 CDS" as it provides a full ability to work with data within the XrmToolBox.

XrmToolBox - is a 3rd party download that has a ton of contributed tools and there are good options for modelling data.  This is a collection of tools that are unbelievably useful and get continually updated and new tools added.

Excel and browser plugins - Can be used to import/export data and there are some nice Edge/Browser plugins to help such as "Level up for Dynamics 365/Power Apps".  Level up for Dynamics 365/Power Apps is a fantastic tool that I encourage any developer to add to Edge or Chrome as an extension.

Level up for Dynamics 365/Power Apps Browser extension

Dynamics 365 Power Pane Browser extension.

Dynamics 365 Power Pane is also a useful extension shown below:

Power Pane options

The built in browser Dataverse management tool is super easy to use to model your Common Data Model further.

Microsoft Dataverse (CDS) - Overview

Overview:  Dataverse is CDS, there is a long story on the naming but ultimately Dataverse is a data store with a advanced security model, Open API's, workflows, pipeline injection...  It is awesome.

It is high performance, and would take considerable effort and components to deliver similar functionality or even semi close functionality.  It does have limitations mainly around performance but don't let that fool you, Dataverse is fast and powerful but for massive industrialized storage it's not the right option.  The costs are also a key consideration.

The biggest mistake I see is people making the same mistakes as they do with relational databases namely: 

Poor Dataverse implementation down to 1) poor entity relationship design, 2) either too many table containing duplicate data or to few table being expanded for a dev teams capability but ignoring existing systems, 3) poor security 4) too many cooks.

Basically, like any Database service, you need to have owners and try keep the structure logical and expand it appropriately.  The idea behind the data model used by the dataverse is to have centralized secure shareable data like customers or account information.  It's simple, treat dataverse as you would your most precious core database, have an owner that needs to understand and approve changes.

Note:  Microsoft have had some trouble naming Dataverse, it was previously known as the Common Data Service (CDS).

Dataverse logo

Overview: Dataverse helps improve processes.  And Dataverse helps reduce time to build IT capability, remove shadow IT, improve security and governance.  Data is the common data store we need to use to be effective.  As part of the Power Platform, it allows us to build custom software fairly quickly.

Updated 07-July-2022

Dataverse provides relation data storage (actually runs on Azure SQL (Azure Elastic Pools), Cosmos DB, and Blobs), lots of tools e.g. modelling tools.  I think of it as a SQL database with lots of extra features.  Most importantly business rules and workflow.

• Dataverse relies on AAD for security
• Easy data modelling and supports many-to-many relationships NB!
• Easy to import data using PowerQuery compatible data sources
• Role-based data (previously called row) and column (previously called fields) level security.  See Dataverse security in a nutshell at the bottom of this post.
• Provides a secure REST API over the Common Data Model, it's awesome
• Easy to generate UI using PowerApps model driven app
• Ability to inject business rules when data comes in or out of the Dataverse (can also use .NET core code)
• Can also stored files (ultimately in blob storage)
• Search that indexes entities and files
• CDS used tables, Dataverse calls them Entities.  Some of the UI still refers to table.  Just assume Entity and Table are interchangeable terms.

Dataverse basically allows you to have a PaaS data hosting service that mimics what we have done for many years with databases and Open API, has advance features and tooling and it is all securely managed for you.  

The cons are basically: is that it is expensive.  So you need to know your size and keep buying add-ons to the plans.  Scaling Dataverse is expensive.

Common Data Model: Collection of tables (previously called entities and most CRM people still call them entities) e.g account, email, customers for a business to reuse.  Comes from CRM originally, the starting point consists of hundreds of entities pre-created.  Common standard for holding data.

Each Power Platform Environment has a single Dataverse associate to it.  It's a good idea to have more than one environment but at it's simplest, use a trial to learn and progress to production.

Once I have a new environment, I can use Power Apps to access my environments Dataverse and model out a new table to store info, I am storing people tax returns.

Go into the Dataverse and model directly

Model the table in you Dataverse instance

Dataverse Security in a Nutshell:

1. A user is linked from AAD to the User entity in the Dataverse.
2. User Entity record is aligned to the AAD User.
3. AAD Users can be part of AAD security groups.
4. Dataverse Teams (Dataverse Group Teams) can have Users and or Security Groups assigned.
5. Dataverse Group Teams are aligned to Business Units.
6. Business Units have roles (rights).

"Security is additive" in Dataverse (generally the whole MS and security world these days).  i.e. no remove actions.  If you have permission in any of the groups you can access the data/behavior.

Business Units used to restrict access to data.  Can be hierarchical i.e. Enterprise > Audit > EMIA > UK (Don't use it like this, keep it simple)

Security Roles define a users permissions across the Dataverse entities i.e role can read only from Accounts entity 

Teams consist of users and security groups.  That get assigned roles.  There are two types of Teams in Dataverse: Owner teams & Access Teams

Field-level security, only allows specified users to see the field data

https://learn.microsoft.com/en-us/power-platform/admin/wp-security-cds (Good clear post on Dataverse security, core concepts are Business, Units, Teams, Roles, Users & OAuth/AAD)