Micro services Pattern without Microsoft Gateway for on-prem. data

PB APIM Series:

APIM Notes

APIM Testing with Postman

APIM Mocking

APIM Mocking part 2

APIM Mocking part 2

Secure APIM with AAD B2C

Handling Security Incidents

Security Incident: An incident that potentially has compromised a companies systems or data.

Goal:  Focus on restoring confidentiality of systems/data and prevent further attack.  Contain the incident and eradicate the issue.  Full resolution target timeline is met for incidents.  These incidents can take up to 100 days but depends on the complexity.  

Examples:  Virus, Trojan Horse, Stolen data, increased unauthorized permissions, compromised server, copying data, DoS, unauthorized system access, ....

Need to record each event and work through the life-cycle (ISO 27035).  Can be dedicated software or modules such as ServiceNow's Security Incident Response (SIR).

1. Plan & Prepare
2. Detection
3. Assessment and Decision - Get logs, review/analyse, document the findings, notify leadership teams.  Impact/Priority e.g. Critical vs Low business impact.
4. Response - limit damage plan, decide on approach, notify if needed and remediate.
5. Lessons Learnt - ensure the threat is removed and potential lessons can help improve the attach surface for similar issues.

https://en.wikipedia.org/wiki/Computer_security_incident_management

Note: Be careful not to delete forensic evidence.

Tip: Organisations must have a Security Incident Plan.  Plan, be ready, know what to do in advance improves the handling of Security incident.

Power BI Notes

Overview:  Power BI is for reporting  and analytics of your data.  There are basically 2 ways to show Power BI Reports: User specific and app specific.

Power BI Embedding Models:

1. User Specific/User Owned Data - Call the Power BI services as yourself/the current user using delegate permissions. 
2. App Specific/App Owned Data - Call the Power BI service using a generic app permissions.  For example a public website, no Power BI licence required and every user of the site has the same access to view Power BI data.

Power Automate Notes

What is Power Automate?
Power Automate previously called Flow.  Power Automate contains "Flows".  Power Automate is workflow including RPA options, refered to a Power Automate Desktop (PAD).  Power Automate is a workflow engine that is based on Azure Logic Apps.  Powerful extendable workflow solution for low code automation.  Allows workflows to be easily automated with 3rd part systems e.g. SAP.

Used for:

1. Personal Workflows e.g. I send an email to all people that have not update the DevOps Scrum board on a in the last day as a scrum master.
2. Business Process e.g. Holiday request form.  If more than 10 days, need senior manager approval.  Generate RFP based on an event.  Historically, used K2 or Nintex or WCF workflows for business processes.
3. Integration: e.g., move twitter posts into my warehouse for data-mining later.

A key concept is who shall the flow run as, a user or a service principal.  Setup a Service Principal in Power Automate » Benedikt's Power Platform Blog (benediktbergmann.eu)

Flows run against the owners account for calculating allowable usage (default), it is common practice to use a service account (normal AAD account with user and Pswd) to own specific flows.  A service account is a normal user account from AAD/Entra's perspective.  A dedicate account ensures that when a specific user leaves your business, the flow continues to run.

Img 1. Usage limits in Flows are counted against the flow owner.

Agree Power Automate Standards:

1. Flow Names: start with a verb  format: Verb + What the Flow does + (trigger) e.g. "Get Tax Rates (Instant)".  I like to also prefix with the Company and Project, but feel free to have a standard to suit your business.  e.g. EY-USTax Get State Taxes (Instant) or EY-USTax Get All US State Tax Rates (Scheduled) or Get SalesForce Data.  Optional, for project specific workflows I also prefix witht he project name e.g. USTax-GetTaxRate. 
2. Description: Short description to help readers understand what the flow does.
3. Actions: Leave the Action desc and add info e.g. Compose: Tax Note.
4. Variables: prefix with "v" e.g. vTaxTotal in camelCase.  e.g. vUserAge.
5. Error handling & Logging: Catch errors and log into to App Insights via an Azure Function or Log using the built in Azure Log Analytics Action.  More logging means better traceability.
6. Scope: Add scope actions for try catch logic.  Add multiple actions inside a "Scope" Action
7. Terminate the flow with the Terminate Action if the flow has failed.
8. Environment Variables: Great for logging as I go thru DTAP.  Also see.
9. Connection Reference Name: Agree a format, does this flow run as  user or as a specified user.
10. Loop Sparingly, use First() for performance.
11. Owner: I like to use a service account in dev, it's a good idea to add tech owners as when it needs updating to support and easily find who they should talk too.  Understand who you are running the flow as, this ties to licencing but is critical.  You need to know you Actions and licencing limits on a project.
12. Comments:  Im not a huge fan as the naming should make most flows make sense/self documented, but for tricky logic, comments are great.  Agree a standard.
13. Retry policy: What happens if an action fails, do you want to try again?  

I borrowed a lot of the standards from "Matthew Devaney"

I found this document later - it is excellent. by Prasad Athalye - Best Practices for Power Automate(Microsoft Flow) Development

Licencing:

1. Seeded licence is part of O365.  Use standard functionality such as standard connectors without needing to pay more for advance.  The advance/premium connectors are not part of the O365 licence.
2. Per User licence -  Allows the  user $15 retail, can get discount with bulk and can use the advanced connectors & on-prem. gateway.  Many users need multiple workflows, normally personal workflows.
3. Per User RPA licence - same as above but also has amazing RPA capabilities.
4. Per Flow/Process - $100 per process per month, min 5 flows per month licences.  Anyone can use as part of the process.  Use for few people but process does a lot of workflows.  Can add a process one at a time after the first 5.

Licencing MS page
Power Automate has some licence add-ons available: AI builder and an unattended RPA add-on.
"Power Apps licenses will continue to include Power Automate capabilities", I don't know what is included but I assume it means any connector I can use in Power Apps, assuming I'm in Power Apps I can make Flows for.

Build workflows:

• Can get a dedicated IDE tool for Power Apps or use the browser (which i always use).
• There are over 350 connectors (in both standard and premium) and you can always use a custom connector to any OpenAPI (Swagger) endpoint.
• Templates have some examples and often help as a starting point to make your own custom Flows in Power Automate.
• Easy clear tracing so you  can see what part of the workflow is working and where you fail, and you can drive into the issue.  Super easy to use.
• Example of an Instant Cloud flow triggered by a canvas Power App...

Query a Dataverse table in a Flow using OData

ParseJSON is fantastic for converting Open API/OData/JSON into an object

Extending - break out for programmatic control, I use C# Functions from my Flows and call them via HTTP triggers.

Retrieving  a row from the Dataverse custom "Subject" table.

Robotic Process Automation (RPA):

• Also known as UI Flows within Power Automate.  Microsoft have purchase and integrated Softomotive for UI flows to add WinAutomation.
• Attend (user logged in) and unattended version (complete tasks without manual intervention)
• Can have multiple instances
• API is generally better than using RPA as it is versioned and generally not changeable, whereas using a website, they website can be changed causing the RPA flow to fail.  Useful for instance when the RESP API is incomplete.
• Recording tool for creating UI flows - Web use Selenium to record.
• 3 Types: 1) Windows /Desktop/Screen reader and 2) web/website (Selenium) and 3) WinAutomation (covers both Windows and Web, easy to use but not as full featured yet).
• WinAutomation has a drag and drop IDE, has error handling.
• UI flows are well priced.  Also get AI builder credits with UI flow licences.
• "Power Automate Per user plan with attended RPA to use UI flows and WinAutomation" Microsoft.

AI Builder:

Cognitive builder e.g recognize forms and extract data. E.g. receive invoices and add to accounting SaaS software.

Other: Zapier is a good tool for end user automation.  Easier than Power Automate but not as structured.  I'd use Zapier to automate in small businesses without O365 or flow licencing and allow end users to do it themselves.

Problem Solving:

Problem:  Another developer created a Flow I need to use from a Canvas page inside a model app.  The Flow is showing up when i say add but i get the error "Unable to add flow"

Initial Hypothesis: The flow is owned by another developer having ownership and the connection is done thru their account, take ownership and change the Connection (Dataverse connection in my case)

Resolution: To use an existing flow, you need: 1) flow in same solution as the canvas app, 2) Ownership and the connection string needs to be switched to the new developer

Problem: Migrating solutions between environments, all the workflows fail when they use the Dataverse connector with 403 errors. Tracing the flows, I can see the error  "Flow client error returned with status code 'BadRequest' and details {error: code: 'XrmApplyUserFailed... UserNotinActiveDirector ... does not exist in user tenantId"

Problem: Using the service Principal account needs to be registered in the new Dataverse environment.

The connector issue was fixed, we had to recreate the connection from scratch making sure it was set up with a/the service principal. Then we added the registered app into the environment as an application. user.

I love Power Apps

No self respecting technologies/programmer/ex-programmer likes simple solutions.  I still chose C# as my go to problem solver (I learnt to use a hammer 18 years ago and now all my problem look like a nail).  Don't tell anyone under 30 I still love C#, they think you are weird if you don't use Javascript for everything.  And 15 years ago I was arguing the same with the C++ golden oldies.  So the circle of life goes on.  Enter Simba, the Lion King...  Well not really but Power Apps is just making programmers look ridiculous.

Power Apps can do a lot...

MS have announce that components and PCF’s are part of Power Apps.  I've been using this for awhile and it's awesome.

https://powerapps.microsoft.com/en-us/blog/announcing-the-public-preview-of-canvas-components-and-power-apps-component-framework/ 

This diagram outlines extending Power Apps using Components:  Developers often say you can’t extend Power Apps so it is not a real language, my answer is really… 

The other argument for PowerApps is if you need to do any complex programming.....  Use an Azure Function (this allows C#, PowerShell, JS, …).  Simple to build in your own language, expose using an http endpoint and you can call the end point using a custom connector.  

So if you need: 

• Complex UI = go PCF, 
• Reusable UI e.g. menu's = use Components, 
• Complex processing or secure code logic = Azure Functions.

Here are some PCF examples that we can use that are already built:

https://powerusers.microsoft.com/t5/Canvas-Apps-Components-Samples/bd-p/ComponentsGallery

So what is stopping Power Apps:

1. Licencing - the licencing is expensive so you need to choose selectively when it is appropriate.  If MS change it's going to just go crazy the usage. IMHO. 
2. Functionality:  MS have introduced automated testing to the Power Apps platform but there are improvements scheduled.  For me the biggest issue is code reuse.  I wish there was a function library that you could write functions to and call in a single line of code, my individual field logic can be insane and tough to amend on complex large Power App UI's.   

Excellent blog on Power Apps

Top 10 Power Apps Articles Of 2021 - Matthew Devaney

Catch Error in Power Apps and App Insight Logging

Error Handling:
App Insights logging: https://sharepains.com/2019/01/24/powerapps-experimenting-with-error-handling/  Replaced as Microsoft have built in telemetry as of 3 Feb 2020.
https://powerapps.microsoft.com/en-us/blog/log-telemetry-for-your-apps-using-azure-application-insights/

Example Error capturing and tracing to Azure AppInsights:
IfError( // Perform API Call , // Fallback so log here! ,

    Trace("Pauls Unique PowerApp",TraceSeverity.Error, {UserName:User().Email,         Role:gblRole, ErrorMsg:ErrorInfo.Message, ErrorControl:ErrorInfo.Control,         ErrorProperty:ErrorInfo.Property});     Notify("Err message ..." & ErrorInfo.Message); // Display the error on the UI

More detail..

Possible Canvas Apps Error Handling Pattern:

1. Ensure AppInsights key is added to each canvas app
2. Use IfError() to check calls and logic
3. Use the Trace method to write info to App Insights
4. Do I want to enable the Experimental error handling features (great to trace by correlationId)
5. Consider all Power Automate that use Power Apps (ensure you use the V2 Connector)
6. Never use IfError to handle business logic

To Review your App Insights Logging:
Open you Azure Portal > Open your App Insights blade >
Click the "Search" navigation option > Free text entry e.g. "Loyalty PowerApp"

App Insights, finding Traces generated in Power Apps

Monitoring Tool within Power Apps
The Monitor tool in Power Apps is great for debugging and tracing.

Start a monitor on the open Power App.

Monitor Tool - Showing a GET via a custom Connector and the returned response

Function/Code Logging:

Server-side code should log to App Insights or you logging framework.

It is ideal with the Trace within Power Apps explained above to be used in conjunction with 3rd party API calls.

Overview: C# code needs to have logging. If an error occurs an appropriate response must be bubbled up for the next lay

Possible C# Error Handling Pattern:

1. All catch write exception to Log analytics or App insights 
2. Calls to data sources, Azure Services and third party API's and complex logic ideally should be wrapped in a try catch and log the error to App insights using the C# App Insights SDK 
3. The catch blocks ideally return the failed information so the caller code can deal with the logic using the output.  If you don't deal with the returned message, simply log the exception and rethrowing the error (this needs to be a conscious decision on each catch) 
4. Catch specific errors: log, if you don't pass info to caller rethrow the error if applicable (bubble), respond accordingly i.e. catch the specific error and lastly use a catch all. - Heavy, but only add to existing code where this happens often or we are having problems, i.e. be specific
5. Don't use Try, Catch to deal with business logic

Thought: Bubble up means: Code must log exceptions and returns appropriate reply to the caller, if you don't send the appropriate reply rethrow the exception after logging it so the caller has to deal with it.

Power Apps Component bug fixed

Problem:  Creating a new component with Input parameters.  When using the component and setting the custom input parameter with a global variable, the value passes thru, as the variable is reset, the control does not update with the variable.  Replication Steps are recorded  in the 2 min 20 sec recording showing the issue.


Initial Hypothesis: I have a large Power App project that uses this technique a lot.  The global variable were working until Fri 14 Feb 2020.  I believe the Power Apps version has been updated and is causing the issue.  I have tried the work on multiple environments and tenants.  It is happening across them all.

https://powerusers.microsoft.com/t5/Building-Power-Apps/Power-App-Components-not-working-with-Global-variables-used-with/m-p/473414

Warning: This affects Power Apps release 3.200221.27 (rolled out to me on Fri 14 Feb 2020), I think 3.20014.23 (latest release as of 13 Feb 2020) was still working correctly.  Don't change your app if it published on the lower versions until Microsoft release a new version.  This only affects Power Apps using Components and using Global variables on the component's input parameters.

Resolution: Power Apps release 3.20022.17 as of 20 Feb 2020 (UK tenant) has been fixed.  Quickly identified and corrected.