Ignite 2023 - Microsoft Fabric - Introduction

GA: Prepare your data for AI innovation with Microsoft Fabric—now generally available | Microsoft Fabric Blog

Everything is brought in and available for analysis in a single Service.  Microsoft Fabric is a unified platform that brings all your analytics under a single service.

OneLake - per Fabric instance.  Stores all data within the SaaS data lake (scales itself), automatically indexes data, and abides by AIP rules/labels.  Intelligent data foundations.

All data is held in the Delta Parquet format (same format for any source).  Data is ready to use.  One copy of data.

SaaS single service, no need to bring pieces together; one data source doesn't need to moved to slice data.  Data stays at the original source but can be worked with, this all falls under the OneLake concept.  Can query using multiple approaches. Create a shortcut to files/folders/databricks, and it becomes part of OneLake while the underlying data resides in the original location that is now linked (only works on Parquet and specific file types).  

Mirroring in MS Fabric - get same benefits of shortcuts, but can connect to databases including SnowFlake, Dataverse, AWS S3 buckets & CosmosDB.  Mirroring is always up to date in real time.  Data is stored in Delta Parquet format so can now use.  With these 2 approaches can use nearly any source. lots of connectors so you could use: Dataverse, Cosmos, Snowflake, SQL Server, blobs on S3,..  Then can write queries across all the data. 

Copilot in Microsoft Fabric will help bring and analyse all the data.

Copilot for Power BI is impressive for building reports, but I need to play with it more.

Power BI is basically becoming Microsoft Fabric. The report generation piece is still called Power BI, but it falls under the MS Fabric product. Licensing for PowerBI Pro is converted to MS Fabric, and you cannot stay on Power BI Premium.

MS Fabric has a new way to access data. It is impressive in that it is fast, real-time, stores data once, carries ACLs/permissions with a lot of the data.  The ETL capabilities are amazing and configured for development.

MS Fabric also supports Real-Tile Intelligence (RTI) and SQL Azure integrated.

Last updated: Feb 2025

Ignite 2023 - Keynotes - Summary

15 & 16 November 2023

Good Keynote: AI is driving a lot of innovation.

Microsoft Fabric in GA (25k instances already).  New feature is 'Mirroring' - copy cosmos/SQL et al into Fabric.  OneLake.  Can bring lots of data from multiple sources into Fabric in near rela time.

MS Teams (320 million users):  Bring everything to the user in one place, not just communications but a canvas for apps.  Good place to build line of business applications.  New teams app - way faster, easier to use.  Teams Premium - intelligent meeting recap is working well, can integrate recap with copilot. 

Copilots - needed for nearly everything you do.  understand context of where you are.  MS has hundreds of copilots.

Copilot Studio - Custom GPT's, can add plugin's to add your own data, can hook into an enterprises unique data.

Copilot for Service - allows agents to get information to provide support, looks interesting.

Personal thoughts: AI is going to be a mega trend that will influence the world hugely, there will be lots of weird decisions on the journey. Currently, it is mainly proving useful as another tool to help improve existing processes.  AI helps me work faster and spend my time on exploration rather than bring base understanding together.

Part 2 Keynote:

Microsoft Graph gives the copilot context within an organisation.  Use plugins to add enterprise data or Open AI GPT's.

Surface Pro Hub 3 released  - looks good, rest of the hardware looks higher spec.

SharePoint Premium - improved knowledge and content management on SharePoint.

Copilot Studio - useful to build internal copilots.  1. Connect copilot to other systems using plugins or GPTs 2. create workflows 3.  Controlled by IT.

Copilot Studio overview

Mesh - Teams can join immersive experience/events, not sure what this means.  GA expected Jan 2024.

Microsoft 365 copilot release to GA 1 November. 

Why Copilot? MS are describing it as a productivity multiplier.  Allows users to be more productive and more creative.  Improves quality of work, avoids searching - as expected.  Makes mistakes but is getting better.

Microsoft Copilot - Bing chat is just MS copilot

So when logging use Entra Id (Azure AD), get contextual enterprise information.  Inherits security and privacy policies.  ACL controlled.  Includes MS graph and Apps.

Try it out: Copilot.microsoft.com  - Chat data is not saved/stored by MS.  Change from "web" to "work".  Also available in Windows taskbar

Ability to use copilot to pull in information, show more graphs, get data.  Good word example getting data from a pptx.

Great example of querying Excel using copilot, created a pivot table. Contrived but looks good, added rainfall from web to look at sales.  Powerful.

Never thought of copilots for being a participant in a meeting - might us amazing.  In teams meeting, takes real time notes, and pulls in info and summaries points for next meeting.  Add as a collaborative partner, can visualize discussions on a meeting whiteboard.

Loop - flexible collaboration, now with copilot.  Not my area but sounds impressive - i don't get it.  People, working with people, now also working with copilot.  Okay.

Copilot for Sales - looks promising.  Hooks into existing CRMs.

Copilot for Service - working with customers, get data that is correct to solve customers problems.  Concise summary, helps craft emails, updates CRM.  Looks very interesting!

Viva - Microsoft copilot dashboard powered by Viva - not sure on this topic.

Summary: People using copilot don't want to loose it.  AI is bringing big changes to many industries.  Promise is to take the grind out of work - sounds great let's see.  Copilot/AI will be a tool and shape how we work.

Thoughts on Logging and Monitoring

Overview:  I mainly work in the Microsoft stack, so my default for logging as Azure Monitor.  Log Analytics/Workspace and Application Insights fall under the term Azure Monitor.  

Going forward MS are storing App Insight logging data within a Log analytics instance.

There are 4 options for displaying/analysis logs in Azure:

1. Azure Dashboards
2. Power BI
3. Grafana
4. Workspaces

SIEM tools take in logs from various sources such as Azure Log Analytics, Defender, other vendors Prometheus logs or Open Telemetry.  

Grafana can be used on most SIEMS including Dynatrace, NewRelic, Microsoft Sentinel, or Azure Monitor.  Grafana supports PromQL and has fantastic dashboarding.

Azure DDoS Sentinel Overview:

Microsoft has the "Azure DDoS Sentinel" service that can help protect your network endpoints from DDoS attacks.  Common DDoS attacks all basically use hundreds of bad actors to flood traffic into you architecture to overwhelm them.  Restricting traffic from the bad actor sources is key.  Mixing the Azure DDoS Sentinel Service with Azure WAF, allows us to identify the bad actions and just block these bad attackers.

DDoS - Increasing, multiple bad actors try overwhelm your resources.  Rate limiting can help, but ideally you want to let thru valid traffic and block bad traffic.  Azure DDoS Sentinel service can be coupled with WAF to protect correctly from DDoS attacks.  Normally UDP flood attacks, also protects HTTP(s) flood and TCP Flood attacks. Covers level 3-4 layer attacks.

Two SKU's:

• DDoS network protection: used on a VNet, service will work out and protect your public nodes.  Can put this in from for Azure WAF are Azure Firewall, After Front Door.  
• Cheaper Alternative is the DDoS IP protection, has most of the features and if only specific IP, like a web traffic IP it's a good option.

More Info:

Radimaging Ltd - Paul Beck's Technical Working Notes for Microsoft Technology: WAF Options (pbeck.co.uk)

Radimaging Ltd - Paul Beck's Technical Working Notes for Microsoft Technology: Azure Application Gateway Basics (pbeck.co.uk)

Mendix Overview

Overview: Mendix is a market leader in low-code app builders. While I predominantly use the Power Platform, Mendix can be a good option.  

The ALM has Version Control. This is intuitive and follows a local checkout version commit back to a main branch (simple version control). It allows you to use branches so comprehensively and flexibly. Checking in small and often is a good idea, or you run the risk of large, complex competing merges. I believe it is git, but from the Mendix Studio Pro IDE, it is seamless.  

Build a local Version using Mendix Studio Pro and deploy it to the cloud. There are several options, including on-prem. The free version is basic and has limitations, but it has proven helpful for learning.

Note: There is also a Mendix Studio (Not Mendix Studio Pro) but it is hardly used by anyone.

Mendix supports sprints and boards, so you can work with User Stories in the Developer Portal for ALM.

An App Package can be stored, and it is a good idea to use this as the base for all projects in your company so that basic branding and naming conventions are consistent.

Deployment anywhere such as on-prem. via Kubernetes deployment, as well as the major cloud platforms i.e. AWS, Azure, GCP, Oracle.

Marketplace - templates, connectors, and components to reuse. 

Domain Modelling is excellent. When creating an app, you can choose your database (not the canned version). Modelling and exposing via OpenAPI contract and generating CRUD screens are easy. 

Get Data Options

Types of Data Sources available in Mendix.

Logic/Programing: In Mendix, we use flows for logic; there is no programming language.

• Microsflows - Server-side logic.
• Nanoflows - Smaller basically client side (think JavaScript logic)

Tip: The variable state is typically stored within the local Postgres database for later use.

Publishing to cloud production versions is very easy, and the local version, which is being developed, is seen on localhost. Mendix has simple, great built-in source control.

Image 1. A high-level overview of the logical components making up Mendix.

Pros:

1. Easy to use.
2. The basics for Low code are included, such as version control, project management, and deployment/publishing.
3. Build native mobile apps.
4. Improve business processes efficiently.
5. Supporting multiple languages is unbelievably simple and easy.

Image 2. Add multiple Languages to your app

Simple exercise: Call a key-secured API and display it on a Mendix page after watching this 7-minute video on API Calls.

The running example has:

1. Various pages and forms show and persist database information. 
2. A REST Call to a 3rd party using the OAuth key.  
3. Publishing a REST API based on a table and an associated entity.
4. Displays an Azure Chatbot

Me playing around with a Mendix App:

1. Get a REST endpoint and verify using postman (using a key for secure access)

Image3. Postman showing the REST call to be used

2. Create a new "microflow" as shown below:

3. Add a new "Action" of type "REST Call"

4. Add a JSON Structure file

5. Decide which attributes to pull out

6. Create an "Entity" in the Domain model to hold the retrieved data.

7. Map Model to the Import as shown below

...

Mendix Series

1.  Overview of Mendix (this post)

2. Mendix - Part 2 - Diving deeper

3. Extensibility for Mendix Studio 

4. Building a Mendix Widget for the Mendix marketplace (Convert text to Audio using Azure AI)

5. Mendix Tips & Thoughts

6. Competitors

Dynamics & Power Platform browser extensions and tools

Key Tools and Browser Extensions for Dynamics and  Power platform Developers:

• Level up for Dynamics (extension)
• Dynamics 365 Power Pane (extension)
• Microsoft power automate Desktop (extension)

Microsoft Azure Artificial Intelligence (AI) - AZ-900 Notes

1. Artificial Intelligence (AI) 

• AI making PC behave like human intelligence.  
• Teach PC to do task for us.  
• PC predicts using patterns and can act.  And good at looking for anomalies.
• PC uses camera/photos to look for patterns.
• Engage in useful conversations, use multiple sources of knowledge.

2. Machine Learning (ML)

• Train PC's to see patterns and see patterns, and look for anomalies.
• Example. predict stock prices by looking at factors that affect stock price.
• Anomaly Detection - Detects unusual patterns e.g. CC used in Asia when normally in Europe, but transactions 10 min apart.  Therefore likely to be fraudulent.  Sort rubbish.
• Predictive models by finding relationships.  Give model data and train the model.  
• Example: flowers have features/characteristics e.g. colour, size, no petals, ...
• Using data to teach machine
• Supervise ML - need quality data including labels.  Avg humidity, hrs sunshine, rainfall, temp, month of year (features), ice creams sold(label/class), so we feed in temp is Regression ML.  Patient has features (weight, sex, age, bmi,...) give value btwn 0-1 of the person developing diabetes.  is Classification ML
• Unsupervised ML - data is not labelled.  Just feature provided, will group into clusters.  Pulls data out and figures out it's own criteria is Clustering ML.  Useful for fraud detection.
• Training - good data based on a training set and a validation set.  Train model, with most data, then check with remaining - allows us to see how close to what happened.  Service tries to figure out relationships.  Model is used by test data - see how close/useful model is.

3. Compute Vision 

• Self driving cars, sorting. Sort rubbish.
• Facial recognition, object recognition,..
• How do computers see?  picture is cut up into pixels, data is pulled and used to find possible ans. 
• Some types on Azure:  object detection i.e. car, bike, car, bus.  Image classification i.e horse, car.  Semantic segmentation i.e. Teams blur background.   Image analysis contect by bring various tougher.  Face detection.  OCR - read image and converts to text.

4. Natural Language Processing (NLP) 

•     interpret e.g grammerly, spam check, Alexia, 
•     Knowledge Mining - Extract info from knowledge and gain insights e.g. social media marketing.

Principals:

1. Fairness - ensure bias is excluded e.g based on gender.
2. Reliable & Safety - need high confidence and in certain systems cannot fail e.g. health systems, autonomous cars.
3. Privacy & Security - Ensure data is protected and not giving away sensitive data.
4. Inclusiveness - should be fair i.e. VI users
5. Transparency - what is the model based on, what could be an issue.
6. Accountability - who is liable for AI decisions

Azure:

1. Scalability & Reliability
2. AI Resources: sit in an Azure Resource Group

AI Services in Azure:

1. Azure Machine Learning - Developers to train, test and deploy ML models.  Within a subscription, create a Azure ML Workspace (consists of: compute, data, jobs, models) can then publish as a service.  Azure ML Designer, used for creating ML pipeline, data in to train model.  Automated Machine Learning user only needs to provide the data and select the model to use, service figures it out. 
2. Cognitive Services - vision, speak, language, decision.  Rest API endpoints - have already been trained, choose the model.  Can deploy multiple parts individually or together.

    3. Azure Bot Service - develop & managing intelligent bots like chat-bots
    4. Azure Cognitive Search - Data extraction, & enrichment for indexing.  Makes data searchable.

Anomaly detector resource - wizard to setup - Add Keys and endpoints to allow access.

Create a new Azure Machine Learning Service, will create a Workspace.  Use multiple azure services such as key-vault, AI, storage accounts.   

• Launch Studio
• Add Compute Cluster
• Add Data (csv, spreadsheets, nearly any form,...)
• AutomateML (figure it out without me) and run job
• Will show trends
• Deploy the model (i.e. to a web service)
• Shows "Endpoints" - get url and a test rig.