Wednesday, 30 April 2025

MS Fabric OneLake Shortcuts

 "Shortcuts in Microsoft OneLake allow you to unify your data across domains, clouds, and accounts by creating a single virtual data lake for your entire enterprise.MS Learn

It allows open storage format data to be stored in the source system, metadata is added to OneLake, and the data can be queried; the load is predominantly performed against the source system, e.g., Dataverse/Dynamics.

Clarification: A shortcut is automatically added to MS Fabric for each Dataverse.  Dataverse creates Parquet files (est 5-10% extra data storage, counts against Dataverse storage).  Via the shortcut, report writers or data engineers can access the Dataverse data as though it is inside MS Fabric's OneLake.

Understand: Dataverse creates Parquet files that MS Fabric can look at to generate dataset data.

"Shortcuts are objects in OneLake that point to other storage locations.MS Learn

External shortcuts (data is held at the source system) supports any open format storage format, including: 

  • Apache Iceberg Tables via Snowflake,
  • Parquet files on SnowFlake,
  • Microsoft Dataverse
  • Azure Data Lake Storage (ADLS), 
  • Google Cloud Storage, 
  • Databricks, 
  • Amazon S3 (including Iceberg tables),
  • Apache Spart (Iceberg)
Internal shortcuts supported:
  • SQL Databases: Connect to SQL databases within the Fabric environment.
  • Lakehouses: Reference data within different lakehouses.
  • Warehouses: Reference data stored in data warehouses.
  • Kusto Query Language (KQL) Databases: Connect to data stored in KQL databases.
  • Mirrored Azure Databricks Catalogs: Access data from mirrored Databricks catalogs.
I think these are also Internal shortcuts:
  • PostgreSQL
  • MySQL
  • MongoDB

Example High Level Architecture

External shortcuts with snowflake and Dataverse.


Tuesday, 29 April 2025

PowerBI reporting against ProjectWise (Project Insights)

Overview: Integration with Bentley is fairly complex but the project ProjectWise Project Insights provides a Power BI Custom Connector.  The connector is not certified by Microsoft but it can be used for gaining reporting insights.

Introduction to ProjectWise's Project Insights

These are the instructions I followed and it still works

Power BI report - Simple - Using ProjectWise Project Insights                                        

ProjectWise Posts:

Introduction to ProjectWise and BIM

ProjectWise from Bentley - extend with Powershell

Postman Collection to connect to Bentley's REST API for ProjectWise (WSG)

PowerBI reporting against ProjectWise (Project Insights) - This post

Postman Collection to connect to Bentley's REST API for ProjectWise (WSG)

Overview: Here is a short recording showing how to connect postman to ProjectWise's API's (WSG).

Here is the postman collection: Tutorial on connecting to ProjectWise APIs using Postman

More Info Added Aug 2025: Postman Quickstart (good resource)

A short video (1.5min) if you want to play with the two endpoints I used:


ProjectWise Posts by me:

Introduction to ProjectWise and BIM

ProjectWise from Bentley - extend with Powershell

Postman Collection to connect to Bentley's REST API for ProjectWise (WSG) (this post)

Power BI reporting against ProjectWise (Project Insights) 

Sunday, 13 April 2025

Mendix - Logging & Tracing

Mendix offers a few integrations with various Application and Performance Monitoring (APM) tools.

There is no integration with Azure Monitor.  The closest I have is log files can be downloaded.

Cloud edition allows you to download the log files.

TBC

Friday, 28 March 2025

Power BI Premium to MS Fabric Primer

Power BI Premium allows all users in your enterprise to consume (use) reports, you buy licences separately for the report builders.

Two methods of getting report data: Import Mode (data is not live/real-time, and retrieval is fast) & Direct Query Mode (real-time data, but retrieval is slow)

All Power BI Premium subscriptions will automatically become MS Fabric during 2025. 

Direct Lake Mode

An additional method of querying, "Direct Lake Mode," combines the best aspects of both old methods for retrieving data.  Direct Lake Mode (real-time and fast data retrieval).

OneLake

  • Storage is in Delta Parquet 
  • Data is stored once, along with permissions, when copied into Fabric; the individual Lakehouses, warehouses, and transformations still rely on the original Parquet file data.
  • Shortcuts create a virtual pointer to a variety of data types such as Snowflake, ADSL, ...

Great Visual Descriptions of the 3 options: Comprehensive Guide to Direct Lake Datasets in Microsoft Fabric

Tuesday, 18 March 2025

Microsoft's AI & Copilots Explained as the naming kills me



Microsoft's Copilots products... starting with M365 Copilot options

Source: Screenshot from a presentation, 6 May 2025 - from MS.

1. Microsoft 365 Copilot versions

1.1. Copilot (Microsoft 365 Copilot Chat) (Is Free)

It allows access to ChatGPT-4 with web content.  AI like ChatGPT, Perplexity, ...

1.2. Microsoft 365 Copilot Pro

(+$20/month full retail if bought as an add-on).  Additional features, but not all that M365 Copilot for Business offers.  All users receive Copilot Builder (maker capability), and consumers require the M365 Pro license to utilise M365 Copilot agents.

1.3. Microsoft 365 Copilot for Business



I believe that many people are grouping "M365 Copilot Pro" and "M365 Copilot for Business" together, and only see the free and paid versions of M365 Copilot. 

M365 Copilot Pro for Business (I know :) ) is grounded in the business data you can access in your enterprise.  Inline copilots, including grounded business data inside the office applications.

Microsoft 365 Copilot is an AI-powered Agent with multiple child Copilots for Microsoft 365 apps like Word, PowerPoint, Outlook, Excel, and Teams. It leverages large language models (GPT-4 and GPT-4Pro) and your enterprise data through the Microsoft Graph. To work with M365 Copilot Pro, you need the M365 Copilot license for each user accessing or creating the new agent, which is pricy, per user per month. 

Tip: Optimise results by preparing content!  Like docx, pptx and pdf files. PDF, DOCX, XLSX (kind of), PPTX. Also works on CSV, text, HTML, MD, and WAV audio files.  Supported file type.

Microsoft-specific app copilots included in M365 Copilot Pro:

  • 2.1. Word Copilot: Helps with drafting, rewriting, editing, summarising, and generating new ideas.
  • 2.2. Excel Copilot: Assists with data analysis, generating insights, creating complex formulas, and automating tasks.
  • 2.3. PowerPoint Copilot: Aids in creating presentations, suggesting layouts, creating slides, and enhancing visual content.
  • 2.4. Outlook Copilot: Supports email management, drafting responses, scheduling, and organising tasks.
  • 2.5. OneNote Copilot.
NoteAs of March 2025, Microsoft Copilot does not currently allow uploading images or pictures, unlike ChatGPT; however, I don't doubt that this feature will be added in the future.  ChatGPT can also redraw (using DALLe).

Ease of use, Options for working with Microsoft AI
Extending the use of M365 Copilot Pro licensing
  1. M365 Copilot takes a no-code approach to building agents with our company data. Each user needs an M365 Copilot Pro licence to access the new bot/agent/copilot.
  2. Copilot Studio is a more advanced way to build copilots and requires a separate license.
Options: Building your own Agents/Copilots with business-aware data.


Agent Builder - all users need the M365 Copilot Pro licence.

2. Microsoft Dynamics 365 Copilots

Overview of Dynamics 365 Copilots:

Source: Microsoft (I lost the reference)

2.1. Customer Engagement (CRM) Apps

  1. Dynamics 365 Sales – Manage leads, opportunities, and customer relationships.
  2. Dynamics 365 Customer Service – Case management, knowledge base, and omnichannel support.
  3. Dynamics 365 Field Service – Manage field operations, work orders, and technician scheduling.
  4. Dynamics 365 Marketing (now part of Customer Insights - Journeys) – Campaign automation and customer journeys.
  5. Dynamics 365 Customer Insights:
    • Data – Unify and analyse customer data.
    • Journeys – Design and automate personalised customer experiences.
  6. Dynamics 365 Customer Voice – Collect and analyse customer feedback.

2.2. Finance & Operations (ERP) Apps

  1. Dynamics 365 Finance – Core financials, budgeting, and global accounting.
  2. Dynamics 365 Supply Chain Management – Inventory, manufacturing, and logistics.
  3. Dynamics 365 Project Operations – Project planning, resource management, and billing.
  4. Dynamics 365 Commerce – Unified retail, e-commerce, and POS.
  5. Dynamics 365 Human Resources (being merged into Finance) – HR management and employee self-service.

2.3. SMB-Focused App

  • Dynamics 365 Business Central – All-in-one ERP for small and medium-sized businesses (finance, sales, purchasing, inventory, and more).

2.4. Industry-Specific Solutions

  • Microsoft Cloud for Industry (e.g., Healthcare, Financial Services, Manufacturing) – Built on Dynamics 365 and Power Platform with tailored capabilities.

Note: Microsoft D365 Copilot/ Microsoft Copilot for Dynamics 365 and Power Platform/ Dynamics 365 AI are in this area.
Dynamics Product Copilot Licenses

3. Copilot Studio

Build copilots and distribute them to the business. This is awesome, but from a naming perspective, it actually makes sense. 

Copilot Studio - requires a Studio Builder licence for each maker, utilising low-code AI for build agents.  Each query is charged using messages as its currency.  There is complex pricing; for instance, a query can easily cost 13 message credits.
 

4. Azure AI Foundry

Organisations and developers can use the AI Foundry platform to build AI-driven solutions.  Can use 1,800 models and 200 Azure Services.  There are numerous AI services available on Azure.  I'm not even going to start that conversation.


AI Foundry - The individual paid services on Azure relate to token usage within AI Foundry for billing.  A simple query is 4 tokens, depending on how the question is broken down.  Once again, it is difficult to estimate the costs. 

5. Security & Governance of Microsoft 365 Copilot 

The M365 copilot will respect the Access Controls you have permission to, so Jim in operations won't see Tim's sales figures in a Word document in SharePoint. Jim does not have access to the document from the sales site collection. Therefore, they can't query Copilot for the data they don't have permission to (done using MS Graph).

Purview: DLP improves what your users can do and promotes better governance.  Ensure M365 copilots are used ethically.


Understand how users are using your M365 Copilot.

6. Code Copilots

GitHub Copilot

GitHub Copilot is an AI-powered tool that assists developers in writing code by suggesting snippets and completing lines in real-time. Developed by GitHub and OpenAI, it learns from public code repositories and supports various programming languages and frameworks. Its goal is to boost productivity and reduce repetitive coding tasks.  It's amazing!! And getting better all the time.

Code Copilots: GitHub Copilot extensions in VS and VS Code.  GitHub comes in three flavours: Individual, Business, and Enterprise.

GitHub explains code, optimises your code's performance, improves readability, generates unit tests, improves error handling, adds new code with requested functionality, ensures coding consistency, and improves modularisation to help with the DRY principle.  GitHub is the diggity bomb.

Note: ChatGPR is owned by OpenAI.  Microsoft is a significant investor but does not own the service.

Amazon Q

Amazon Q Developer is an AWS code generator designed for building code and solutions.

Cursor, Windsurf.io, ...

7. DSPM (Purview)

DSPM stands for Data Security Posture Management. It's a cybersecurity approach focused on identifying, monitoring, and securing sensitive data across cloud and hybrid environments.  Copilot integrates seamlessly with DSPM in MS Purview, controlling access through DLP and various other options, such as SIT (Sensitive Information Types). It offers numerous out-of-the-box features but can also be customised to help reduce the exposure of organisational data using AI.  Document Fingerprinting also helps reduce exposure. 

Content Explorer in Purview is handy.  

To access all this valuable information, ensure that all your data sources are OAuth-enabled.

7. Summary

Naming and complexity with Microsoft are tough. The AI parts are massive. The M365 copilots break down into Dynamics or O365 worlds, each with many options.  I wish Microsoft had a hierarchical, sensible naming system, which no doubt the Marketing department would hate.  Sources must be secured correctly, and purview can help you manage access.

My AI Posts



Microsoft AI Landscape Pie.



Amazing Summary..

Sunday, 16 March 2025

SAP LeanIX Intro

  • LeanIx is an SAP Saas product that facilitates Enterprise Architecture (EA) in businesses, utilising dashboards and diagrams to support informed decision-making.   
  • To create the roadmap, align business goals with IT strategy by generating 'As-is' and 'To-be' architectures.
  • Improve integration, reduce complexity, and eliminate duplicate systems to make better decisions.

Note: Using Leanix is an all-in or don't bother tool.  LeanIx provides a comprehensive view of your applications, business processes, and interactions with IT assets.

3 Products make up LeanIX:

  1. Application Portfolio Management: The base product for recording the application landscape allows for inventory dependencies and identifying redundancies to optimise the application landscape.
  2. Architecture and Roadmap: Using the Application Portfolio, visualise the 'As Is' and 'To Be' architectures and visualise the impacts of change.
  3. Tech Risk and Compliance: We verify technology standards and can fund projects to help address data residency rule breaches.

Twelve fact sheets, such as the application fact sheet template, are essential. The meta model is the plan that shows how information in the fact sheets is interconnected.  

The "LeanIX Meta Model" at a high level shows the 4 EA architectural views:  

  1. Data Architecture
  2. Application Architecture
  3. Business Architecture
  4. Technical Architecture 

On top of the four architectural pillars sits Strategy & Transformation, covering:

  • GAP & Roadmap, 
  • Principles, 
  • Requirements and Assumptions.

What I Like:

  1. Inventory of Products, dependencies and link to documentation
  2. Nice predefined reports and the ability to collaborate with stakeholders
  3. Visualise dependencies, excellent reporting capabilities

Note: LeanIX is based on The Open Group Architecture Framework, i.e. TOGAF.  TOGAF's Architecture Development Method (ADM).

Technology Landscape

I love this picture, unfortunately, I don't know where it comes from..