NoSQL Document Database options on Azure - CosmosDB

Overview: Azure has a plethora of options for using NoSQL, I have used RavenDB and DocumentDB a couple of years back.  Both are easy and great tools for the right situation, DocumentDB now falls under CosmosDB as a product at Microsoft. However, I feel that CosmosDB would be anyone's default choice today on Azure as DocumentDB is really a feature subset of CosmosDB.

CosmosDB: "Azure Cosmos DB is a global distributed, multi-model database (db) that is used in a wide range of applications and use cases. It is a good choice for any serverless application that needs low order-of-millisecond response times, and needs to scale rapidly and globally."  CosmosDB is used by Microsoft's Skype, MSN, Xbox, Office 365, Azure products.

Def: CosmosDB is a Planet scalable NoSQL JSON database that has multiple API support (including SQL(Core)).  Multiple copies/instance around the world (think SQL AOAG).

• Encrypted on Azure at Rest and in Motion.
• Multiple API's supported including SQL API (DocumentDB) and multiple other JavaScript and Table.
• A logical breakdown of CosmosDB API
• Partitions are managed transparently and users are routed based on geographic location and usage.
• One write db and multiple reads.  Or can setup multitple write databases  Can set automatic failover so if the write db is unavailable, one of the read db's becomes the write db.
• CosmosDB Migration tool is great at bringing in data from JSON, SQL, CSV, MongoDB, Amazon DynamoDB.

Concerns:

• Determinant geo-replication - Use to be 1 master and multiple read copies of the data.  Not all copies can be written to but if you have country data residency rules you can't configure data to be within specific regions.  I.e. I can't specify certain bits of data are only stored in a specific region.  You can specify a region/location for a container, but not split a container.   : Check!  Not a fact.
• Backup and Recovery - Point in time recovery and MS ticket needs to be raised.  Can't structure complex backup plans.  Take it or leave it approach.
• Limited LINQ support
• SQL API is very limited compared to SQL relational databases, offering no joins or aggregation capability such as GROUP BY.
• Temporal Tables don't exist, there are good auditing options such as the "Change Feed" where all changes can be streamed into an external database/system.
• Entity Framework support is limited. Consider a PoC before using.
• Consistency (copy data to other read-only debs) 5 options: "Strong" (commit to all dbs and acknowledge state, so slow to align but all reading same data but it may be stale.  "Eventual", reads what is in the local db you are going to.   The default is "Session".  As always, it depends on the requirement.

Tip: Amazon DynamoDB is AWS's equivilant to Azures CosmosDB

More Info:
NoSQL options - https://www.nebbiatech.com/2017/02/09/exploring-the-nosql-options-on-azure/ 

SharePoint Online Property Bag SPWeb Properties are not indexed by default

Problem:  Moving an on-prem SharePoint solution to SPO, I realised that SPO does not automatically index property bag values.

Initial Hypothesis:  The Search schema looks correct and automatically created the correct Managed Properties.  Asked our Microsoft representative and they sent us a link to enable property bag values in the search index.

Resolution: Be aware that you need to do some Powershell commands on your tenant and site collections when using SharePoint Online to make property bag settings appear in the search results.

More Info:
https://blog.kloud.com.au/2018/04/26/how-to-make-property-bag-values-indexed-and-searchable-in-sharepoint-online/

ShareGate User Migration Gotcha

Problem:  Migrated an Extranet site with a large user base, and multiple users have the same name.  When a user is removed from AD, and running migration to the new farm, the AD automatically picks a different user and gives them the user that lefts permissions.

Example:
John Smith (john.smith@contoso.com) has been added to a site collection.
John Smith (@contoso) is removed from AD but still exists in the site collection permissions.
Ran Sharegate to move the content including user permissions to a new farm.
John Smith is added to the same SharePoint groups however, it has added john_smith@clientA.com

Initial Hypothesis: Sharegate tries to resolve the user and is incorrectly resolving the user's name and not the name in AD.  As the user has left the firm, the other user is being resolved and we end up with permission inconsistency.

I got this reply from Sharegate and can see that my issue happens at step 8.

"How Sharegate resolves users from the source to the destination"

"We look at the whole account name available, for matches to users at the destination through the SharePoint people picker.

Once we have a list of potential matches for your user, we go through the list of values below (in the specified order). We consider the account a match when we find the same values for one of these properties:

1.    Exact same account name

2.    Same normalized account name (without claims header)

3.    Same login and domain

4.    Same login

5.    Same login and domain (source login read from display name - this can happen when importing from file system because the account name is set as the display name)

6.    Same login (source login read from display name - this can happen when importing from file system because the account name is set as the display name)

7.    Same email address

8.    Same display name

9.    PrincipalType is not set or is a Security Group and same display name without domain"

Somewhat related:
https://sharegate.com/blog/unresolved-user-when-preserving-created-modified-sharepoint-migration

SharePoint Online Geo-Replication SPO/O365

Geo-replication/Multi-tenancy

Mid 2018 I outlined the state of Multi-geo on O365, the easier parts of Geo-Replication are already well handled and the changes are discussed in the the link.  This post focuses on SSO options today and the likely road-map.

O365 is moving towards multi-tenancy that will allow multinational companies to store data in compliance with country rules.  For instance EU data may not be allowed to be stored outside the EU but you already have your O365 tenancy based in the US.

Historically, most larger companies have chosen either the US or EU to base their data storage in.  If you wanted data to be stored in another region you had to buy another tenant with Microsoft strongly discouraged.

Microsoft, are working towards supporting O365 in multi geo-locations.  Basically, their are 2 parts: 1) User specific data (email, OneDrive) where we know where a user is based and their data is encrypted and stored in that country. and 2) group/team/country specific data (SharePoint) where the data itself may have residency rules.

This post looks at SharePoint data that is required to be stored in a specific country.

Options today:
1. On-Prem. : Have a SharePoint farm in each geo location, this requires a fair amount of thought to deal with SSO, Search, MMS, Content Types and UPA.
2. O365: Have multiple tenants (non are connected) in each location and connect your authentication up to each tenant.  The problem with option 2 is that each O365 tenant requires a separate Azure Active Directory.  This means that you will need to hook each O365 tenant up to a single MMS, Search service and poly-fill in the SSO process.  Imaging if you have 8 regional tenants for regulatory purposes.  To achieve SSO, you will need to create a central AAD, then connected each regional AAD to the central AAD.  Azure directory sync is needed, inviting members and guests, other companies AAD becomes and issue.  The image below outlines a possible pattern to solve this complex problem.

Coming Q1 2019 : Multi Geo tenant, that shall be the answer.  A lot of the multi-tenant is still in  preview so I shall be interesting to see mutil-geo tenancy when it goes into General Availability (GA) next year (+-Feb/March 2019).

MSIgnite tour London updates 27-Feb-19:

Brent Alinger

Sovereign geos:

US Gov

China (21Vianet)

Germany

Coming new geos: South Africa, UAE, Norway o365 data regions coming soon.  See office.com/datamaps

UK: Cardiff, London, Durham are 3 data centres in the UK.

Note: some services such as AAD, planner, yammer, Sway are not uk based either Europe or US based.

US has 8 data centres

Can get default region moved, it’s difficult.

Phase 1:  oneDrive and exchange April 2018 delivered

Phase 2: o365groups and SharePoint private preview Oct 2018.  Good feedback so far.  Keen ferry, Cott dimension data.

Multi-geo is not for solving:

GDPR

PERFORMANCE enhancer - rather align with MS Global Network.  

pining data to a specific country

Cost:  $2 per month extra per user in satellite locations, go thru account manager to set it up.  Once approved shows in admin centre and provisioned, take less than 30 days but can be 2 days.

Need a domain name per geo location for OneDrive and SPO e.g. https://emeia-radimaging.sharepoint.com

Preferred Data Location (PDL) - used to specify in AAD to show where a user is stored.  Not for travelling user but long term office assignment.  Users of exchange online are seemlessly moved.  ODfB requires a PS cod to move the user data.  

Phase 2: SPO March into GA by 30 March 2019 confirmed.  DLP per satellite geo.  Hub sites can span multi geos.  

Aka.ms/multi-geo

Update: 2020-06-30.  Multi-geo is available in
Australia, 
Asia Pacific, 
Canada, 
European Union, 
France, 
India, 
Japan, 
Korea, 
United Kingdom, 
United States, 
United Arab Emirates, 
South Africa, and 
Switzerland.

O365 AAD - Federation B2B options

Problem: Using O365 as an Extranet.  A basic analysis before starting is a minimal requirement.  The existing Extranet will make a lot of the questions fairly easy to clarify.  You can cover this in tremendous detail but to avoid information paralysis, I recommend a decision maker, and preferably someone that already works on Extranet.  A committee is cool if you have the cash but it's so hard to guess at the future, my preference is to get the broad strokes right and amended once we are in the weeds.  These four points can be answered with the right people in a meeting or may take months for complex organisations especially if there is no clear leader to make decisions.

Consideration Point:
1. Who is using the Extranet?  Clients, partners, vendors, ..., or Client Users
2. How will Client and Company users authenticate? O365 options including ADFS, another federation service e.g. Ping, Passport/Live, Google, Facebook,...
3. Self-registration or known approved Client Users?  Try to figure out what the process for on-boarding your Client User will be.
4. Client User Profile Usage?  Will the client users amend content, have the ability to share permissions or old school, they will read web published pages (read-only).  Will client users have OneDrive, use teams, only SharePoint or other O365 applications.

2.> O365 authentication
The most basic option is to allow O365 to have client users (guests), as long as a user has an O365 account they can be a Client User.  You can also use any Microsoft account for a client user.
Azure has a service that allows for you to connect users as guests, the user shall use their own AAD or ADFS or any federation service including Google and Facebook to authenticate.  Microsoft allows 5 guest accounts on AAD for every 1 member (licence user).

4.> Client Usage Profiles
O365 can share a document anonymously in a link within an email.  Obviously, this means anyone can potentially access the file.  However, to replace attachment in an email and wide distribution this is a great step forward, as you can control versions and retract the access at any point.  Additionally, the link settings can be customised to control who can use the link.  For example, you can set the specific people who get the link or you could specify only internal people get the link.  Once it is set to "Anyone" the email or link can be forwarded and literally anyone can get access.

Governance:  Manage O365 to apply the businesses rules so users comply with governance.  O365 has an easy straight forward configuration to make this happen.  When configuring sharing governance you need to ensure it is done at the O365, SharePoint Admin and Site Admin levels.  If 1 of these says no external sharing you can't share so it is a fairly granular approach.  This allows Extranet and Intranet to live on the same O365 tenant.

Licensing: As a general rule, there tends to be no cost for External users, as 5 client Users for every internal O365 user is allowed for the O365 extranet scenario.  Check with Microsoft as business scenarios play out differently.

Thoughts:

• O365 uses Azure Active Directory (B2C), there is a 1-to-1 relationship between your tenant AAD and you O365 instance.
• External accounts can be connect as guests e.g. Another AAD tenant, Micsrosoft accounts (passport), ADFS or any auth provider (SAML), Facebook, Google+, AAD B2C (separate service from AAD).  There is also a One Time Passcode option.

Securing SharePoint O365

Microsoft outline how they treat access to your company data, how your data is kept secure and audit and availability, read this post.  The information below notes possible settings and configuration to secure 0365.

Azure AD is the key, ensure auth is 100%.  e.g. MFA for some or all accounts.  Use the "Identity Secure Score" to check possible problems.  Consider Microsoft Authenticator for MFA.

O365 Settings use:

1. Secure Score - Overview of my tenant settings and how they should be set.  Check my tenant again set MS best practices for O365. 
2. Validate setting meet governance and are not merely defaults.
3. Review SPO audit logs - ensure it is turned on (default is to have it turned off).
4. Security and Compliance Dashboard - Good email checker/analysis.  Low value for SPO.

Cloud App Security (CAS) - service looks for security on O365 tenants, improving constantly.  CAS Overview.  Add-on or included in E5 plans.

Office 365 Advanced Threat Protection (ATP) - service to identify threats.  "ATP analyzes content that's shared and applies threat intelligence and analysis to identify sophisticated threats.", Microsoft.

To manage document use IRM on SPO and AIP on documents.

"Azure Active Directory Identity Protection is a feature of the Azure AD Premium P2 edition that enables you to 1) Detect potential vulnerabilities affecting your organization’s identities 2)
Configure automated responses to detected suspicious actions that are related to your organization’s identities 3) Investigate suspicious incidents and take appropriate action to resolve them".  More info.

SAML, OAuth and OpenID Connect

Rough Notes - Fix

Overview:  SAML has been around for a fair amount of time  (roughly 8 years) and still is widely used for authentication and authorisation of end-users on the Internet.  OAuth2 is used to allow internet users to give internet/web-based applications access to the user's information without the user password.  OpenID Connect is an extension to OAuth2 and generally the way to go instead of SAML for user authentication.

Azure Active Directory Supports:

1. SAML 2.0
2. OAuth 2.0,
3. OIDC, and
4. WS-FED.

SAML:

1. SAML is an umbrella standard that covers federation, identity management and single sign-on (SSO); and
2. SAML is an open standard for exchanging authentication and authorization data between parties, in particular, between an identity provider and a service provider. SAML is an XML-based a markup language for security assertions.

Limitations of SAML:

1.       It was launched in November 2002 which supports SSO but has now been deprecated by a lot of auth suppliers.

2.       SAML is not supported with native Mobile Applications. Hybrid mobile apps can work with it.

3.       As it has now deprecated, finding it may not be supported by the latest technologies applications and technologies. 

OAuth 2.0:
1. Derived from OAuth, Auth2 uses 2 passwords and is more secure.
2. (Open Authorization) is a standard for authorization of resources. It does not deal with authentication. It was released in 2006.

Limitations of OAuth:

1.       It only deals in Authorization so we have a limitation that we cannot verify the user's identity, i.e. Authentication.

OpenID Connect:

1. OpenID Connect (OIDC) is a protocol to verify user identities and get the user profile information. OIDC enables devices/apps to verify identities based on the authentication done by an authentication server
2. It was launched in February 2014
3. OpenID Connect (OIDC) is an authentication layer on top of OAuth 2.0, an authorization framework
4. OpenID Connect is built on top of OAuth 2.0, specifies a RESTful HTTP API, and uses JSON as a data format (JWT). It has a specialized set of predefined data types and endpoints for exchanging user information between the identity provider and the application.
5. There are two different types of flows in OIDC, pick the appropriate flow, normally Code flow over implicit.

Advantages of OpenID Connect

1.       It supports SSO and federation.

2.       Has good support with .Net Core.

3.       It supports a wide range of clients like web applications, mobile apps and JavaScript applications.

4.       Have support with Azure AAD B2C as per Microsoft guidelines and others like Google+.

Comparing Differences:

1. https://www.gluu.org/resources/documents/articles/oauth-vs-saml-vs-openid-connect/
2. https://stackoverflow.com/questions/7699200/what-is-the-difference-between-openid-and-saml.
3. https://www.quora.com/What-is-the-difference-between-OAuth-OpenID-and-OpenID-Connect

Support with Azure b2c

1. https://docs.microsoft.com/en-us/azure/active-directory-b2c/active-directory-b2c-setup-oidc-idp

Web sign-in with OpenID Connect in Azure Active Directory B2C

1. https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/development/enabling-openid-connect-with-ad-fs
2. https://docs.microsoft.com/en-us/aspnet/core/security/authentication/ws-federation?view=aspnetcore-2.2

Using Postman to Generate an OAuth Token Example

Using the Generated Bearer token in a get Request

Tip:  To examine a Bearer token use the website https://jwt.io