SharePoint Search to extract SharePoint list Data into SQL

Problem:  I have multiple lists that I need to get data changes updated into SQL on multiple site collections.  Think 1000 site collections with 5 lists in each so 5K lists are being updated (my actual requirement is much larger).

Initial Hypothesis: In SP 2010 or SP 2013 I would use a Full Trust Event Receiver and register each list using a feature but now we are in the new world of no full trust code.  The 1st thought is RER (Remote Event Receivers), I need to register 5,000 of these and they are notoriously unreliable for delivery.  Search has to pick up all changes to list items for indexing and I can break into the crawl pipeline during Content Enrichment.

Proposed Solution:

Points to Consider:

1. I need to create a Content Enrichment Web Service (CEWS) that I shall call from the crawl component.  I can only register a single web service on a crawl SSA so consider using the toolkit/Microsoft CEWS Framework on the Web Service endpoint this allows multiple Web Servies to be strung together.  Good idea to implement a toolkit so multiple pieces of logic can be applied.  So if BA Insights registered, then can't add another CEWS Web service. CEWS toolkit provides a pipeline of stages to add custom logic later.  So write our own stages and hook into the CEWS framework.
2. CEWS does not process item deletes so a possibility is using the crawl log to identify deletes
3. There is no CEWS on SharePoint Online/O365.

Note: Properties returned from CEWS has minimal data input (common custom fields such as Title) and need to augment with registered MP's to get the data in the CEWS web Service.  Do not get a dump of properties/need to register and ask for the properties.

27/04/2017 Note: An idea I had to deal with "Deletes", I could put the item into a status of "About to Delete" and the crawl would pick up the status and the CEWS could delete from he SQL database and then the Web Service could delete the list item from the SharePoint list.  It doesn't work, the CEWS pipeline does not pickup and item updated and then deleted.  So the crawl is smart but I can't enhance using this approach so more work to fix the delete ...

CSOM for Deployment of SharePoint sites and Assets

Problem:  Historically we built Site Templates and WSP to build up of SharePoint solutions.  With the current state of SharePoint it's not advisable to release Full Trust Code (FTC).

Initial Hypothesis: So I often tend to write CSOM for custom provisioning and asset deployment.  I utilise the Tenant Admin API for provisioning site collections and then CSOM SharePoint in C# to provision and deploy my assets.

Possible Resolution:
PnP has a great library to provision assets to base you solution on, customising and building XMLfiles allows you to quickly build repeatable SharePoint solutions that are easily configurable.
Also InstantQuick has a solution called IQApp that is worth a look at.

Displaying SharePoint site pages in an iFrame

Posts on the iFraming issue Series:
IE11 not saving field data iframe Issue
Term Store/MMS musing when using iFrames (WebTaggingDialog)
Displaying SharEPoinnt Site Pages in iFrames (this post)

Problem:  I need to display a SharePoint 2013 document library within an iFrame of another SharePoint 2016 site.  SharePoint by default does not allow iFrame itself.

Hypothesis: IE and the other browser respect different rules for allowing iframing.  The iFraming rules are dictated by http headers output from SharePoint.  SP 2010 and MOSS did not have iFraming restrictions.  Furthermore, chrome and safari do not follow the X-Frame-Options http output but use CSP to control ancestor framing.

Resolution:
1.> Remove the SharePoint generated X-Frame-Options header generated by SharePoint - step 2 below.
2.> Add the custom X-Frame-Options http header to allow a specific domain to iframe the site (keep the attack surface as small as possible) - step 1 below.

3.> Add the Content-Secuirty-Policy frame-ancestors entry to ensure Chrome and the other browsers will allow framing as shown below.

Posts on the iFraming issue Series:
IE11 not saving field data iframe Issue
Term Store/MMS musing when using iFrames (WebTaggingDialog)
Displaying SharEPoinnt Site Pages in iFrames (this post)

Create a Lookup column from a list in a separate web

Problem:  I have a site collection (SPSite) with multiple sites (SPWeb) and I have a reference list in the SPSite (root web) that I want to use from multiple lists in the sites (SPWeb).

Resolution:  I recorded this short video that shows how to create a lookup site column in the site collection and then at the list within a sub web, you can associate the lookup.

MMS hybrid between SP2013 and O365 and SP2016 farms

Problem:  A lot of large enterprise customers have the Management Meta Data Service including the Content Type Hub that SharePoint farms subscribe to.  You are on-prem. with this centralised MMS and CTH.  Now you want search to work on your O365 public tenant and to use SP2016 on-prem.  It may even be more complicated with SP2016 installed on Azure and there is no direct access onto the on-prem SP2013 CTH.


Initial Hypothesis:  You want to have a central production MMS service that all SP farms subscribe to,  You can't subscribe from a SP2016 farm to the SP2013 central MMS service.  O365 can't subscribe to a different MMS, you need to use the MS MMS and sync the term store using CSOM or a tool that shall use CSOM.  Crossing domains such as in a DMZ that does not allow inbound connections look at chaining for CTHub solutions.

Restoring MMS to another farm also see moving the MMS database (think Prod for development workstations) is straight forward if you merely want another copy of the MMS, use the Export-SPMetadataWebServicePartitionData to get the MMS info and then import the MMS proxy using the PS Import-SPMetadataWebServicePartitionData.  Best post is here on exporting and importing ensuring GUIDs are maintained.  Andrew Connell has a great series on MMS and one of his post looks at the Copying the MMS instance from Prod to Development.  A normal backup and restore of the Content Hub Site Collection how to bring a copy of the CTHub back.


In Progress....

Performance Testing SharePoint

Problem: Once again performance testing has raised caused concerns on a project.  There are various methods for calculating how many users a system can deal with.

Description:

Non-function Requirements are key to determining how "performant" the SharePoint farm needs to be to deal with peak loads.

Load testing allow us to mimic various users and see when the site/farm performance starts to degrade.  A good idea is to identify all the possible actions the users will perform  and items like Search are far more resource intensive than clicking on a link in.

Average visits per hour = (5,000 average visitors/day) / 10 hours = 500
Page Request per hour = (Ave visits/hr * 5 ave page request/hr) = 500*5  = 2,500
An example can be further broken down as follows by assuming of the users 5 request, 3 are for pages, 1 is a search and the last is viewing a document.
Recording this scenario with wait times provides a basic load test whereby the user numbers can be increased at 5 minute intervals.

Extranet Authentication Options for SharePoint 2013

Overview: Most large enterprises using SharePoint have implement Extranet solutions and these vary in complexity greatly.  Many implementation I have seen have morphed into bazaar solutions generally due to the tactical solutions implemented over time and were not caused by poor architecture.  It is the nature of these projects to get something out and with the rapid change in authentication over the past 5 years tons of business have landed in complex scenarios.

Office 365 has grown quickly and using Office 365 is generally a good idea however a lot of organisations still are resistant due to a variety of concerns such as regulatory compliance and trust.  Microsoft is definitely removing these barriers and I'd lean towards hosting the SharePoint Extranet in the cloud in the majority of situations.  The biggest barriers to moving to the cloud are Executive level buy in followed by senior IT folks that are bias to sticking to what they knew 10 years ago.   So a lot of the change is around education and providing a clear road-map.  The biggest technical hurdle will be around identify management.

Pretty much every organisation I deal with used Active Directory and then you may have a Federation Service normally ADFS.  You may have you external users in the same AD, a dedicate DMZ AD, or any other user directory including SQL or other LDAP provider.

Using Office 365/SharePoint Online I need to get both my internal and external users to be work with Office 365 and depending on the client setup I need to work thru both scenarios and think about the ramifications.

Note:  Ramifications are: resetting user passwords, does search work for all users and where does the data reside.

Possible Options:

• AzureAD - Azures ACS for user accounts
• Federated Identifies - use ADFS and build trust with ACS, identity and password is under our company control
• AD sync to AzureAD - Think DirSyng, tooling is ADConnect

On an on-prem. SharePoint farm, the following Authentication methods are supported at the Web Application Level:

1. Classic (Windows (Basic/NTLM/Kerberos)), 
2. CBA - Claims Based Authentication backed by either Windows Claims (Windows (Basic/NTLM/Kerberos)) or SAML Claims (ADFS or SiteMinder or Ping or ThinkTexture, ....)
3. FBA - Forms Based Authentication, and 
4. Anonymous (none)

Notes:
http://www.sharepointeurope.com/blog/2015/10/identity-management-in-a-saas-based-world