Tuesday 16 December 2014

SharePoint 2013 Public Website Check list

Ux:
  1. Responsive design vs Device channels - Does the site switch resolutions and browsers gracefully.  RWD vs AWD (Adaptive Web Design)
  2. Broken Links: Check My Links 3.3.4 is a plugin for Chrome to check a page for broken links (go over main pages at least)
  3. Fiddler - Use for 404, and other errors, look for dodgy urls and headers being passed around.
  4. Charles is a similar tool - helps with broken links, size of files, shows web calls, review response headers, size of files and speed of execution.
  5. Minification - is the minification of JavaScript and CSS.
  6. Alt labels, WCAG, valid html checker
SEO:


Testing:
All devices and browsers (1. PC/laptop (IE 11-IE7, Chrome, Firefox, Opera, Mac/Safari), 2. Phones(iPhone, Android OS, Windows OS), 3. Tablets (Android, MS/Surface, iOS/iPad).

Helper Tools:
AddThis.com - Nice tool to add Social bookmarking service for your websites. Collects stats TypeKit - Nice for Fonts, review the licensing needed.

Security:
  1. Check Internal Search is not returning passwords
  2. Check google is not picking up passwords/confidential data 
  3. Remove response headers:
  4. MicrosoftSharePointTeamSiteServices(versio), X-Powered-By. X-SharePointHealthScore, X-aspNet-Version) Performance X-SharePointHealthScore
  5. Check XSS and SQL but with SharePoint you are testing the product
<script>alert();</script>
<img src=;; onerror=alert();>
<iframe src=javascript:alert();>

Wednesday 3 December 2014

SharePoint 2013 workflow

Overview:  SharePoint 2013 has a new workflow engine, you can still use SP2010 workflows.  Not that SP2013 workflow works as a separate install and then a services is setup (same change as for Office Web Apps).


Overview of SP 2010 and SP2013 workflow
Created workflows using Visual Studio or SharePoint designed for SP2013 workflows (Visio also I believe)

Thursday 27 November 2014

One Drive terminology in a picture

Problem: Confused by SkyDrive / OneDrive / My Site One Drive for business and MySite now call OneDrive with OneDrive labelling?
http://technet.microsoft.com/en-us/library/dn167720(v=office.15).aspx





Friday 17 October 2014

SharePoint Hosted Apps vs Embedded JS


Overview: The use of Apps (specifically SPHA) in SharePoint seems to be misunderstood, developers and architects often want to use the App model for functionality that folks have built using previous versions of SharePoint.  Apps are reusable pieces of custom logic akin to a specialised document library.

The app needs to be deployed to the catalogue store and permissions granted to leverage SP functionality.

SharePoint Hosted Apps (SPHA) are the internal sub web created with SharePoint, that can use JavaScript to perform customisation.

For example I want to read values from a term set, you can simply embed JavaScript and using the current users context get the term set data you want.

Permissions in SPHA run in the context of the current user as opposed to Provider Hosted Apps that can run in either: current user context, app context or app and current user context.

Deployed JavaScript will perform exactly the same when called from a page or from a SharePoint page or from within the SPHA (app web).  JavaScript runs in the context of the current user for both approaches.

The following embedded JavaScript works both in a web part page or in a page inside a SPHA (app web):

<script type="text/javascript">
var termSetName = //document.getElementById('termsetID').value;
var locale = 1033; // your locale. Here is English
var context  = SP.ClientContext.get_current();  //User the current users context.
var taxonomySession = SP.Taxonomy.TaxonomySession.getTaxonomySession(context);
var termStore = taxonomySession.getDefaultSiteCollectionTermStore();
var termSets = termStore.getTermSetsByName(termSetName, locale);
var termSet = termSets.getByName(termSetName);
var terms = termSet.getAllTerms();
context.load(taxonomySession);
context.load(termStore);
context.load(termSet);
context.load(terms);
context.executeQueryAsync(function onSucess(){
  var termEnumerator = terms.getEnumerator();
  var termList = "Terms: <br/>";
while(termEnumerator.moveNext()){
var currentTerm = termEnumerator.get_current();
termList += currentTerm.get_name() + "<br/>";
}
Windows.alert(termList);// Output to the screen                                 
                },function onFailure(args){
                    // Notify user of error
                });         
}

The user only needs to be a visitor to have read access to the term store.  JS works in the same way whether inside an SPHA or within a page on a SharePoint site.

“Apps that do not make OAuth authenticated calls (for example, apps that are only JavaScript running in the app web) cannot use the app-only policy. They can request the permission, but they will not be able to take advantage of it because doing so requires passing an app-only OAuth token. Only apps with web applications running outside of SharePoint can create and pass app-only tokens.”  MSDN article

JavaScript inside a SPHA can only run within the context of the current user.
Provider-Hosted Apps (PHA) can use either:
  • context token (user context)
  • user+app access token
  • app-only access
This was spoon fed to me from some good folks I'm working with Nick, Sachin & Peter- thank-you.

Thursday 16 October 2014

Cross Cutting Concerns for SharePoint 2013

Overview:  Last week I was speaking to a smart chap and he dropped the term Cross Cutting Concern as we were discussing SharePoint Host Apps (SPHA) and JavaScript.

Problem:  When creating apps for SharePoint 2013 multiple solutions need to address cross cutting concerns.  In the past I deployed a SharePoint library with caching, logging, lazy loading and various other "Cross Cutting Concerns", now for Provider Host Apps (PHA), SPHA and JS embedded within pages and Single Page Apps (SPA) we need frameworks for clients to address common components.

Hypothesis:
Caching for Client Side Code: In JavaScript you can either cache using the client cookie which is small or in HTML 5 based browsers use the JavaScript local store. 
Caching on the Server: All the normal Caching of C# or Azure are available.  Also look at Redis.

References:
http://en.wikipedia.org/wiki/Cross-cutting_concern
Update 27/01/2015:
http://channel9.msdn.com/blogs/OfficeDevPnP/SharePoint-Apps-and-client-side-caching

Wednesday 10 September 2014

SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews

SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews (This Post)

Refine your results to a specific site or part of a return result set.
Search Query Example


CSOM Search
JavaScript API CSOM search query








Setting up Pdf Previews for Search

You will need a Office Web App (WCA) Farm (1 or more servers), the WCA needs to have any patch after the original WCA product release.

 Perform a full Crawl and..

SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews (This Post)

SharePoint 2013 Search Series - Post 4 - Search Result Removal

SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Removal (This Post)
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews






The image below shows how to restrict the results displayed using a "Result Source" at the site collection level to display a subset of data.  You can also refine the results displayed using the search result web parts and reducing the result source set.

The test button is useful to see if your refinement/filtering is working.

The 2 screen below allow me to create new Result Sources.  The result source creates a subset of results that can be consumed by search results web parts.


SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Removal (This Post)
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews
Tips:

Thursday 28 August 2014

Monitoring SharePoint Public Websites

Overview:  This post is applicable to public website and not just SharePoint, I have used it for SharePoint and feel it is a good product.  The principle will apply to other monitoring products and services.

AlertFox is a SaaS monitoring service.  It allows me to monitor various websites using http posts or complicate macros to perform various steps such as logging into a website using ACS.  This differs from an internal monitoring service such as Solar Winds but it definitely has it's place.  I discuss various monitoring options in this post.

The benefits are:
  1. You are notified when the site is down and what the issue is from a web request point of view.
  2. You are monitoring externally so you can see what you customers see.
  3. You can see if your response time are slowing down.
  4. You keep the IIS webservers warmed up (so if you have an app pool recycle).
  5. Easy to monitor and you can setup alerts.
  6. Complex scenarios can be accounted for in testing so you know the complex parts of your site are working.
Image 1. See when you have problems, what the issue is and when it occurred.


Image 2. Verify the performance from around the world

Image 3. Check uptime

 

Tuesday 19 August 2014

SharePoint 2013 on-prem using Windows Live Id via Azure ACS

Overview:  I have a pretty simple requirement to allow users to register on my customers public SharePoint 2013 web sites.  I have setup custom providers and thought ACS was going to make me a hero at my client.  The whole experience is terrible and I can't see why anyone would use the default of LiveId via ACS due to the implementation.

Opinion: I hate the way ACS works with Windows Live Id, it is so bad I can't see a scenario when a client would use it.

Anyway, I have SP2013 SP1 on-prem and I want to hook at ACS allowing customer to register on the site and get elevate permissions on the site.  I'd like them to use multiple 3rd party authentication providers such as Facebook, Windows Live Id, LinkedIn and Google.  In my PoC I decide to simply use Live Id as it is the default on ACS and as both services are Microsoft owned it must be the easiest. 

I worked thru Wictor Wilem's post series and as usual Wictor has provide a great resource.  I had to make minor adjustments to get it to work for me on SP2013 but overall, Wictors series of posts is a good place to start.

The 1st issue I got was when logging in using Windows Live ID, I was continuously redirect back to the /_login/default.aspx page.

After bashing my head trying to figure out what the issue was I realised in Wictor's common issues post in the series, he mentioned the claims mapping/rule needs to be adjusted for Live ID authentication. 

I now was getting an access denied, which at least told me the claim was hooking up.

The next issue was now I was getting the message you are not authorised "Sorry, this site hasn't been shared with you."

Give all authenticated users access to the site as shown below.

Once you login you will notice a horrible looking user that is logged in.   You can assign permissions using the "Friendly Username".


Common public Federation (IdP) Identifcation provides are:
  • LiveId (MS - not where you would expect the MS offering to be)
  • Google (constantly changing - easy hookup)
  • FaceBook
  • LinkedIn
Common Enterprise IdP Servers/Services are:
  • Microsoft ADFS (best default option for greenfield SP)
  • PingFederate (pretty expensive but a comprehensive solution, use if already in place or the advanced features really suit the business at an enterprise level)
  • ThinkTextures IdentityServer (Great for customisation, difficult support but for the hardcore tecky type organisation a good option)
  • CA-SiteMinder (Good product, used in enterprises and hooks up well to SP.  Has a large set of tools and options).  Update: 19-Nov-2015, seen another large implementation of SiteMinder, it has expensive add-in modules and extremely problematic.  SP agent needs AD groups.
  • RSA Federated Identity Manager (No experience)
  • Entrust GetAccess (No Experience)
  • IBM Tivoli (CAM) (Had a hard time with this a few years back)
  • ComponentSpace (Good for .NET customisations, not a large Federation service Server)

Saturday 9 August 2014

PowerShell to Create and Remove Promoted Search Results in SharePoint 2013

Overview: I want to manage promoted results programatically.  PowerShell is a good candidate for automating the creation of "Promoted Results" previously/also known as "Search Best Bets".

In this post I provide PowerShell to create promoted results at the site collection.  The image below shows that my search has picked up 2 pages in my site collection.  I want to display a promoted result when a user types in certain terms in y case the search term is "Messi".  The picture bellows explains what I'm achieving through PS using promoted results.

To manually create Promoted Results:
  1. On the Site Collection, go Site Settings > Search Query Rules
  2. On the page select "All Sources" for the qu "For what context do you want to configure rules?"
  3. And select "Promoted Results Contains", you can add Promoted Results/Best Bets thru the UI at this point.
Or Open PromoteResults.ps1 and edit the Powershell to create the promoted results for you, comment out the DeletePromoteResults, as it is used to roll out the changes.
Run PromoteReults.ps1. 
Search for the term "Messi" and you will see the promoted result.
 
 

Friday 25 July 2014

Office Web Apps 2013 for public facing websites

Overview:  A couple of weeks ago I told a customer that Microsoft offers a service to display Office documents thru Office Web Apps (WCA).  I thought I had read this or seen it on twitter but I was confident that viewing website office based documents was a free service offered by Microsoft.

Initial Hypothesis:  I looked on the web and could not find anything and I had to hastily tell the customer I had made a mistake as I could not find anything about it on the web.  The customer decided to setup a public facing Office Web Apps Server to feed up office documents and pdfs for their websites.

Fast forward a few weeks and the customer is installing a 1 server office web apps farm to display pdfs and word document from their public websites and I have been informed that there actually is a public service.  http://blogs.office.com/2013/04/10/office-web-viewer-view-office-documents-in-a-browser/

"Do you have Office documents on your website or blog that you want your readers to view even if they don’t have Office installed?  Would you rather view a document before downloading it?  To give your audience a better experience, try the Office Web Viewer."

To use the service there are a couple of considerations.  The service only supports office documents like word and excel, it doesn't support pdfs which is the clients preferred method of providing downloads.

Resolution:  You do not need to perform WOPI binding to use the service. 
The document is located at: http://calibre-ebook.com/downloads/demos/demo.docx

All you need to do is prefix the url link as follows:
http://view.officeapps.live.com/op/view.aspx?src=http://calibre-ebook.com/downloads/demos/demo.docx
 
Summary: A nice service offered by Microsoft for viewing Office documents "on-the-line" O Wilson, V Vaughn 2013 (The Internship).  If won't work for pdfs and you will need to call the service in the html call.  Good to know it is available but it won't meet my clients needs.

Tip: Ensure the link opens a new tab as the opened pdf will lose the clients context on yor site and force them to use the browsers back button.

Note: To get WCA to open public documents on the web using the web viewer, you need to setup the SharePoint farm to use "external-https".  If you are using it both internally and externally, which I have not done, you need to use external-https and use Alternate Access Mapping (AAM).

Example:
Below is a single WCA server farm that I am using for both internal e.g. document libraries & external i.e. public SharePoint 2013 websites.  Note the Internal and External URL are the same, you could also use AAM as suggested earlier.

FarmOU                            :
InternalURL                       : https://wca.demo.co.uk/
ExternalURL                       : https://wca.demo.co.uk/
AllowHTTP                         : False
SSLOffloaded                      : False
CertificateName                   : wca.demo.co.uk
EditingEnabled                    : False
LogLocation                       : E:\OfficeWebApps\Logs\ULS\
LogRetentionInDays                : 30
LogVerbosity                      :
Proxy                             :
CacheLocation                     : E:\OfficeWebApps\Working\d\
MaxMemoryCacheSizeInMB            : 5000
DocumentInfoCacheSize             : 5000
CacheSizeInGB                     : 40
ClipartEnabled                    : False
TranslationEnabled                : False
MaxTranslationCharacterCount      : 125000
TranslationServiceAppId           :
TranslationServiceAddress         :
RenderingLocalCacheLocation       : C:\ProgramData\Microsoft\OfficeWebApps\Working\waccache
RecycleActiveProcessCount         : 5
AllowCEIP                         : False
ExcelRequestDurationMax           : 300
ExcelSessionTimeout               : 450
ExcelWorkbookSizeMax              : 10
ExcelPrivateBytesMax              : -1
ExcelConnectionLifetime           : 1800
ExcelExternalDataCacheLifetime    : 300
ExcelAllowExternalData            : True
ExcelWarnOnDataRefresh            : True
OpenFromUrlEnabled                : True
OpenFromUncEnabled                : True
OpenFromUrlThrottlingEnabled      : True
PicturePasteDisabled              : True
RemovePersonalInformationFromLogs : False
AllowHttpSecureStoreConnections   : False
Machines                          : {EXT-WEBDEMO1}

I am using a SSL certificate installed on the WCA box as I don't have an SSL termination device (F5, Kemp).

On the WCA VM I run the PS > Set-OfficeWebAppsFarm -OpenFromUrlEnabled:$True

This gives me the generate.aspx page on the WCA farm to provide pdfs and office documents via the web viewer.  Mine is https://wca.demo.co.uk/op/generate.aspx

SharePoint 2013 OOTB lookup list filtering

Problem
I don't want to start customising my solution and I need a lookup column with restricted options.  I read this post from April Dunnam via Google and it's awesome and I wish I had thought of it first.

I have 2 lists: Countries (Country, Continent) & Business (Name, LocationInAsia, LocationInEurope).  I want to be able to specify their head office in each continent.  I could do a lookup list to country and show all the countries or I can use Aprils approach and only show Countries for the specific region.  This is a contrived example but it's a simple way to explain it.

Resolution:
On the Countries list create a Calculated column/field, call it "AsianCountries". 
The formula is =IF([Continent]="Asia",Country,"")
On the Business list, create a lookup called "LocationInAsia" that points to the "AsianCountries" field in the Countries list.

http://www.sharepointsiren.com/2013/05/sharepoint-2007-2010-2013-filtered.html

Summary:
This is pretty useful in that I don't need to write and custom code to have lookup lists that only show the appropriate content.  Sure I need to have extra calculate columns but a good no code way to restrict lists lookups.


 

Saturday 12 July 2014

SharePoint 2013 Search Series - Post 3 - Search Box Web Part to only query Managed Properties

Search Series:
SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties(This Post)
SharePoint 2013 Search Series - Post 4 - Search Result Source Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews

Overview:  In this post we are looking at using a Display Template to change how the Search input box works.  Specifically the example will look at making the search box only search on the Title property of documents.  It will search for any part of the title.  If my word document is called "Std-634-HealthandSafety", using the search term "634" we get the document.

Steps to Change the way the Search Box works:
1.> Copy the Display Template "Control_Display Template", edit & rename the new template and upload to the Display Template gallery.
2.> Change the Title element tag in the Display Template and amend the html as shown below to change the behaviour of the search box.
3.> When the html display template is uploaded, you will notice the js file with the same name is updated.  Remember to Publish the html display template.
4.> On the Search Page edit or add the Search Box Web Part.  Select the Display Template, in my case it is "Demo Title Search Box" and save.
Search now only does a wild card search on the Title of the uploaded documents (assume the content has been crawled).

Search Series:
SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties(This Post)
SharePoint 2013 Search Series - Post 4 - Search Result Source Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews

Sunday 6 July 2014

SharePoint 2013 Search Series - Post 2 - Refiners

Search Series:SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners (This Post)
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Source Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews

Overview:  Refiners are a powerful way of drilling into your search result data.  This post aims to look at configuring the search refiner web part.  Any Managed Property can be marked as a refiner and then used on the Refinement Web Part.

Steps to setup a refinement panel:
1.> Add the "Refinement" search web part to your search page.
2.> Edit the Web part and under the "Choose Refiners" button select the appropriate Refiners.

3.> Add custom managed properties as shown below, mark the managed property as "refineable" and it will show up in the "available Refiners" screen.

4.> Add your custom "Available Refiners"
5.> Save and publish the web part and check the refiners are working.

Search Series:
SharePoint 2013 Search Series - Post 1 - Configure Custom Search
SharePoint 2013 Search Series - Post 2 - Refiners (This Post)
SharePoint 2013 Search Series - Post 3 - Search Box Web Part query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Source Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews

Saturday 5 July 2014

SharePoint 2013 Search Series - Post 1 - Configure Custom Search


Overview: This series of posts examines SharePoint 2013 search.  It looks at configuration to get Search working for you.  There are other posts on this blog about configuring your Search architecture and farm.

This post looks at 2 main points: 1.> crawling content and 2.>  Result Sources.

Search Series:
SharePoint 2013 Search Series - Post 1 - Configure Custom Search (This Post)
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part to only query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Source Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews

1.> Crawling Content:  The UI in CA has changed but if you understood crawling in previous versions of SharePoint, it still works the same way.  You are looking to create indexes by crawling content.  The image below shows the Search Administration screen after choosing the "Content Sources" option.  This will allow you to create content sources and configure them so SharePoint can get data into the indexes and have permission awareness (ACL).

After setting up your Content Sources run a full crawl to populate the indexes.

The "Crawl Log" is extremely useful to see when crawls where run and what content has been included. 

Once all the content is crawled you are ready to look at the Results Source formally referred to as Search Scopes.

Tip: The crawler must run through all content to be searchable, I still see clients trying to restrict data by no crawling it.  Crawl everything and restrict on the query side what comes out. 
Tip: Don't create to many Content Sources.  This can be a tough item to figure out, my advice is use as few Content Sources as possible and break them up by how often the content needs to be crawled.

2.> Result Sources: You can create Result Sources at the SSA/farm level down to the web/SPSite level.  This example is looking to return data from a specific path for a particular content type, so I created the Result Source at the site level.

Tip: The "Search Result Preview" is really useful for checking you are doing this correctly.

Once I have a "Result Source" setup I go to my search page and add a "Search Results" web part to output the results to my user.  I  can edit the "Search criteria" on the web part to use my new "Result Source".  The picture below is annotated to explain setting up the Search Results Web Part.


Search Series:
SharePoint 2013 Search Series - Post 1 - Configure Custom Search (This Post)
SharePoint 2013 Search Series - Post 2 - Refiners
SharePoint 2013 Search Series - Post 3 - Search Box Web Part to only query Managed Properties
SharePoint 2013 Search Series - Post 4 - Search Result Source Removal
SharePoint 2013 Search Series - Post 5 - Restricting Results & Enable pdf previews

Thursday 12 June 2014

Migration test of Sharegate to SP2013

Problem: I am looking at migrating some basic content into an SP2013 farm from SP2010.  I would normally opt for Metalogix or DocPoint if the client already has the licences but today I needed to use another tool namely Sharegate.  I used Sharegate.Migration.4.6.3

Scenarios:
SP2010 blog posts to an existing SP2013 blog.
SP2010 lists into SP2013.

Screenshot of the tool:


Verdict:  A great tool, easy to use, mapped over data cleanly.  Absolute winner! 

Updated: 2018/10/23
I also like Metalogix but Sharegate is still a brilliant tool for SharePoint data migration.

Wednesday 11 June 2014

CAML Designer for SharePoint 2013 is Brilliant

I always used U2U for checking my CAML queries in SharePoint.  I have just used CAML Designer for SharePoint 2013 and it is brilliant.

Link to the CAML Designer tool
http://karinebosch.wordpress.com/2012/12/13/caml-designer-for-sharepoint-2013-released/

Simple Example of using CAML to query a list:
string riskListCAMLQry  = "<Query><Where><Eq><FieldRef Name='Status' /><Value Type='Text'>Scheduled</Value></Eq></Where><View><RowLimit>1000</RowLimit></View></Query>"

Call using C# CSOM
CamlQuery camlQuery = new CamlQuery();
camlQuery.ViewXml = riskListCAMLQry;  
// Add a where clause on the status on the list to restrict the result set, only return the first 1000 rows

Saturday 7 June 2014

Content database Sizing & Cleanup

Problem:  At a customer site, a content database is massive.  Various Site collections are using the same content database. 

Initial Hypothesis:  Smaller site collections can be moved to new separate content databases.  This reduces the size to some degree.  The SQL log (ldf) is in good shape.  The excessive size is due to 3 unavoidable issues, multiple versions of large blobs (we need the versioning so not an option).  The recycle bin is set to the default 50% and my dumping older content is brings down the size and lastly, delete content databases are still sored within SQL. 

Resolution: Remove the previously deleted site collections fro the content database.   Using CA, I tried to run the "Gradual Site Delete timer job", it no difference, the delete site was still lingering about. 

Used PowerShell to remove the deleted site collections as shown above.  E.g. PS> Remove-SPDeletedSite -Identity ""

More Info:
http://blogbaris.blogspot.co.uk/2013/01/delete-sharepoint-2010-site-collection.html
 

Wednesday 4 June 2014

SharePoint 2013 urls

I am often having to go to Google to find SP url formations, to work with the product as I don't memorise them.  Ahmed Farag has a good post on this:
"Famous SharePoint URLs & Locations"

My common ones are:
Login using Windows credentials
/_windows/default.aspx
Web Part Maintenance Page
?Contents=1
Sign in as a different user
/_layouts/closeConnection.aspx?loginasanotheruser=true
Change Site Master Page
/_layouts/ChangeSiteMasterPage.aspx
Master Pages library
/_catalogs/masterpage/Forms/AllItems.aspx
User Information list per Site Collection
/_catalogs/users/simple.aspx 
/_catalogs/users/detail.aspx  (also see)
/_layouts/userdisp.aspx?id={UserID}&Force=True
List all users whi have accessed the Site collection
/_layouts/15/people.aspx?MembershipGroupId=0
Get the custom property bag propeties on the Root SPWeb of the site collection
/_api/web/allproperties
Get the version of the SharePoint server (Patch level)
 /_vti_pvt/Service.cnf
SharePoint Workbench for SPFx
/_layouts/15/workbench.aspx

Also see:
http://blogs.msdn.com/b/how24/archive/2013/05/23/famous-sharepoint-urls-amp-locations.aspx

Tuesday 3 June 2014

SharePoint 2013 publishing site blog won't allow anonymous access

Problem:  I have a public facing website that has a blog.  I have been hitting my head trying to make it anonymous when eureka Google saved me on a topic I had totally forgotten. 

Initial Hypothesis: I have anonymous access enabled on the web application and on the site collection.  I started thinking it was the blog not inheriting permissions so I tried applying anonymous access at the site level followed by examining permissions on the posts list.  All erroneous so I reverted to full inherited permissions.  I Google-ed the symptoms and viola, it is the ViewFormPagesLockDown feature.

Resolution: Turn off the ViewFormPagesLockDown feature for the site collection and anonymous access users will not get the credential prompt when accessing the post.
PS> $lockdown = Get-SPFeature viewformpageslockdown
PS> Disable-SPFeature $lockdown -url http://sitecollectionURL
and confirm the change.

More Info:
http://blogs.msdn.com/b/varun_malhotra/archive/2012/01/13/can-t-access-a-list-on-an-anonymous-site-sharepoint-2010.aspx
http://blog.mastykarz.nl/preventing-authenticated-visitors-browsing-system-pages/ (Important)

Anonymous REST JS call error using SP2013

Problem: I am making an anonymous JavaScript REST call to a image library.  I get the error "Request failed.  The method GetItems of the type List with id ... is blocked by the administrator on the server.nundefined".
 
Resolution:  Change the ClientCallbackSettings on the web application as shown below.
 
 

$wa = Get-SPWebApplication http://www.demo.dev
$wa.ClientCallableSettings .AnonymousRestrictedTypes.Remove([microsoft.sharepoint.splist],"GetItems")
$wa.Update()


More Info:
http://www.sharepoint-zone.com/search/label/ClientCallableSettings.AnonymousRestrictedTypes
 

Saturday 31 May 2014

Content Type Hubs - Thoughts on running Global Content Types for an Enterprise - Part 1

Rough Notes - Publish once cleaned up

Problem: Content Types are a misunderstood and often underutilised set of functionality for syndicating metadata.  Management of the syndication's and ramifications of decisions often lead to unpleasant scenarios.

Notes:  Not a lot has changed between SP2010 and SP2013.

General if you need to reuse a content type in more than 1 site collection it is a good candidate to go into the content type hub.

It is a good idea to ensure that each DTAP environment has identical Content Types in the Content Type Hub.  Concept Search has a product that synchronises Content type across env.  So changes made in production can be pushed to pre-production and then onto your UAT environment.  Writing code to do this is fairly simple.

Concept Search is a third party solution designed to auto-classify content that’s dependent on metadata in the MMS Term Stores.

If is a good idea to have a service owner for Content Type across the enterprise, as this role is very closely related to the Taxonomy/Term store, it is a good idea for this service owner to manage both of the MMS relevant functions. 

A good break down is to work out what the required columns on all data is and this improves search considerable.  For example all document should have a country.  This could be a Term Set with Global, regions (e.g. EMIA), countries (e.g. United Kingdom) and provinces/states (e.g. )  allow each document to be tagged to a country.

I like to group Global Content Types into 2 or 3 layer namely (Enterprise CT's, Division Specific CT's and application specific CT's).  The CT's created in any of these layers inherit from the layer above.  For example, all our documents have an additional field country that applies to all documents (Enterprise global CT), at a division Marketing (A marketing medium media e.g. Newspaper) is appended.  Lastly in our contrived example we have built a specific application for tracking our marketing and this would add the fields (product Type, Audience age range) shall be added.  This allows for consistent tagging/metadata of data within an enterprise.

A good example of search working with meta data is not on the enterprise search you can add a filter on country and allow folks to easily filter down to country specific documents quickly.

How it works:
  • Site Collections can subscribe to CT in the CTH
  • CTH is a site collection
  • Site Collection consume the CT's in the CTH
MMS Term Groups are closely related to the Content Type Hub:




SharePoint 2013 SEO Note

Overview: This post looks at optimising a public facing website built on SharePoint 2013.  This version of SharePoint has additional SEO capabilities built into the product.

WebCEO  - http://www.webceo.com

http://www.awesomium.com/ - Nice for displaying html from a custom windows application.

PageSpeed Insights:  This is a Google service to check the speed of your public website and provides a nice summary of items you can optimise.  Useful for SEO also.

Friday 23 May 2014

SharePoint 2013 Licencing

Overview:  I have looked at licencing in the past and SP2013 licencing seems

  • There is only 1 server licence (not longer, standard, enterprise or Internet as was the case in SP2010), pricing for the server licence has come down significantly.
  • Cals are still broken into Standard or Enterprise.  Pretty similar pricing but you can mix and atch cals depending on what your users use.
  • OWA is not paid for at the server level, no cals needed in read-only mode but if you are editing documents you need office cals licences.

Disclaimer:  These are my note and I'm not an expert on licencing.  This is my rudimentary understanding.  If any one with good info pls comment or let me know so the post has real value.

SP2010 licencing
SP2010 licencing ore info on this site 

Sunday 18 May 2014

SharePoint 2013 Zurb Foundation Publishing site tips

Advanced Banner


Tabs



Hover over image display
<h4>Executive Directors</h4>
<style>
#tooltip1 { position: relative; }
#tooltip1 a span { display: none; color: #FFFFFF; border-style: inset; border-color:#FFFFFF;  }
#tooltip1 a:hover span { display: block; position: absolute; width: 150px; background: #aaa url(/PublishingImages/150x230/pbeck.jpg); height: 224px; left: 300px; top: -40px; color: #FFFFFF; padding: 0 0px; z-index:1; }
#tooltip2 { position: relative; }
#tooltip2 a span { display: none; color: #FFFFFF; border-style: inset; border-color:#FFFFFF; }
#tooltip2 a:hover span { display: block; position: absolute; width: 150px; background: #aaa url(/PublishingImages/150x230/rlarkin.png); height: 224px; left: 330px; top: -40px; color: #FFFFFF; padding: 0 0px; z-index:1; }
</style>
<ul>
<li id="tooltip1"><a href="#">Paul Beck, Chief Executive Officer<span> </span></a></li>
<li id="tooltip2"><a href="#">Richard Larkin, Commercial and Strategy Director<span> </span></a></li>
</ul>


 

Thursday 1 May 2014

SSRS 2012 with SP2013 component diagram

I like diagrams as they help me understand faster.  I wrote a post about 9 months ago about installing SSRS for SP2013, and recently a friend called me to moaned about the documentation available and I remember it being very poor.  My posts cover the topic be re-reading the posts and it could be clearer.  The key is to understand the pieces and where they sit.  He had seen my posts but some related to SP2010 and the 1 specific post relating to SP2013 using SSRS 2012 is not instantly understandable.  It has valuable info but where the parts/components reside is not perfectly clear so I put this diagram together.


There are only 4 parts, the fewer servers to have the easier it is to do theses steps.  I.e. on 1 server farms like dev it's pretty easy as all the SQL components are installed hopefully during the initial SQL Server 2012 SP1 install.

Part 1, you need a SP 2013 farm
Part 2, you need to have a database with the SSRS components/functionality install.  Now if you are using the same database as your SP database ensure the SSRS components were installed at the initial build.  This diagram assumes you have a separate SQL instance which is pretty reasonable.
Part 3, you need to run a core SQL install (only needs the SQL component relating to SSRS) on the SharePoint App/web server.  This 1 or more servers containing the SharePoint binaries.  This is the step that is new/different and most people don't do.
Part 4, run some Powershell to create the SSRS SSA and hook up the relationship.  This is done on any of the SharePoint servers such as your main CA box.

This post couple with the overview should help you understand the components needed for large or automated installs.
http://blog.sharepointsite.co.uk/2013/08/installing-ssrs-2012-on-sp2013-within.html


====================


Tip: If you need to move your Reporting Services database to another server, you need to manually add the RSexecRole to the new SQL Server.

Wednesday 30 April 2014

OWA intermittently not returning office documents in Office Web Apps 2013

Problem: Intermittent requests are not returning the pdf/word documents. Most requests are working and occasionally 1 request doesn't work. Every 4th request tries to get the pdf to display on Office Web Apps for a few minutes without any error message and then stops trying and displays the message "Sorry, Word Web App can't open this ... document because the service is busy."

I have 4 OWA/WCA servers on a stretched farm being used by SP2013 etc.

Initial Hypothesis: Originally I thought it was only happening to pdfs but it is happening to word and pdf documents (I don't have excel docs in my system). My monitoring software SolarWinds is badly configured on my OWA servers as the monitor is showing green, drilling down into the servers monitoring; the 2 application monitors are both failing. The server should go amber if either of the 2 applications monitoring fails and in turn red after 5 minutes. At this point I notice that I can't log onto my 4 OWA/WCA server. Web request are not being returned. I look at my KEMP load balancer and it says all 4 WCA servers are working, I notice the configuration is not on web requests but on ping (not right) and the NLB/KEMP is merely redirecting every 4th request to the broken server.

Resolution:
  1. Reboot the broken server, once it comes up I can make http requests directly to url http://wca.demo.dev/hosting/discovery on the rebooted server.
  2. SolarWinds monitoring is lousy - need to fixed the monitoring.
  3. Kemp hardware load balancing needs to be changed from checking the machine is "ON" to rather checking each machine using a web request.
SolarWinds Monitoring is not configured correctly

 

Sunday 20 April 2014

Backup and Restore Site Collection in SP2013

Overview:  I have seen several customers us Backup and restore to help speed up the development process and have the ability to deploy between DTAP environments.  So the basic premise is create the site collection on dev/a and use backup and restore to promote the site collection including customisations and code in the next environment.

SharePoint 2010:  In SP2010 this worked assuming the env you are going to has a higher patch level than the source environment.  So if you went from SP2010 + cu to SP2010 + SP1 in production backing up and restoring the site collection works.  The trick was to package all assets into the site collection and to ensure all environments were on the same edition/patch level (or at least the destination farm was patched to a higher level than the source farm).

SP2013:  You can use PS backup and restore to move site collections but it is further restricted.  The source and destination environment need to be the same edition.  My issue is I can't move a troublesome production environment back to UAT as my UAT has been patched and is a later/newer version of SP2013 on prem.

I learnt this when restoring the site collection from 15.0.4481.1005 (SP2013 + Mar CU) on the source and trying to go to 15.0.4569.1000 (SP203 + SP1) y destination farm.

Restore-SPSite : 0x80070003
At C:\Users\SP_install\AppData\Local\Temp\5ae5fd1c-86ac-4032-8975-c739f39b6f36.ps1:3 char:1
+ Restore-SPSite –Identity "http://uat.futurerailway.org" –path "C:\Software\Deplo ...
+ CategoryInfo : InvalidData: (Microsoft.Share...dletRestoreSite:SPCmdletRestoreSite) [Restore-SPSite], DirectoryNotFoundException + FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletRestoreSite  

Conclusion:  To move Site collection between farms or to different content databases, the SP farms need to be using the exact same version of SP.

Sunday 6 April 2014

Writing SharePoint PowerShell Modules or snap-ins

A module is a self-contained set of code that once loaded can be called/used.  I use modules to keep similar SharePoint functionality that I want to reuse.  There are 2 basic types of modules:
1.> Script modules, this is the easiest and most common type of module.  Take you .ps1 file and rename it .psm1
2.> Code/Binary modules are compiled dll's that can be loaded and used.  You can also create a snap-in with binary code.
3> Snap-ins, are compiled code dll's.  The snap-ins can be loaded and used within your PS console.


Snap-ins are not code modules.  I prefer to keep my files as ps1 files and load them in a controlling .ps1 files before using them or if I need to write code that uses the CSOM I would use C# and create a snap-in that I can port and consume.


More Info:
http://zimmergren.net/technical/sp-2010-how-to-create-a-powershell-snapin-part-1
http://www.wrox.com/WileyCDA/Section/Extending-Windows-Powershell-with-Snap-ins.id-320555.html



Monday 24 March 2014

Installing CU1 for SharePoint 2013

Overview: I need to upgrade from SP2013 CU June 2013 to SP2013 SP1. 

Tip: SP1 does not require the March 2013 PU to be installed.  In my situation it was already installed.

Steps:
1.> Check there are no upgrades pending.
2.> Run the SP1 upgrade on each machine in the farm containing the SP binaries.
3.> Ensure the Upgrade is required PS>get-spserver $env:computername).NeedsUpgrade
if True on all SP machines (can also verify on a large farm using CA as shown below) then
4.> PS> psconfig.exe -cmd upgrade -inplace b2b -force  (This will upgrade the SharePoint databases and update the binaries on the 1st machine).
5.> Run psconfig on all the remaining SharePoint servers in the farm.

Result:  The farm should upgrade, my dev farms upgrade however my UAT and Prodcution farms did not complete the upgrade, the fix is shown below.

More Info:
http://blogs.msdn.com/b/sambetts/archive/2013/08/22/sharepoint-farm-patching-explained.aspx

***************

Problem:  The Usage and Health database cannot be in an AOAG when upgrading.
 ERR          Failed to upgrade SharePoint Products.
An exception of type System.Data.SqlClient.SqlException was thrown.  Additional exception information: The operation cannot be performed on database "SP_UsageAndHealth" because it is involved in a database mirroring session or an availability group. Some operations are not allowed on a database that is participating in a database mirroring session or in an availability group.
ALTER DATABASE statement failed.
System.Data.SqlClient.SqlException (0x80131904): The operation cannot be performed on database "SP_UsageAndHealth" because it is involved in a database mirroring session or an availability group. Some operations are not allowed on a database that is participating in a database mirroring session or in an availability group.
ALTER DATABASE statement failed.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler,


Tip: Any CU, PU or SP will not perform the upgrade if the Usage and Health SharePoint database is a AOAG database.  You need to remove the db and perform the upgrade.

Initial Hypothesis:  The error message is pretty clear that the problem is the UsageandHealth database can't be modified in the upgrade process if it is part of the availability group.  I use an aliase so I could repoint the aliase to the primary database do the upgrade and then update the SQL aliase back to point to the listerner or the approach I use is to remove the AOAG listener for the usage database, perform the upgrade to SP and readd the AOAG for the HealthandUsage database.

Resolution:
1.> "Remove the UsageAndHealth database from the Availability Group",

2.> Perform the SP1 upgrade
3.> Change the Recovery model to "FULL" and perform a Full backup.
4.> Add the database back in as part of the availability group.

***************

Problem: When running PSConfig to upgrade my SP2013 farm to include SP1, the upgrade fails and the PSConfigDiagnostic log informs me of the problem:
WRN Unable to create a Service Connection Point in the current Active Directory domain. Verify that the SharePoint container exists in the current domain and that you have rights to write to it.
Microsoft.SharePoint.SPException: The object LDAP://CN=Microsoft SharePoint Products,CN=System,DC=demo,DC=dev doesn't exist in the directory.
at Microsoft.SharePoint.Administration.SPServiceConnectionPoint.Ensure(String serviceBindingInformation)
at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()






More Info
http://onpointwithsharepoint.blogspot.co.uk/2013/06/configuring-service-connection-points.html
http://sharepointfreshman.wordpress.com/2012/03/02/1-failed-to-add-the-service-connection-point-for-this-farm/
http://gallery.technet.microsoft.com/ScriptCenter/af31bded-f33f-4c38-a4e8-eaa2fab1c459/
http://blogs.technet.com/b/balasm/archive/2012/05/18/configuration-wizard-failed-microsoft-sharepoint-spexception-the-object-ldap-cn-microsoft-sharepoint-products-cn-system-dc-contoso-dc-com-dc-au-doesn-t-exist-in-the-directory.aspx

Thursday 20 March 2014

Data Protection and Regulation

Update 2023 December - Accessibility

  • European Accessibility Act (EAA) EU directive will be law in member states by June 2025.
  • The Disability Equality Act came into force 2022.
  • General Equal Treatment Act (AGG) came into force in 2006, prohibits discrimination based on disability.

Update 2022 Mar 16: Privacy Management | OneTrust implement Privacy Management

Update 2021 Nov 20:  Applies to organisation handling personal EU and UK data member data.  Limits what organisations can do with peoples personal data.  It's enforce but the key is to only use data you need, protect personal people's data basically how any company should behave.

UK is still part of UK law since Brexit Data protection Act 2018.  Differences:

  • Territorial applicability of UK GDPR
  • International transfers of personal data

Overview:  Data protection in relation to SharePoint is a large body of information.  This post outlines my notes on holding data within SharePoint and generally applicable to various regulations I have come across.  Also, see my post on Compliance for O365 and SharePointLast updated: 18 July 2019

Records Management:  Data needs to be disposed of depending on the applicable rules, the rules depend on the industry, country, the category of data.  AvePoint has good records management and governance tools to help with the disposal/cleanup of data.

Search: Request for Information (Freedom of Information (FOI)).  SharePoint can be used to traverse over multiple systems/LOB to determine where information is held about individuals.  Configure to generate reports or as a starting point in trawling data in the enterprise.

United Kingdom:
Updated 24 May 2016 - The European Union (EU) General Data Protection Regulation (GDPR) "intends to strengthen and unify data protection for individuals within the European Union (EU). It also addresses export of personal data outside the EU" Wikipedia 
The EU GDPR applies to EU member states such as the UK, Germany et al. and covers personal data held by companies in the EU and extends to companies holding EU citizens data.  Of interest for the GDPR is that companies can be fined up to 4% of turnover.  SharePoint and Office 365 holds a lot of company assets and data and appropriate protection needs to be in place.  Part of any companies active Defense needs to include SharePoint.  Of note here is Office 365 have fantastic capabilities in defense and I believe will increase the speed enterprises move to the cloud.  

http://www.computerweekly.com/news/2240114326/EC-proposes-a-comprehensive-reform-of-data-protection-rules New EU Data Protection Directive not yet legally binding.  Companies in the UK are bound by the Data Protection ActFreedom of Information Act 2000  also plays a part with personal data. DPA 1998 explained.  DPA 2018 alignes with GDPR.

Purpose of GDPR (started 25 May 2018):
  • Protect personal data
  • Consistency legislation in the EU
  • Encourage competition between EU countries
GDPR is concerned with EU citizens personal data and protecting it.

DLP has a module for Health Records that adheres to the U.K. Access to Medical Reports Act 

G-Cloud allows public sector organizations to buy cloud services, from a range of suppliers on a validated secure network.  In effect, it is cloud services for local and central government.  G-Cloud in effect offers the cloud (think AWS & Azure type services) to government bodies.  Updated: 16 March 2016, the G-Cloud has been abandoned.

Dealing with Breaches:
SharePoint holds a ton of company data and needs to be part of any companies Active Defence strategy.  Still need the old school basic defences: Firewall, Intrusion detection, and anti-virus. Do you have a list of critical applications and data within SharePoint?  Do we know who we do business with (client or HR could compromise our data)?  Who is likely to attach?  Employee, organised crime, ... and what happens when we are compromised?  (Do we shut down or restrict,  how do we identify, legal and forensics, communication plan). DLP can help with breaches:
PII data
Theft - are employees mining SP data looking for highly confidential data, IP or client lists  
Security Centre helps with:
  • Investigation
  • Forensic collection
European Union (including the UK):
  • Companies will be required to appoint data protection officers if more than 250 employees.
  • Organisations will have to notify citizens in plain language what information is collected and how it is used as well as explicitly get consent before using any personal information.
  • Users of online services must also have the right to be forgotten, which means they must be able to remove or delete personal information from an online service.
  • Clear rules for data transfer across borders within multinational corporations with a streamlined process that once approved by one data authority, will be accepted by all others.
  • Requiring organisations to notify the national data protection authority and all individuals affected by a data breach within 24 hours.
  • Businesses operating in more than one EU country will, however, welcome the fact that they will be subject to oversight from one supervisory authority rather than multiple authorities
  • Once the directive is accepted companies will have 2 years to comply.
  • Organisations will only have to deal with a single national data protection authority in the EU country where they have their main establishment. Likewise, people can refer to the data protection authority in their country, even when their data is processed by a company based outside the EU. Wherever consent is required for data to be processed, it is clarified that it has to be given explicitly, rather than assumed.
  • People will have easier access to their own data and be able to transfer personal data from one service provider to another more easily
  • A ‘right to be forgotten’ will help people better manage data protection risks online: people will be able to delete their data if there are no legitimate grounds for retaining it.
  • EU rules must apply if personal data is handled abroad by companies that are active in the EU market and offer their services to EU citizens.
  • Penalties of up to €1 million or up to 2% of the global annual turnover of a company.
South Africa:

What is POPI?
Protection of Personal Information (POPI) is the legal requirement in South Africa for holding, collecting, distribution, amending and destruction of information involving people and companies. POPI controls how your personal information is used by organizations, businesses or the government.
With so much personal data held by an increasing number of companies, there needs to be some benchmark for companies to follow if they are to ensure that data is handled legitimately. POPI provides the laws/framework to guide how companies must store personal data relating to people and companies that it holds in either electronic or paper form.
In a nutshell, when holding parties personal data POPI attempts to enforce:
  • transparency
  • only collect information that you need
  • ensure the data is protected/secure
  • ensure the personal data help is correct, required and up to date
  • discard data when it is no longer needed
  • ensure the end person/subject has given his/her explicit consent to keep and use their personal data
  • allow the end person/subject to see their own data that you hold if they request it

Why is should you adhere to POPI?

  • Customer confidence is improved
  • No superfluous data is stored
  • Data is more secure, accurate and old data is expired
  • Avoid criminal and civil actions

What you need to do?

POPI applies to all IT and paper-based data that your company holds.  Your company will take steps to ensure the security of personal data which are held in electronic and paper form.  You must prevent the unauthorized disclosure of data to third parties, and loss or damage to data that may affect the interests of end person/subjects.  You will also ensure that data processors your organization uses to provide an appropriate level of security for the personal data which they are processing on your behalf.  Any data must be restricted to the appropriate person and your company needs to take steps to ensure it is not allowing unauthorized access to data and information.

What happens if you Violate POPI - EY South Africa
United States
FATCA requires a financial institution to identify and report US customers. 

Safe Harbour  - US companies storing EU customer data would self-certify that they adhere to 7 principles to comply with the EU Data Protection Directive and with Swiss requirements. Overturned in 2015.  The EU-US Privacy Shield is an agreement between the European Union and the United States to enable US businesses to store EU citizens personal data that complies with EU privacy laws.  EU-US Privacy Shield in effect the replacement to safe harbour agreement.  

Patriot Act - Greatly affects companies as the US can request access to data.  This leads to multinationals choosing to host data outside of the US.

Internal State Laws - Each federal state may have localized laws that your business needs to adhere too. For example California Data Privacy Protection Act (CDPA)

Brazil 
LGPD - General Data Protection Law - Basically GDPR for protecting personal data and peoples user privacy.

China has the PRC Cybersecurity Law relating to protecting personal data.
Hong Kong has the Personal Data (Privacy) Ordinance (Ap.486)
Middle East: Bahrain - Data Protection Law (Law No. 30 of 2018), Qatar - Data Protection Law (Law No. 13 of 2016), UAE has Digital payment Regulation and Data protection laws specific to each emirate
Turkey - Law on the Protection of Personal Data 6698 (LPPD)
South America, all the major countries have Data protection laws including Argentina (Law 25.326)
Canada - Various state laws pretty much PIPA.
Australia has a host of Data privacy laws including the "Australian Privacy Principles"
Japan has APPI.
South Korea has PIPA

Other:
Common Reporting Standard (CRS), same idea as FACTA but not just US customers, heavier and most of Europe and others.  "CRS is a globally coordinated approach to the disclosure of income earned by individuals and organizations outside their country of tax residence", KPMG.com.

Pharma and Medical:
  1. HIPAA - "Health Insurance Portability and Accountability Act of 1996) is United States legislation that provides data privacy and security provisions for safeguarding medical information"
  2. DSP is broadly similar to HIPAA but from the UK, it is a toolkit for compliance for the NHS.
  3. HL7 - "Health Level-7 refers to a set of international standards for the transfer of clinical and administrative data between software applications used by various healthcare providers"
  4. FHIR V4 - replaces HL7 for exchanging data exchange and information modelling standards for over 20 years. FHIR is a new specification like HL7
  5. GxP - Good x Practice e.g. GCP, GMP (manufacturing), CLP, 
  6. GCP - Good Clinical Practice - Guidelines and regs used in the pharma industry
  7. GMP - Good Manufacturing Practice
  8. GAMP - Good Automated Manufacturing Practices - applies to software development.  GAMP has 4 categories.  Part 5 is the most hardcode for companies.  Cat 1 is infrastructure Software, Part 4, for instance, is Configurable software, PArt 5 is for custom software.  So depending on the software you are providing to a customer, you need to be audited by external parties and clients to be compliant.  If you have written your own Software, you need to be GAMP5 compliant.  GAMP is GxP but for IT systems.  Build-in quality, this requires following procedures and principles when building software products.  ISPE - Run GAMP and check qualifications.
  9. Eudralex - Pharma industry in EU guidelines for dev, manufacture and control of medicinal products.  Rules governing medical products in the EU.
  10. EMA - European Medicines Agency, same as FDA but covers Europe.
  11. FDA - Food and Drug Agency out of the US.
  12. MHRA (Medical and Healthcare Products Regulation Agency), same as Eudralex but for the UK.
  13. Title 21 CFR part 11 - FDA reg so that electronic records and signatures equivalent to paper hand signed reconds and consent.  About storing e-records including securing and signatures.  Code of Federal Regulations (CFR) Title 21 Part 11.  Ensure the system is secure, audit logs of all transactions with timestamps maintain the integrity of the open or closed system.  Signatures must ensure non-repudiation (the signer can't claim it wasn't him).  E-signatures can be biometric-based, this is hard for web-based systems without specific hardware.  E-signature that are not biometric-based requires that on the first sign-in all components (general means sign in with username and password) on the first signature assuming the user is already logged into the system.  See Section 11.200 Electronic Signature Components and controls go to point a) (1)(i).  Subsequent e-signatures can 1 component meaning the username. If using biometric signatures each signature uses the biometric method again.
  14. ISO 27001 - ISO is best practices guidelines, not regulations.  27001 is concerned with info security and info assets.  Asses and treatment of security risks.  Also see ISO27002 & ISO27017.
  15. ISO 9001 - Quality Management, checklist / process orientated.
  16. FDA (Food and Drug Agency) is equivalent to EMA (European Medicines Agency) in Europe.  

ISO 27001 for SaaS:

It is pretty easy to get ISO 27001 certified for SaaS companies and it brings huge benefits if implemented correctly.  Azure provide fantastic documentation and if your product is based on Azure.  It is really easy as the technology infrastructure is validated.  

Azure have a fantastic set of documentation for certifications called Blueprints

https://docs.microsoft.com/en-us/microsoft-365/compliance/offering-iso-27001?view=o365-worldwide

https://docs.microsoft.com/en-us/azure/governance/blueprints/samples/iso27001/

ISAE3402 (SOC):

ISAE 3402 is similar to ISO 27001

Accounting and Tax:

XBRL (eXtensible Business Reporting Language) - XML based format for exchanging business information.  iXBRL is a derivative of XBRL used in the UK for submitting company accounts, VAT, self assessments.  iXBRL is also used to submit annual accounts to companies house each year.

ISO 21378 - Audit data