Stretched farms, Geo-replication and options

Stretched farm: A stretch farm is where you put a single farm into 2 data centres.  This provides redunancy across locations improving HA and DR.   Generally the 2 data centres will be located relatively close to one another geographically as the ping time needs to be less than 1 ms from any WFE to the primaray SQL Server database and at least 1 gigabit per second bandwidth.  hospitals with multiple sites / data centres that have good network connectivity and are located close together is a perfect example.

Geo-replication: SharePoint 2013 like previous versions does not have Geo-replication built into the product.  You can use PowerShell to provide this functionality.  A more achievable approach is to use a 3rd party tool such as DocAve to achieve and maintain Geo-replication.

SharePoint 2013 Geo-Replication

Another option, without the hassel to GEO-Replication is to use Riverbed devices distributed around your corprate network.  This decreates network requirments and improves the end user experience and does not involve syncronising data to multiple farms.  Riverbed's website.

Finding Correlation Errors on a SP2013 farm

Background: SP 2013 has rich and expansive logging/tracing capabilities.  Logging is done via the Unified Log Service (ULS).  This will add logs to the tracelogs (often refered to as the ULS logs or ULS trace logs or ULS, it doesn't matter except you need to understand the ULS service is not only the trace log) and the Windows Event Viewer.  Anything logged in the Event Viewer log will also be in the ULS trace logs.

It is worth check how your logging is setup on your farm.  I change my default location for my ULS trace logs.  Change the logging so it matches your farms requirements.

On a small farm, it's normally pretty easy to take a Correlation Id / the unique GUID generate for the SharePoint request, open the trace log using notepad and find the error.  The default is to create a trace log every 30 minutes, these log files have a lot of data in them on busy production farms, and as you may have a large farm you also have multiple logs to check. I use Microsofts'd unsupported ULSViewer to look at all my logs regardless of farm size.  You can trace the logs in a live format and then filter out what you need.  Another option is to open existing errors to get historical issues.  If you know the datetime and server where the error occured, you open the correct log file (it is labled with a datetimestamp) and then either filter for the correlationId or look around the time the error occured.

Lastly, timer jobs ship entries from the ULS logs into the SharePoint Logging Database (SP_UsageandHealth).  You can directly query the SP_UsageandHealth database using T-SQL.

Tracing Correlation Errors on a SharePoint 2013 Farm.
User passes you a correlation Id and the date/time when the error occured, find the apprioate ULS trace log.  Open the log using ULSViewer and filter for the CorrelationId.  If you can reproduce the bug, you have the developer dashboard that can be turned on (performane penalty) selectively, their is a new SP2013 tab "ULS" this will show you the ULS trace snippet relating to this request.

On a big farm you may want to1st find out which server in the farm had the error:

Merge-SPLogFile -Path ".\error.log" -Correlation "5ca5555c-8555-4555-555b-f555af4d5555"

Tip: Be aware this is a heavy process, so restrict which logs you will merge.
 

Use ULSViewer to find the correlationId and review the logs.

Use IT tools or Fiddler to examine the http response from SharePoint to get the correlation Id, this is the SPRequestGUID (assuming it is not show on the error message).

 

More Info:

Tobias Zimmergren has a great post on working with Correlation Id's.

http://www.sharepointblog.co.uk/2012/09/logging-capabilities-of-sharepoint/

http://habaneroconsulting.com/insights/An-Even-Better-Way-to-Get-the-Real-SharePoint-Error

List of the ULS viewers

http://www.sharepointblog.co.uk/2012/09/logging-capabilities-of-sharepoint/

SP2013 AutoSPInstaller Error Correction

This post contains alist of issues I have had to correct on multiple farm builds. 

Problem:  After a remote offline install the main CA box shows an upgrade is required.  CA > Manage servers in this farm :Upgrade Required
Resolution: Run cmd > C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\BIN>PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures

Problem: PU or CU is not getting installed.
Resolution: Ensure the .exe are extracted out e.g. ubersrv2013-kb2817414-fullfile-x64-glb.exe this is the June 2013 CU.  Ensure the March 2013 PU is extracted and in the same folder ubersrvsp2013-kb2767999-fullfile-x64-glb.exe.  Ensure they are placed in the correct location for me it is ..\2013\Updates\...  CU are cumulative so you only need the PU and then the latest CU.  Don't rename the CU - this caught me out.

Problem: I kept receiving the error: 17303 Extracted file error for the June CU (ubersrv2013-kb2817414-fullfile-x64-glb.exe).
Resolution:  Looks like the download was corrupt, re downloaded the CU and extracted the contents which fixed the issue.

Problem: I built a 4 SP server 2013 farm.  I had 2 Search nodes setup via AutoSPInstaller and added another 2 search nodes (index partitions with it's corresponding replicated index).  After a reboot I got into the following state and could not get WFE2 to join the search farm.

Using CA and looking at the Search topolgy on WFE2, I can see the error "Administrative status  The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel." and "Unable to retrieve topolgy component health states. This may be because the admin component is not up and running.".

 

Resolution: Check the routing, my CA box is WFE, I also had a CA role on WFE2, while playing with the farm I added a host entry to work with CA on WFE2.  The issue is that this is the same host entry as where my Admin component sits.  Removing the host entry on WFE2 fixed the status of the index and query components.

 

 

Installing OWA 2013

Series of Posts for Installing and Configuring Office Web Apps (OWA) 2013 to work with SharePoint 2013.

Overview: Manually Install OWA 2013 on Windows 2008 R2 SP1.

Steps:

1. Download and install .Net 4.5 (dotNetFx45_Full_setup.exe)
2. PowerShell 3 (Windows Management Framework 3.0)  Windows Server 2008 R2 SP1 (64-bit versions: Windows6.1-KB2506143-x64.msu)
3. Install KB2670838 (Windows6.1-KB2670838-x64.msu)
4. Run PS>Import-Module ServerManager
   PS > Add-WindowsFeature Web-Server,Web-WebServer,Web-Common-Http,Web-Static-Content,Web-App-Dev,Web-Asp-Net,Web-Net-Ext,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Includes,Web-Security,Web-Windows-Auth,Web-Filtering,Web-Stat-Compression,Web-Dyn-Compression,Web-Mgmt-Console,Ink-Handwriting,IH-Ink-Support
5. Install OWA 2013 (wacserver.img)

 

Load balancing in SharePoint

Background: SharePoint 2010 required sticky sessions.  In effect when the 1st request from a user comes into SharePoint 2010 to the load balancer, the request will be directed to a particular SharPoint WFE server that will then serve the subsequent requests (the user now has session affinity/sticky session).

SharePoint 2013 has distributed cache so you no longer need to use sticky sessions, the load balancer merely needs to direct the user to any server (preferably the quietiest).

Possible options for Load Balancers in SharePoint:
F5/BIG-IP (Level 2, 3, 4 &7)
KEMP (Level 4 & 7)

Barracuda (Level 4)
Cisco - I don't think Cisco build dedicate load balancers and more however "Cisco IOS-based router product has load balancing capabilities".
Not familiar with:
http://www.coyotepoint.com/products/compare
http://www.radware.com/Products/ApplicationDelivery/AppDirector/load_balancing.aspx
http://www.a10networks.com/resources/files/DG_MS-SharePoint2013.pdf - NLB and SSL termination.

Key features needed off a Load Balancer:
Distributed Denial of Service (DDoS) attack protection
Http compression
SSL acceleration/ and termination
Persistance/Sticky sessions
HA via distribution of requests and verifying that SP is running.
 

Identifying page load times using Fiddler

Problem:  I have poor performing pges and I need to see which pages are causing my issues, I have the Developer dashboard or web testing in Visual Studio 2012 Ultimate but a nice quick way to identify the issue is to use fiddler with a slight customisation.

Resolution: There are 2 options in Fiddler to monitor performance:

1.> Highlight the requests and click the Timeline tab

2.> Customise Fiddler to show request times for each request as shown below.

Process to Customise Fiddler:

Create a TTL column in fiddler to see how long each request is taking.  In Fiddler press Ctrl + R, this opens the editor CustomRules.js.  Edit as shown below.



In my scenario the page is taking 6 seconds to load as shown in yellow highlight.  My SQL database hosting my SP Config database could not grow.  Once I correct this, my TTL is under 0.1 seconds on a blank default publishing page.

More Info:
See other related Fiddler posts
Enable the developer dashboard post.

SharePoint 2013 Search Overview

Overview:  This post explains how SharePoint 2013 Search works.  SharePoint 2013 search is the latest search within the SP product and replaces SP and Fast search that was used with SP 2010.

6 Components:

1. Query
2. Index
3. Analytics
4. Content Processing
5. Crawl
6. Admin

Tip:  Group Query & Index roles on the same server.  Then group Analytics & Content processing.  Group Crawl & Admin (you can also add content processing to these servers).

Installing search on SP 2013 creates 4 database (all db's by default use the 'Simple' recovery model):

1. Search Admin,
2. Search analytics Reporting,
3. Search Crawl, and
4. Search Links.

A guideline from Microsoft is: "Add one index partition for every 10 million items in the search index."  This depends on how you are using search, so if you have more documents per index but have few queries or can live with longer response times or your kit can deal more queries this can be increase or decreased.
Index partitions are splitting the data vertically, so if you have 25 million search items and want less than 10 million per partition, you will need 3 index partitions on the 3 index servers.  You do not have redundancy, so if any index goes down your search is broken.  Index replicas as the name suggests is a copy.  I think of this as horizontal scaling.  So if you want HA on your 25 million item search farm, you need another 3 index (Replica) partitions.  You have 6 index servers.  Using index replicas will improve query results speed.

Search Architectures for SharePoint 2013 - From MS (Kavindra Palaraja out of his oit2013-model-sharepoint-search-architecture.pdf document).  This is not my diagram but it explains the components nicely.

Search Components in SharePoint 2013

Replicas and Index partitions explained are refereed to as Rows and Columns. 

Add a new Index partition when the number of documents in the index exceeds 30 million.

SharePoint 2013 supports 3 types of Crawls:

1. Full (SP2010) 
2. Incremental (SP2010) and
3. Continuous.

• The continuous crawl on works on SP2013 content and shall display content in the crawl results as soon as data has been crawled and run thru the content processing component (CPC) (it doesn't wait for the crawl to complete).  Note: security changes are only picked up after incremental search is run.  There are no crawl logs for continuous crawls, so for troubleshooting go to SQL Search Service DB for the table MSSMiniCrawls (verify).
• "multiple continuous crawls can run at the same time. Therefore, even if one continuous crawl is processing a large content update, another continuous crawl can start at the predefined time interval and crawl other updates. Continuous crawls of a particular content repository can also occur while a full or incremental crawl is in progress for the same repository."  Technet 
• It is a good idea to run incremental crawls as they index more data than just SP2013 data and continuous crawl does not process or retry items that return errors and the incremental crawl shall clean theses items up.

Tip: Results can also be security trimmed at Query time, this is FTC (Full Trust Code) that must be deployed on the query role search server on-prem.


More Info:
Capacity management and sizing overview for SharePoint Server 2013
http://www.microsoft.com/en-us/download/details.aspx?id=30383
http://www.microsoft.com/en-gb/download/details.aspx?id=30374
SP2013 Stretch Farms
SP2013 Database types and desc

Design Goal - Index partitions and Index replicas.

Partitions marked in Red.

Note: 2016/11/16 - Adding email messages e.g. msg to SharePoint has always crawled the data however in MOSS and SP2010, the attachments do not get crawled.  SP2013 (it may be since SP1) and Office 365 will also index the attachments of messages saved in SharePoint.