AutoSPInstaller configuration for SharePoint 2013 Search

Overview:  AutoSPInstaller is a great tool for building SharePoint 2010 ans 2013 farms.  One section of AutoSPInstaller relates to installing SharePoint 2013 search.  This post examines how to configure the xml that drives AutoSPInstaller and supplemental Ps to build the Search farm you require.

 

Below is an example of a search xml config that will setup Search with full redunancy on 2 servers in the SP farm.  This design provides High Availaibility (HA).  MS recommend keeping each index partition under 10 million items.  So in this scenario if you follow MS recommendations you are limited to having 10 million items in your index (hardware, moving outher roles to more servers, usage of search to query all affect how quickly results will be returned so with tuning you can get your indexes to be larger and provide a good end user experience).



 

Tip: Create a seperate SQL aliase fo your 4 Search databases, this will alow you to move the dbs to a new instance or at 40mill documents MS recommend you consider a dedicate search farm.  The SQL Alaise gives you room to adapt.

Tip: Group your search roles onto servers such as:

• Index & Query Processing
• Analytics & Content Processing
• Crawl, Content processing & Search Admin 

Summary:  As of version 3.87 of AutoSPInstaller for an SP2013 installation, you only have 1 index partition.  To create additional partition indexes use PowerShell.  The example below is use the PS code given to me by Joseph Saad.

$WFE2="WFE2"
$WFE4="WFE4"

$ssa = Get-SPEnterpriseSearchServiceApplication #Assuming you have a single Service application
$active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa -Active
$clone = New-SPEnterpriseSearchTopology -SearchApplication $ssa -Clone –SearchTopology $active

New-SPEnterpriseSearchIndexComponent -SearchTopology $clone -SearchServiceInstance $WFE2 -IndexPartition 1
New-SPEnterpriseSearchIndexComponent -SearchTopology $clone -SearchServiceInstance $WFE4 -IndexPartition 1

Set-SPEnterpriseSearchTopology -Identity $clone

More Info:
https://autospinstaller.codeplex.com/discussions/453501

Example:

 4 Server Search farm created using AutoSPInstaller.  AutoSPInstaller version 3.87 only supports creating a farm using one partion index as shown below.

To add "Partion Index 1" to the farm using WFE1 and WFE2 to hold the index components use the following PowerShell:

$ssa = Get-SPEnterpriseSearchServiceApplication #Assuming you have a single Service application
$active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa -Active
$clone = New-SPEnterpriseSearchTopology -SearchApplication $ssa -Clone –SearchTopology $active
New-SPEnterpriseSearchIndexComponent -SearchTopology $clone -SearchServiceInstance "WFE1" -IndexPartition 1
New-SPEnterpriseSearchIndexComponent -SearchTopology $clone -SearchServiceInstance "WFE3" -IndexPartition 1
Set-SPEnterpriseSearchTopology -Identity $clone

 

Results in a farm that can handle up to 20 million documents due to the partioned index.

 Another Example:

 

SharePoint 2013 Search Overview

Overview:  This post explains how SharePoint 2013 Search works.  SharePoint 2013 search is the latest search within the SP product and replaces SP and Fast search that was used with SP 2010.

6 Components:

1. Query
2. Index
3. Analytics
4. Content Processing
5. Crawl
6. Admin

Tip:  Group Query & Index roles on the same server.  Then group Analytics & Content processing.  Group Crawl & Admin (you can also add content processing to these servers).

Installing search on SP 2013 creates 4 database (all db's by default use the 'Simple' recovery model):

1. Search Admin,
2. Search analytics Reporting,
3. Search Crawl, and
4. Search Links.

A guideline from Microsoft is: "Add one index partition for every 10 million items in the search index."  This depends on how you are using search, so if you have more documents per index but have few queries or can live with longer response times or your kit can deal more queries this can be increase or decreased.
Index partitions are splitting the data vertically, so if you have 25 million search items and want less than 10 million per partition, you will need 3 index partitions on the 3 index servers.  You do not have redundancy, so if any index goes down your search is broken.  Index replicas as the name suggests is a copy.  I think of this as horizontal scaling.  So if you want HA on your 25 million item search farm, you need another 3 index (Replica) partitions.  You have 6 index servers.  Using index replicas will improve query results speed.

Search Architectures for SharePoint 2013 - From MS (Kavindra Palaraja out of his oit2013-model-sharepoint-search-architecture.pdf document).  This is not my diagram but it explains the components nicely.

Search Components in SharePoint 2013

Replicas and Index partitions explained are refereed to as Rows and Columns. 

Add a new Index partition when the number of documents in the index exceeds 30 million.

SharePoint 2013 supports 3 types of Crawls:

1. Full (SP2010) 
2. Incremental (SP2010) and
3. Continuous.

• The continuous crawl on works on SP2013 content and shall display content in the crawl results as soon as data has been crawled and run thru the content processing component (CPC) (it doesn't wait for the crawl to complete).  Note: security changes are only picked up after incremental search is run.  There are no crawl logs for continuous crawls, so for troubleshooting go to SQL Search Service DB for the table MSSMiniCrawls (verify).
• "multiple continuous crawls can run at the same time. Therefore, even if one continuous crawl is processing a large content update, another continuous crawl can start at the predefined time interval and crawl other updates. Continuous crawls of a particular content repository can also occur while a full or incremental crawl is in progress for the same repository."  Technet 
• It is a good idea to run incremental crawls as they index more data than just SP2013 data and continuous crawl does not process or retry items that return errors and the incremental crawl shall clean theses items up.

Tip: Results can also be security trimmed at Query time, this is FTC (Full Trust Code) that must be deployed on the query role search server on-prem.


More Info:
Capacity management and sizing overview for SharePoint Server 2013
http://www.microsoft.com/en-us/download/details.aspx?id=30383
http://www.microsoft.com/en-gb/download/details.aspx?id=30374
SP2013 Stretch Farms
SP2013 Database types and desc

Design Goal - Index partitions and Index replicas.

Partitions marked in Red.

Note: 2016/11/16 - Adding email messages e.g. msg to SharePoint has always crawled the data however in MOSS and SP2010, the attachments do not get crawled.  SP2013 (it may be since SP1) and Office 365 will also index the attachments of messages saved in SharePoint.

SharePoint 2010 enable the scope immediately

Problem:  After creating a new scope, you need to wait 15 minutes for it to update the new scope (default SP2010 setting is: Automatically scheduled).

Initial Hypothesis: Gary Lapointe use to have a stsadm cmd to push thru the scope but I wanted to do it using PowerShell.  You can also use the Central Admin UI:

Resolution:
PS> $ssa = Get-SPenterpriseSearchServiceApplication
PS> write $ssa.name
PS> $ssa.StartScopesCompilation()
 

Technet has a post on forcing the scopes to update.

More Info:
http://gallery.technet.microsoft.com/scriptcenter/Force-to-update-Search-cf101b62
http://sharepointfieldnotes.blogspot.com/2011/07/sharepoint-2010-search-scopes-explained.html
http://blog.falchionconsulting.com/

SP2010 Enterprise Search Crawl log error

Problem: Using enterprise search after a clean SP2010 install using autoSPInstaller, I preform my initial full crawl of the content source and receive the following error in the crawl log "The account password was not specified. Specify the password."

Resolution: Edit the Search Service Application to use the appropriate password.

FAST Search Overview

Overview:  Researching FAST for SharePoint 2010 Enterprise, I am logging my findings to provide a basic overview for using FAST with SP2010.

FAST for SP2010 OOTB Search Results

Tips:

• Install FAST on it's own Hardware x64 Windows 2008 R2, need 4GB RAM and 4CPU's min should use 16GB RAM and 8 CPU's;
• Min disk 50GB, 1TB with multiple spindles recommended;
• Install FAST on seperate Hw (not on SP2010 or DC machines);
• Neeed Internet access port 80 and IP adress should be static;
• FAST need SP2010 Enterprise Edition;
• SP2010 search must be installed it is still used for the People search results;
• FAST uses a db for configuration of FAST so back-off your existing SQL Server farm used by SP2010;
• The indexed data out of the content databases is stored in FAST indexes on the file system not in the SQL Server db;

Technical Overview:
Main components are: Crawler (examines the data to be made searchable), Web Analyser and Indexer (performs the search queries).


Crawler servers maximum of 30 million docs per node (server), crawler produces 2 databases. Sizing is roughly 3GB per million docs in the log file and 4GB per million docs in content.

Web Analyser servers has a maximum recommendation of 30 million per node. Storage of 5GB per million docs.

Indexer servers the queries back and recommend staying under 15 million docs per node. Need roughly 120GB/million documents crawled. Due to the high IOPS required by the index servers it best to keep these as physical servers. VMWare experts and tune VM’s for high IOPS but a specialist is highly recommended.

Licensing is done per server or VM and is roughly 14K/server/VM (Not verified)

People search is still surfaced using SP2010 Search so don’t remove it from the SP2010 farm. I believe you can use FAST to do people search also but it doesn’t support phonetic searches so probably a good idea to leave people search with SharePoint’s enterprise search.

Virtualization: Don’t virtualize the SQL Server, use a SAN. Don’t virtualize the Indexer servers.

Advantages of FAST:

• Higher performance and scalability

• Facetted searches are provided OOTB

• Improved meta data extractors

• Previews and thumb previews for PowerPoint, word and pdf documents

• Federation has exact number counts (I love this)

• Programmatic hook into the content publishing pipeline

Setting up FAST for SP2010 on a devloper VM  This is based on articles on technet about FAST for SP2010 and is not my original work, it has been addapted to my specific requiremnt for FAST on a development VM.
References:

Simple Logical Architecture http://www.social-point.com/sharepoint-2010-search-and-fast-search

FAST for SharePoint 2010 Troubleshooting

1.> Check FAST servers are running.  PS>nctrl status
2.> Ensure OSearch14 Windows service is running as 1 of the 2 FAST installed specified accounts 

3.> Check the Certificate Connection using PS (SharePoint)> Ping-SPEnterpriseSearchContentService -HostName FS4SP1.demo.dev:13391

PS> Restart-Service -Name "OSearch14"

4.> Authorisation crawl errors.  Check the account that is performing the crawl has permissions

• FAST logs are found in <>\FastSearch\var\log
• \syslog\all.log is the best log for fault finding.
• \querylogs shows all the logs for queries
• Use Perfmon to monitor FAST, fast has it's own set of counters. 

Updated 2017-04-05:  SP 2013 and SP2016 allows breaking into the Search pipeline on the crawl  using CEWS (Content Enrichment Web Service).  Office 365/SharePoint 2013 does not support CEWS.  Also watch out for how Deletes go thru the Web Service!  Also CEWS only has one registration of a CEWS Web Service allowed per query pipeline so look at the Microsoft CEWS toolkit if you need more than 1 web service on the crawl).  

http://www.netwoven.com/2014/07/using-multiple-endpoints-as-a-content-enrichment-web-service-in-sharepoint-2013-search/

SharePoint 2010 Search

SharePoint 2010 Edition comparison

Problem: Setup search securely from scratch
Initial Hypothesis: The setup wizard is not precise enough so it is a better idea to setup search using Powershell or even by creating a new Search Service Application using the UI.  The crawl account used by your Search Service Application can be seen under the "Default content access account" property.
When you create the new Search Service application you loose your existing search data such as indexing, scopes and search settings.
Resolution:
Corey Roth has a good post on permissions for the crawl account.

Security Trimming: Search only returns the results the current user has access to in SharePoint.  File servers work the same way.  Any outside content source you crawl using a specified account will not have security permissions and therefore will show all results.