Adding Additional Search Crawl database to a SP2013 Search farm

Problem: By adhereing to Microsoft recommendations for search farms, a threshold that is passed fairly quickly is that the Search Crawl database should not exceed 20 million items per database.  I have a 12 server search farm with four partitions (recommendation 10 million items per index partion).  I need to add a second Crawler DB.

Initial Hypothesis: Figure out your current crawl databases.

Add additional Search Crawl databases

 Review the existing crawl search databases

I proved this on a test system by resetting my index and recrawling, the results after adding 3 content db's are shown below.

Resolution:  Add more crawl database to the search service application.

$SSA = Get-SPEnterpriseSearchServiceApplication$searchCrawlDBName = "SP_Search_CrawlStore2"
$searchCrawlDBServer = "SP2013-SQL3" # SQL Aliase could be the Conn str
$crawlDatabase = New-SPEnterpriseSearchCrawlDatabase -SearchApplication $SSA -DatabaseName $searchCrawlDBName -DatabaseServer $searchCrawlDBServer
$crawlStoresManager = new-Object Microsoft.Office.Server.Search.Administration.CrawlStorePartitionManager($SSA)
$crawlStoresManager.BeginCrawlStoreRebalancing()

Powershell to see what is going on:

cls
$SSA = Get-SPEnterpriseSearchServiceApplication
$crawlStoresManager = new-Object Microsoft.Office.Server.Search.Administration.CrawlStorePartitionManager($SSA)
Write-Host "CrawlStoresAreUnbalanced:" $crawlStoresManager.CrawlStoresAreUnbalanced()
Write-Host "CrawlStoreImbalanceThreshold:" $ssa.GetProperty("CrawlStoreImbalanceThreshold")
Write-Host "CrawlStoresAreUnbalanced:" $crawlStoresManager.CrawlStoresAreUnbalanced()
Write-Host "CrawlPartitionSplitThreshold:" $ssa.GetProperty("CrawlPartitionSplitThreshold")
$crawlLog = New-Object Microsoft.Office.Server.Search.Administration.CrawlLog $SSA
$dbInfo= $crawlLog.GetCrawlDatabaseInfo()
Write-Host "Number of Crawl Databases:"
$dbInfo.Count$dbInfo.Values

More Info:
http://blogs.msdn.com/b/sharepoint_strategery/archive/2013/01/28/powershell-to-rebalance-crawl-store-dbs-in-sp2013.aspx

Remote Debugging SharePoint 2013

Overview:  Object reference not set to an instance of an object is all the information I have in my customers ULS logs.  I need to debug an application in production that does not contain Visual Studio 2012.

My Steps:

• Ran the Remote Debugguer (msvsmon.exe) on 1 of the production servers

• Added a host file to point to my production server "SP-WFE1" (optional depending on where the web requests originate from)
• Opened Visual Studio 2012 on a VM containing the projected (hooked up to TFS 2012) and opened the solution (.sln).
• Select Tools > Attach to Process

..... Finish when time permits


More Info:
http://msdn.microsoft.com/en-us/library/ff649389.aspx
http://www.dotnetmafia.com/blogs/dotnettipoftheday/archive/2008/03/05/how-to-remote-debugging-a-web-application.aspx
 

SharePoint Online Random Tips

1.> SPO and PowerShell:
To connect and use SharePoint online you cannot have a small business plan (P).  The Medium and enterprise Office 365 plans allow the SPO PowerShell scripts to run (very few so far).
Steps to use PS against SPO:
1.1.> Download and install the SharePoint Online Management Plugin

 1.2.> Add the snap-in/module to your PowerShell (or use the console with the snap-in already loaded)

 1.3.> Connect to your SharePoint site and validate it works.

=============================================================

2.> OOTB SPO Workflows:
P Plan only has three state workflow
E Plan has 5 basic workflows: signature, 3 state, approval, ..

Sandbox does not support Visual Studio based (code) workflows but you can create declaritive workflows via SPD.

SharePoint 365 Workflow Creation Process:

• Prototype in Visio 2010 by using the SharePoint specific workflow, this can be demonstrated with the end user as they understand Visio workflow diagrams to explain there business process.
• Export Visio 2010 diagram to SPD
• Implement the workflow using SPD can integrate InfoPath forms
• Custom actions are created using VS and can be deployed, the custom actions can be integrated into the SPD created workflow.

=============================================================

DevOps and SharePoint

Components of DevOps

• IT Automation
• Agile Development
• Operation and dev teams working together
• Service vitalisation
• Continuous releases
• Automated testing (Recorded Web UI & Unit testings)
• Performance

Why DevOps

• Collaboration btwn Dev & Ops teams; reduce unknowns early on, info easily available, better decision making, improved governance.
• Increase apps faster; faster time to market
• Improve App quality; environments available and not tedious to deploy
• Reduce costs; fewer staff required as tasks are automated

DevOps Maturity Model
There are different levels of maturity and your final goal will depend on your company.  Operations departments are very different in a bank as opposed to Facebook.
The 4 main stages to DevOps Maturity are:

1. Early - Manual traditional approach to software delivery.  Little or no automation.
2. Scripted - Moving toward more automation.  Some scripting is used to assist the tracked recorded deployment process thru DTAP environments.
3. Automation - Reusable common automation approach to releasing applications.  Workflow/orchestration using the automation capabilities used over all the DTAP environments.
4. Continuous Delivery - end-to-end application delivery process from dev to production environments.  Probably includes Continuous Integration, nightly builds

Presentation at SharePoint Saturday 2013 on of Infrastructure as Code (IaC)
http://blog.sharepointsite.co.uk/2013/11/iac-presentation-for-sharepoint.html

Audit and documenting a SharePoint on prem farm

Overview:  I have had and done several SharePoint infrastructure audits over the past few years.  These can vary from running through checklists of best practices to see what the customer setup has and finding out why they may of chosen to diverge. 

There is a lot of good tooling and checklists to help you audit your farm such as the built in Health analyser, central admin, PowerShell scripts and a great tool to help document your farm SPDocKitSetup from Acceleratio.

Tooling:

• PowerShell
• Health analyser
• Central Admin
• ULS logs and event logs
• IIS (IIS RealTime Log Viewer)
• Fiddler
• SQL Management Studio, SQL profiler
• Tool to audit and document your SharePoint farm: SPDocKitSetup
• Third party tools like: DocAve and Metalogix (Ideara had a decent tool previously) can help audit and document your system.

The basic steps are:

1. Document the farms topology;
2. Verify versions and components installed;
3. Compare MS recommendations to the farm settings, provide findings and allow the customer time to explain why these setting differ; and
4. Monitoring, troubleshooting and performance.

Below are a few items to review:

SharePoint:
SharePoint Logs - I put these into my d drive
Multiple install accounts
List all customisations especially wsp and custom code
External Access - DMZ, network,

IIS:
Change the IIS default log location to a separate disk
Reclaim old IIS logs (Backup and remove from WFE's)
IIS RealTime Log Viewer - good tool for reading IIS logs

SQL basic checks:
I/O, CPU, Memory on the base database machine/s
MaxDop = 1
Initital mdf sizing and ldf sizing
ldf on fast disk, mdf of content db's on slowest disk (or just put it all on fast disk)
Backups

Maybe Reminders:
Remove all certificate files that are not needed (*.cer & *.pfx)
Ensure local account user name/passwords are changed & secure (I have a local admin account on my Windows template used to create all my VM's ensure this account is disabled or secured).

More Info:
http://nikpatel.net/2012/03/11/checklist-for-designing-and-implementing-sharepoint-2010-extranets-things-to-consider/
https://habaneroconsulting.com/insights/Do-you-need-a-SharePoint-infrastructure-audit
http://blog.muhimbi.com/2009/05/managing-sharepoints-audit.html
http://www.portalint.com/thoughts-on-sharepoint-audit/

Search stops working and CA Search screens error

Problem: In my redundant Search farm, search falls over.  It was working, nothing appears to of changed but it suddenly stops working.  This problem has caused itself to surface in several places:
1.> In  CA going to "Search Administration" displays the following error message: Search Application Topology - Unable to retrieve topology component health states. This may be because the admin component is not up and running.

2.> Query and crawl stopped working.
3.> Using PowerShell I can't get the status of the Search Service Application
PS> $srchSSA = Get-SPEnterpriseSearchServiceApplication
PS> Get-SPEnterpriseSearchStatus -SearchApplication $srchSSA
Error: Get-SPEnterpriseSearchStatus : Failed to connect to system manager. SystemManagerLocations: net.tcp://sp2013-srch2/CD8E71/AdminComponent2/Management

4.> In the "Search Administration" page within CA, if I click "Content Sources" I get the error message: Sorry, something went wrong
The search application 'ef5552-7c93-4555-89ed-cd8f1555a96b' on server SP2013-SRCH2 did not finish loading. View the event logs on the affected server for more information.
I used PowerShell to get the ULS logs for the correlation Id returned on the screen via the CA error message
PS> Merge-SPLogFile -Path "d:\error.log" -Correlation "ef109872-7c93-4e6c-89ed-cd8f14bda96b" Out shows..
Logging Correlation Data       Medium Name=Request (GET:http://sp2013-app1:2013/_admin/search/listcontentsources.aspx?appid=b555d269%255577%2D430d%2D80aa%2D30d55556dc57)
Authentication Authorization   Medium Non-OAuth request. IsAuthenticated=True, UserIdentityName=, ClaimsCount=0
Logging Correlation Data       Medium Site=/ 05555e9c-5555-555a-69e2-3f65555be9f4
Topology  Medium WcfSendRequest: RemoteAddress: 'https://sp2013-srch2:32844/8c2468a555594301abf555ac41a555b0/SearchAdmin.svc' Channel: 'Microsoft.Office.Server.Search.Administration.ISearchApplicationAdminWebService' Action: 'http://tempuri.org/ISearchApplicationAdminWebService/GetVersion' MessageId:
General    Medium Application error when access /_admin/search/listcontentsources.aspx, Error=The search application 'ef155572-7555-4e6c-89ed-cd8f14bda96b' on server SP2013-SRCH2 did not finish loading. View the event logs on the affected server for more information.  Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter) at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Ob General  Medium ...(IMethodCallMessage methodCall, ProxyOperationRuntime operation)  at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)  Exception rethrown at [0]: at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.ErrorHandler(Object sender, EventArgs e) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.OnError(EventArgs e) at System.Web.UI.Page.HandleError(Exception e) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) 
General      Medium ...em.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) 05b5559c-4215-0555-69e2-3f656555e9f4 Runtime   tkau Unexpected System.ServiceModel.FaultException`1[[System.ServiceModel.ExceptionDetail, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]: The search application 'ef109555-7c93-444c-85555-cd8f14b5556b' on server SP2013-SRCH2 did not finish loading. View the event logs on the affected server for more information.   Server stack trace:      at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)     at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) 
Runtime     tkau Unexpected ...s, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)  Exception rethrown at [0]: at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.ErrorHandler(Object sender, EventArgs e) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.OnError(EventArgs e) at System.Web.UI.Page.HandleError(Exception e) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Runtime       tkau Unexpected ...ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously
General    ajlz0 High Getting Error Message for Exception System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: The search application 'ef10-555-a96b' on server SP2013-SRCH2 did not finish loading. View the event logs on the affected server for more information. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is: System.Runtime.InteropServices.COMException: The search application '7c93-555c-89ed-cd8f555da6b' on server SP2013-SRCH2 did not finish loading. View the event logs on the affected server for more information. at icrosoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T]General        ajlz0 High ...   at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(String applicationName)     at Microsoft.Office.Server.Search.Administration.SearchAdminWebServiceApplication.GetVersion()     at SyncInvokeGetVersion(Object , Object[] , Object[] )     at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)     at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMess...) Micro Trace uls4 Medium Micro Trace Tags: 0 nasq,5 agb9s,52 e5mc,21 8nca,0 tkau,0 ajlz0,0 aat87 05b16e9c-4215-00ba-69e2-3f656eabe9f4
Monitoring   b4ly Medium Leaving Monitored Scope (Request (GET:http://sp2013-app1:2013/_admin/search/listcontentsources.aspx?appid=be28d269%255577%2D430d%2D555a%2D30d55556dc57)). Execution Time=99.45349199409
Topology    e5mb Medium WcfReceiveRequest: LocalAddress: 'https://sp2013-srch2.demo.local:32844/8c2468a555943555bf42cac555f05b0/SearchAdmin.svc' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://tempuri.org/ISearchApplicationAdminWebService/GetVersion' MessageId: 'urn:uuid:c2d55565-bb59-4555-b34b-6555ef1d79a5'

I can see my Admin component on SP2013-SRCH2 is not working, so I turned off the machine hoping it would resolve to the other admin component, it did not change and keeps giving me the same log errors.  I turned the Server back on and reviewed the event log on the admin component Server (SP2013-SRCH2).  Following errors occured:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance + Reason: The device is not ready. 
The Execute method of job definition Microsoft.Office.Server.Search.Administration.IndexingScheduleJobDefinition + The search application + on server SP2013-SRCH2 did not finish loading.

Examining the Windows application event logs on SP2013-SRCH2 and I noticed event log errors relating to permissions:
A database error occurred. Source: .Net SqlClient Data Provider Code: 229 occurred 0 time(s) Description:  Error ordinal: 1 Message: The EXECUTE permission was denied on the object 'proc_MSS_GetConfigurationProperty', database 'SP_Search'
Unable to read lease from database - SystemManager, System.Data.SqlClient.SqlException (0x80131904): The EXECUTE permission was denied on the object 'proc_MSS_GetLease', database 'SP_Search', schema 'dbo'.
The event log showed me the account trying to execute these stored procs.  It was my Search service account e.g. demo\sp_searchservice

Initial Hypothesis: It looks like the Admin search service is not starting or failing over.  By tracing the permissions I can see the demo\SP_SearchService account no longer has execute SP permissions on the SP_Search database. 

By opening the SP_Search database and looking at effective permissions I can see the "SPSearchDBadmin" role has "effective" permissions over the failing stored procs (Proc_Mss_GetConfigurationProperty).

If I look at the account calling the Stored Proc (demo\SP_SearchService), I can see it is not assigned to the role.  This examination leads me to my conjecture that the management tool or someone/something on the farm has caused the permissions to be changed

Resolution: Change the permissions on the database.  Give minimal permissions so go to the database and give the service account (demo\sp_SearchService) SPSearchDBAdmin role permissions.

All the issues recorded in the problem statement were working withing 5 minutes on my farm with the issue.

Note: I have a UAT environment in exact sync with my PR environment; UAT has the correct permissions in place already.

Fixing SSRS on SP2013

Error: "For more information about this error navigate to the report server on the local server machine, or enable remote errors".  When opening a report in SharePoint I get the error shown above.
Perform these 2 steps to view the error:

1.> In IE, navigate to the "Reports Library", click "Site Settings" click "Reporting Services Site Settings".  Select "Enable remote errors in local mode".
2.> Open CA > "Manage service applications" > Select the "SSRS" service application > System Settings > "Enable Remote Errors".
 TBC