Automating SharePoint Environment Builds using Team City

"TeamCity is a closed source, Java-based build management and continuous integration server from JetBrains" - Wilipedia.

And for SharePoint build automation it is a nice tool.  I have build scripts for various SharePoint environments and these are automated to run daily or on an adhoc basis.  TFS could achieve all the build pieces I have put together for automating my environment creation consisting of:

• Create VM's using PowerCLI on VMware ESXi 5.0;
• Create SQL 2012 databases servers with Always on high availability groups;
• SharePoint 2013 or SharePoint 2010 with FAST using AutoSPInstaller;
• Deploying the SharePoint code and creating the sites, webs, activating the features and deploying web parts. 

Below are the screen shots of Team City.

 

Digital Signatures and Install Software gotcha

Problem: In automating SQL Server and SharePoint images, the actual installation is taking a long time on my managed environment whereas my developer laptop is fast.  All installations are done without Internet access (offline).

I have a dev environment build on my laptop that runs SSD and I run 3 VM using VMware workstation 9 (all use Windows 2008 R2 SP1).  I create an 1) AD with 1GB or RAM and 1 CPU 2) SQL 20012 with 10GB RAM and 4 CPU's 3) SP2010 CU Aug 2012 10GB 4 CPU's.  All the installation is automated using slip streamed images.

So for simplicity on the CI environment I will explain a simplified comparable setup. I have 3 machines with the same roles however the SQL 2012 and SP2010 install take considerably longer.  The CI environment is on ESX (Cisco blades & chassis, and Violin (SSD) storage.  The CPU/compute is connected to the storage via SAS/Fibre channel made no difference either).  I have summarised the results below:

                                                                 SQL2012 (duration)      SP2010 (duration)
Laptop(VMworkstation Workstation)           15 min                              16 min
CI (ESX)                                                        22 min                              92 min 

Finding: My hardcore/good ESX infrastructure is taking 9 minutes longer to install SQL Server 2012 on beter hardware and an amazing 70 minutes longer to install SP2010.

Update 21 Feb 2013: Don't use PowerShell 3 with AutoSPInstaller (including using the version switch i.e. -version 2), it doesn't work and even changing AutoSPInstallers internal web call fail.  It can be made to work with the version 2 switch but it isn't worth the effort.

Initial Hypothesis:
After many many hours between service providers managing the infrastructure, it was not hardware or ESX configuration/setup.  However if the network card on the VM is disabled, the performance change improves to:

                                                          SQL2012 (duration)            SP2010 (duration)
CI (ESX)                                                     13 min                       5 min and 5 seconds

Pretty hefty improvement.  Using netstat is looks like there are requests to the Internet.  After adding Wireshark to monitor all traffic.  I can see requests being sent to crl.mirosoft.com (certificate revocation lists) and ctldl.windowsupdate.com

Issue shown in Wireshark

Issue Shown in Fiddler

This is the 1st time I have seen this issue in a clients production environment.  If the WFEs/SP servers have internet access (less preferable) or the servers don't have access the install work in a timely fashion.  The symtoms of the issue are when the WFE's/SP Servers don't have internet access but think they do.  All the binaries are digitally signed and the install will try validate the signatures despite this being an offline install.

I confirmed the problem being how the networking is setup.  My issue shows up on the VM NIC adapter, Originally the IPv4 Connectivity has a status of "No Internet Access", once I ping google I get a reply and the status changes to "Internet".  I can ping google but not browse to it.

Resolution:  The problem is that executable code is digitally signed.  This is good, all code should be digitally signed so it can be authenticated.  However in this situation a lot of requests are being sent out from the VM as the install tries to verify all the SharePoint complied code.  The install on the local VM acts as if there is an Internet connection (which there is not).

It takes unique networking to get into this issue and SP/any digitally signed code will check the digital certs.

There are a few fixes such as:
1.> Allowing the servers to get out to the Internet, so open the firewall or set a proxy on the local VM.
2.> Add host entries to the cert fails immediately but will continue installing (This is not working for me).
3.> Make the following registry change:

set-ItemProperty -path "HKCU:\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944

set-ItemProperty -path "REGISTRY::\HKEY_USERS\.Default\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944
get-ChildItem REGISTRY::HKEY_USERS | foreach-object {set-ItemProperty -ErrorAction silentlycontinue -path ($_.Name + "\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing") -name State -value 146944}


More Information:

Certificates for installing sofware is cause slow install:
http://joelblogs.co.uk/2011/09/20/certificate-revocation-list-check-and-sharepoint-2010-without-an-internet-connection/

http://ddkonline.blogspot.co.uk/2010/05/fix-sharepoint-very-slow-to-start-after.html

If you want to verify if a machine is having problems with a poarticlar process Process Explorer (Usefule if a machine has high memory, CPU or IO issues)

PowerShell to Create User Accounts for SP Install

Problem:  I keep building this script to setup accounts with permissions to put a SharePoint farm using AutoSPInstaller.  I have decided to post so I don't have to go look for this each time.  My list is based on the accounts for AutoSPInstaller recommended install accounts per Tobias Lekman's blog post series.

Use Powershell to create the accounts (This script was originally given to me by Mark Slavik)

Download the PS file here (rename to be a ps1 file)

Note: ThePowerShell file creates tha accounts in the right groups.  The User Profile Service/Synchronisation Account needs "Replicating Directory Changes" permissions, this can be done in various ways and depends on if the NETBIOS name and domain name match. 

Steps to add "Replicating Directory Permissions" to the User Profile synchronisation account:
1.> Open "Active Directory Users and Computers".  Right click on the domain name in the management console and select "Delegate Control..."

2.> On the "Delegation Control Wizard" click "Next" > On the "Users or Groups" screen used to delegate control.  Click "Add" and add your User Profile Sync account.  Click "Next".

3.> On the "Tasks to Delegate" screen select the option "Create a custom task to delegate" > "Next".

4.> On the "Active Directory Object Type" screen accept the default settings and click "Next".

5.> On the "Permissions" screen check the box to allow "Replicate Directory Changes" and Click "Next".  The last screen is for review and select "Finish".

Check your account has permissions using PowerShell.  I needed to amend Tobias Lekman's script
http://lekman.codeplex.com/releases/view/65930  to make it work for me; this is 99% Tobias's work.  I also check if the account is a domain administrator as if they are you won't need to add the special permission (not recommended).  Your other option is to make the User Profile Synchronisation account a local administrator on the VM where the User Profile Service is running.

Alternatively check the permissions thru the AD User and groups UI:

Summary: Add 10 (or as many as you decide to use) accounts.  SP_Install needs administrator domains permissions all the others just need domain user account access.  The SP_Install account needs SQL roles DBCREATOR and SECURITYADMIN. Lastly, ensure the SP_ProfileSync account has "Replicating Directory Changes" permissions.  These permissions are implicit if the SP_ProfileSync account is a local admin or part of the domains administrators group.

Tip: The Execute method of job definition Microsoft.SharePoint.Diagnostics.SPDiagnosticsMetricsProvider (ID ..) threw an exception. More information is included below. An update conflict has occurred, and you must re-try this action. The object SPWebService was updated by demo\sp_farm, in the OWSTIMER (8140) process, on machine... 

 

Prerequisites for SP2013 Offline Install

RE: http://autospsourcebuilder.codeplex.com/

Background: This is a codeplex project that will slipstream SharePoint 2013 & SP2010.  It flattens out the binary install files, gets the latest CU, language packs & pre-requisites.  This will then allow you to install SharePoint to the latest build pref using AutoSPInstaller.

Doing an Offline install is useful as most production environments don't allow internet access to download the pre-requisites.  It's also useful as you don't need to wait for the download on each server.  In my current scenario it's useful as I am building over 100 VM's in My DTAP environment and a lot of these are full CI daily rebuilds.

Overview:  I have been doing SP2013 installs using AutoSPInstaller and I have put this post together to help people use Bian Lalancette's (@brianlala) AutoSPInstaller tool. Pretty obvious but AutoSPInstaller is a great tool so use it or Gary Lapointe install scripts to automate SharePoint installations.  I have been using the tool to install 2013 and this post shows all the pre-requisites to SP2013 RTM for an offline install.

Findings:  There are 13 files I am downloading to install SP2013 without an Internet connection.  Brian Lalancette has a codeplex project that you should checkout, I hadn't seen it until today.
It helps build the slipstreamed images for SharePoint both 2010 and 2013 so you have a full patched version of binaries to run offline installs.
Add the prerequisites to "C:\Software\SP\SharePoint\PrerequisiteInstallerFiles"

List of Files to download for SP2013 RTM prerequisites:

1. http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix354400/7600/free/427087_intl_x64_zip.exe
2. http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix402568/7600/free/447698_intl_x64_zip.exe
3. http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix368051/7600/free/433385_intl_x64_zip.exe
4. http://download.microsoft.com/download/9/1/3/9138773A-505D-43E2-AC08-9A77E1E0490B/1033/x64/sqlncli.msi (http://go.microsoft.com/fwlink/?LinkId=228086)
5. http://download.microsoft.com/download/E/7/6/E76850B8-DA6E-4FF5-8CCE-A24FC513FD16/Windows6.1-KB2506143-x64.msu (http://go.microsoft.com/fwlink/?LinkId=233187)
6. http://download.microsoft.com/download/b/a/4/ba4a7e71-2906-4b2d-a0e1-80cf16844f5f/dotnetfx45_full_x86_x64.exe (http://go.microsoft.com/fwlink/?LinkId=225702)
7. http://download.microsoft.com/download/D/7/2/D72FD747-69B6-40B7-875B-C2B40A6B2BDD/Windows6.1-KB974405-x64.msu (http://go.microsoft.com/fwlink/?LinkId=226830)
8. http://download.microsoft.com/download/0/1/D/01D06854-CA0C-46F1-ADBA-EBF86010DCC6/rtm/MicrosoftIdentityExtensions-64.msi (http://go.microsoft.com/fwlink/?LinkId=252368)
9. http://download.microsoft.com/download/E/0/0/E0060D8F-2354-4871-9596-DC78538799CC/Synchronization.msi (http://go.microsoft.com/fwlink/?LinkId=224449)
10. http://download.microsoft.com/download/A/6/7/A678AB47-496B-4907-B3D4-0A2D280A13C0/WindowsServerAppFabricSetup_x64.exe (http://go.microsoft.com/fwlink/?LinkId=235496)
11. http://download.microsoft.com/download/7/B/5/7B51D8D1-20FD-4BF0-87C7-4714F5A1C313/AppFabric1.1-RTM-KB2671763-x64-ENU.exe (http://go.microsoft.com/fwlink/?LinkId=251471)
12. http://download.microsoft.com/download/9/1/D/91DA8796-BE1D-46AF-8489-663AB7811517/setup_msipc_x64.msi (http://go.microsoft.com/fwlink/?LinkId=219568)
13. http://download.microsoft.com/download/8/F/9/8F93DBBD-896B-4760-AC81-646F61363A6D/WcfDataServices.exe (http://go.microsoft.com/fwlink/?LinkId=247921)

Files I add to my SP2013 binaries for an Offline install

Rather use the new tool "AutoSourceCodeBuilder"

My slipstreamed SharePoint 2013 binaries

 Update 07 November 2012: For my Windows 2008 R2 SP1 OS installation, the WCFDataServices.exe prerequisite will look for trusted certificates on the Server, if it can't find them it will try get them from the Internet.  I downloaded "Rootsupd.exe" (I'm not sure about support for "Update for Root Certificates For Windows XP [April 2012] (KB931125)" for my Windows 2008R2 SP1OS install from Microsoft).
The PowerShell to install the certificates is (this can be incorprated into your autobuild scripts if you need it or just run it (be warned there is not feedback on the install)):
PS> & "c:\rootsupd.exe"

Update 15 July 2013: Below are the latest pre-requisites I downloaded for SP2013 using AutoSPSourceBuilder.

Note:  I like to have a large disk for my c rive for the system files and also the Windows "page file", when calculating my disk space for my c drive assuming my page file is going to my c drive, I add 3 times the possible memort upgrade per machine.  So it is common to start with 32GB on a SQL box, when their is a bottleneck, it is often SQL and memory is key to SQL performance.  So my page file portion of my c drive would by on 64 GB multiplied by 3.  So My c drive in this case would be 64GM x3 plus program files, so at least 250 GB.  I'd probably go for 300GB.  this is my preference and I'm sure I'm going overkill but in controlled wel managed environments this will aloow for growth.

More Info:
WCFDataService install issue: http://schoennie.blogspot.co.uk/2012/10/installing-sharepoint-2013-preview.html

Office Web Apps on a DC

Problem: Office Web Apps on a developer machine which has it's own domain controller.  Can view word documents in the browser, editing of docx files stangely works.

http://technet.microsoft.com/en-us/library/ff431687.aspx

FAST previews/thumbails need Office Web Apps running.

Summary: Install Office Web Apps pre AutoSPInstalller is run.  Then activate in the autospinstaller.  Otherwise install the office web apps but only do the config of the web apps, don't run the SP config wizard.

SP2010 install using AutoSPInstaller from PowerGUI

Problem: I want to install SharePoint 2010 using Brian Lala's AutoSPInstaller codeplex project however, I want to be able to see values and debug using PowerShell

Resolution:
Ensure AutoSPInstaller is setup correctly and PowerGUI is installed.  Correct setup is shown below.  Ensure that the xml file is named correctly.

Tip: Windows has a built in graphical debuggger for PowerShell called ISE Editor
http://blogs.msdn.com/b/powershell/archive/2009/01/19/debugging-powershell-script-using-the-ise-editor.aspx

PS> c:\SP2010\AutoSPInstaller\AutoSPInstallerMain.ps1 c:\SP2010\AutoSPInstaller\AutoSPInstallerInput-WIN-2C5PI8TSRPA.xml

Press F5 or the arrow to debug.  I have all the .ps1 files in the PowerGUI editor and add break points. 

Note.  If you hit a break point and press F5 once you are debugging, PowerGUI will not stop at the next break point.  F10 will take you to your next breakpoint or where the next error is trapped.

SP2010 CU April 2011

Overview: Cumulative Update for SharePoint 2010 is the latest CU for SP2010 Server. The CU can be applied to an existing far or added be part of the install using AutoSPInstaller for new installations.

SharePoint Server 2010 cumulative update package  can be found here.  http://support.microsoft.com/default.aspx?scid=kb;EN-US;2512800

Installation:  You need to request the file from the link list above.  Download the file to your local machines (It's about 600MB).
For an update simple extract the from the file 432209_intl_x64_zip.exe using the cmd>432209_intl_x64_zip.exe /extract:.\

 If you want to include the CU in a new install using AutoSPInstaller, you will need to extract the .msp files from office2010-kb2512800-fullfile-x64-glb.exe /extract:.\

And place the msp files inside the approriate directory in you AutoSPInstaller directory.



Installing SharePoint using a dedicated Installation account

Problem: Best practice is to install SharePoint servers using a dedicated install account.  This account is not the farm account.  Once the installation is complete, the installation account should be disabled.  You need to install in the role of the admin account to install correctly. 

Note:  If your install and farm account are the same as is often the case, this post does not apply to you.

Initial Hypothesis:  It is only worth following this post if you are using at least 5 or more accounts for your farm install.  The idea is that the installation account is disabled after installation.

Resolution:  The farm account needs the 2 SQL Server security roles namely: dbcreator and securityadmin.  The farm account still needs local admin permission rights on each Web Front End (WFE) server.  The Installation account does not need any SQL Server permissions.

Using the codeplex AutoSPinstaller.  Launch the installer using a cmd prompt using the runs cmd to run in the farm domain admin priviledge.
cmd> runas /user:demo\farm_admin %windir%\system32\cmd.exe
cmd> D:\SP2010\Script>Launch.bat


You can also do the runas shotcut to change the account installing SharePoint.
Update 6 Dec 2010 - Hold down the shift key and right click the cmd prompt program menu, you are given the "run as .." menu option.

Update: 22 June 2011 - the current version of AutoSPInstaller is 2.5, additionallly the codeplex project is called AutoSPIntaller not SPAutoInstaller as previously named.

AutoSPInstaller - Step-by-step guide: http://blog.lekman.com/2010/11/automated-sharepoint-2010-installations.html

Installing Sharepoint 2010 options & Basic SP2010 manual installation tips

You have 4 options for installing SharePoint farms:

1. Manually sun the setup and follow the installation wizard (this is discussed below);
2. Deploy SharePoint 2010 via a slipstream install, this was my prefered method for MOSS.  I ran the install from a batch file that got it's configuration from an xml file;
3. PSConfig installation (sic); or
4. Use PowerShell to Install SharePoint. and technet scripted deployment

 Summary: For environments such as live the PowerShell/Slipstreamed options are best as they allow for recreation and input is always identical.  Manual install is fine for development servers however their is no advantage except for a lower learning curve for the IT admin.

Post below is a Manual Installation:
SP2010 install video
Install the pre-requisites

• Prerequisits will install roles and software you need internet access on the server to fetch the prerequisits software (this can be put on the server to stop the machine going to the Internet).
• Preferably have seperate instance of SQL 2008 R2 but for dev/demo machines. If 1 machine rather setup SQL devleoper or a instance (I dislike using SQL express).

Setup / SP 2010 install tips

• Install "Server farm" option not standalone
• "Complete" installs all component prefered option
• Connect to a new farm
• Database server name us name rather than IP (incase it changes)
• DB account (must already exist in AD)
• Passphrase used to connect new servers to this server farm (remeber/keep it)
• Kerbros - if your network supports it but use NTLM if you aren't sure.
• Wizard - follow screens, services can be heavy so add them when you need them, however for demo I select all services and create a new site collection - a good options is to use the Team Site Template.
• Need 3 accounts for min Best practices: 1) Managed Service account (domain user account) that SQL Server runs in, 2) Managed Service Account (domain user account) all services will be installed on this account (MS suggests using a seperate managed account for each service) on small farm s/dev I use 1 account,  and 3) Farm install account (domain account) this needs to be a local admin on each SP2010 server and have creator & dbsecurity accouts on SQL.
• 5 Accounts is a better option excluding the SQL services account namely:

1. SP-Install - domain account with admin local rights on each WFE also need SQL dbcreator and securityadmin roles (used to login and install binaries, use this account for add new servers to the farm),

2. SP-Farm - domain account no permissions, will be the account to run timer job and other key roles,
3. SP-Web-App-Pool - Content Web app account - Domain account only,
4. SP-Services - Install all services to use the same domain account, this can be seperate for each services but for easy of setup and mainentance use 1 account.  Exception is the User Profiles service, setup seperately using Spence Harbors post as the user domain account needs unique security, and
5. SP-Crawl - Used to crawl SP content.

Additional Info on accounts:

1. SQL Server needs to run as a windows service, you need an account, I would use a managed account in AD with no permissions called SP2010-SQLService.
2. Farm Installation account, you need to create a domain user account in AD, give the account local admin access to each SP2010 machine.  Call it SP2010-Admin.
3. SP2010 Service account/s, you need to create a managed service account with zero permissions in AD.  You can use 1 account or create a seperate account for each service (MS Best Practice).  I call my 1 account SP2010-Services. 

Use slipstreaming for SharePoint it's faster and consistant.
Use:

1. Windows 2008 R2 x64
2. SQL 2008 x64
3. On HyperV/VMWare except the db which should be a seperate physical machine/SAN

Update 08 November 2010:  Notes on deploying a 3 server farm consisting of 2 WFE's that are NLB using Windows NLB.  Installation done using AutoSPInstaller. 
Installation Notes for a 3 server NLB SharePoint 2010 farm

Update 10 November 2010: SharePoint install account - Todd Klindt.
Update 11 May 2011: SharePoint 2010 database management article