DTAP for Office 365

Overview: Someone has asked me about DTAP environment's for Office 365 after reading my last post on DTAP for SharePoint. I should only comment on SharePoint Online's DTAP setup but the simple answer for Office 365 is to create a separate Office 365 account for each of your DTAP environments. So I would have at least 2 or 3 environments hosted on Office 365. In order of priority: Production, UAT, Pre-production, Testing. Development is done locally so CI and development machines are on-premise. SharePoint Online should use DTAP roughly as outline below.

SharePoint Online DTAP:
SharePoint 365 development should be done using a local development solutions using "SharePoint Sandbox Solutions" as this is as close to SP365 as you can get on a development environment. You can use a solution validator (as provided by multiple vendors & MS) or write your own to ensure your wsp deployment files meet all the additional SP365 requirements.
My Order would be:

1. Development Environment - local Sp2010 dev machine where all code is deployed via wsps as a SharePoint Sandbox solution. On-premise hosted.
2. Continuous Integration Environment - perform daily builds of TFS controlled code, TFS code must comply to the custom solution validator that will verify the code will run on SP Online. Candidate for removal as value can be fairly low in smaller deployments. On-premise hosted.
3. UAT, testing and pre-production - for a small farm I would make this the environment as this will be identical to the setup of production, larger implementations could have 3 separate environments. This machine. Office 365 hosted.
4. Production - only perform solution deployments once it has been tested on the pre-production environment.

Summary: Just because you are using the cloud, you should still follow good change control practices. You need to have more than a developer doing his best guess while deploying directly onto your SharePoint online production environment.

Additioanl Info:
Alan Richard's article on Lync 2010 via Office online for messaging, voice and video calls and meeting conferences

DTAP for SharePoint

Overview: Development, Test, Acceptance and Production (DTAP).

To implement SharePoint solutions it is a good idea to have DTAP environments. DTAP can be as simple as 4 environments but I would recommend 5 or more separate environments for SharePoint. 
Development
The base development machines can be standalone VM's with self contained SQL databases, AD, FAST, and SharePoint.  Additional software would include: InfoPath, SharePoint designer, U2U, Fiddler, firefox, IE8 and more.  You should have TFS or a source control repository for all developers to check compiling code into. 

Continuous Integration (CI)

This box can range from very simple to a complex complete tear down rebuild on a daily basis with unit test.  At a minimum all code should be deployed daily.  All changes to CI and subsequent environments should be deployed via wsp's and PowerShell scripts.  Sometimes it is time efficient to write administration manual steps, this should be avoided where possible and if it is absolutely required, the documentation must be explicit and testable.

Testing
This environment is a build release, it is stable so testers can perform testing against a specific build.  Bugs should be documented (preferable in tied to source control changes).

Acceptance
End users should test the system pre go-live in this environment.  This phase can also be split into 2 environments namely: Acceptance and pre-production environments.  Used for user acceptance testing.

Production
This live system is only changed once the changes have gone thru change request management and have been deployed and tested on all the environments.  Pre-production should be as close a copy of the production environment as possible.  Production and pre-production must be kept as close as possible throughout the life time of the SharePoint farm.

Summary:

• Never do changes directly on the production environment. 
• Build a formalised change control process for SharePoint. 
• Minimum of 3 environments for small farms is my base guideline. 
• Environment changes should be done using wsp's and PowerShell not the UI where possible.

Powershell Setup for SharePoint Developers

Overview:  PowerGUI (from Quest Software) provides a PowerShell(PS) GUI to perform actions with minimal customisation you get Powershell with Intellesense and colour coding that is SharePoint aware.  Additionally you can plug in a Visual Studio (VS) plug-in (VSIX) that allows you to have PowerGUI functionality directly within you VS IDE.

1.> Install PowerGUI
1.1.> Download PowerGUI at http://www.powergui.org/ I installed version 2.4.0
1.2.> Install the PowerGUI msi on your development server

2.> Configure PowerGUI
2.1.> Open the PowerGUI application
2.2.> Add the Microsoft.SharePoint.PowerShell module as shown below:

3.> Write a PS script
3.1.> Open the "PowerGUI Script Editor" by clicking Tools > PowerGUI Script Editor
3.2.> Ensure your intellesense is working by typing "Get-SP", you will see the intellisense pick up the GET-SPSite cmdlet.

3.3> Add the following command and debug with a break point to see PowerGUI in action.  This will get all the Content Types associated to the RootWeb of a specific Site Collection.  Note the variables windows on the right hand side of the PowerGUI interfaces allows you to examine each of the variable objects declared in the script.

It's not full intellesense but it is a huge help to get you most of the way to writing custom PS.

4.> Add PowerGUI Visual Studio 2010
4.1.> Add the "PowerGUI VSX" through extension manager (VSIX) (Tools > Extension Manager > Online Gallery > Search for .. PowerGUI > Download.

4.2.> VS2010 will reboot, ensure VS2010 is open, click "View" > "PowerGUI Console".  You can use the console to write PS.
4.3.>  The VSIX also provides an SPI for adding PS Scripts as shown below:

Error msg:
PS E:\Windows\system32> Add-PSSnapin -Name Microsoft.SharePoint.PowerShell
Add-PSSnapin : The Windows PowerShell snap-in 'Microsoft.SharePoint.PowerShell' is not installed on this machine.

References:
PowerGUI and SharePoint 2010

Update: 19/07/2012 - I downloaded the latest version of PowerGui 3.2.0.2237 and tried to call SharePoint scripts arfer adding the SP add-in and got the following error:
Get-SPFarm : Microsoft SharePoint is not supported with version 4.0.30319.237 of the Microsoft .Net RuntimeFix navigate to: C:\Program Files (x86)\PowerGUI\ScriptEditor.exe.config and edit the xml.  I comented out the line telling PowerGui to use .NET 4.0.

FAST Search Overview

Overview:  Researching FAST for SharePoint 2010 Enterprise, I am logging my findings to provide a basic overview for using FAST with SP2010.

FAST for SP2010 OOTB Search Results

Tips:

• Install FAST on it's own Hardware x64 Windows 2008 R2, need 4GB RAM and 4CPU's min should use 16GB RAM and 8 CPU's;
• Min disk 50GB, 1TB with multiple spindles recommended;
• Install FAST on seperate Hw (not on SP2010 or DC machines);
• Neeed Internet access port 80 and IP adress should be static;
• FAST need SP2010 Enterprise Edition;
• SP2010 search must be installed it is still used for the People search results;
• FAST uses a db for configuration of FAST so back-off your existing SQL Server farm used by SP2010;
• The indexed data out of the content databases is stored in FAST indexes on the file system not in the SQL Server db;

Technical Overview:
Main components are: Crawler (examines the data to be made searchable), Web Analyser and Indexer (performs the search queries).


Crawler servers maximum of 30 million docs per node (server), crawler produces 2 databases. Sizing is roughly 3GB per million docs in the log file and 4GB per million docs in content.

Web Analyser servers has a maximum recommendation of 30 million per node. Storage of 5GB per million docs.

Indexer servers the queries back and recommend staying under 15 million docs per node. Need roughly 120GB/million documents crawled. Due to the high IOPS required by the index servers it best to keep these as physical servers. VMWare experts and tune VM’s for high IOPS but a specialist is highly recommended.

Licensing is done per server or VM and is roughly 14K/server/VM (Not verified)

People search is still surfaced using SP2010 Search so don’t remove it from the SP2010 farm. I believe you can use FAST to do people search also but it doesn’t support phonetic searches so probably a good idea to leave people search with SharePoint’s enterprise search.

Virtualization: Don’t virtualize the SQL Server, use a SAN. Don’t virtualize the Indexer servers.

Advantages of FAST:

• Higher performance and scalability

• Facetted searches are provided OOTB

• Improved meta data extractors

• Previews and thumb previews for PowerPoint, word and pdf documents

• Federation has exact number counts (I love this)

• Programmatic hook into the content publishing pipeline

Setting up FAST for SP2010 on a devloper VM  This is based on articles on technet about FAST for SP2010 and is not my original work, it has been addapted to my specific requiremnt for FAST on a development VM.
References:

Simple Logical Architecture http://www.social-point.com/sharepoint-2010-search-and-fast-search

FAST for SharePoint 2010 Troubleshooting

1.> Check FAST servers are running.  PS>nctrl status
2.> Ensure OSearch14 Windows service is running as 1 of the 2 FAST installed specified accounts 

3.> Check the Certificate Connection using PS (SharePoint)> Ping-SPEnterpriseSearchContentService -HostName FS4SP1.demo.dev:13391

PS> Restart-Service -Name "OSearch14"

4.> Authorisation crawl errors.  Check the account that is performing the crawl has permissions

• FAST logs are found in <>\FastSearch\var\log
• \syslog\all.log is the best log for fault finding.
• \querylogs shows all the logs for queries
• Use Perfmon to monitor FAST, fast has it's own set of counters. 

Updated 2017-04-05:  SP 2013 and SP2016 allows breaking into the Search pipeline on the crawl  using CEWS (Content Enrichment Web Service).  Office 365/SharePoint 2013 does not support CEWS.  Also watch out for how Deletes go thru the Web Service!  Also CEWS only has one registration of a CEWS Web Service allowed per query pipeline so look at the Microsoft CEWS toolkit if you need more than 1 web service on the crawl).  

http://www.netwoven.com/2014/07/using-multiple-endpoints-as-a-content-enrichment-web-service-in-sharepoint-2013-search/

Customer Care Accelerator for SharePoint

A few days ago I was in an Architecture Design Session with Microsoft Consulting and they showed me the Customer Care Accelerator (CCA) for Microsoft CRM dynamics 2011.

CCA simply allows you to maintain context between disconnected systems.  If I select a customer using my Windows application that stores my customer orders, I can take the selected CustomerId and use it directly in SharePoint to say retrieve all documents related to a customer. 

In summary, CCA allows me to use data from various desktop applications, such as web forms, crm, SharePoint or websites.  This is pretty useful for sticking together historical disparate systems.

More Info:  http://dynamics-crm.pinpoint.microsoft.com/en-GB/applications/customer-care-accelerator-for-microsoft-dynamics-crm-2011-12884914795
http://community.dynamics.com/product/crm/crmtechnical/b/crmukblog/archive/2011/05/11/getting-started-with-cca-for-crm-2011.aspx
http://blogs.msdn.com/b/ukcrm/archive/2011/05/11/getting-started-with-cca-for-crm-2011.aspx

Scanning, Storage & RBS

Problem:  The client has millions of physical documents, that need to be available via SharePoint, additionally documentation still arrives in physical form and needs to be scanned and classified.

Initial Hypothesis: SP2010 can store documents in the SQL database in blob format however, it's not really made for large blob storage performance wise, additionally SQL storage is expensive (RAID, HA). Remote Blob Storage (RBS) helps with storing blobs but does not get around limitations imposed by MS guidanace.  RBS can reduce storage and improve performace if you data storage involves a lot of large blobs (over 256kb  is a good size).  My rough sums show a huge data requirement so for example 600,000 customers transact with the client.  On average each customer has 3 physical documents a year.  So we are talking 1,8 million scanned documents a year.

Documents need to be scanned in at 300 dpi so they can be printed and stored adequately.  With compression and converting these files into tiff/pdf files we are assuming an average of 1 MB per file. So our storage requirement per year would require 1.8 million scanned documents at 1MB per file meaning my storage on 1,800GB

As we have a restriction of 200GB per content database in SP2010 (threshold that MS will support up to).  So we would require 9 new site collections on a new content db per year to meet this requirement. 

Tip: Also worth considering are thresholds and bounderies provide by the SharePoint team.  Site collections max size is 100GB, this scenario has a caviet in that a single Site Collection using a single document library/site supports up to 1TB in the Content Database.  You can have subsites nested in a site collection but 2000 per view is recommended.  Max of 300 content db's per Web Applicaion.  Max 5000 site collections per content database.

Our storage cost is much higher as our disks are RAID so at a minimum we would use 3 times this in actual physical disk space.  On top of this my indexes will be about 25% of the storage requirement.  So price, performance are getting out of control pretty quickly.

Resolution: Using RBS my estimate on these blobs is will will reduce the content database by 90% however content database size is calcualte including RBS so our storage requirement will be cheaper using RBS that is resilient however the content database sizing will not be reduced by using RBS. 

Updated: 21/07/2011 - RBS sizing Calc

Scanning tips for SP:

• Tiff or pdf are the common base storage file type;
• 300dpi is good print quality most requirements can be lower;
• Black and white is far smaller then grey scale scanning.
• Pdf's if stored correctly can be indexed by the search crawler.

More Info Scanning:
http://www.psigen.com/ - scanning and capture for SP2010.
Capturx from www.adapx.com/sharepoint is a pen that automates data capture on forms.
CoSign does digital signatures and looks to have pretty decent integration with http://www.arx.com/digital-signature/sharepoint
www.kodak.com/go/sharepoint
http://www.goscan.com/connectors-sharepoint.php
http://www.kofax.com/solutions/microsoft.asp

More Info Sizing:
HP Sizer for SP2010

Capacity management for SP2010 - Sw boundries

SQL & Storage requirements

http://www.sharepointjoel.com/Lists/Posts/Post.aspx?ID=332

http://www.azaleos.com/resources/SharePoint-Calculator

http://www.sharepointpromag.com/article/sharepoint-server-2010/sharepoint-2010-deployment-calculator-helps-future-proof-your-migration

http://tihomirignatov.blogspot.com/2010/12/sharepoint-2010-capacity-planning-and.html

Building a customers Taxonomy

Problem: I need to create a taxonmy for my client

Initial Hypothesis: Either start by interviewing stakeholders and building up a taxonomy that is offered as a Service application or
Buy an exisitng industry taxonomy http://blog.wandinc.com/2011/02/list-of-taxonomies-for-sharepoint-2010.html and offer it via a Service Application

Resolution: My prefered option would be to buy a taxonomy and try amend the taxoanmy with key stakeholders.  WAND have good taxonomys I believe: http://www.wandinc.com/