Agile Software development for SharePoint

Overview

SharePoint 2013 brings new software development challenges to organizations.  Anyone architecting solutions involving SharePoint is faced with more critical decision points than ever before.  Traditional approaches to requirement gathering used on SharePoint projects, often include a long winded structure analysis phase.  It is expensive and difficult to get the relevant analysis clarified and Agile offers a different approach to working out what you are trying to achieve. 

The familiar IT Project Management Triangle gives one a clear visual representation of the constraints facing projects.  By using Agile management of projects can remove beaurocrartic redundant steps and focus on delivering what the customer is looking for.  This post provides an overview of using Agile methodologies such as SCRUM to help alleviate some of the problems we face when working with SharePoint 2013 development projects.



IT Project Management Triangle.

I am looking at SCRUM and Agile practices from a practical SharePoint delivery perspective.  SCRUM, like all methodologies, has it’s time and place.  SharePoint 2013 has many visual quick win so it is easy to get visual ideas in front of the end client very quickly in an iterative approach.

Supplementing project management with Agile practices and techniques to move a project towards successful delivery should overrule any historic dogmatic approaches.  

Methodologies

For simplicity, I break down the various software development methodologies into two main camps:

• The traditional approach that the Systems Development Life Cycle (SDLC) falls under and
• Agile Methodologies (SCRUM being the most common framework of Agile, other frameworks include Agile Kanban, Extreme Programming). 

SharePoint projects are all too often run using traditional approach that can result in “analysis paralysis”.

Agile methodology is implemented normally using an Agile Framework such as SCRUM which is by far the most popular framework.  As with anything, context is all-important.  The approach and techniques need to make sense to your organization and the project.  A good technical lead or Solutions Architect picks and borrows specific parts of different framework or methodologies when providing SharePoint solutions in order to give the best return for effort. 

Agile Manifesto, know the four values and 12 principles. 

4 Values:

1. individuals and interactions over processes and tools;
2. working software over comprehensive documentation;
3. customer collaboration over contract negotiation; and
4. responding to change over following a plan.

12 Principles:

1. Satisfying customers through early and continuous delivery of valuable work.
2. Breaking big work down into smaller tasks that can be completed quickly.
3. Recognizing that the best work emerges from self-organized teams.
4. Providing motivated individuals with the environment and support they need and trusting them to get the job done.
5. Creating processes that promote sustainable efforts.
6. Maintaining a constant pace for completed work.
7. Welcoming changing requirements, even late in a project.
8. Assembling the project team and business owners on a daily basis throughout the project.
9. Having the team reflect at regular intervals on how to become more effective, then tuning and adjusting behavior accordingly.
10. Measuring progress by the amount of completed work.
11. Continually seeking excellence.
12. Harnessing change for a competitive advantage.

SCRUM however, is gaining momentum as the defacto project management framework and should be a tool in your arsenal for SharePoint development.  Agile principles tend to skip some of the more formalized phases of SDLC and I am not suggesting that planning is superfluous; the trick for the technical development team is do not let process get in the way of building your product.

Experience has shown it is likely that you will face significant challenges in trying to implement SCRUM.  The team stakeholders may resist, your organization may resist, and the techniques may not be appropriate.  A simple and practical tip for overcoming these challenges is to try to find someone that can work you through the SCRUM process on suitable projects.  By bringing Agile practices to SharePoint development projects you can aim to reduce time, deliver what the client really needs and make software development more rewarding.  You don’t need to use every facet of a methodology, start with the easier options that will immediately add value.  As your SCRUM team maturity evolves add more pieces. 

Tip:  Microsoft Team Foundation Server (TFS) 2012 has 3 project management templates, one of which is the SCRUM template and this is a good place to start with minimal project management overhead.  The nice thing about TFS 2012 is you can easily amend these templates to suit your needs as your experience grows. 

Traditional approaches such as the SDLC often have a lengthy requirements gathering phase.  This long winded analysis phase is often ingrained into IT folks psyche that people find hard to break away from.  Projects involving a lengthy requirement gathering phase can easily suffer from excessive documentation if there is poor governance.  This documentation can be erroneous, contradictory, outdated, incomplete, challenging to follow and result in documentation overload. 
As a metaphor, SharePoint analysis is like the children’s game “Chinese whispers” or “Telephone” where one person whispers a message to another person, which is passed through a succession of people until the last player announces the message to the entire group.  This lost-in-translation is all too common as excessive noise in documentation or the analysts gloss over the expert’s main point.  Whereas a developer performing analysis generally picks up when something needs detailed explanation as they are “solutioning” the project as they directly communicate with the expert.
The more people in the chain, the more potential for errors in the relayed messages. Moreover, these business processes are often extremely complex, so that the actual intended meaning of the relayed message is lost.  SharePoint is a complex platform and even on the biggest of projects, the success of the technical delivery will often come down to a few key individuals.  Increasing team sizes does not often help and after a certain size becomes a real hindrance.  Agile teams work best when they are relatively small (SCRUM recommends team between 5-9 team members).  If the project is too big, get better quality people; don’t make the commonplace management mistake of adding more resources blindly.

Not all projects or SharePoint teams are ready to work within Agile methods however current commonplace SDLC approaches have problems we hear about repetitively such as change to software cost money and the SDLC is extremely inflexible, Agile based projects lend themself to change. 

Another argument is that that traditional methodologies can be used to give you fixed price quotes.  Vendors operating on a fixed price basis tend to promise the world and try cut every corner to deliver on time (and I hear the vendors screaming already “that is how the other vendors work not us”). 

Nevertheless, many of us realize that software consultancies operate work on the simple maxim ‘sell high, ship quickly’.  Once a software consultancy is in, the inevitable changes that will need to be made will cost the earth.  SCRUM can also run away but with the demonstrable iterations it quick to identify when you are off track and remedy the situation.  Agile methodologies use a time and material model whereas most financial budget decision makers prefer a fixed price (despite this being almost always wrong and they don’t seem to learn).     

Internal software teams often run over in time, deliver the wrong product and buggy solutions due to promises made based on poor initial guestimates.  SDLC can work if managed properly, Agile allows for better feedback controls to verify you are going in the right direction.

SharePoint projects often lend themselves to a ‘time-and-materials’ model as opposed to the ‘fixed-cost’ approach.  If you are unhappy with a project’s development, remove the vendor without the full budget being used up before discovering this project will be successful.  With SCRUM the team have to show you the deliverable product at the end of each sprint (2-4 week delivery phase) and replacing a team members is possible between sprints.

ALM for SharePoint 2013

SharePoint’s developer tooling has matured throughout the iteration of SharePoint.  Tooling is primarily based on Visual Studio and TFS.  ALM covers many roles with the team including project managers, testers, architects and stakeholders.  In this section I mainly focus on ALM for the developer.  Third-party tools assist with building the solution, communication within the team and technical guidance that easily plug into TFS and VS.  An example is CKSDev for Visual Studio.

Tip:  Visual Studio 2012 and TFS 2012 have a large number of purposes.  Always start with the basics of ALM and continue adding the more sophisticated features as your ALM maturity increases.   Source control is definitely the first important feature of TFS.  The next step would be bug tracking.  Build automation is about when you know you are really starting to get going properly with ALM.  With builds your developer isolate code is now integrated into the solution allowing verification of that the individually developed parts fit together.

TFS 2012 comes in 2 on-prem. varieties, don’t bother with the express edition.  It is free but limited to 5 users, if you are this small rather get a hosted TFS 2012 solution or use TFS preview.  TFS has an excellent web user interface portal, which is user friendly for team members that don’t necessarily use Visual Studio to enter Product Backlog Items (PBI’s).  TFS contains the following key components: source control, work items (PBI’s, tasks and bug tracking in SCRUM terminology), build automation and reporting. 

Visual Studio 2012 comes in 4 flavours as a developer you can use the Ultimate, Premium or Professional edition.  Depending on your MSDN subscription use the most feature rich edition you have available.  There is also a Test edition version of Visual Studio 2012 that is not suitable for development.

A good opinion on development machines is that each developer must have a completely standalone environment hosted on visualized PC/laptop infrastructure.  VMware workstation is a good option to provide this virtualized environment to developers.  The chapter in this book on DTAP and automation strongly examines setting up the SharePoint developers’ development environment.  Using automation builds are triggered by developers or are schedules using FSP policy, these builds can include smoke test or full testing scenario allowing for high levels of confidence in that the build work correctly.

Microsoft tooling for SharePoint is good and wide ranging.  Starting with Visual Studio 2012 there are item templates for achieving common tasks.  The community writes many Visual Studio Extensions such as CKSDev.  Consider adding this to the developer machine base builds so when the image is rebuild or the images are distributed the developers are up and running quickly.
Code needs to be in a source code repository such as TFS 2012, using the SCRUM template is the good option.  At the most basic level it allows for easy management of the project using the SCRUM template with the appropriate reports with a source code repository and bug tracking facility.
Assembling all the constituent parts of your SharePoint project using the build functionality available through TFS allows you to have confidence that it all works together and can be deployed into production accurately. 

Build tab within the SCRUM template.

 Build process templates are used for automating the build in TFS.  The figure below shows the default templates.

Built in Build Process Templates

Daily automated builds are a good idea to ensure code is working against your automated tests.

Tip: Don't build your daily automated SharePoint 2013 environment from scratch. Rather use snapshots or cloned images for the base images and merely deploy and test your code through automation.  Chris O’Brien has good information on Continuous Integration for SharePoint check out his blog. 

PowerShell is your friend.  Automate everything in the build.  This allows team members to get the latest code and setup the appropriate databases quickly - the invested time pays for itself almost immediately.  The developers will often use a baseline content database so they know that they have the correct test data.

ALM allows for easy migration onto your Development, Testing, Acceptance and Production (DTAP) environments.  Automated of deployment pieces bring efficient delivery through to the production environment.  I can't stress enough that you need more environments than development and production.  Even the smallest project should have at least three separate environments and remember to keep the production and staging (also referred to as pre-production or acceptance testing) in sync.

Summary

SCRUM is a commonly used Agile framework for delivering projects.  SCRUM lends itself well to a lot of SharePoint projects as it allows for focused collaboration.  Product feedback is on-going and the team demonstrates the project sprints in multiple short iterations to the business which ensures the projects direction is focused.  SharePoint 2013 is a complex product and having smaller teams with superior resources can greatly improve returns. Agile does not mean no documentation, it advocates using a bottom-up self-managing team that is organized.  Agile is not anarchy but allows us to focus on business problems creatively utilizing all team members aiming to work together to manoeuvre SharePoint to meet business needs.  Agile and SCRUM try move away from overdrawn project analysis phases.

Microsoft has a 1st class stack of ALM tools for SCRUM comprising of Visual Studio 2012 and TFS2012 that hook nicely into SharePoint 2013 development projects.  TFS is not just for developers, it can be used by the whole team.  TFS has source control for your code as well as a plethora of features to help your SCRUM based projects such as the SCRUM template, reporting/ project management, requirements and acceptance gathering artifacts, the ability to automate your solution via the build process templates and testing.

Agile for SharePoint (This post)
Scrum for SharePoint - Part 1
Scrum for SharePoint - Part 2
Scrum - Part 3 - Scaling Scrum
Scrum - Part 4 - Kanban and Scrum
Scrum - Part 5 - Certification PSM 1