3 Apr 2017

SharePoint Development - SharePoint 2013

This post is the fourth in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look the SharePoint 2013 release.

More posts in the series:

 

SharePoint 2013

The SharePoint 2013 release was considerably less ambitious than the 2010 release. It was relatively minor upgrade from an IT perspective, with no major architectural changes. This wasn’t a major surprise. SharePoint was by this stage a mature and robust product that had been around for over 12 years, and was,by 2013, used by some 100 million users and 80% of Fortune 500 companies. The 2013 release was simply an incremental change on the existing software that focused on improving usability for end users, particularly around the social experience.

The different versions available for SharePoint 2103 remained unchanged from SharePoint 2010 (Foundation, Standard and Enterprise). The first technical preview of SharePoint 2013 was released in early 2012, and the public beta was released in July 2012. The public release of the finalized version was on 11th October, 2012.

The main features in this release were:

  • A major redesign of the user interface, including:
    • Improved cross browser support
    • Drag and drop documents from your desktop into SharePoint libraries, and also to move content within and between SharePoint libraries (also cross browser)
    • Ability to create and edit lists without leaving the page you’re currently editing
  • Sync document libraries to your computer using OneDrive for Business (previously known as SkyDrive Pro, it was simply an updated version of the Groove tool).
  • Additionally, the MySites feature was rebranded as ‘OneDrive for Business’. In fact, the existing MySite feature was kept, and each MySite now came with a Document Library specially built for the OneDrive for Business tool. As end users unsurprisingly associated it with Microsoft’s OneDrive cloud storage solution, this caused endless confusion. Chalk up yet another win for Microsoft Marketing.
  • Improved task management features, such as:
    • Updated Project Site template, and an improved Tasks list that now included a timeline and Gantt views.
    • Ability to assign multiple resources to tasks, and to create subtasks
    • Improved integration with Microsoft Project
    • Option to view all tasks (across the entire SharePoint farm)  assigned to you in your personal site, without the need for custom or third party solutions.
    • Tasks could also be synchronized with Outlook, and could also be pulled in from other providers (such as a CRM or TFS) using the extendable framework provided.
  • The search features in SharePoint 2013 were considerably improved, with an eye to the search features used in Google that users were already familiar with. These new features included:
    • Type ahead (auto-complete) when entering search terms
    • Previous search terms are now displayed as query suggestions at the top of the search results page
    • Use of Boolean operators in search queries
    • Wildcard searches
    • Quick preview of documents within the search results, allowing the user to preview the document without leaving the results page.
    • Additionally, users could now see how many times an item has been viewed from within the search results
    • A new one click option to view search item in their parent library 
    • Users could follow, and be updated about changes made to, a document from the search results page
    • The new continuous crawl meant that there was no lag period following the addition of new documents to the documents appearing in the search results
    • Improved discussion list, with a more ‘Facebook’-like activity feed.
  • The most important search related improvement was the ability for a piece of content to be surfaced in different locations, across site collection boundaries. This included a number of new category of web parts called ‘Search-driven content’, where the web parts were all pre-configured variants of the Content by Search Web Part (e.g. “Items Matching a Tag”). Similarly, any list/library in SharePoint could be nominated as a ‘catalog’ and then shared across site collections as part of the ‘Cross-Site Collection Publishing’ Feature.
  • Improved social features. The social networking features in SharePoint had been poor, and organisations were forced to use third party add-ons, like Social Sites from Newsgator. The 2013 release had a number of improved social features:
    • The introduction of micro-blogging with (searchable) #hashtags and @mentions
    • Ability to follow and be updated about changes to people, sites and documents, as opposed to the older style alerts
    • New Community site template focused on conversations between users. This was very different to the existing team sites - users could observe conversations and then “join” a community when they want to post. The user’s photo was shown beside all posts they had made, and there was gamification (ratings, badges) to encourage participation
    • An interactive newsfeed to aggregate and follow a user’s activities across all their sites, and to view the activities of users and content that they follow
    • One click sharing to easily share documents and sites with others
  • Mobile
    • In SharePoint 2010, it was possible to create a mobile views for sites, but it was very constrained, and didn’t allow for any site customization for specific mobile devices or tablets. With the SharePoint 2013 release, the mobile browser experience was optimized, and allowed the creation of device channels to render SharePoint content in a design for a specific device type
    • SharePoint Newsfeed apps were released for Windows Phone and iOS devices, and later for Windows 8
    • SkyDrive Pro (later re-branded as OneDrive for Business) applications were also available for Windows Phone, and later for Windows 8 and iOS, allowing for offline viewing of SharePoint lists and document libraries
  • Business Intelligence
    • The BI options in SharePoint 2013 enhanced the existing features, such as Excel integration and the PerformancePoint service (including a new Dashboard designer)
    • Improved support for rendering dashboards on iPads and Windows 8 tablets
    • A new Business Intelligence Center site template to manage reports, scorecards, dashboards, and data sources in a single location
  • Miscellaneous
    • OneNote integration with team sites (when configured with Office 365)
    • Improved integration with Microsoft Outlook
    • Improved support for playing video (HTML5 and Silverlight players) and searching for videos
    • New eDiscovery (electronic discovery and audit) site template
    • Use of Metro styling for the UI

As always, this is an incomplete list, but it covers the major improvements in user features for SharePoint 2013.

Developing for SharePoint 2013

As already noted above, there were no major architectural changes in SharePoint 2013. Despite this, the upgrade process from SharePoint 2010 to SharePoint 2013 was still problematic, due to deprecated features and new or modified features (such as the updated discussion forums) which could cause existing SP2010 solutions to stop working following the upgrade. Additionally, for this release, there was only one supported path, from 2010 to 2013; you could no longer skip a release and go from 2007 to 2013.

The visual and in-place upgrade options were discontinued in this release, with only the database attach upgrade method being now supported. However, there was now an option to retain a content database as a 2010 content database and to perform a “deferred site collection upgrade”. In practice, this means that the 2010 site collection continues to be a 2010 site collection, just running on a 2013 farm. The solutions for the 2010 site collection continued to be deployed to the 14 hive, while solutions for 2013 site collections were deployed to the new 15 hive. In theory, the existing 2010 sites would continue to work when migrated to the SP2013 farm, including your customizations. When you performed the deferred site collection upgrade, the site collection now uses the 15 hive, which is when the custom solutions might break. As someone who has recently upgraded a major SharePoint farm to 2013, this was a nice feature that allowed the migration to proceed in phases as the existing 2010 customizations were made 2013 compatible.

At the time, Microsoft stated that SharePoint 2013 would be the last version released on a 3 year upgrade cycle. In future, both the on-premise and online versions will be updated much more frequently, with SharePoint Online to be upgraded every 90-120 days. Future releases of the on-premise release would simply be snapshots of current SharePoint Online.

The main changes to the administration of SharePoint were:

  • Improvements in the ability to manage multi-tenancy deployments (again, based on Microsoft’s experience of running the SharePoint Online service)
  • Significant improvements in the User Profile Synchronization Service with the re-introduction of the AD Import (faster than the User Profile Sync), faster performance for full synchronizations, and the ability to sync with more directory services (such as the Java System Directory Server, Novell eDirectory and IBM Tivoli). The User Profile Sync was still difficult to setup, but it was considerably improved on the 2010 release.
  • Introduction of AppFabric to give a distributed, in-memory object cache to scale out high-performance .NET applications. AppFabric was installed as a prerequisite for SharePoint 2013 and could only be configured via PowerShell. While an important new feature, it was difficult to configure SharePoint’s Distributed Cache Service (a wrapper around AppFabric) to work across a high availability SharePoint farm with redundant servers.
  • Request Management to control how the SharePoint farm handles incoming requests by evaluating logic rules against them in order to determine which action to take, and which machine(s) in the farm (if any) should handle the request. This wasn’t to replace load balancers, but to provide more flexible configuration options for the SharePoint farm.
  • Internet Explorer 7 was no longer supported
  • Various improvements in the Web Content Management (WCM) features, including:
    • Ability to copy and paste directly content from Word, without having to strip out the formatting
    • Improved support to upload and embed videos in pages, including a new video content type, and the auto-generation of thumbnail preview images
    • Ability to insert iframe elements into a page, allowing external content to be embedded
    • Improved support for multilingual sites
    • The introduction of cross-site publishing, allowing specified site collections to be published to other publishing site collections. Also, the new Product Catalog (simply terrible naming) site template allowed you to create lists and libraries whose content could be published to other sites
    • Introduction of managed navigation, new feature that allowed site admins to define and maintain the navigation on a site collection by using managed metadata term sets
    • Improved SEO for public facing websites, such as user friendly URLs, allowing verification of site ownership, the ability to edit sitemap settings (such as the robots.txt) for search engines, and the ability to create SEO meta tags.
    • Improved branding options, allowing the use of standard HTML, CSS and JavaScript designs and workflows to brand SharePoint, rather than constraining designers to using only Visual studio or SharePoint Designer
    • As mentioned above, the introduction of device specific channels
  • Related to the above improvements in WCM, SharePoint Internet sites no longer required an expensive licence. The use of cheaper standard server licence for public facing sites meant that organisations could host their internet and intranet on the same platform, reducing costs and providing more flexibility.

The new features for developers included:

  • The new App Model (see below)
  • Major improvements in SharePoint search, with the core SharePoint 2010 Search product and Fast Search for SharePoint 2010 being merged and extensively overhauled into a single product called SharePoint 2013 search that was common between the Foundation and Standard versions.
  • Introduction of Shredded Storage to store large files in the database as small shreds instead of a single BLOB (binary large object). The file shreds are combined into a single document when retrieved from the database. This aimed to reduce the I/O of updates to existing files (only the file shreds that are modified are uploaded). So if only a file’s metadata is updated, the file shreds are not updated (in SharePoint 2010, a copy of the document’s BLOB was created). Unfortunately, this feature was poorly implemented, and could result in increasing rather than decreasing file storage in many common scenarios, as well as increasing the time required to retrieve documents from SharePoint. Additionally, this also caused some issues for admins, as they could no longer retrieve documents directly from content database backups as was possible in SharePoint 2010.
  • Addition of OAuth authorization (used in the new App Model)
  • As already mentioned above, the introduction of Device Channels for mobile/tablet specific page rendering.
  • The JavaScript Link property (JSLink) that allows the user interface of fields and lists to be modified using a JavaScript file, as opposed to using modified XSLT views as in previous versions. A small but very useful change that allowed easy customization of the UI.
  • Introduction of the Minimal Download Strategy (MDS) to download only the differences between pages when navigating between them. This improves rendering performance when browsing content where large parts of the page do not change. MDS is enabled by default on team and community site collections.
  • Support for HTML5
  • Significant improvements to Workflow. As well as being backwards compatible with SharePoint 2010 workflows (based on Windows Workflow Foundation 3.5), SharePoint 2013 used the new Workflow Manager service, based on the Windows Workflow Foundation in .NET Framework 4.5. The new Workflow Manager is external to SharePoint, and can run on completely separate server(s), allowing for greater performance and scale. The Workflow Manager and SharePoint WFEs communicate using a REST API secured by OAuth. Note, it isn’t possible to automatically upgrade from older SharePoint 2010 workflow to the new workflow types.
App Model

For developers, the main change with the release of SharePoint 2013 was the introduction of the new App Model (later renamed the Add-in Model). Previously, developers wrote custom server-side code that was run within the SharePoint process (either as a fully trusted solution or within the sandboxed user code service). SharePoint Apps, however, are run external to the SharePoint process, running within the browser as a client-side solution, or remotely on some other server or service external to SharePoint.

What did this mean in effect? Developers would no longer develop SharePoint solutions using C# to run on top of SharePoint, but would instead develop apps (essentially web applications) using HTML5, CSS and JavaScript using current web development best practices. The apps would be loosely coupled with SharePoint, consuming data using the SharePoint CSOM, JSOM and REST APIs, rather than using the server side API. These new apps would be compatible across both on-premise SharePoint and SharePoint online instances, and would provide users with ‘No Code’ (as in no server side SharePoint code) customization solutions.

The new App Model had 3 separate deployment models:

  • SharePoint Hosted App – the solution components are hosted on either an on-premise or Office 365 SharePoint farm. SharePoint hosted apps are installed on a SharePoint 2013 website, called the host web. They have their resources hosted on an isolated subsite of a host web, called the app web.
  • Provider Hosted App -  also known as a self-hosted app, this solution include components that are deployed and hosted outside the SharePoint farm. They are installed to the host web, but their remote components are hosted on another server. The separately hosted components are typically an ASP.NET application, but they can be developed in any server side technology (such as Node.js, PHP) that can use OAuth and consume the SharePoint REST API.
  • Automatically Provisioned Azure App – also known as auto-hosted apps, these were cloud-hosted apps whose remote components are provisioned and deployed on Windows Azure. As with the provider hosted app, it could interact with a SharePoint website but also uses resources and services that are located on a remote site that is hosted by Windows Azure. This deployment option was only available to SharePoint Online customers.

As app stores were all the rage (Apple’s App Store, Google Play for Android, the Windows Phone Store, Windows 8 Store), SharePoint also got one (the Office Store). Corporations got an internal app store, the internal app catalog where only apps approved for use within the organization were available.

SharePoint Online/Office 365

The SharePoint Online service was updated to use the SharePoint 2013 version at the end of February 2013. Initially, it was identical to the on premise version. However, Microsoft quickly started to add new features to SharePoint Online/Office 365 that were not available with the on-premise installations. These unique features included Yammer integration (for Enterprise subscriptions), and then, later, Delve, a personal search and discovery tool (think Cortana/Sir for work). The quick iterative nature of the updates to Office 365/SharePoint Online reflected the new 90-120 day release cycle that Microsoft was using. Other changes for SharePoint Online included the new App Model and the Office Store mentioned above.

The use of a hybrid architecture, with some SharePoint sites and services running in Office 365/SharePoint Online, and some on-premises, gained popularity after the release of SharePoint 2013. In particular, many organisations opted to host their MySites with SharePoint Online, while retaining the majority of their data in an on-premise farm. For those organisations, this removed the need to police the contents of an employee’s MySite and simply offshore the problem to Microsoft’s data centres.

Conclusions

For end users, SharePoint 2013 was a major improvement with its improved UI and focus on usability. For those administering SharePoint, there were significant improvements,  particularly the new Web Content Management features. For developers, however, the release was more of a mixed bag.

The notable improvements were the introduction of JSLink, and the new Search and Workflow engines. Unfortunately, there were quite a few ‘meh’ features, including the new Shredded Storage which is simply sloppy in implementation. Another was Device Channels – why base this feature solution on User Agent strings, and completely ignore CSS3 media queries? Similarly, HTML5 support was great, but being able to build components for SharePoint with ASP.NET MVC rather than legacy Web Forms would have been even better.

In addition, I felt in this release that there were several major failings. The first, and the most surprising given that the trouble that Microsoft had went to to improve the UI and to introduce Device Channels for mobiles and tablets, was that there was no official Microsoft SharePoint app for iOS or Android (or even Microsoft Phone, not that anyone actually owned one…). There was the newsfeed app, but if you want a native app to browse and view content on a SharePoint site, you had to use third party offerings.

The second major fail was the new App Model. It sounds like a brilliant idea. No longer would developers be forced to use clunky old C# full trust solutions to customize SharePoint that would risk the stability and performance of the SharePoint server (and more importantly, Microsoft’s SharePoint Online service). Instead, they could deliver solutions across both on-premise and hosted SharePoint sites using the latest and greatest web development practices. Developers obviously rushed to use the new App Model to develop their SharePoint solutions, right?

Well, actually, no. In fact, practically no one used the new app model, as SharePoint developers could see it for what it was – a desperate attempt by Microsoft to solve the no server code customization issue for their SharePoint Online service. They had been burnt by Microsoft pushing Sandboxed solutions in SharePoint 2010, in a previous attempt to solve the same issue, and were understandably wary of jumping on another development model that could be killed off with the next release.

The App Model wasn’t helped by the fact that it was more complex to implement and deploy than full trust solutions, or that it didn’t provide the same range of customization options as either full trust or the more restrictive Sandbox solutions. In particular, it was unable to:

  • Deploy resources to the SharePoint hive, such as master pages, application pages, CSS, JavaScript or image files
  • Provide any delegate control elements or user controls (*.ascx files)
  • Deploy custom Site Definitions
  • Perform Custom Action Groups & Hiding Actions
  • Deploy Custom themes
  • Create a Timer job

Effectively, if it wasn’t possible to do something with the CSOM API, it wasn’t possible to do it with SharePoint Apps.

In time, with the help of the excellent Patterns and Practices project (PnP), some of these major gaps were (partially) filled in, but for some problems, you would still have to use full trust solutions. This meant that when customising for an on-premises solution, you were being asked to support 3 different development models: full trust code, Sandboxed solutions and the App Model.

Of the 3 different deployment types for the App Model, the Provider Hosted App was considered the most useful. Very few people used the SharePoint Hosted App due to issues with authentication and upgrading, and so few used the Azure Hosted Apps that Microsoft removed the deployment option within 2 years of release, stating scalability and manageability concerns.

In the end, most SharePoint developers simply ignored the App Model, despite all of Microsoft’s marketing. The ideas behind the App Model were correct. The SharePoint architecture and development methods were still based on ASP.NET 2.0 Web Forms, which was a decade old. Customers wanted SharePoint solutions that made use of web development best practices and frameworks, and a UI that was responsive. Unfortunately, the App Model simply wasn’t going to deliver that.

A third failing was the Yammer integration. While it was possible to integrate it with an on-premise SharePoint 2013 farm, there was never a clear reason to do so. Yammer never offered enough value to business to be used instead of the existing SharePoint social features.

The Hybrid approach was heavily promoted by Microsoft, as a way of introducing organisations to Office 365. But actually implementing a hybrid architecture was difficult, and was the most complex option (compared to simple on-premise or SharePoint Online solutions). Hybrid really only offered:

  • Search across both environments (but with a very poor user experience, as the search results from the 2 environments weren’t merged together)
  • BCS allowing access to data in on-premise and Office 365
  • Duet integration allowing access to data in on-premises SAP from Office 365

There was no shared global navigation across the 2 environments, or a global site directory. The branding across the 2 environments could not be synchronized, and likewise, configuration settings for one environment were not replicated across to the other. The link between the 2 environments was really quite minimal.

A Hybrid architecture was, however, attractive to Microsoft, as you were effectively paying twice for the same solution – the licencing costs of your on-premise servers, and the monthly subscription for Office 365.

Overall, SharePoint 2013 was worth upgrading to, due to the real improvements for end users, and a number of notable new features such as the WCM improvements. Would the next SharePoint release, which Microsoft hinted would be available in under 2 years due to their new release cycle, be as useful? I’ll look at this in my next post in this series.

6 Feb 2017

SharePoint Development – SharePoint 2010

This post is the third in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look the SharePoint 2010 release.

More posts in the series:

 

SharePoint 2010

The 2010 release built on the growing popularity of SharePoint following MOSS 2007, and it was clear that Microsoft had ambitious goals for SharePoint. Notably, the Microsoft CEO Steve Ballmer referred to it as "like an operating system" in the keynote presentation at the Microsoft SharePoint Conference in 2009. This release was seen as positioning SharePoint as a business collaboration platform that would fill all the needs of an information worker. As a result, there was huge marketing drive for the SharePoint 2010 that targeted end users. These changes were reflected in the updated functional pie chart, from the emphasis on IT Solutions in SharePoint 2007, to the emphasis on end user and business solutions in SharePoint 2010.

Comparison of SharePoint 2007 and 2010 features (taken from https://nikpatel.net/2010/02/14/sharepoint-2010-end-users-improvements-and-new-features-overview/)

In this release, SharePoint was positioned separately to both Office and Windows brands, with references to both removed from the version names. There were 3 versions:

  • Foundation (free, corresponding to WSS in the earlier releases)
  • Standard (with additional search functionality)
  • Enterprise (additional content management capability)

SharePoint 2010 was launched globally in May 2010. There were a large number of new features in this release:

  • An enhanced user interface:
    • Introduction of the fluent ribbon to mimic the changes to the Office UI, minimizing the number of mouse clicks required
    • Multi-browser support
    • Use of Wiki pages rather than web-part pages in templates
    • New themes to customise a site's appearance
    • Mobile Device Access (to render mobile views of the default SharePoint templates)
  • Improved ways of integrating with existing business data
    • The InfoPath Forms service was updated to allow customization of the default SharePoint list forms
    • Introduction of PowerPivot, a special Excel workbook that can be shared and updated in a SharePoint site, as part of a drive to add Business Intelligence capability to SharePoint
    • Two way Business Connectivity Service (BCS , formerly the Business Data Catalog in MOSS 2007) to read and write data from external systems
      • Use of External Content Types to define external data sources, and to allow creation of external lists that can be used in the same way as SharePoint's own lists
  • Improved workflow options
  • Updated user profile and introduction of social networking features:
    • User profiles now included photos and presence indicator, status updates
    • Tagging of both user profiles and content
    • Users could now create their own blogs
    • Colleague suggestions
    • An Organisation chart to show a user's place in the organisation hierarchy
  • Records Management (retention and auditing policies, records repository, email content as records, legal holds)
  • Improved integration with Microsoft Office, including the ability to open content and edit in SharePoint directly in Word and Excel
    • Office Web Apps (OWA) was introduced as an add-on to SharePoint as the online companion to Word, Excel, PowerPoint and OneNote applications that enables users to access and edit documents from anywhere. This allowed for the replacement of the traditional desktop Microsoft Office clients.
  • Improved governance and IT management tools, to control what end users can do with SharePoint. Notably, this was to prevent duplication of content and to also prevent users creating sites without any oversight, as these were major problems with the MOSS 2007 release.
  • Improved Content Management
    • Document Sets
    • Shared Content Types
    • A new Managed Metadata Service was introduced to allow for tagging of content, and the use of taxonomy hierarchies for implementing logical data architectures
    • Rich Media management

This is by no means a comprehensive list of the new features in SharePoint 2010 (but you can find a full list of the new features and comparisons to previous versions here, here and also here).

Developing for SharePoint 2010

The SharePoint 2010 release involved massive changes for developers. This wasn’t just an incremental release on top of the previous 2007 release. There were significant architectural changes, mainly aimed at removing the scalability and performance issues that had limited MOSS 2007 as an enterprise platform. Notably, Microsoft had experienced these problems when managing their online SharePoint service, Business Productivity Online Services (BPOS). These architectural changes included:

  • Moving from a 32-bit to 64-bit architecture, giving a significant performance boost. This also meant that upgrading from MOSS 2007 involved significant changes to the hosting infrastructure.
  • The replacement of Shared Service Providers by Service Applications to share resources across sites in different web applications and farms. The new service application architecture allowed for the use of dedicated application servers, and also lent itself to multi-tenancy scenarios, which was particularly important for Microsoft’s online offering.
  • Improved the boundaries and thresholds for scalability, including:
    • Increased the list view threshold from 2000 (in MOSS 2007) to 5000 (20,000 for administrators)
    • Increased the number of content databases supported per web application from 100 to 300
    • Maximum content database size increased from 100 GB to 200 GB
    • The full list of boundaries and limits for SharePoint 2007 can be found here, and for 2010, here.
  • The introduction of the Sandbox Solutions to allow site collection administrators to monitor and control the resources consumed by web parts and other  solutions in a site collection. Sandboxed solutions were hosted in the new SharePoint user code solution worker process (SPUCWorkerProcess.exe), run code that can only affect the site collection of the solution. As the code was not run in the IIS worker process (as in the full trust farm solutions that had previously been used), the sandboxed code could not affect the performance or stability of solutions in other site collections. This had been a common issue for Microsoft when running their multi-tenancy BPOS solution – poorly coded web parts in one tenant’s site collection would affect the availability of other tenants solutions running on the same web application. The new Sandbox Solution framework were created to solve this problem. Microsoft was so insistent on Sandbox Solutions being used by developers that they made them the default solution type when creating a new SharePoint solution in Visual Studio, and SharePoint Online (the BPOS successor, see below), only accepted Sandbox Solutions.

The changes in the architecture are highlighted by below between the two diagrams:

SharePoint 2007 Architecture

SharePoint 2010 Architecture

In addition to the Sandbox Solutions, Microsoft had a major push to educate developers on common programming pitfalls and performance issues when developing SharePoint solutions, and released the SPDisposeCheck tool (a static code analyser) to help identify poorly coded solutions.

There were a number of important changes to improve the administration of SharePoint:

  • The Central Administration application interface was updated, including the introduction of the ribbon and wizards to step through common configuration tasks (including the initial configuration of a farm).
  • In addition to the STSADM executable, PowerShell was introduced to provide command-line administration. As an early adopter of PowerShell, I particularly welcomed the addition of PowerShell to the SharePoint developer’s toolbox.
  • Managed Accounts were added, allowing for the management of service accounts directly in Central Administration, instead of having to manage these accounts directly in the AD.

The tooling and best practices for developers changed radically with the release of SharePoint 2010. Notably, there were now 3 separate APIs available to develop against:

  1. Server-side Object Model – the existing .NET API, updated to use .NET Framework 3.5.
  2. Client Object Model – this was a new API designed to allow developers to produce solutions for use in remote client-side solutions that run on computers where SharePoint 2010 is not installed. This included a Client Class Library (for use in  .NET managed or Silverlight applications) and a JavaScript class library (an unmanaged ECMAScript object model for use in the browser).
  3. SharePoint 2010 Web Services- updated to have a RESTful interface.

The changes to the API model reflected the improvements in web development techniques and a new emphasis on creating rich Internet applications (RIAs) in the browser for an improved user experiences using Ajax and Silverlight. It also reflected the new emphasis on SharePoint as an enterprise platform – the new APIs allowed integration with other enterprise systems.

In addition to the new APIs, the Business Connectivity Services (BCS), the successor to the Business Data Catalog, had significant new features to connect to external data sources outside of SharePoint and literally treat them as a SharePoint List. These connections were now two way (read/write) connections. Out of the box, the BCS could connect to SQL databases and web services, and also provided a pluggable Connector Framework to allow connectors for new external systems to be developed.

Similarly, the User Profile service was revamped in SharePoint 2010, with the associated synchronization service also made a two way connection, so that changes made in a SharePoint user profile could be written back to an organisation’s AD. Personally, I came to dread dealing with it (and the associated Microsoft Forefront Identity Manager (FIM) engine). The initial documentation on TechNet was very poor, so I came to rely on Spencer Harbar’s excellent Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization.

For developers, there were significant improvements in the tooling for SharePoint 2010, including the introduction of a number of different SharePoint project templates in Visual Studio 2010. It was now possible to deploy and upgrade features and solutions from within Visual Studio, instead of having to rely on third party tools and extensions.

Developers were encouraged to start using the new LINQ to SharePoint provider, building on the popularity of the LINQ to SQL ORM that was released in 2007. LINQ (Language Integrated Query) is a .NET Framework component that adds native data querying capabilities that was introduced with C# 3.0. The LINQ to SharePoint provider promised to abstract away the complexity of writing CAML to query SharePoint lists. Sadly, the implementation of the first release was poor – you can read the details in Bjørn Furuknap's article. There were also a number of other open source libraries that provided similar functionality, such as Camlex.NET.

Accessibility was improved in SharePoint 2010, after being famously dire in MOSS 2007. This had been a major issue in the take-up of SharePoint solutions by UK public sector organisations who had a duty to provide accessible websites to their users. SharePoint 2010 was marketed as being WCAG 2.0 AA compliant, and also standards (W3C XHTML) and cross browser complaint. In reality, while still a major step forward compared to MOSS 2007, SharePoint 2010 was not fully compliant to the WCAG 2.0 AA standard. I detailed the issues in a past article (the article is no longer online but is still available via the Internet Archive), but to develop a fully accessible web application using SharePoint 2010 required a significant developer effort, using custom control adaptors and JavaScript.  SharePoint 2010 could now support Firefox, Chrome and Safari in addition to Internet Explorer, but the differences in browser support for various features and web parts (mainly due to the use of ActiveX) caused a lot of work for developers. Many organisations were also forced to work around the lack of (official) support for Internet Explorer 6.

Other notable changes for developers included:

  • In addition to the change to a 64-bit architecture already mentioned, SharePoint could also be installed on developer workstations (using Windows 7 or Vista OSs), instead of only on servers as previously.
  • Improved search functionality with an emphasis on promoting information discovery across both structured and unstructured data. Additionally, the FAST search application was integrated into SharePoint only 18 months after Microsoft had acquired the company. Unfortunately, there were 3 separate search options available in SharePoint 2010 (Basic Search Center, Enterprise Search Center and FAST Search) which was needlessly confusing. This was finally resolved in the later SharePoint 2013, when the FAST search become the standard search engine in SharePoint 2013 and the SharePoint Online.
  • Addition of web templates and provisioning providers, as well as updates to the existing SharePoint deployment approaches, site definitions and site templates (detailed here).
  • The introduction of claims based authentication, using the Microsoft Identity framework (code name Geneva), allowing non-AD users to be authenticated (such as users using Hotmail and Gmail accounts).
  • Major improvements in the information architecture, with improved ways of adding metadata to documents with managed terms and keywords.
  • Introduction of Remote Blob Storage (RBS) to store large binary files such as video.

I don’t have space or time to do justice to these new features, or the many others that I have failed to list. But even from this brief overview, you can see that SharePoint was fundamentally rebuilt and extended with the 2010 release.

BPOS/SharePoint Online

As already discussed, many of the changes made in SharePoint 2010 were made to improve Microsoft’s BPOS offering. This led to to an identity crisis that Microsoft still hasn’t fully resolved - is SharePoint an online service or on-premise software? While Microsoft’s direction of travel is now clear (more on this later in the series), the contradictions between these two views of SharePoint have certainly affected how developers see the platform. However, it has been evident that SharePoint Online has contributed significantly to driving new features in SharePoint and in also reducing the product release cycle time.

With the development of Sandbox solutions, custom code could now be deployed to SharePoint Online. However, the feature set for SharePoint Online lagged considerably behind SharePoint 2010 on-premise. Following the release of Office and SharePoint 2010, the BPOS services were refreshed and re-launched as Office 365 in October 2010. Microsoft chose the name ‘Office 365’ to reflect "dependability every day of the year", following a very poor BPOS uptime track record.

Existing BPOS customers were given 12 months to upgrade to the new online service, with possibly complex migrations involving updating the client Office applications. This was an issue for many existing BPOS customers who were predominately small or medium-sized businesses. The demand for BPOS/Office 365 was seen as low, and the uptake for SharePoint Online at the time was even lower.

Conclusion

As described above, SharePoint 2010 was a massive improvement from MOSS 2007, both in terms of features for end users and organisations, and also for developers. But there were some notable failures. SharePoint attempted to do everything, but did very few things well. The new blogging and wiki features were underwhelming. Similarly, the accessibility story fell considerably short of the marketing hype. I recall repeatedly having to spend considerable amounts of time developing SharePoint solutions that were fully WCAG 2.0 AA  compliant for UK public sector clients.

SharePoint 2010 was widely marketed as a suitable platform for Enterprise development, and I worked on a number of large scale projects (with 20+ development teams) to develop various platforms. Some were successful, most were not. One notable failure I recall was an attempt to develop an ecommerce platform on top of SharePoint 2010. This was a clear case of believing marketing hype over reality, as SharePoint was never suitable for developing this type of solution.

With the huge growth in SharePoint usage (detailed in my previous post), a large number of ASP.NET developers started to develop solutions in SharePoint, with mixed results. At one point in my career, I was employed in a large SharePoint team in a major consulting firm that was responsible for cleaning up SharePoint solutions gone wrong. Without fail, the root cause of these failures were architects and developers attempting to develop SharePoint solutions without prior experience. Reading one of the many developer books published on SharePoint 2010 didn’t make you a SharePoint expert, though I came across many who believed it did.

One common anti-pattern was attempting to use SharePoint as a relational database, with developers using SharePoint lists as data tables for massive numbers of records. Another was attempting to install and configure SharePoint using only 1-2 service accounts, instead of the 15-20 accounts actually required. While fine for creating a developer environment, this would cause no end of problems in Production, particularly with the User Profile Synchronization Service. And yet another common failure was re-using service accounts used in an existing MOSS 2007 farm for your new SharePoint 2010 installation – this would always result in constantly locked accounts when password changes failed to synchronize across the two farms.

Despite the major improvements in SharePoint 2010, I now see the release as damaging Microsoft’s relationship with developers. While the new APIs were significant, Microsoft’s push to have developers build Silverlight clients for SharePoint would rebound when the company would side-line Silverlight in favour of HTML5 towards the end of 2010. Likewise, the LINQ to SharePoint provider was killed off along with LINQ to SQL, when Microsoft settled on Entity Framework as their chosen ORM.

However, probably the biggest setback was the introduction of Sandbox solutions. Introduced to overcome performance and instability issues introduced by badly coded farm solutions, particularly in Microsoft’s BPOS service, they were made the default solution type in Visual Studio. However, these solutions offered fewer features but were significantly more complex to implement, especially when attempting to use APIs that were not by default supported within the sandbox (full-trust proxies and the dreaded service locator, anyone?). As a result, very few developers bothered to use them.

The introduction of Sandbox solutions was clear case of Microsoft’s SharePoint identity crisis – was it online or on-premise, service or software? Some very quickly came to realise that the future of SharePoint was to be subsumed in the Office 365 brand. At the time, despite being aware of the major architectural changes going on, I tended to sneer at SharePoint Online as a toy offering, given the very low take-up of the service.

These and other issues (like the completely unnecessary debate over the future of .NET in Windows 8)  would lead to SharePoint developers becoming reluctant to adopt new frameworks and features being touted by Microsoft - more on this trend later in the series.

In the next post in the series, I’ll look at SharePoint 2013. Apologies for the delay in publishing this series – the combination of the Christmas holidays and the baby has meant that I’m running significantly behind in my blogging schedule. Also, thank you to all those who have sent emails about this blog series. All comments or feedback are gratefully received.

13 Jan 2017

Goodbye to Delicious, Hello to Pocket

After some 6 years of using Delicious to store my bookmarks, I've finally admitted that the service is effectively dead, and have now moved to use Pocket instead. I’ve updated my Links page accordingly, and I have also deleted the old bookmarks RSS feed on FeedBurner (another service in terminal decline).

The final straw for me with Delicious was finding that the export option was no longer available, and the developer API was gone. This meant that I had no way to export and back-up my some 5000 bookmarked links. Luckily, I had an export of my Delicious bookmarks from last year. I went through my most recent Delicious bookmarks and manually transferred the most useful to my Pocket reading list.

I’ve been using Pocket for over 8 years, back when it was know as Read It Later and it was a simple reading list add-on for Firefox. Since then it has developed considerably. I’m particularly fond of the excellent iOS app, something I use daily. The premium subscription is a bit steep ($45/£37 per annum), but given I’ve been using their service for so long for nothing, it seems only right to pony up for it. I did consider combing Pocket with PinBoard (and this may happen at a later date), but Pocket alone is sufficient for the moment. I’m not interested in saving 1000s of links that I’ll never look at again, as I did with Delicious.

Again, I’m struck how important it is that you keep control of your data, even something as minor as a list of bookmarks. This is something I touched on earlier last year when I wrote about the shutdown of the Shelfari service. Often, you will lose this control when using a free/freemium service, as the service provider gradually attempts to lock users in.

As a result of thinking about this, I plan to move to use either paid services that give me more control of my data, or to use self-hosted open source applications. I’ll be writing more on this soon. If you have any thoughts on this, I would love to hear from you.

7 Dec 2016

SharePoint Development - MOSS 2007

This post is the second in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look the SharePoint 2007 release.

Series Outline:

 

MOSS 2007

The 2007 release (actually released in 2006) attempted to build on SharePoint 2003, and to integrate it more closely with the Microsoft Office suite. This version fixed many of the shortcomings of the 2003 product but also vastly expanded the platform’s core functionality. A major focus was placed on improving document and records management, alongside web content and portals. This version had a fully functional and integrated content management system, a result of the feedback on Content Management in the 2003 release. This gave rise to the infamous pie diagram for the SharePoint 2007 release:

Image showing pie chart of the MOSS 2007 Features

End users were given the option to navigate the platform to create team sites and manage workflows. As a result, SharePoint moved to become a more collaborative platform, and started to resemble the product we know today.

As in 2003, the product came in 2 different versions:

  • Windows SharePoint Services v3 (WSS 3.0), a basic free version for Window Server.
  • Microsoft Office SharePoint Services (MOSS 2007), the premium version built on top of WSS, with the new MOSS designation indicating the greater level of integration with the Office suite. This version was bundled with the Office 2007 suite and added enterprise search and people search, document repositories, additional Web parts, workflow, and content syndication. MOSS targeted organizations storing more than 500,000 documents (large enterprise customers).

The new features included with MOSS 2007 were:

  • Business Data Catalog
  • InfoPath Form Services
  • Excel Services
  • Content types
  • Use of SharePoint Designer to develop portals (replacing FrontPage)
  • Rights Management
  • Shared Service providers
  • Updated security model
  • Improved administration (now create and configure IIS web sites via Central Admin)
  • Improved authoring and site management experience
  • Updated MySite UI
  • Introduced minor versioning and document check out/in in document libraries for editing

 

Developing for SharePoint 2007

WSS 3.0 was built on top of ASP.NET 2.0 and IIS 6.0. Unlike WSS 2.0, requests to WSS 3.0 were routed through the ASP.NET runtime before reaching the WSS layer. This allowed the use of AJAX with SharePoint web parts and pages in WSS 3.0, as well as wrapping custom ASP.NET user controls (.ascx files) in web parts for deployment. Additionally, it allowed the use of  ASP.NET authentication (Forms Based Authentication). As portal sites no longer had to be created at the root of the IIS web site, hundreds of portal sites could be hosted inside a single IIS web site (in current SharePoint terminology, at the web application level).

SharePoint 2007 Stack Architecture (taken from http://dotnetslackers.com/articles/net/MOSS-2007-Architecture-and-Requests.aspx)

The stacked architecture was meant to encourage existing ASP.NET developers to start developing for SharePoint, as it meant that their existing skills and tools could be re-used for SharePoint development. However, developers new to the SharePoint platform still struggled to cope with SharePoint’s particular development methodology (such as features) and gotchas.

As a result of these fundamental architectural changes in the SharePoint architecture, upgrading from 2003 to 2007 was a painful  and laborious process. Additionally, updating custom site definitions (.stp files) when upgrading to 2007 was to be avoided - something that would be repeated in the next release as well. The use of custom site templates for application development, although available since the 2003 release, was heavily promoted in MOSS 2007 by Microsoft. This gave developers a means of creating custom templates for creating new sites in SharePoint with specific site or web features automatically activated. This led to a lot of messing around with GUIDs for features in onet.xml files, and using the Visual Studio Extensions for Windows SharePoint Services (VSeWSS) to package and deploy the custom templates.  In general, you could add custom site definitions easily enough, but it was very difficult to edit or delete an existing site definition.

Additionally, in MOSS 2007, an existing site could be saved as a site template (.stp file) for new sites. It is a package containing a set of differences and changes from a base site definition. It wasn’t as performant as the site definitions, as it was stored in the content database, while the site definition files were cached on IIS on the web front end servers. Site templates also weren’t as portable as site definitions, as to move them between farms, the farms had to be at the same version and patch level. This was because the site templates were dependent on the original base site definitions they were created from.

In WSS 3.0, features and farm (full trust) solutions were introduced. A feature is simply an XML definition for a component deployed to the SharePoint farm, with a specified scope (Farm, Web Application, Site and Web). Features were introduced in WSS 3.0 to address the failings in WSS 2.0 with regard to defining functionality used across multiple sites. Feature Receivers and Feature Stapling were also introduced, allowing modification of features on activation, and allowing features to be automatically enabled on specific site templates. Features were deployed in the new Windows SharePoint Services Solution Package (WSP) files that replaced the .cab file format used previously, and were deployed either using the SharePoint UI, or more commonly using the STSADM.exe tool (and a plethora of .bat files).  Again, I seem to recall a lot of messing around with DDF files to package and deploy web parts correctly.

In addition to all this, developers could now use:

  • Content Types: metadata used to describe the attributes and actions applied to a particular category of information, and used  to develop a logical information architecture. Content types are in practice the schema definition of a site or list, and allow us to encapsulate a data schema and make it independent of a location on a SharePoint site.
  • Integrated workflow and out of the box workflow templates
  • InfoPath Forms Services and Excel Services to integrate with existing business processes
  • Business Data Catalog
  • Integration with SQL Server Reporting Services

With the release of MOSS 2007, there was major growth in the use of SharePoint, particularly across the enterprise. In 2008, Microsoft reported more than 100 million SharePoint users across over 17,000 client organisations, and total SharePoint sales of more than $1 billion. The product and associated tools had matured enough to be a realistic platform for internet and intranet usage in large organisations. As a developer, it was better than the 2003 release to work with (a very low bar), but it was still an extremely frustrating product. The tooling in Visual Studio 2007, whilst improved since Visual Studio 2003, was still clunky and slow. Developers instead looked at third party tools to help with packaging and deploying solutions, such as:

There are probably many other tools that I can’t recall – feel free to call out any in the comments below.

SharePoint Online (BPOS)

In October 2007, the Business Productivity Online Suite (BPOS) was released. This was Microsoft's first attempt at establishing a SaaS offering. BPOS included SharePoint Online, Exchange Online, Office Communications Online and Live Meeting. The SharePoint Online offering was based on the SharePoint 2007 release. Depending on the subscription chosen, this would be either a Dedicated (single tenant) or a Standard (multi-tenant) instance.

While the BPOS offering was successful, Microsoft encountered a number of issues running high performance multi-tenanted instances of SharePoint Online using MOSS 2007, and this experience fed directly into the development of SharePoint 2010.

Next week, I’ll look at the release of SharePoint 2010 and how the developer story changed significantly.

29 Nov 2016

SharePoint Development - Origins

This post is the first in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look back at SharePoint's origins, up to and including the release of SharePoint 2003.

Series Outline:

 

SharePoint's Origins

SharePoint evolved from two separate Microsoft projects, named "Office Server" and "Tahoe", that were developed during the development of Office XP. It was also influenced by the Platinum project that introduced the concept of digital dashboards, and the TeamPages project that allowed users to create and edit simple web-based lists (the origin of the CAML markup language). These projects were all aimed at accessing and sharing information across organisations. The initial development on these products started back in March 1998, prior to the release of the .NET framework (the initial beta versions of the .NET framework were released in late 2000).

SharePoint 2001

The initial SharePoint release was actually two separate products, the Microsoft SharePoint Portal Server (SPS, released 2001) and the Microsoft SharePoint Team Services (STS, released in 2002). The SPS product was based on "Tahoe", offering top-down portals, search and document management. Tahoe was a collection of different technologies from various companies acquired by Microsoft that was built on top of the Exchange data store. STS was based on "Office Server", and was a bottom up team collaboration product.

Thankfully, I have never had to work with this early version of SharePoint, which was very basic.

SharePoint 2003

In 2003, SharePoint as we would recognise it today was released. It combined STS and SPS together to offer collaboration, search, content management and portal capabilities in a single product. It had an improved user interface, better personalization, and a collaboration store.

A lot of the criticism aimed at the initial SharePoint 2001 release was around the under-powered web store that limited the functionality of the product and a digital dashboard that was outside Microsoft’s core development platform, limiting support options for users. As a result, significant work was put into making SharePoint 2003 more reliable and scalable, and to improve support options by making use of the same developer tools as other Microsoft products.

The product came in 2 different versions:

  • Windows SharePoint Services v2 (WSS 2.0), a basic free version that was included with Windows Server 2003.
  • SharePoint Portal services 2003 (SPS 2003), a premium version was built on top of WSS that included additional functionality for document management and search.

The main features included:

  • Alerts
  • Site templates
  • Calendars
  • Surveys
  • Document Libraries
  • Lists
  • MySite
  • Site Directory
  • User Profile
  • Improved search and indexing (SPS)
  • Improved user interface and personalization options
  • Taxonomy
  • Document collaboration and versioning features
  • Single Sign-on
  • Audiences
  • Web Parts (.cab)

 

Developing for SharePoint 2003

The major developer features in SharePoint 2003 were the introduction of server-side API, and the replacement of the Exchange data store with SQL Server. The server-side API made use of the new integration with the .NET framework, and allowed the development of custom code solutions, including web parts. In addition to the default web parts (Content Editor, Image, Form and Contacts), developers could develop custom web parts using Visual Studio 2003 (using the .cab file format).

Note that in SharePoint 2003, and all subsequent versions of SharePoint, COM components (unmanaged code) are used for core features, with a (smaller) managed layer wrapped around it for the .NET API. This is the cause of the dispose issues that affects fundamental SharePoint objects such as SPSite and SPWeb (here and also here).

I did have to work with SharePoint 2003, and it was a pretty horrible experience. Even in it's second release, it was a very immature product, and the developer tooling (based around Visual Studio) was lacking.

In the next post, I’ll look at at the release of MOSS 2007.

SharePoint Development - Past, Present and Future

With the recent release of SharePoint 2016, and with details of the latest development framework (the SharePoint Framework) for SharePoint becoming available, I thought it would be useful to produce a series of blog posts looking at SharePoint development over the years, and how it is likely to change in the future, based on trends from past releases and the current focus on cloud based solutions from Microsoft. The series will consist of 6 articles, published weekly:

This series will be focused primarily on on-premise SharePoint development. While Office 365/SharePoint Online are growing rapidly and are influencing the development of the SharePoint platform greatly, the majority of large organisations are still using on-premise, and are (rightly) wary of having all their data in a third party data centre, even if that third party is Microsoft. I'll point out trends in Office 365 that I think will affect future SharePoint development, but it isn't the main focus for these articles.

I'll be drawing heavily on my own experiences of developing for SharePoint. I first started developed solutions for SharePoint back in 2006, when as a recent graduate I was part of a large team building integration products with SharePoint 2003 (a hideous project). I then spent over 3 years working full time on SharePoint solutions (MOSS 2007 and SharePoint 2010), at which point I took a break to spend several years lost in rewriting large .NET legacy applications. Last year, I started working with SharePoint once more. In this new role, I am currently part of a small team working on a phased SharePoint migration (2010 to 2013) that will be completed in the new year (January 2017). Over all this time, I've worked on a wide variety of SharePoint solutions (intranets, various organisational platforms, eCommerce sites, HR applications and currently a Virtual Learning Environment).

So, lets start by looking at the origins of the SharePoint platform.

5 Oct 2016

My Son

My son, Matthew Oiyan Parkhill, was born on Monday 26th September at 04:18, at the Royal Jubilee Maternity Hospital, Belfast. He weighed in at 5 lbs 9 oz. Mum and baby are doing well.

Mei and I would like to pass on our thanks to everyone at the Royal Jubilee Maternity Hospital for all their help during Mei’s & Matthew’s stay, especially the midwives Patricia, Deirdre and Heidi. We won’t forget their many kindnesses to our little family.

Matthew comes from the Hebrew name Matityahu, meaning ‘Gift of God’, and Oiyan comes from the Cantonese for ‘Love and Grace’.