29 Jun 2017

SharePoint Development - SharePoint 2016

This post is the fifth in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look the recent SharePoint 2016 release.

More posts in the series:

 

SharePoint 2016

There was considerable discussion about the SharePoint 2016 release – notably, whether an on-premise version of SharePoint would even be released. This reflected the constant focus by Microsoft on the Office 365 brand and SharePoint Online over the on-premise offering. This only intensified with the appointment of Satya Nadella as Microsoft CEO in February 2014. He brought a fresh focus on mobile and cloud offerings, and a move away from the traditional Office and OS products.The future of SharePoint on-premise, despite being a product that generated $3 billion for Microsoft in 2015, was seen to be seriously under threat.

In the end, Microsoft did release an on-premise version, after realising that many of their large corporate customers were not prepared to consider migrating their data to a third party cloud service. Microsoft’s “Evolution of SharePoint” blog post in February 2016 confirmed this decision. But even this blog post focused mainly on SharePoint Online/Office 365 features, and how SharePoint 2016 could integrate with them. The hybrid capabilities of SharePoint 2016 were heavily marketed by Microsoft in the lead up to the release, in an effort to entice large organisations to start using cloud based services alongside their existing on-premise installations.

The new on-premise version was an incremental update on the 2013 release, with the major improvements being around the hybrid capabilities already mentioned. The hybrid options required having an Office 365 subscription, in addition to paying for the on-premise licensing. Some of the new features (such as durable links) also required having Office Online Server (the updated and renamed Office Web Apps Server 2013), with Office Online Server now being seen as essential to deploy alongside SharePoint Server 2016.

With this version, the free Foundation edition was dropped, and it was only available in the Standard and Enterprise editions. SharePoint 2016 was released to market on the 14th March 2016, and was finally released to general availability on the 4th May 2016, after missing the planned release date at the end of 2015. As described at the release event, the four areas of innovation in SharePoint 2016 were:

  • File sharing on any device
  • Intranet/Team sites on any device
  • New extensibility options for developers
  • Enterprise features around security, compliance and privacy

The new features in SharePoint 2016 included:

  • Improved mobile experience
    • Support for touch screens on the web application
    • Mobile apps for Android iOS and Windows devices promised within the next year
  • Integration with the new PowerApps and Flow services, again not available upon release, but to be available within the year.
    • PowerApps is a set of developer tools and templates that are simplified for ordinary business users to create business applications. The announcement that they will be integrated with SharePoint means that users will be able to create cross-platform PowerApps that utilize SharePoint lists and libraries as a data source.
    • Microsoft Flow is part of PowerApps and is a clone of the well known IFTTT service, allowing users to mash up data from various services, and set up certain actions to specific defined events. 
  • Hybrid features, including:
    • SharePoint Insights, a reporting service that combines “usage and compliance data from on-premises and cloud into the Office 365 Reporting Center”. In other words, the analytics logs from your SharePoint on-premise farm are automatically uploaded to Office 365 to give a unified audit view. Again, this feature was not available on the release of SharePoint 2016, but was to be available for the end of 2016.
    • The new Hybrid Search allows users to search across both on-premise and online SharePoint sites for relevant content with a single combined list of search results (and not separate results for on-premise and online sites as before). This uses a new Cloud search service application for SharePoint 2013 and 2016 that generated a unified cloud-based search index.
    • A Single Sites View for users to view any on-premise and SharePoint Online sites they have access to.
    • A new enhanced UI, so that the look and feel is consistent across both online and on-premise sites for a consistent user experience, particularly for hybrid scenarios. This included the introduction of the App Launcher to SharePoint 2016, to allow users to access Office 365 applications such as Exchange Online, Delve and OneDrive from on-premise.
    • OneDrive Redirection – use of the user’s Office 365 My Site instead of hosting it on-premise. OneDrive for Business aims to bring users to one place to help them work with their files, regardless of where they are stored. Note, this feature was available in SharePoint 2013 with SP1.
    • Integration with Office 365 Delve and the Office Graph (see Office 365 section below) with on-premise, to surface relevant content and activity for individual users
    • Users can have a single profile in Office 365 where all their profile information is stored. This profile can then be used for both SharePoint on-premise and SharePoint Online.
  • Compliance features were significantly improved, a major selling point for Enterprise customers.
    • Data Loss Prevention (DLP) features, similar to those found in Microsoft Exchange Server 2014 and Exchange Online, were introduced to both SharePoint Online and SharePoint 2016.
    • The Compliance Center uses the DLP features to allow users to identify and search for sensitive content in both SharePoint Server 2016 and OneDrive documents. 51 built-in sensitive information types, such as credit card numbers, passport numbers and bank account numbers, are defined and used to identify sensitive data in documents. Once found, following a crawl by the search service application, sensitive documents can be reviewed in the eDiscovery Center, and the sharing permissions can be changed, the data removed from shared sites, or the documents can be exported for further review
    • The In-Place Hold Policy Center allows users manage policies defined across site collections and OneDrive for Business, and to establish time-based holds on data for a specified time.
  • Miscellaneous
    • Improved sharing options, with a share button on every page, and the option to view who already has access to an item.
    • Support for Open Document Format (ODF)
    • The file names can now be longer than 128 characters, and character restrictions are being removed, so that the & ~ { } characters, GUID, leading dots are now allowed in file names.
    • About Me profile page to include content from Delve and Office Graph API
    • Image and video previews are generated on hovering the mouse over a a file in the document libraries.
    • Durable links - make it easier to move files without having to update links.

While this is not a comprehensive list of all the end user features in SharePoint 2016, it confirms that the release was incremental and didn’t offer many significant improvements for end users. This wasn’t particularly surprising, given the maturity of the SharePoint platform.

Developing for SharePoint 2016

This was the first release of SharePoint to be based on the SharePoint Online codebase, with some capabilities backported for the on-premise version (such as PerformancePoint). This meant that with a single unified code base for both the SharePoint Online and on-premise versions, there was a now a single roadmap for the product. Features would come first to the cloud, and when appropriate, would be added to the on-premise version.

Microsoft made much of the fact that new features would be constantly be added to Office 365, and that these will be made to on-premise farms available more quickly using the new Feature Packs (for those customers with Microsoft's Software Assurance volume-licensing plan), rather than the traditional Service Packs. Clearly, most of the future effort and innovation would be directed at Office 365, with relatively few new features coming specifically for on-premises.

The move to a single codebase based on SharePoint Online meant a number of architectural changes:

  • As already mentioned, there would be no Foundation version of SharePoint 2016. This was understandable, given the minimal capabilities and very low take-up of Foundation in previous releases.
  • As noted above, several new features required having Office Online Server (the renamed Office Web Apps Server) installed. Office Online Server is now seen as being essentially a dependency for SharePoint Server 2016.
  • Excel Services has been removed from SharePoint 2016, and is instead included as part of the Office Online Server.
  • Microsoft Project Server 2016 (PWA)  is now integrated to SharePoint 2016, instead of requiring a separate install as previously. As a result, MS Project Server content is now consolidated into the SharePoint Content databases and not in a separate database.
  • Forefront Identity Manager is no longer included for bi-directional synchronization of user profile information with Active Directory. Instead a uni-directional sync using AD Import is available. Microsoft Identity Manager 2016 can be configured separately to enable bi-direction sync if required.This is part of a move away from Windows based identity management to cloud based identity management (such as Azure Active Directory, which was now trusted by default).

This version of SharePoint was primarily an infrastructure release, with relatively few improvements for end users. Instead, Microsoft was focusing on improving how SharePoint is installed, configured and maintained, and in particular, offering more options for integrating on-premise SharePoint farms with Office 365 services. The changes included:

  • Installation & Topology
    • The initial configuration of Farm and Server Roles was simplified in the installer wizard, due to the new MinRole server configuration.
    • A new MinRole topology. Based on the experiences gained by Microsoft in running SharePoint Online, this allows administrators to optimize the configuration of each server in a Share Point farm based on the role it has been assigned. The different server roles supported are Front-End, Application, Distributed Cache, Search, and the new Specialized Load role. The Specialized Load role is used for services that are to be isolated from the default SharePoint services (such as 3rd party services or PerformancePoint), and servers with this role are excluded from compliance and reporting. The SharePoint health analyser is used to enforce the min-role topology by scanning all servers except those used for the specialized load role.
    • SharePoint 2016 can no longer be installed in standalone mode. Farm Administrators would have to choose one of the 6 Mini Roles available during the configuration phase. There is a Single Server Farm option where everything is installed on the same computer for development purposes.
    • As there is no ‘standalone’ installation option to install SQL Express, SQL server must already be installed and running prior to installing SharePoint.
  • Upgrade Options
    • You could only upgrade to SharePoint 2016 from SharePoint 2013,  using the database-attach upgrade method. There were no supported direct upgrade paths from SharePoint 2010 or earlier.
  • Improved performance and reliability
    • The Zero Downtime Patching strategy will make use of simplified patch management to significantly reduce the size and number of update packages. Rather than deploying large cumulative updates that contain all patches up to that point in time, as currently, the new patching strategy will mean patches are distributed through smaller packages (~100 MB). As the new patches contain smaller, more targeted changes, SharePoint administrators can now choose which patches to install and skip, so that patching with zero downtime is possible.
    • The introduction of Fast Site Collection Creation to speed up the creation of a new Site Collection. This uses a master site collection template at the Content Database level to provision a new site, and avoids the overhead of Feature activation. Because of this, the time to create a site collection fell from 40 seconds to just 2 seconds.
    • Reliability improvements to the Distributed Cache. This still relies on AppFabric 1.1, which, despite being deprecated as a Windows feature, will continue to be supported for the SharePoint 2013 and 2016’s lifecycles.
    • Resilient file system (ReFS) storage protects data from common errors that normally cause data loss. When ReFS is used in conjunction with a mirror space or a parity space, detected corruption can be automatically repaired using the alternate copy provided by Storage Spaces. ReFS also prioritizes data availability so that if corruption occurs, the repair process is both localized to the area of corruption and performed online, requiring no volume downtime. ReFS includes the Salvage feature that removes the corrupt data from the namespace on a live volume and prevents good data from being affected by non-repairable corrupt data.
  • Scalability
    • Content databases will scale into Terabytes, up from a maximum of 200 GB per content database in SharePoint 2013.
    • A maximum of 100,000 Site Collections per database (the limit was previously 20,000)
    • The list view threshold will be greater than the existing 5,000 items limit. In reality, the threshold still exists, but SharePoint 2016 will automatically create Indexed Columns on lists with more than 5000 items, so that while the threshold is still in place, it no longer impacts end users.
    • Maximum file upload size has increased from 2GB to 10GB, allowing for large media (video) files to be stored in SharePoint 2016. The file BLOBs will still be stored in the content database and will make use of the Shredded Storage implemented in SharePoint 2013.
  • Security
    • Encrypted connections (TLS 1.2) supported by default 
    • Supports sending email to SMTP servers that use STARTTLS connection encryption, and using non-default SMTP ports
  • Miscellaneous
    • SharePoint Logging API (SLAPI) allows easier ability to record and report on analytics and telemetry across a whole range of objects in the Farm
    • Some 115 new PowerShell cmdlets have been added to help support SharePoint, and STSADM is finally officially deprecated

For developers, the main changes were:

  • The announcement of a new development model, the SharePoint Framework (SPFX, see details below).
  • A new simplified Page model, to give a improved end user experience. The new page model was required for the new SharePoint development model, and gives rise to a new page canvas that gives a simplified editing experience for end users (no more clunky ribbon), and made it easier to add and remove web parts (no web part zones as in classic pages). However, classic web parts would not work in the new page model, so new web parts would be provided to replace the existing OOTB SharePoint web parts. The new page model gives rise to new client application types:
    • Page-based apps – an alternative to provider-hosted apps with server side code, these are implemented in JavaScript, but have benefits such as having full page context.
    • List-based apps – an alternative to JS Link for transforming the display/edit/new experience around list items
  • The new modern page model is used in the updated team and publishing sites, which can mean that when migrating content from existing SP2013 sites, you can end up with mixed content (classic and modern pages) that have significantly different web parts and editing experiences available.
  • A new version of SharePoint Designer would not be released with SharePoint 2016. The existing version SharePoint Designer 2013 would continue to be supported in SharePoint 2016.
  • Similarly, a new version of InfoPath would not be released. The previous version InfoPath 2013 will continue to be supported in SharePoint 2016. The InfoPath Form Services will continue to be supported to 2023 but there will be no further enhancement to InfoPath. Despite some talk of next-generation forms for SharePoint, Microsoft currently has no replacement story for InfoPath. Instead, PowerApps is being touted as a possible replacement.
  • Search Improvements
    • SharePoint 2016 also offers the option to crawl current and legacy SharePoint versions such as 2007, 2010 and 2013, without needing to upgrade those versions.
    • Supports indexing of up to 500 million items per Search Server application
SharePoint Framework

For developers, the main talking point with the release of SharePoint 2016 was the introduction of (yet another) development model, the SharePoint Framework (SPFx). This was a SharePoint client-side framework that would allow developers to use the latest web development technologies and tooling to create pages and web parts for both SharePoint Online and SharePoint on-premise.  As for many of the new features announced for SharePoint 2016, it wasn’t actually ready for release in May 2016. Instead, a preview was available in August 2016, and the framework was released to general availability in February 2017. This was for SharePoint Online only; it is expected to be released for SharePoint 2016 on-premise via Feature Pack 2 in the second half of 2017.

Key features of the SharePoint Framework include:

  • Runs in the context of the current user and connection in the browser. The new framework does not use iFrames, unlike the previous App Model.
  • The controls are rendered in the normal page DOM.
  • The controls are responsive and accessible by nature.
  • It's framework agnostic - React, Handlebars, Knockout, Angular, and other JavaScript frameworks can be used..
  • The toolchain is based on common open source client development tools like npm, TypeScript (required), Yeoman, webpack, and gulp.
  • Solutions can be deployed in both classic web part and publishing pages and modern pages.

SPFx is effectively a JavaScript based client-side UX framework for SharePoint. While it is possible to use any JavaScript framework with SPFx, the clear preference from Microsoft is for developers to use TypeScript (which makes sense for SharePoint developers coming from the legacy server-side APIs), and the React framework. The new framework would use RESTful API’s to communicate with the SharePoint (Online or on-premises) backend.

This was a major change to SharePoint development which historically was based around full trust C# code running server-side. With the SharePoint Framework, Microsoft was saying that the future of SharePoint development was using client-side JavaScript. As the new framework was client based, the files for any web parts of apps that use SPFx can live anywhere (on a CDN or a separate website), not just in SharePoint. Developers were no longer tied to using Visual Studio (or even the Windows OS) for development, but could instead use lightweight code editors like VS Code or Sublime.

I haven’t been able to play with the new framework, as it is currently only available for Office 365. But I have been following a useful blog series that is asking prominent SharePoint developers to describe their experiences of SPFx in their own words. Some common themes emerge:

  • The framework is a good start, but it has a long way to go until it gains parity with full-trust server side code or the App Model. Currently, the framework really only offers web parts; developers are looking to use the framework to:
    • Create SPAs built on top of SharePoint.
    • Provide more granular customizations (giving alternatives to custom actions and display templates)
    • Provision lists and sites (developers are still reliant on PnP Guidance, which is to use Add-ins)
    • Automate deployment of SPFx solutions
    • Give alternatives to JS Link and script injection
    • Ability to target specific site collections, not just the tenant
  • The positives about the framework include:
    • You no longer need to develop on a SharePoint server. The “local development” model is very different – IIS no longer hosts files on your local machine, since Gulp and node.js are used to serve files instead.
    • The use of unit tests (automatically generated when creating a new SPFx solution) will lead to higher quality applications.
    • There were mixed views about the use of TypeScript, but most agreed that this would help legacy SharePoint developers make the jump to the new framework.
    • The framework, combined with the new page model, helps to simplify end user experience, and gives rise to improved rendering on mobile devices
  • Common complaints include:
    • Lack of documentation (to be fair, I personally think the documentation is quite good compared to previous releases, but there is a definite lag on documenting the new features, simply due to the large number being released).
    • The steep learning curve for Enterprise developers who are unfamiliar with the modern web development stack.
    • Lack of support for the framework in Visual Studio (currently there is only a community extension that allows developers to create SPFx projects in Visual Studio).
    • SPFx doesn’t provide guidance on how to bundle dependencies, or how to deal with an updated version of solutions.
    • The Framework is not as agnostic as it purports to be: all the existing first party web parts and documentation use TypeScript and the React framework, making it difficult for developers who prefer vanilla JavaScript and Angular. This issue was mentioned by almost every developer in the blog series.
    • As SPFx is a client-side framework, there is no security model to work against, as in the server-side API. The lack of a security model means that  Provider-hosted add-ins (using iFrames) are a still a valid developer option as they are isolated on the page. Alternatively, a server side component (eg Web API) may be required.


SharePoint Online/Office 365

As noted in my blog post on SharePoint 2013, Microsoft were quickly developing new features for SharePoint Online/Office 365. The pace of change had only accelerated since 2013, and new features and services being delivered almost weekly. Recent features include:

  • Microsoft Graph is an underlying Office 365 service maps the relationships among people and information (such as email, Yammer, meetings, recently accessed documents), and allows more relevant and personalized information to be presented to each individual user. The Office Graph uses machine learning techniques to connect people to the relevant content, conversations and people around them. It is based on the Yammer enterprise Graph. In another win for the folks at Microsoft Marketing, Microsoft Graph was known previously as the Office Graph (renamed in November 2015), and before that, known as the Office 365 Unified API (see below).
  • Delve is a data visualization and discovery tool that surfaces interesting content to users. It is based on the Office Graph, and in effect, tries to show documents and content to the user that it thinks they will need, before they know they need themselves.
  • Sway for Business, an interactive multimedia tool that allows users to quickly create and share interactive reports, personal stories and presentations.
  • Modern team sites with a responsive UI.
  • Office 365 Groups - a way to centralize membership for multiple Microsoft products (such as SharePoint Online, Exchange, Teams, Yammer) in one place, and apply policies at the project or team level instead of applying to each separate product.
  • Improved integration between SharePoint Online and OneDrive, with users being able to access SharePoint Online document libraries using the OneDrive mobile app.
  • Office 365 Public CDN, allowing a library in SharePoint Online to be become a repository for assets to be stored in a CDN.

One of the major changes in Office 365 for developers was the launch of the Unified APIs, subsequently renamed as the Microsoft (Office) Graph. Due to the rapidly increasing number of services in the Office 365 brand, Microsoft recognized that unified API was required to bring together all the various APIs that access the Office platform. This Office 365 Unified API subsequently evolved into the Microsoft Graph, giving developers a single endpoint to access data from Microsoft cloud services.

Conclusions

SharePoint 2016 was a pivotal release, not so much in the new features it offered, but in that it effectively set SharePoint Online as the modern SharePoint platform. This new modern SharePoint is built on top of Azure and uses modern web development technologies, in direct contrast to the legacy SharePoint on-premises platform it evolved from. Microsoft's effort and innovation will clearly be directed at developing SharePoint Online and Office 365, with few new features coming specifically for the legacy on-premise platform.

For the end user, the main features were the improved mobile experience (and improved UI overall), and the increasing number of Office 365 services that they could use from SharePoint on-premise. Apart from this, the 2016 release offered end users very little. For administrators, the improved hybrid support in SharePoint 2016 made integrating with Office 365 significantly easier, and there were significant improvements in installing and maintaining on-premise farms (such as the MiniRole topology, Zero Downtime Patching and ReFS). For developers, this was a significant release, with the new codebase being based on SharePoint Online, and yet another development framework, and a simplified Page and publishing model.

I have mixed feelings about the new SharePoint Framework (SPFx). Obviously, as the fourth development model to be introduced, why should I or others bother investing in this one? Microsoft is selling it as the one and true way to develop solutions for SharePoint, but they pushed Sandbox Solutions and the App Model in the same way, and they’re now as dead as the dodo. Why should we assume that there won’t be yet another development model in the next SharePoint release?

One reason is that Microsoft has actually released the new model correctly. They had learned from their many mistakes in releasing the previous development models. Instead of just laying down the law to developers that this was the development model to use, the process has been more open. Microsoft had been discussing the framework with developers and vendors for over a year before SharePoint 2016 was released, in a series of Dev Kitchen events organised by the SharePoint Product Group. Previews of the framework were available for almost a year before it was released in February to general availability. The use of the SharePoint Framework Roadmap has been rightly praised – the goals for SPFx for the next 12-18 months are openly discussed and prioritised using community feedback, and new framework features and functionality are rapidly being released.

Another reason is that Microsoft has embraced the modern web development stack. The new framework attempts to standardise development for SharePoint around web development best practices and tooling, and to move to client side development. SPFx gives us a clean break from legacy SharePoint, and the increasingly anachronistic server-side development model that we have all been using for too long. While this means a huge learning curve for existing SharePoint developers, it does represent a chance to encourage other (non-Microsoft) developers to build solutions using SharePoint/Office 365.

Finally, the framework is additive: it doesn’t replace anything or make the legacy development models obsolete. Organisations can continue to build solutions using the existing development models, but as the framework matures, customers will begin to use it to build new solutions and enhance existing solutions.

I suspect adoption of the new framework will be very slow. It is still not available for on-premise SharePoint 2016, and the majority of existing on-premise customers are still on SharePoint 2013 or even older versions. The use of the web development stack will be a barrier to existing SharePoint developers making the jump. I suspect it will take at least another 2-3 years before we see any significant uptake of the framework, as organisations start to migrate to 2016, and as developers gain confidence that Microsoft is not going to pull the rug from under their feet (yet again).

Overall, I’m actually positive about the SharePoint framework. It represents a break from the legacy SharePoint development practices that have built up, and allow us to embrace web development techniques that can be used outside of the SharePoint ghetto. Obviously Microsoft is hoping it will encourage greater use of SharePoint from non-Microsoft developers. Regardless of whether the SharePoint Framework survives or not, the future of SharePoint development is clearly client-side. I’m looking forward to developing solutions with it soon!

To be clear, with the advent of the new SharePoint Framework, the App/Add-in model is dead. It will kept around for a while until SPFx gains parity, but it will then be deprecated to death, just as Sandbox Solutions have. I doubt if full trust server side solutions will ever stop being an option for on-premise customization, simply because they offer a way to provide a custom backend API for SPFx solutions. Additionally, I suspect that if you’re serious about developing for SharePoint (as opposed to just consuming a SharePoint / Office 365 API), you will still need to understand the underlying object model.

While the SharePoint Framework was a positive feature of the SharePoint 2016 release, there were also a few negatives. For developers, the lack of investment in new tools to replace SharePoint Designer and InfoPath was disappointing. After initial hinting at a replacement for InfoPath (Forms on SharePoint Lists), this was cancelled, and PowerApps was parachuted in to (partially) replace InfoPath. As there is no full replacement for InfoPath, the InfoPath 2013 client and InfoPath Forms Services will continue to be supported until 2023.

The much heralded new iOS app for SharePoint, released in June 2016, was also underwhelming. As someone who had to use and support it, I’m aware of issues where users belonging to a large number of site collections were unable to access their sites via the app. As can be seen from the iTunes rating graph below, until the release of v2.9.3 last week, the ratings for the app were around 3.2-3.4, which is relatively low. Hopefully, the new version of the app will have resolved many of the issues that Microsoft appears to have ignored for the past year.

Another major failing in the release of SharePoint 2016 was the failure to address what we should be using Yammer for. After all the hype for Yammer in the SharePoint 2013 release, it wasn’t mentioned at all at the release of SharePoint 2016. What is the use case for Yammer? Where does it stand amongst the other conversation tools like Skype, Office 365 Groups and traditional SharePoint discussions?

The major takeaway from this release was that Microsoft wanted developers to stop viewing SharePoint as a platform, and to start viewing it as a service. This conclusively resolved Microsoft’s SharePoint identity crisis. With the new SharePoint Framework and the new modern page model, Microsoft were effectively ignoring the legacy SharePoint development models and were rebooting using the latest web development technologies.

Microsoft now appears to see SharePoint on-premise as a way of promoting hybrid solutions. The improved hybrid support in SharePoint 2016 aims to push the current on-premise customers (typically larger organisations reluctant to store their data in a third party cloud-based solution) into using the hybrid model and increasing their exposure to the rapidly growing range of Office 365 services. As part of the move away from being a platform, SharePoint (Online and on-premise) is now being built more for an out of the box experience for end users, as opposed to being a platform for developers to build customized solutions.

In attempt to quell any of the uncertainty around the SharePoint 2016 release, Microsoft has already told us that it will not be the last on-premise release. As well as summarizing this series so far, I’ll look ahead to the next release of SharePoint and how we will develop for it in the next (and last) article.