AppFabric Distributed Caching and SharePoint 2013

Windows Server AppFabric is installed as a prerequisite for SharePoint 2013. AppFabric is installed as an extension to the Windows Server Application Server Role and consist of two main components:

1. Distributed caching – Previously codename Velocity
2. Service and workflow management – Previously codename Dublin 

This blog will focus on the distributed caching component and how this is leveraged in the context of SharePoint 2013. The following elements will be described:

1. What is AppFabric distributed caching and how does it work?
2. How does SharePoint 2013 utilize AppFabric distributed caching? 

What is AppFabric distributed caching and how does it work? 

The concept of distributed caching is to provide caching across multiple servers. This enables highly scalable and available caching environments which results in more scalable, available and high-performance applications.

AppFabric implements this caching methodology by using a cache cluster containing a number of caching servers. This cache cluster automatically handles the complexity of load balancing, scaling and failover. Each one of the servers in the cache cluster has AppFabric installed and is running an instance of the AppFabric Caching Service. As an addition to storing data in the cache cluster AppFabric Caching Service also has the ability to store data in a local cache on the server. This can be used to provide faster access to certain data which can be configured to automatically synchronize with cached items in the cache cluster.

When a cached item is being requested the local cache is checked first (if it is being used). If the item is found it is returned to the client. If the item does not exist the query is sent to the cache cluster and returned if it exists. This interaction with the caching service is transparent to the client which only sees the cluster and local cache as a single entity. This way the client does not have to worry about which server the caching is stored on. AppFabric distributed caching implements the cache-aside architecture (Explicit Caching) which means that all items must be explicitly added to the cache. The client can therefore not expect that the data is available in the cache and should implement functionality to query and add the data when needed. AppFabric now also supports the ability to ‘Read-Through and Write-Behind’ by implementing a Read-Through and Write-Behind provider. This provider is able to integrate with a backend system and perform read and writes when data is queried from the cache cluster. ‘Read-Through’ implements support for automatically reading and caching the data from the backend system if the data is not available in the cache cluster. This way the client never have to worry about where the data comes from but always expect to receive the data when requesting it from the cache. ‘Write-Behind’ supports periodically and asynchronously writing added or updated data from the cache cluster to the backend system.

All data in the cache cluster is stored in memory on the cache servers. By default the data is stored in the memory on only one server in the cache cluster. By enabling the high-availability option in the AppFabric Caching Services a second copy of each cached item is created on another server in the cache cluster. The first cached item is called the primary item and the second copy is the secondary item. In case a server in the cache cluster becomes unavailable the data is available on one of the other servers. The secondary item is then promoted to the primary item and a new secondary item is created on another server in the cluster.

In the above example the cache cluster consist of three servers each containing a primary and a secondary item. If cache server one was shut down S1 in cache server two would be promoted to the primary item and a new secondary item would be created on cache server three. Since S2 on cache server one is also lost, a new secondary item would be created on cache server two. The result would be the following:

AppFabric distributed caching is designed to be used with .Net and supports caching of any serialized .Net object. Cached items can be updated and deleted explicitly or removed by the caching service through expiration time-out periods or to make room for more frequent accessed data. The AppFabric caching service also provides built-in support for the ASP .Net session objects which can be configured to be stored in the cache cluster by simple configuration. This means that existing applications can get the benefit of AppFabric distributed caching without changing any code.

The communication between clients and the data cluster can be configured to be digitally signed and encrypted, and administrators can configure which users should have access to each cache in the cache cluster.

Caching components 

The logical architecture of a cache cluster is made up of the following components:

• Named Caches
• Regions
• Items

When items are stored in the cache cluster they are always added to a specific cache. When installing the cache cluster a default cache is always created which is used if no specific cache is selected. Other caches can be created and configured independently through PowerShell in order to divide the data into separate cache containers. These cache containers are referred to as ‘named caches’. Named caches are persisted even if the cache cluster is restarted however the content of the named caches will be deleted.

Regions are containers within named caches and are optional. Regions are created at runtime through code and make it possible to search through items stored inside the region. This search is performed by using tags which can be added to the cached items. The downfall of using regions is that regions are not able to span over multiple servers and hence not able to take advantage of the benefits of using distributed caching.

An item is the data that is stored in the cache cluster. An item can be stored in a named cache or a region and is always stored using a key-value approach. An item is always returned as an Object and must be cast to the correct type at runtime.

AppFabric Concurrency Model 

The AppFabric distributed caching supports two different concurrency models:

• Optimistic
• Pessimistic

The optimistic concurrency model does not use locks to update items in the cache but instead adds a version number to the cached item every time it is updated. When an item is requested from the cache, the item and its version number is returned. When updating the item the version number must be supplied and the cached item is only updated if the version number matches the current version number in the cache. If the version number does not match, the item has been updated since it was received and a DataCacheException is thrown. The pessimistic concurrency model uses locks to ensure no other users update or delete the cached data while it is locked. When an item is requested a lock handle is returned which is later used to unlock the item. If the lock is requested while the item is locked a DataCacheException is thrown. Locked items can be configured to automatically release their locks in case the client application ends before releasing a locked item. This concurrency model assumes that all clients follow the rules for locking. Any item in the cache can be updated and deleted by an operation not using locks whatever the item is locked or not.

Comparison 

The following table compares distributed caching to some of the commonly used caching techniques in .Net.

How does SharePoint 2013 utilize AppFabric distributed caching? 

The distributed caching is used by SharePoint 2013 for caching social elements and FedAuth tokens. Every time a user accesses a front end server the user needs to be authenticated which can become an issue when using multiple front end servers. In SharePoint 2010 this was handled by enabling load balancing affinity (Sticky Sessions) to ensure that the user stayed on the same front end server. By caching the FedAuth token in the distributed cache it is no longer necessary to enable sticky session since the authentication token is now available from all front end servers through the cache cluster.

Implementing the Service Locator Pattern

The idea of a service locator is to have a single point of reference to obtain a service needed by an application. The service locator acts as a registry for services and provides functionality for registering and retrieving references to services.

A service locator implementation consists of three components:

1. Service interface(s)
2. Service implementation(s)
3. Service locator

The picture below illustrates how the service locator, the interfaces (I1, I2 and I3) and the service implementations (Service 1, Service 2 and Service 3) are connected.

The following illustrates an example of how the service locator pattern can be implemented.

All interfaces implement a global interface called IService:

public interface IService { }

This example uses two services Products and Users. For each service an interface is defined:

// Interface for product service
public interface IProductService : IService
{
    // Get all products
    List<Product> GetProducts();
}

// Interface for user service
public interface IUserService : IService
{
    // Get all products
    List<User> GetUsers();
}

The service implementations implement the interfaces:

public class ProductService : IProductService
{
    // Get all products
    List<Product> GetProducts()
    {
        // Return List of products
    }
}

public class UserService : IUserService
{
    // Get all products
    List<User> GetUsers()
    {
        // Return list of users
    }
}

The service locator is implemented using the singleton pattern and provides a registry for registering and retrieving service references. Services a registered using the service interfaces:

public class ServiceLocator
{
    // Service Locator
    private static ServiceLocator _serviceLocator;
        
    // Lock to make singleton pattern thread safe
    private static object _lock = new object();

    // Dictionary of all available services
    private Dictionary<Type, IService> _services;

    // Constructor to initialize all services
    private ServiceLocator() 
    {
        _services = new Dictionary<Type, IService>();
        _services.Add(typeof(IProductService), new ProductService());
        _services.Add(typeof(IUserService), new UserService());
    }

    // Get service locator
    public static ServiceLocator GetServiceLocator()
    {
        lock (_lock)
        {
            if (_serviceLocator == null)
                _serviceLocator = new ServiceLocator();
        }
        return _serviceLocator;
    }

    // Get service of type T
    public T GetService()
    {
        return (T)_services[typeof(T)];
    }
}

To receive a reference to a service through the service locator, the application calls the generic GetService method on the service locator:

var serviceLocator = ServiceLocator.GetServiceLocator();
var productService = serviceLocator.GetService<IProductService>();
var products = productService.GetProducts();

The service locator pattern reduces the dependency on the implementation and makes it easy to substitute service implementation without changing the entire application. In this case if products were read from a database, the implementation could be changed to read from an XML file by simply changing the service implementation.

Extension Methods for SharePoint fields

By combining the blog on 'Reading and writing SharePoint field values' and 'Extension Methods' the following extension methods for reading and writing data to SharePoint fields have been implemented:

namespace SharePointExtensions.ListItem
{
    public static class ListItemExtensions
    {
        // Get value from text/multiline/single-choice field
        public static string GetStringValue(this SPListItem item, string columnName)
        {
            return item[columnName] as String;
        }

        // Get value from number field
        public static Double? GetNumberValue(this SPListItem item, string columnName)
        {
            return item[columnName] as Double?;
        }

        // Get value from currency field
        public static Double? GetCurrencyValue(this SPListItem item, string columnName)
        {
            return item[columnName] as Double?;
        }

        // Get value from datetime field
        public static DateTime? GetDateTimeValue(this SPListItem item, string columnName)
        {
            return item[columnName] as DateTime?;
        }

        // Get value from multiple choice field
        public static SPFieldMultiChoiceValue GetMultipleChoiceValueCollection(this SPListItem item, string columnName)
        {
            if (item[columnName] == null) return null;
            return new SPFieldMultiChoiceValue(item[columnName] as string);
        }

        // Set multi choice field to the given choices
        public static void SetMultipleChoiceValueCollection(this SPListItem item, string columnName, string[] choices)
        {
            var value = new SPFieldMultiChoiceValue();
            foreach (var choice in choices)
                value.Add(choice);
            item[columnName] = value;
            item.Update();
        }

        // Add choice to multi choice field 
        public static void AddChoiceToMultipleChoiceValueCollection(this SPListItem item, string columnName, string choice)
        {
            SPFieldMultiChoiceValue value;
            if (item[columnName] != null)
                value = item[columnName] as SPFieldMultiChoiceValue;
            else
                value = new SPFieldMultiChoiceValue();

            value.Add(choice);
            item[columnName] = value;
            item.Update();
        }

        // Get value from lookup field
        public static SPFieldLookupValue GetLookupValue(this SPListItem item, string columnName)
        {
            if (item[columnName] == null) return null;
            return new SPFieldLookupValue(item[columnName] as string);
        }

        // Set value of a lookup field
        public static void SetLookupValue(this SPListItem item, string columnName, int itemId)
        {
            item[columnName] = itemId;
        }

        // Get value from multiple lookup field
        public static SPFieldLookupValueCollection GetLookupValueCollection(this SPListItem item, string columnName)
        {
            if (item[columnName] == null) return null;
            return new SPFieldLookupValueCollection(item[columnName] as string);
        }

        // Set value of multiple lookup field
        public static void SetLookupValueCollection(this SPListItem item, string columnName, int[] itemIds)
        {
            var value = new SPFieldLookupValueCollection();
            foreach (int i in itemIds)
                value.Add(new SPFieldLookupValue(i.ToString()));

            item[columnName] = value;
        }

        // Add value tos multiple lookup field
        public static void AddLookupToLookupValueCollection(this SPListItem item, string columnName, int itemId)
        {
            SPFieldLookupValueCollection value;
            if (item[columnName] != null)
                value = item[columnName] as SPFieldLookupValueCollection;
            else
                value = new SPFieldLookupValueCollection();

            value.Add(new SPFieldLookupValue(itemId.ToString()));
            item[columnName] = value;
        }

        // Get value from true/false field
        public static Boolean? GetTrueFalseValue(this SPListItem item, string columnName)
        {
            return item[columnName] as Boolean?;
        }

        // Get value from user field
        public static SPFieldUserValue GetUserValue(this SPListItem item, string columnName)
        {
            if (item[columnName] == null) return null;
            var userField = item.ParentList.Fields.GetFieldByInternalName(columnName);
            return userField.GetFieldValue(item[columnName] as string) as SPFieldUserValue;
        }

        // Get value from multi user field
        public static SPFieldUserValueCollection GetUserValueCollection(this SPListItem item, string columnName)
        {
            if (item[columnName] == null) return null;
            return item[columnName] as SPFieldUserValueCollection;
        }

        // Set value of multi user field
        public static void SetUserValueCollection(this SPListItem item, string columnName, int[] userIds)
        {
            var web = item.Web;
            var value = new SPFieldUserValueCollection();
            foreach (int i in userIds)
                value.Add(new SPFieldUserValue(web, i.ToString()));
            item[columnName] = value;
        }

        // Add value to multi user field
        public static void AddUserToValueCollection(this SPListItem item, string columnName, int userId)
        {
            var web = item.Web;
            SPFieldUserValueCollection value;
            if (item[columnName] != null)
                value = item[columnName] as SPFieldUserValueCollection;
            else
                value = new SPFieldUserValueCollection();

            value.Add(new SPFieldUserValue(web, userId.ToString()));
            item[columnName] = value;
        }

        // Get value for url and image field
        public static SPFieldUrlValue GetUrlValue(this SPListItem item, string columnName)
        {
            if (item[columnName] == null) return null;
            return new SPFieldUrlValue(item[columnName] as string);
        }

        // Set value for url and image field
        public static void SetUrlValue(this SPListItem item, string columnName, string url, string description)
        {
            var value = new SPFieldUrlValue { Url = url, Description = description };
            item[columnName] = value;
        }

        // Get value for Managed Metadata field
        public static TaxonomyFieldValue GetTaxonomyValue(this SPListItem item, string columnName)
        {
            return item[columnName] as TaxonomyFieldValue;
        }

        // Set value for Managed Metadata field
        public static void SetTaxonomyValue(this SPListItem item, string columnName, Guid termId)
        {
            var field = item.Fields[columnName] as TaxonomyField;
            var session = new TaxonomySession(item.Web.Site);
            var term = session.GetTerm(termId);
            field.SetFieldValue(item, term);
        }

        // Get values for Managed Metadata field
        public static TaxonomyFieldValueCollection GetTaxonomyValueCollection(this SPListItem item, string columnName)
        {
            return item[columnName] as TaxonomyFieldValueCollection;
        }

        // Set values for Managed Metadata field
        public static void SetTaxonomyValueCollection(this SPListItem item, string columnName, Guid[] termIds)
        {
            var field = item.Fields[columnName] as TaxonomyField;
            var session = new TaxonomySession(item.Web.Site);
            var terms = session.GetTerms(termIds);
            field.SetFieldValue(item, terms);
        }
    }
}

Extension Methods

Extension methods were introduced with C# 3.0 and allow the developer to extend an existing type with new functionality without modifying the original type. Extension methods are implemented as static methods that reference the target type by using the ‘this’ keyword. Visual Studio automatically provides intellisense support and compile time checking for all extension methods.

The following example adds an extension method to the string type that determines if the length of a sting is equal to or larger than a given length:

namespace ExtensionMethods
{
    public static class ExtensionMethods
    {
        public static bool HasMinLength(this string s, int minLength)
        {
            return s == null ? false : s.Length >= minLength;
        }
    }
}

To use the extension method the namespace of the extension method must either be imported or part of a parent namespace of the current class:

using ExtensionMethods;

namespace MyProgram
{
    class Program
    {
        public static void Main(string[] args)
        {
            var s = "MyString";
            
            if(s.HasMinLength(5))
                Console.WriteLine("Length is not to short");
            else
                Console.WriteLine("Length is to short");
        }
    }
}

Extension methods support inheritance of classes which means that an extension method for a parent type is also available to inherited types. By creating an extension method for the object type the extension method is available to all types since the object type is the base class of every type:

public static bool IsOfType(this object o, Type t)
{
    return o.GetType() == t;
}

This method can now be used on all objects to determine if an object is of a particular type:

var s = "MyString";
var i = 11;

s.IsOfType(typeof(string));  // Returns true
i.IsOfType(typeof(double));  // Returns false

There are no security risks by using extension methods since extension methods cannot be used to override instance methods and the extension method is not able to access any private variables of the class. This means that if an extension method contains the same name and signature as an instance method, the instance method will always be called during execution.

Reading and writing SharePoint field values

When working with SharePoint lists and libraries through the object model data is often retrieved or saved to and from a number of different types of fields. This blog entry will go through the different types of fields and describe how to read and write data using the object model. For each type it is defined which type of object is used to store the value, if validation is performed on the input, and how values are read and written to the field.

General info

As a general rule when working with the object model, all limits and restrictions on fields are ignored except unique value and column validation formula.

If the field does not contain any value the field value is null. This can be checked before reading the value from the field. The value can be cast to the object type that the value is stored as before it is checked for null, however this is only possible for nullable types.

Single Line of Text

The text field can contain up to 255 characters and is stored as a string object.

Read

var value = item[columnName] as string;

Write

item[columnName] = "text";
item.Update();

Multiple lines of text

This field is of type 'Note' and can be configured to contain either plain text or rich text. The content is stored as a string object.

Read

var value = item[columnName] as string;

Write

item[columnName] = "text";
item.Update();

Choice

When setting the value of a choice field there is no validation on the value, which means that the field can be set to any string (max. 255 characters). The value is stored as a string object.
Reading and writing to this type of field varies between single and multiple-choice fields.

Single choice
Read

var value = item[columnName] as string;

Write

item[columnName] = "MyChoice";
item.Update();

Multiple choice
Read

SPFieldMultiChoiceValue value = null;
if (item[columnName] != null)
    value = new SPFieldMultiChoiceValue(item[columnName] as string);

Write
To write new value:

var value = new SPFieldMultiChoiceValue();
value.Add("Choice 1");
value.Add("Choice 3");
item[columnName] = value;
item.Update();

To add a choice to selected choices:

SPFieldMultiChoiceValue value = null;
if (item[columnName] != null)
    value = new SPFieldMultiChoiceValue(item[columnName] as string);
else
    value = new SPFieldMultiChoiceValue();

value.Add("Choice 3");
item[columnName] = value;
item.Update();

Number

The number field contains a number stored as a double. Validation on minimum and maximum value is ignored when writing to the field.

Read
The value is cast to a nullable double type in order to be able to hold the null value if the field is empty.

var value = item[columnName] as Double?;

Write

item[columnName] = 11;
item.Update();

Currency

The currency field works the same way as a number field. The value is stored as a double and minimum and maximum values are ignored when writing to the field.

Read
The value is cast to a nullable double object in order to be able to hold the null value if the field is empty.

var value = item[columnName] as Double?;

To get the value including currency use the following code:

var value = item.GetFormattedValue(columnName);

Write

item[columnName] = 11;
item.Update();

Date and DateTime

Date and DateTime fields are both used the same way through the object model. When writing to the fields the full date and time is stored for both, however for the date field only the date part of the stored value is displayed. The data is stored as DateTime objects.

Read
The value is case to a nullable DateTime object in order to be able to hold the null value if the field is empty.

var value = item[columnName] as DateTime?;

Write

item[columnName] = DateTime.Now;
item.Update();

Lookup

Lookup fields are stored as string values using the format 'id:#value'. All additional fields added are accessed in the same way as the 'main' lookup field.

Reading and writing to this type of field varies between single and multiple-choice fields.

Single lookup
Read

SPFieldLookupValue value = null;
if (item[columnName] == null)
     value = new SPFieldLookupValue(item[columnName] as string);

Write
When writing simply set the value to the ID of the referenced item:

item[columnName] = 2;
item.Update();

Multiple lookup
Read

SPFieldLookupValueCollection value = null;
if (item[columnName] != null)
    value = new SPFieldLookupValueCollection(item[columnName] as string);

Write
To write new value store the IDs of the referenced list items:

var value = new SPFieldLookupValueCollection();
value.Add(new SPFieldLookupValue("2"));
value.Add(new SPFieldLookupValue("3"));
item[columnName] = value;
item.Update();

To add a lookup value to selected values add the ID of the new item to the collection:

SPFieldLookupValueCollection value;
if (item[columnName] != null)
    value = item[columnName] as SPFieldLookupValueCollection;
else
    value = new SPFieldLookupValueCollection();

value.Add(new SPFieldLookupValue("3"));
item[columnName] = value;
item.Update();

Yes/No

This type of field stores the value as a boolean.

Read
The value is case to a nullable Boolean object in order to be able to hold the null value if the field is empty.

var value = item[columnName] as Boolean?;

Write

item[columnName] = true;
item.Update();

Person or Group

Person or Group fields store their value as strings. When writing to the field user/group is not validated against the constraints defined on the field. This means that any user or group can always be stored in the field. The user from the SPFieldUserValue object can be accessed through the User property. If the User property is null the value is a group.

Reading and writing to this type of field varies between single and multi-person/group fields.

Single person/group
Read

SPFieldUserValue value = null;
if (item[columnName] != null)
{
    var userField = item.ParentList.Fields.GetFieldByInternalName(columnName);
    value = userField.GetFieldValue(item[columnName] as string) as SPFieldUserValue;
}

Write
When writing the value can be set to either the SPUser/SPGroup object or the ID of the person or the group. The following would all be valid assignments:

item[columnName] = SPContext.Current.Web.CurrentUser;
item.Update();

item[columnName] = SPContext.Current.Web.CurrentUser.ID;
item.Update();

item[columnName] = SPContext.Current.Web.Groups["MyGroup"];
item.Update();

item[columnName] = SPContext.Current.Web.Groups["MyGroup"].ID;
item.Update();

Multiple people/groups
Read

SPFieldUserValueCollection value = null;
if (item[columnName] != null) 
    value = item[columnName] as SPFieldUserValueCollection;

Write
To write new values store the IDs of the persons/groups:

var web = item.Web;
var value = new SPFieldUserValueCollection();
value.Add(new SPFieldUserValue(web, "1"));
value.Add(new SPFieldUserValue(web, "6"));
item[columnName] = value;
item.Update();

To add a person/group add the ID to the collection:

var web = item.Web;
SPFieldUserValueCollection value;
if(item[columnName] != null)
    value = item[columnName] as SPFieldUserValueCollection;
else
    value = new SPFieldUserValueCollection();
value.Add(new SPFieldUserValue(web, "6"));
item[columnName] = value;
item.Update();

Hyperlink or Picture 

The 'Hypelink or Picture' field stores an URL and a description of a hyperlink or picture. The URL must either start with the protocol e.g. 'HTTP://' or with a forward slash if it is a relative url.

Read

SPFieldUrlValue value = null;
if (item[columnName] != null)
    value = new SPFieldUrlValue(item[columnName] as string);

Write
When writing the URL can be directly written to the field. This way the description will be the same as the URL:

item[columnName] = "http://www.MyFavoriteSite.com";
item.Update();

To define both the URL and the description use the following:

var value = new SPFieldUrlValue { 
                   Url = "/_layouts/ApplicationPage.aspx", 
                   Description = "Application page link" 
            };
            
item[columnName] = value;
item.Update();

Managed Metadata

Reading and writing to this type of field varies between single and multi-managed metadata fields.

Single managed metadata
Read

var value = item[columnName] as TaxonomyFieldValue;

Write
When writing the value is set by getting the Terms from the Term Guid:

var field = item.Fields[columnName] as TaxonomyField;
var session = new TaxonomySession(item.Web.Site);
var term = session.GetTerm(termId);
field.SetFieldValue(item, term);
item.Update();

Multiple managed metadata
Read

item[columnName] as TaxonomyFieldValueCollection;

Write
When writing the value is set by getting the Terms from their Term Guids:

var field = item.Fields[columnName] as TaxonomyField;
var session = new TaxonomySession(item.Web.Site);
var terms = session.GetTerms(termIds);
field.SetFieldValue(item, terms);
item.Update();

For a complete set of extension methods for reading and writing to SharePoint fields see the blog on 'Extension Methods for SharePoint fields'.