Monday, 7 March 2016

Automapper, Dynamics CRM and excluding fields

In the past when I've built web based applications Automapper was always one of those libraries that I both loved and hated at the same time. In the Dynamics CRM world it can often be a bit of a dangerous tool to unleash on a website, especially when utilised by developers that don't know CRM very well. Let me explain why.

Attribute Collections

Most CRM Developers will already know what I'm talking about here, but for those of you not privy to this - the properties of an entity from a CRM database table are not actually surfaced quite like EntityFramework or nHibernate, they are surfaced using a dictionary. This is subsequently wrapped in an Attribute Collection. One of the main advantages of this is you can control partial updates/retrieves without worrying about the state of the entire entity. On the negative side this plays havoc when libraries like Automapper are used to populate the entities, especially when using early bound objects.

Here's a sample to give you and idea of what I'm getting at. Firstly, let's presume our domain models are Early Bound objects exported from dynamics. (self plug! Personally I use this open source tool : https://github.com/conorjgallagher/Dynamics.ExtendedSvcUtil).

Now, let's say we have an REST service and we want to utilise it to update an account. So we build a view model like this:

public class AccountViewModel
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public string Address1_Line1 { get; set; }
}

And we wire up automapper to map that across to our domain entity. In the latest version I believe that would look like this:

CreateMap<Account, AccountViewModel>().ReverseMap()

In this example we can create a new account via the this REST service with the following JSON:

{
    Id: "aef7b4c1-98f6-4f53-9be3-2fa72d1e319d",
    Name: "Hello",
    Address1_Line1: "Home!"
}

Within our controller we will receive an AccountViewModel populated with all the relevant data. We can then use Automapper to push this View Model into an early bound Account entity using the same method as above. As long as all the field names match we're good to go. Even though you are mapping to strongly typed fields what you end up with under the hood is an dictionary like this:

"accountid"="aef7b4c1-98f6-4f53-9be3-2fa72d1e319d"
"name"="Hello"
"address1_line1"="Home!"

Null vs Unset

A really nice feature of CRM is it differentiates between null and unset. If you exclude a field from the attribute collection it will also be excluded from the update statement that hits SQL. This is very useful for performance and limiting what plugins / workflows fire.

Back to Automapper and our view models above - a problem arises when you exclude fields from the JSON.  For example, if you subsequently send the following after the previous update:

{
    Id: "aef7b4c1-98f6-4f53-9be3-2fa72d1e319d",
    Address1_Line1: "Home line 1!"
}

This will hit our view model as this:

{
    Id = "aef7b4c1-98f6-4f53-9be3-2fa72d1e319d",
    Name = null,
    Address1_Line1 = "Home line 1!"
}

Which I guess is expected, because how else can you represent an excluded value in the view model? If we don't deal with this we hit a more fundamental issue further down the chain in that our attribute collection will end up like this:

"accountid"="aef7b4c1-98f6-4f53-9be3-2fa72d1e319d"
"name"=null
"address1_line1"="Home line 1!"

And we'll blank our account name in CRM. Not good!

Excludable

This takes me on to a pattern I would like to propose for this type of issue: the concept of an excludable property. Looking at how nullable value types work surely we can come up with a similar concept! I won't take you through the entire evolution, but here is the interface and struct I now propose:

    public interface IExcludable
    {
        bool Include { get; set; }
    }

    public struct Excludable<T> : IExcludable
    {
        private bool hasValue;
        internal T value;
        private bool include;


        public Excludable(T value)
        {
            this.value = value;
            if (value != null)
            {
                this.hasValue = true;
            }
            else
            {
                this.hasValue = false;
            }
            this.include = true;
        }

        public bool HasValue
        {
         
            get
            {
                return hasValue;
            }
        }

        public bool Include
        {
            get { return include; }
            set { include = value; }
        }

        public T Value
        {
            get
            {
                return value;
            }
        }

   
        public T GetValueOrDefault()
        {
            return value;
        }

     
        public T GetValueOrDefault(T defaultValue)
        {
            return hasValue ? value : defaultValue;
        }

        public override bool Equals(object other)
        {
            if (!include || !hasValue) return other == null;
            if (other == null) return false;
            return value.Equals(other);
        }

        public override int GetHashCode()
        {
            return hasValue ? value.GetHashCode() : 0;
        }

        public override string ToString()
        {
            return hasValue ? value.ToString() : "";
        }

        public static implicit operator Excludable<T>(T value)
        {
            return new Excludable<T>(value);
        }

        public static explicit operator T(Excludable<T> value)
        {
            return value.Value;
        }

        public static bool operator ==(Excludable<T> x, T y)
        {
            return x.Equals(y);
        }

        public static bool operator !=(Excludable<T> x, T y)
        {
            return !x.Equals(y);
        }
    }

Overriding all the comparisons were required to get it to perform both value and null comparisons correctly. Finally, we can change our view model to use this instead:

    public class AccountViewModel
    {
        public Excludable<Guid> Id { get; set; }
        public Excludable<string> Name { get; set; }
        public Excludable<string> Address1_Line1 { get; set; }
    }

Json Converter gotcha

There's 1 final problem we need to solve that I'll highlight now. If we run our object without the address in it all seems to work just fine. Our property comes through as excluded as expected. But, if we instead try set it to null it is marked as included=false! The reason is down to how the JSON formatter deserializes the data into the given object. It doesn't actually call your constructor but does something dirty under the hood. How do we fix this? Create a custom json converter

    public sealed class ExcludableConverter : JsonConverter
    {
        public override bool CanConvert(Type objectType)
        {
            return objectType.IsGenericType && objectType.GetGenericTypeDefinition() == typeof(Excludable<>);
        }
        public override bool CanWrite { get { return false; } }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            return Activator.CreateInstance(objectType, reader.Value);
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            throw new NotImplementedException();
        }
    }

And set this up on application start within your global.asax:

            var formatters = GlobalConfiguration.Configuration.Formatters;
            var jsonFormatter = formatters.JsonFormatter;
            var settings = jsonFormatter.SerializerSettings;
            jsonFormatter.SerializerSettings.Converters.Add(new ExcludableConverter());

Now, when a null comes in we explicitly tell the converter to create a new object for us.

Wiring up Automapper

The final step is getting automapper to utilise this new way of excluding items. In automapper you can set a condition when mapping so that you can do conditional ignores. I found the simplest way to add this rule was write an extension method:

public static void AddExcludableRule<S, D>(this IMappingExpression<S, D> m)
        {
            m.ForAllMembers(opt => opt.Condition(
                    s =>
                        !(s.SourceValue is IExcludable) || ((IExcludable)s.SourceValue).Include
                    ));
        }

Now, whenever I want to wire up an object that has excludables I just add that call to the end:

CreateMap<Account, AccountViewModel>().ReverseMap().AddExcludableRule();



Thursday, 25 February 2016

Dynamics CRM Web API - Pluralization bug

I started investigating the Web API while it was in preview mode so have noticed it move quite quickly over time. Mainly this is changes but sometimes it's bugs. For example, a change I spotted since my previous post (Web API Preview - Unrecognized 'Edm.String' literal...) is how it deals with lookup and regarding references. In the preview you performed the following:
/api/data/appointments?$select=subject&$filter=regardingobjectid eq a199a199-a199-a199-a199-a199a199a199

This has change to the following in the v8.0 release:
/api/data/v8.0/appointments?$select=subject&$filter=_regardingobjectid_value eq a199a199-a199-a199-a199-a199a199a199

The second thing I spotted is in fact a bug. This may have existed in the preview, but I never spotted it and it had me scratching my head for quite some time. We have the concept of an "event day" which we have name "inv_eventday". When creating this entity CRM creates a collection schema to allow you to reference it via the web api. For example the appointment entity has an equivalent collection schema called "appointments"; notice the pluralisation.

So, on to my event days. What you would expect to find as a collection schema is this:
/api/data/v8.0/inv_eventdays

Unfortunately, this is not the case. You get an error "Resource not found for the segment 'inv_eventdays'." I wondered if I had a case sensitivity issue or something, but every combination of captitals / camel case / pascal case just didn't seem to work.

After some time I decided I should check out the metadata to see if I could uncover the correct casing. For reference the following is how you get a list of all entity metadata:
/api/data/v8.0/EntityDefinitions

This uncovered a definition with the following schema name for the entity:
"SchemaName":"inv_eventday"

Which seems to match my expectations on what it should be. So what's the problem? Reading further down I find the Collection Schema name:
"CollectionSchemaName":"inv_eventdaies"

That's some funky plural we've got there! Looks like the algorythm decides that anything ending in "y" is pluralised as "ies". Obviously this isn't correct for all words. E.g. the word "sky" is indeed "skies", but "day" is in fact "days".

Now... where do you log CRM bugs...


Thursday, 3 December 2015

Web API Preview - Unrecognized 'Edm.String' literal...

Today I was playing around with the new Web API Preview to try and check out how the new REST API was going to work. If you're interested take a look here: https://msdn.microsoft.com/dynamics/crm/webapipreview. At a first glance this looks really good. It's much cleaner and does away with a long unnecessary url and the case sensitivity that always bugged me.

The first issue I hit was when trying to filter by a regarding object on an appointment. My first thought was is this not supported in the preview? I took a standard oData query from the 2011 endpoint and converted it so that this:
/xrmservices/2011/OrganizationData.svc/AppointmentSet?$select=Subject&$filter=RegardingObjectId/Id eq (guid'a199a199-a199-a199-a199-a199a199a199')

became this:
/api/data/appointments?$select=subject&$filter=regardingobjectid/id eq (guid'a199a199-a199-a199-a199-a199a199a199')


But I get presented with the error
Unrecognized 'Edm.String' literal 'guid'a199a199-a199-a199-a199-a199a199a199''


So what's going wrong? I tried many different variations, such as:

/api/data/appointments?$select=subject&$filter=regardingobjectid/id eq guid'a199a199-a199-a199-a199-a199a199a199'
/api/data/appointments?$select=subject&$filter=regardingobjectid eq (guid'a199a199-a199-a199-a199-a199a199a199')

etc... I always got 1 of 3 errors, the one previously mentioned, or
Can only bind segments that are Navigation, Structural, Complex, or Collections.

A binary operator with incompatible types was detected. Found operand types 'Edm.Guid' and 'Edm.String' for operator kind 'Equal'


The final error mentioned was by doing a direct comparison
/api/data/appointments?$select=subject&$filter=regardingobjectid eq 'a199a199-a199-a199-a199-a199a199a199'


And it got me thinking, maybe they've changed something. A quick look at oData 4.0 and I find that they've done away with both specifying the object type and the quotes for guids. The query is simply this:
/api/data/appointments?$select=subject&$filter=regardingobjectid eq a199a199-a199-a199-a199-a199a199a199


A sample on this query would have been nice (maybe there is one, I couldn't find it!)... But I'm liking these changes a lot!

Tuesday, 14 July 2015

Dynamics CRM - Developing faster code

We're deep in the days of large corporate multinational CRM systems, integrating with other systems, being interrogated daily by websites and portals, and generally being poked and prodded by many different systems. Because of this we regularly hit performance issues in our systems that in a worst case scenario can grind CRM or the related access points to a halt. Outside of throwing servers at the problem what as a developer can you do? Here are my top 5 tips.

1. Always specify a column set

This is a no-brainer, or at least it should be, always specify a column set when performing a query otherwise you're pulling back every column in the table every time. The amount of times I see the following in code is not funny:
var cols = new ColumnSet(true);

I've even been a perpetrator myself! We really need to get better at specifying our columnsets. Breaking it down to basics, specifying a column set can mean the difference between this:
select * from contact

and this
select contactid, fullname, birthdate from contact

I have seen companies hit the max columns on either contact or account, so that might just give you an idea of what sort of impact specifying all columns will have.

2. Nolock = true

Unfortunately, this is only available on the QueryExpression or via FetchXml in a FetchExpression object. Be aware that this tip isn't necessarily as black and white as "always do this", as there are situations where it will not be feasible.

The upside of setting NoLock = true is that it won't request a record lock during the SQL read and therefore should neither block nor get blocked from making the call. This not only makes the call faster, but makes load balanced servers more efficient as they can read the same records without waiting on the data to be freed from a lock.

There is a downside to setting this which is the possibility of reading uncommitted data or even duplicate data. Depending on what you are reading you may or may not care about this, so you'll need to make a judgement accordingly.

For example, imagine I was reading a settings table to find out if I should display a field in red or blue. I happen to read some uncommitted data that changed the colour and was subsequently rolled back. This usually wouldn't be considered detrimental. In fact, this is probably a good example of when you should consider using NoLock.

On the other hand what if you were reading something more crucial? For example you read an uncommitted price of a stock on the market that in turn caused you to price an option incorrectly due to uncommitted data being rolled back. Probably an unrealistic scenario, but definitely something you would consider as a bigger issue!

The question you need to answer before applying this is - which is worse...
  • A deadlock
  • Inaccurate data
If inaccurate data isn't a huge problem then NoLock is going to offer a favourable solution.

3. OrganizationService.Retrieve locks the record!

This leads on from the previous point, if you perform an OrganizationService.Retrieve you'll end up locking the record on the database. If the retrieve doesn't need to lock then use a QueryExpression (or FetchExpression) to perform a NoLock and retrieve the record that way instead. This will return an entity collection which you'll just need the first record from, but if performance is king this shouldn't be a problem.

4. LINQ locks too!

Again, as with Retrieve, LINQ doesn't allow you to specify a NoLock. Which means you need to use a Query Expression (or FetchExpression) instead. This is particularly annoying as LINQ is such a beautiful way to access CRM, but such a pity you can't gain access to the QueryExpression it creates under the hood. (Well... you can... but not without performing some reflection to invoke a private method... let me know if you'd like to know how to do this)

5. Indexing

Finally, slightly beyond the realm of coding, but still something you could do - take a look at indexing in the database or lack thereof. If you have a DBA team these guys will be able to generate reports on missing indexes for you. On the other hand, if you know of fields that you are constantly searching / filtering then consider adding them to the quickfind views which will in turn create indexes in the background. Getting a DBA to add coverage is ultimately better, but quickfind is a quick (be it slightly dirty!) way to get a field indexed.

Take care though, indexes can deplete performance on other operations, such as inserts. So don't start adding indexes willy nilly all over the database! Something to be aware of.

Wednesday, 27 May 2015

The new LocalPluginContext (and new BasePlugin class)

As a .net developer who often embarks on CRM plugins there's 1 thing I have always took for granted and that is the code provided for plugins when you use the CRM Developer Toolkit for Visual Studio. I have often reviewed the code, performed some minor refactoring on it, and pretty much just overlooked the general logic of it. It's boiler plate stuff that you just learn to accept. But today things were different. Today I decided to read it properly. And let's all admit it, LocalPluginContext and Plugins from the CRM Developer Toolkit need a serious refactor.

Don't get me wrong, I have used and loved this class for years. It wraps up all the objects I need very nicely and provides them in properties for me. But you know what else it does? It makes your plugins un-unit-testable, if I am allowed to create such a word. This is of course if you don't change / rewrite the class yourself.

There are a few mistake that are made by the developer toolkit:

Firstly, it embeds the LocalPluginContext class as a protected class and makes all the important properties internal. Worse again, it privatizes all the property setters on those most used interfaces. Now I cannot mock them with most mocking libraries!

Secondly, it creates a list of registered events to check if the plugin should run for the given message (the infamous tuple that dirties your constructors and has driven me daft for years). This is like a double plugin registration and really isn't required. Think about it, you register your plugin to fire in the Plugin Registration Tool, but if you forget to register it a second time in the constructor it will refuse to execute!

Finally, it needs a serious code review with resharper installed. There are a few redundant statements and lines that could be simplified or even removed.

I think it's time for new Plugin and LocalPluginContext classes and here is my suggestion:

Rework the base plugin class

I don't like the idea of inheriting from a Plugin class that implements IPlugin. It feels wrong to me that we have a class we can register as a plugin, but it's not really a plugin! So I want to rework this as a proper Base Plugin class. And this is my version of it:


    public abstract class BasePlugin : IPlugin
    {
        private readonly string _className;

        protected BasePlugin()
        {
            _className = GetType().Name;
        }

        public void Execute(IServiceProvider serviceProvider)
        {
            // Construct the Local plug-in context.
            var localContext = new LocalPluginContext(serviceProvider);

            localContext.Trace("Entered {0}.Execute()", _className);

            try
            {
                Execute(localContext);
            }
            catch (FaultException<OrganizationServiceFault> e)
            {
                // Trace the exception before bubbling so that we ensure everything we need hits the log
                localContext.Trace(e);

                // Bubble the exception
                throw;
            }
            finally
            {
                localContext.Trace("Exiting {0}.Execute()", _className);
            }
        }

        public abstract void Execute(ILocalPluginContext localContext);
    } 

So we still have the Execute method as before, but instead of an over complicated execution process (using the dreaded tuple) I just expect you to implement an Execute method accepting an ILocalPluginContext when inheriting this class.. So it will work almost exactly as before. Also, although our BasePlugin implements IPlugin it is abstract, so it's not going to appear in the list of available plugins when you load the DLL.

You also may have noticed I have introduced an ILocalPluginContext. This brings me on to my next suggestion.

Rework LocalPluginContext

Another area that always concerned me was how this internal class was written. I'm going to simplify it a little. Rather than pulling everything out of the necessary areas in a constructor let's just create lazy loaded properties. At least then we're only pulling out an IOrganizationService if it's used, a tracer if it's used, etc. I generally expect that all plugins will use a tracer, but quite often you won't need an IOrganizationService, e.g. validation plugins.

I also want to add an interface to this to allow easier mocking while unit testing.

So here's my idea for the LocalPluginContext class:

    public interface ILocalPluginContext
    {
        IOrganizationService OrganizationService { get; }
        IPluginExecutionContext PluginExecutionContext { get; }
        ITracingService TracingService { get; }
        void Trace(string message, params object[] o);
        void Trace(FaultException<OrganizationServiceFault> exception);
    }

    public class LocalPluginContext : ILocalPluginContext
    {
        private readonly IServiceProvider _serviceProvider;
        private IPluginExecutionContext _pluginExecutionContext;
        private ITracingService _tracingService;
        private IOrganizationServiceFactory _organizationServiceFactory;
        private IOrganizationService _organizationService;

        public IOrganizationService OrganizationService
        {
            get
            {
                return _organizationService ?? (_organizationService = OrganizationServiceFactory.CreateOrganizationService(PluginExecutionContext.UserId));
            }
        }

        public IPluginExecutionContext PluginExecutionContext
        {
            get
            {
                return _pluginExecutionContext ??
                       (_pluginExecutionContext = (IPluginExecutionContext)_serviceProvider.GetService(typeof(IPluginExecutionContext)));
            }
        }

        public ITracingService TracingService
        {
            get
            {
                return _tracingService ?? (_tracingService = (ITracingService)_serviceProvider.GetService(typeof(ITracingService)));
            }
        }

        private IOrganizationServiceFactory OrganizationServiceFactory
        {
            get { return _organizationServiceFactory ?? (_organizationServiceFactory = (IOrganizationServiceFactory)_serviceProvider.GetService(typeof(IOrganizationServiceFactory))); }
        }

        public LocalPluginContext(IServiceProvider serviceProvider)
        {
            if (serviceProvider == null)
            {
                throw new ArgumentNullException("serviceProvider");
            }

            _serviceProvider = serviceProvider;
        }

        public void Trace(string message, params object []o)
        {
            if (PluginExecutionContext == null)
            {
                SafeTrace(message, o);
            }
            else
            {
                SafeTrace(
                    "{0}, Correlation Id: {1}, Initiating User: {2}",
                    string.Format(message, o),
                    PluginExecutionContext.CorrelationId,
                    PluginExecutionContext.InitiatingUserId);
            }
        }

        public void Trace(FaultException<OrganizationServiceFault> exception)
        {
            // Trace the first message using the embedded Trace to get the Correlation Id and User Id out.
            Trace("Exception: {0}", exception.Message);

            // From here on use the tracing service trace
            SafeTrace(exception.StackTrace);

            if (exception.Detail != null)
            {
                SafeTrace("Error Code: {0}", exception.Detail.ErrorCode);
                SafeTrace("Detail Message: {0}", exception.Detail.Message);
                if (!string.IsNullOrEmpty(exception.Detail.TraceText))
                {
                    SafeTrace("Trace: ");
                    SafeTrace(exception.Detail.TraceText);
                }

                foreach (var item in exception.Detail.ErrorDetails)
                {
                    SafeTrace("Error Details: ");
                    SafeTrace(item.Key);
                    SafeTrace(item.Value.ToString());
                }

                if (exception.Detail.InnerFault != null)
                {
                    Trace(new FaultException<OrganizationServiceFault>(exception.Detail.InnerFault));
                }
            }
        }

        private void SafeTrace(string message, params object[] o)
        {
            if (string.IsNullOrWhiteSpace(message) || TracingService == null)
            {
                return;
            }
            TracingService.Trace(message, o);
        }
    }

One extra feature I have added is how it traces. The new LocalPluginContext contains a function that performs better tracing of FaultExceptions (ToString does not cut it) and I've matched the standard trace to how the tracing service works and included a param of objects.

What the new plugins look like!

A basic plugin looks like this which in my opinion is a lot cleaner:

    public class MyPlugin : BasePlugin
    {
        override public void Execute(ILocalPluginContext localContext)
        {
            // do what needs to be done!
        }
    }

One big upside to doing it this way is we have a much more testable framework where we simply pass in a mocked up ILocalPluginContext.

Friday, 23 January 2015

CRM 2011/2013 - SQLAgent service is not running

Quick one from me today. Quite often I come across this issue where it tells me the SQLAgent service is not running on the server when trying to import an organization or install a CRM server.

9 times out of 10 this comes down to user rights. I believe the right it is being denied is interrogating the the windows server to get a list of services running so that it can check for SQL Agent. Which means adding the user as a sys admin to SQL often won't help.

Try adding the user to the Local Admins group on the SQL box as well and this error should go away.

Friday, 24 October 2014

I'm done with story points, velocity and sprints

Ok, saying I'm done with it might be harsh, but there has to be a better way to run a project. I've been fairly committed to agile for quite a few years now and one of my main bugbears is slowly becoming story points, velocity and sprints. I have multiple problems with all of these, but my main gripe is that on most projects they break agile. Let's take each and analyse what they are supposed to give you versus what you actually witness in a project

Story points

Story points are a method to size a piece of work without having to get bogged down in how long it will take. Let's say the story is "Filling a glass of Orange Juice".

  • Time based approach: how long will it take to pour a glass of orange juice?
  • Story points approach: "let's estimate what size the glass is". 


With both approaches you have inaccuracies. The first might incorrectly estimate the speed of the pour, while the second might not size the glass correctly. But the reason we moved away from a time based approach is due to developers inability to provide accurate time estimates, we just seem to be better at sizing something as opposed to giving it a time estimation. You've all heard different equations for estimating a story or task length - take a developers estimate and double it, or even multiply it by 2.5. We are notorious underestimaters even given this knowledge! Points based estimates are supposed to fix this because you estimate the size or complexity of the story instead. The theory is it's easier to estimate roughly how big a glass of orange juice is than to estimate roughly how long it takes to fill.

Also, some developers are faster than others. So if you quote a glass of orange filling at 2 days, it might take a junior developer 3 or 4 days to do the same story. If we had it estimated in points then that solves the problem. 2 points to a junior developer can mean a different thing to a senior developer and we can account for that by giving the junior devs (as a resource) a lower sprint velocity. If we lose them from the sprint then it's easier to account for what the loss in points will be.

This sounds fantastic, but doesn't work quite as awesome as you'd think. You've been in those planning meetings where the developers are thinking about a story, prodding the BAs for answers and eventually come to their conclusion. They all raise their cards and the points average out at 5. BAs and project managers shout "How on earth can that be a 5 point story?!?". This completely misses a couple of points of what the estimate is. It's a story size, as in agreement by the developers. What exactly is 5 points anyway? It's just a number that defines story complexity. But quite often you'll get argued down and the story ends up being 3 points. And suddenly you have your 50 point sprint full of underestimated stories and only complete 75% of the sprint. It's almost like the reverse of the stereotypical developer estimates happens, developers give their estimates and rather than the old school "double it!" we get a "half it!" approach.

There are more problems than this, but is better thought of in terms of Velocity.

Velocity

As I said, what exactly are story points anyway? And what exactly is velocity? To give you an idea of the misconceptions I have witnessed I will paraphrase a statement I head in a recent project I was involved in:

'We should have a velocity of 50 points per sprint.'

This instantly rose suspicion with me as we hadn't even hit 3 sprints and were actually performing a little lower than that. It's also a remark usually associated with a misunderstanding of what points and velocity means. I poked about with a few questions and eventually discover that the reason the sprints should have a 50 point velocity is due to another similar-ish project in the company that had a 50 point velocity.

So, let's look at the question again, what exactly are story points anyway? Let's go back to our story "Filling a glass of Orange Juice":

  • Team A estimate this as 5 points
  • Team B estimate this as 3 points
  • Team C estimate this as 8 points


Sprint starts:

  • Team A fill the glass of orange juice. It takes them 2 days (it was a very big glass!). 
  • Team B, also start to fill a glass of orange juice. It also takes them 2 days even though they pointed it lower! 
  • Team C fills their glass. Add a dash of mango to it because they have a pretentious client and then paint some fancy designs on the glass. They were probably right to point it at 8


There's 2 things I'm trying to get across here.

  • Firstly Team A and B quoted different points for what was effectively the same complexity of work. This really doesn't matter. I would expect Team B to also maintain a lower velocity, but that doesn't mean they actually produce any less work. They will have the same output, they just estimate and process points at a different rate.
  • Also, look at Team C versus the other 2 first. Although a story might seem like the same piece of work sometimes it's not. It can depend on technology, who your developers are, etc.

Effectively, as far as different projects are concerned velocity is only relevant to that project. You shouldn't try to impose another projects velocity (or even points) on another independent project.

Up until this point I can actually work with all the concepts. The issue I have with points and velocity is a misunderstanding of the concept as opposed to something more fundamental. But I would add that I'm frustrated of having to re-educate people every time I join a new agile project.

Sprints

Sprints are something I have really become less fond of as time has elapsed. Taking a look at the definition of a sprint on the face of it all they seem like a good idea (taken from wikipedia):

'A sprint (or iteration) is the basic unit of development in Scrum. The sprint is a "timeboxed" effort; that is, it is restricted to a specific duration. The duration is fixed in advance for each sprint and is normally between one week and one month, although two weeks is typical.

'Each sprint is started by a planning meeting, where the tasks for the sprint are identified and an estimated commitment for the sprint goal is made, and ended by a sprint review-and-retrospective meeting, where the progress is reviewed and lessons for the next sprint are identified.

'Scrum emphasizes working product at the end of the Sprint that is really "done"; in the case of software, this means a system that is integrated, fully tested, end-user documented, and potentially shippable.'

The problem I have is when the following 3 concepts are bundled into the same process:

  • Timeboxed
  • Story commitment
  • Shippable product

My first issue is this, if you're going to "timebox" the work you are presuming that every single deliverable can be developed in your timeboxed period. In every project I have worked on to date about 10% of the sprints have truly delivered something shippable. Most sprints finish with incomplete stories being moved into following sprints, or partially delivered features due to your sprint periods being over-restrictive.

My second issue is with the commitment to sprint tasks/stories. Generally, when the sprint content has been defined or stories "accepted" into the sprint, you're not really supposed to change it. We all throw this concept out fairly early, because that's the first bit that really breaks agile. The reason behind not being allowed to change the sprint when it has already started is you'd never get any work done if people kept changing the stories and work from beneath your feet. But there has to be some kind of flexibility here. Priorities change frequently in fast moving projects so you frequently have to move stuff in and out. This means you have to either:

  • Estimate not just stories that can fit in a sprint, but above and beyond that
  • Or estimate new stories as and when we need to push stuff in and out of sprints

Now when the new stories that come in... do they "fit" with our deliverable? Maybe they do, maybe they don't. Either way, to be truly agile we really should allow priorities to change and allow work to get completed when it's needed. But we are dancing a fine line with our shippable product here!

Which takes me on to my final issue - the "Potentially shippable product". Sprints don't help this, they hinder it. What you end up doing is shoehorning X points worth of stories into a sprint and calling it your "shippable" piece of work. When really what you frequently end up delivering is partially complete features that no end user would really want to use (yet).

Alternative?

While writing this post I came across this blog post Stop using story points. Looks like I'm not the only one frustrated with this, and I'm a little late to the game! I have to say I love the concept of estimating in Apples and Oranges. I also love the idea of variable length iterations. I wish I could claim that idea, but alas I cannot!

Flexible iterations is probably the alternative I will seek from here on. One of the biggest advantages is you can define what you're going to ship rather than allowing your iteration to define this for you. Pick something small and manageable, like 2 or 3 features (depending on your feature sizes) that will complete a few pages, or give you a functional service. This doesn't mean you have to stop using points. If this is something you're already comfortable with then keep using them. You can also still calculate velocity, but maybe create a "weekly" velocity so that you can easily calculate your iteration lengths. But the point is pick something shippable you will complete and then estimate the iteration.

Basically, what I'm suggesting is, let's get back to being agile. Let's not taint such a great methodology with unagile practices and in result giving it a bad name.