Monday, 23 May 2016

Why you should use CRM recommended values for optionsets

Sometimes when I'm on a project I encounter CRM consultants determined to use less obscure values for their optionsets than the CRM recommended for your publisher. I.e. 1, 2, 3, 4... You might think this won't cause massive headaches, but I will give you 3 pretty big reasons why I recommend you shouldn't do this.

Conflicting solutions

The most obvious reason is another company also wants to use the same values as you. This will go unnoticed in custom fields, but as soon as you both customise the values of an out of the box optionset, like accountcategorycode, you'll quickly run into problems. Let me show you what happens. In the following example I set up a couple of new options on accountcategorycode and changed the value of "New Option 1" to 3. I have left "New Option 2" as it's recommended value for the purpose of this sample:



Another publisher is also using these same values rather than the recommended in a managed solution. What you end up with is an overwrite on the options with the shared values:



As you might notice our option "New Option 1" has been renamed to "Other 1". This might not seem detrimental in this particular example, but what if the change was from "Bad Debt Customer" to "Most Awesome Customer" or something to that effect. This should be enough to prevent people from doing this, but unfortunately it's usually swept under the carpet as a really unlikely scenario.

Status Codes

The next issue is inconsistent usage introduced due to fields like status code. You may not realise this when you embark on your adventure to ignore the CRM recommended values, but you cannot apply this standard to all option sets, in particular status codes:



CRM disables the value field so you're stuck with what it recommends and you've got an inconsistent standard. So should you just make an exception to use recommended values for status codes, but continue on your own standard for other fields? Or do you extend this exception to all out of the box fields? There are few things I hate more than inconsistency in naming and value conventions so I say just stick with the CRM recommended for all fields.

Annoying message boxes

The next big reason you shouldn't do this is it becomes incredibly frustrating to add new options to an optionset if you want to maintain this standard. Even more so when you're adding a lot of new options. You will get bombarded with a message like the following every time you change the number for an option:



Ultimately why not just stick with the CRM recommended. I know the values are sometimes unreadable and a pain to copy & paste in javascript / code, but these values are not visible to end users, only to us developers/technical consultants. So the impact of leaving them as the recommended values is relatively minimal.

Friday, 20 May 2016

CRM 2016 - Synchronous Workflow bug

Today I spent quite some time working on a bug in one of our workflows which ended up being a bug within CRM. The error I was getting was
The given key was not present in the dictionary.
Let me show you how to recreate and how to spot that this is infact an issue within CRM as opposed to your code.

In our CRM we have a concept of an Attendee which is how we link people to a meeting. We also have the concept of a Target entity which is a parent of an Attendee. The lookup to Target on Attendee is not mandatory as some attendees are just regular people (i.e. not linked to a "Target"). When an Attendee is linked to a meeting I want to check a target has been set and and if it is do some "stuff" based on Target information. So I create a workflow with just this check in it:



For now I am not going to add anything else as this is all I need to highlight the bug. Next, as with a lot of systems, we have bad data and not all Targets have a name. For example:



I run my workflow in its current state against this record and I get an error

:

Downloading the log file gives you a trace like this:

[Microsoft.Crm.ObjectModel: Microsoft.Crm.Extensibility.InternalOperationPlugin]
[46f6cf4c-14ae-4f1e-98a1-eae99a37e95c: ExecuteWorkflowWithInputArguments]
Starting sync workflow 'CG - workflow with bug', Id: e717769d-8c1e-e611-80f4-5065f38aa981
Entering ConditionStep1_step: Check Attendee.Target contains data
Sync workflow 'CG - workflow with bug' terminated with error 'The given key was not present in the dictionary.'

I'm not sure what the CRM workflows code is doing under the hood here, but it must be trying to reference the name field in code. If you set a value for Name on the Target the problem goes away.
Pro tip: Put comments in your workflows! They are included as part of the trace



Thursday, 19 May 2016

Automapper, Dynamics CRM and excluding fields - Part 2

In my previous post, Automapper, Dynamics CRM and excluding fields, I introduced a concept of an "Excludable property". This is just 1 side of the call - POSTing/PUTtting records using a REST API. What about a GET? If you are using something like excludables you'll notice that the JSON returned does not look like the proposed JSON you POST or PUT. In fact, it looks something like this:

{
    Id:
    {
        Include: true,
        Value: "aef7b4c1-98f6-4f53-9be3-2fa72d1e319d"
    },
    Name:
    {
        Include: true,
        Value: "Hello"
    },
    Address1_Line1:
    {
        Include: true,
        Value: "Home!"
    }
}

Which is how our Excludables map to JSON. Here do we stop this?

Extend the IExcludable interface

To convert our excludables correctly we need to intercept the conversion and handle these properties manually. The first problem we hit is although we know it's an Excludable<> we don't know what the raw type is. The best way around this is to expand the IExcludable interface to allow exposing of a raw value, like this:

    public interface IExcludable
    {
        bool Include { get; set; }
        object RawValue { get; set; }
    }

The Excludable<> class just implements it on top of the existing value field, like this:

        public object RawValue
        {
            get
            {
                return value;
            }
            set { this.value = (T) value; }
        }


View Model Json Converter

Now that we can find the raw value without needing to know the underlying generic type we can intercept any excludable and convert it. This is the full converter class that results:

public class ViewModelJsonConverter : JsonConverter
    {
        public override bool CanRead => false;

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            JObject o = JObject.Parse(JsonConvert.SerializeObject(value, Formatting.Indented,
                new JsonSerializerSettings {ReferenceLoopHandling = ReferenceLoopHandling.Ignore}));
            foreach (var propertyInfo in value.GetType().GetProperties())
            {
                if (propertyInfo.CanRead)
                {
                    var currentValue = propertyInfo.GetValue(value);
                    IExcludable excludable = currentValue as IExcludable;
                    if (excludable != null)
                    {
                        if (excludable.Include)
                        {
                            if (excludable.RawValue == null || excludable.RawValue.GetType().IsValueType || excludable.RawValue.GetType().Name == "String")
                            {
                                o.Property(propertyInfo.Name).Value = new JValue(excludable.RawValue);
                            }
                            else
                            {
                                o.Property(propertyInfo.Name).Value = JObject.FromObject(excludable.RawValue);
                            }
                        }
                        else
                        {
                            o.Remove(propertyInfo.Name);
                        }
                    }
                }
            }
            o.WriteTo(writer);
        }

        public override bool CanConvert(Type objectType)
        {
            if (objectType.BaseType == typeof (ViewModel))
            {
                return true;
            }
            return false;
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            throw new System.NotImplementedException();
        }
    }

In the above code our view models code had inherited a ViewModel class. Also worth pointing out is how you handle the both value and object types (with strings needing an extra helping hand as they're a bit different!). For value types and strings you need to create a JValue where as for reference types you need to expose them as a JObject.

Now, just add this converter like so in your global.asax exactly how you added your excludable converter from the previous post:

GlobalConfiguration.Configuration.AddJsonConverter(new ViewModelJsonConverter());


This code effectively flattens the excludable class into the generic types and outside of the world of your REST api nobody is any the wiser.

Monday, 7 March 2016

Automapper, Dynamics CRM and excluding fields

In the past when I've built web based applications Automapper was always one of those libraries that I both loved and hated at the same time. In the Dynamics CRM world it can often be a bit of a dangerous tool to unleash on a website, especially when utilised by developers that don't know CRM very well. Let me explain why.

Attribute Collections

Most CRM Developers will already know what I'm talking about here, but for those of you not privy to this - the properties of an entity from a CRM database table are not actually surfaced quite like EntityFramework or nHibernate, they are surfaced using a dictionary. This is subsequently wrapped in an Attribute Collection. One of the main advantages of this is you can control partial updates/retrieves without worrying about the state of the entire entity. On the negative side this plays havoc when libraries like Automapper are used to populate the entities, especially when using early bound objects.

Here's a sample to give you and idea of what I'm getting at. Firstly, let's presume our domain models are Early Bound objects exported from dynamics. (self plug! Personally I use this open source tool : https://github.com/conorjgallagher/Dynamics.ExtendedSvcUtil).

Now, let's say we have an REST service and we want to utilise it to update an account. So we build a view model like this:

public class AccountViewModel
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public string Address1_Line1 { get; set; }
}

And we wire up automapper to map that across to our domain entity. In the latest version I believe that would look like this:

CreateMap<Account, AccountViewModel>().ReverseMap()

In this example we can create a new account via the this REST service with the following JSON:

{
    Id: "aef7b4c1-98f6-4f53-9be3-2fa72d1e319d",
    Name: "Hello",
    Address1_Line1: "Home!"
}

Within our controller we will receive an AccountViewModel populated with all the relevant data. We can then use Automapper to push this View Model into an early bound Account entity using the same method as above. As long as all the field names match we're good to go. Even though you are mapping to strongly typed fields what you end up with under the hood is an dictionary like this:

"accountid"="aef7b4c1-98f6-4f53-9be3-2fa72d1e319d"
"name"="Hello"
"address1_line1"="Home!"

Null vs Unset

A really nice feature of CRM is it differentiates between null and unset. If you exclude a field from the attribute collection it will also be excluded from the update statement that hits SQL. This is very useful for performance and limiting what plugins / workflows fire.

Back to Automapper and our view models above - a problem arises when you exclude fields from the JSON.  For example, if you subsequently send the following after the previous update:

{
    Id: "aef7b4c1-98f6-4f53-9be3-2fa72d1e319d",
    Address1_Line1: "Home line 1!"
}

This will hit our view model as this:

{
    Id = "aef7b4c1-98f6-4f53-9be3-2fa72d1e319d",
    Name = null,
    Address1_Line1 = "Home line 1!"
}

Which I guess is expected, because how else can you represent an excluded value in the view model? If we don't deal with this we hit a more fundamental issue further down the chain in that our attribute collection will end up like this:

"accountid"="aef7b4c1-98f6-4f53-9be3-2fa72d1e319d"
"name"=null
"address1_line1"="Home line 1!"

And we'll blank our account name in CRM. Not good!

Excludable

This takes me on to a pattern I would like to propose for this type of issue: the concept of an excludable property. Looking at how nullable value types work surely we can come up with a similar concept! I won't take you through the entire evolution, but here is the interface and struct I now propose:

    public interface IExcludable
    {
        bool Include { get; set; }
    }

    public struct Excludable<T> : IExcludable
    {
        private bool hasValue;
        internal T value;
        private bool include;


        public Excludable(T value)
        {
            this.value = value;
            if (value != null)
            {
                this.hasValue = true;
            }
            else
            {
                this.hasValue = false;
            }
            this.include = true;
        }

        public bool HasValue
        {
         
            get
            {
                return hasValue;
            }
        }

        public bool Include
        {
            get { return include; }
            set { include = value; }
        }

        public T Value
        {
            get
            {
                return value;
            }
        }

   
        public T GetValueOrDefault()
        {
            return value;
        }

     
        public T GetValueOrDefault(T defaultValue)
        {
            return hasValue ? value : defaultValue;
        }

        public override bool Equals(object other)
        {
            if (!include || !hasValue) return other == null;
            if (other == null) return false;
            return value.Equals(other);
        }

        public override int GetHashCode()
        {
            return hasValue ? value.GetHashCode() : 0;
        }

        public override string ToString()
        {
            return hasValue ? value.ToString() : "";
        }

        public static implicit operator Excludable<T>(T value)
        {
            return new Excludable<T>(value);
        }

        public static explicit operator T(Excludable<T> value)
        {
            return value.Value;
        }

        public static bool operator ==(Excludable<T> x, T y)
        {
            return x.Equals(y);
        }

        public static bool operator !=(Excludable<T> x, T y)
        {
            return !x.Equals(y);
        }
    }

Overriding all the comparisons were required to get it to perform both value and null comparisons correctly. Finally, we can change our view model to use this instead:

    public class AccountViewModel
    {
        public Excludable<Guid> Id { get; set; }
        public Excludable<string> Name { get; set; }
        public Excludable<string> Address1_Line1 { get; set; }
    }

Json Converter gotcha

There's 1 final problem we need to solve that I'll highlight now. If we run our object without the address in it all seems to work just fine. Our property comes through as excluded as expected. But, if we instead try set it to null it is marked as included=false! The reason is down to how the JSON formatter deserializes the data into the given object. It doesn't actually call your constructor but does something dirty under the hood. How do we fix this? Create a custom json converter

    public sealed class ExcludableConverter : JsonConverter
    {
        public override bool CanConvert(Type objectType)
        {
            return objectType.IsGenericType && objectType.GetGenericTypeDefinition() == typeof(Excludable<>);
        }
        public override bool CanWrite { get { return false; } }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            return Activator.CreateInstance(objectType, reader.Value);
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            throw new NotImplementedException();
        }
    }

And set this up on application start within your global.asax:

            var formatters = GlobalConfiguration.Configuration.Formatters;
            var jsonFormatter = formatters.JsonFormatter;
            var settings = jsonFormatter.SerializerSettings;
            jsonFormatter.SerializerSettings.Converters.Add(new ExcludableConverter());

Now, when a null comes in we explicitly tell the converter to create a new object for us.

Wiring up Automapper

The final step is getting automapper to utilise this new way of excluding items. In automapper you can set a condition when mapping so that you can do conditional ignores. I found the simplest way to add this rule was write an extension method:

public static void AddExcludableRule<S, D>(this IMappingExpression<S, D> m)
        {
            m.ForAllMembers(opt => opt.Condition(
                    s =>
                        !(s.SourceValue is IExcludable) || ((IExcludable)s.SourceValue).Include
                    ));
        }

Now, whenever I want to wire up an object that has excludables I just add that call to the end:

CreateMap<Account, AccountViewModel>().ReverseMap().AddExcludableRule();



Thursday, 25 February 2016

Dynamics CRM Web API - Pluralization bug

I started investigating the Web API while it was in preview mode so have noticed it move quite quickly over time. Mainly this is changes but sometimes it's bugs. For example, a change I spotted since my previous post (Web API Preview - Unrecognized 'Edm.String' literal...) is how it deals with lookup and regarding references. In the preview you performed the following:
/api/data/appointments?$select=subject&$filter=regardingobjectid eq a199a199-a199-a199-a199-a199a199a199

This has change to the following in the v8.0 release:
/api/data/v8.0/appointments?$select=subject&$filter=_regardingobjectid_value eq a199a199-a199-a199-a199-a199a199a199

The second thing I spotted is in fact a bug. This may have existed in the preview, but I never spotted it and it had me scratching my head for quite some time. We have the concept of an "event day" which we have name "inv_eventday". When creating this entity CRM creates a collection schema to allow you to reference it via the web api. For example the appointment entity has an equivalent collection schema called "appointments"; notice the pluralisation.

So, on to my event days. What you would expect to find as a collection schema is this:
/api/data/v8.0/inv_eventdays

Unfortunately, this is not the case. You get an error "Resource not found for the segment 'inv_eventdays'." I wondered if I had a case sensitivity issue or something, but every combination of captitals / camel case / pascal case just didn't seem to work.

After some time I decided I should check out the metadata to see if I could uncover the correct casing. For reference the following is how you get a list of all entity metadata:
/api/data/v8.0/EntityDefinitions

This uncovered a definition with the following schema name for the entity:
"SchemaName":"inv_eventday"

Which seems to match my expectations on what it should be. So what's the problem? Reading further down I find the Collection Schema name:
"CollectionSchemaName":"inv_eventdaies"

That's some funky plural we've got there! Looks like the algorythm decides that anything ending in "y" is pluralised as "ies". Obviously this isn't correct for all words. E.g. the word "sky" is indeed "skies", but "day" is in fact "days".

Now... where do you log CRM bugs...


Thursday, 3 December 2015

Web API Preview - Unrecognized 'Edm.String' literal...

Today I was playing around with the new Web API Preview to try and check out how the new REST API was going to work. If you're interested take a look here: https://msdn.microsoft.com/dynamics/crm/webapipreview. At a first glance this looks really good. It's much cleaner and does away with a long unnecessary url and the case sensitivity that always bugged me.

The first issue I hit was when trying to filter by a regarding object on an appointment. My first thought was is this not supported in the preview? I took a standard oData query from the 2011 endpoint and converted it so that this:
/xrmservices/2011/OrganizationData.svc/AppointmentSet?$select=Subject&$filter=RegardingObjectId/Id eq (guid'a199a199-a199-a199-a199-a199a199a199')

became this:
/api/data/appointments?$select=subject&$filter=regardingobjectid/id eq (guid'a199a199-a199-a199-a199-a199a199a199')


But I get presented with the error
Unrecognized 'Edm.String' literal 'guid'a199a199-a199-a199-a199-a199a199a199''


So what's going wrong? I tried many different variations, such as:

/api/data/appointments?$select=subject&$filter=regardingobjectid/id eq guid'a199a199-a199-a199-a199-a199a199a199'
/api/data/appointments?$select=subject&$filter=regardingobjectid eq (guid'a199a199-a199-a199-a199-a199a199a199')

etc... I always got 1 of 3 errors, the one previously mentioned, or
Can only bind segments that are Navigation, Structural, Complex, or Collections.

A binary operator with incompatible types was detected. Found operand types 'Edm.Guid' and 'Edm.String' for operator kind 'Equal'


The final error mentioned was by doing a direct comparison
/api/data/appointments?$select=subject&$filter=regardingobjectid eq 'a199a199-a199-a199-a199-a199a199a199'


And it got me thinking, maybe they've changed something. A quick look at oData 4.0 and I find that they've done away with both specifying the object type and the quotes for guids. The query is simply this:
/api/data/appointments?$select=subject&$filter=regardingobjectid eq a199a199-a199-a199-a199-a199a199a199


A sample on this query would have been nice (maybe there is one, I couldn't find it!)... But I'm liking these changes a lot!

Tuesday, 14 July 2015

Dynamics CRM - Developing faster code

We're deep in the days of large corporate multinational CRM systems, integrating with other systems, being interrogated daily by websites and portals, and generally being poked and prodded by many different systems. Because of this we regularly hit performance issues in our systems that in a worst case scenario can grind CRM or the related access points to a halt. Outside of throwing servers at the problem what as a developer can you do? Here are my top 5 tips.

1. Always specify a column set

This is a no-brainer, or at least it should be, always specify a column set when performing a query otherwise you're pulling back every column in the table every time. The amount of times I see the following in code is not funny:
var cols = new ColumnSet(true);

I've even been a perpetrator myself! We really need to get better at specifying our columnsets. Breaking it down to basics, specifying a column set can mean the difference between this:
select * from contact

and this
select contactid, fullname, birthdate from contact

I have seen companies hit the max columns on either contact or account, so that might just give you an idea of what sort of impact specifying all columns will have.

2. Nolock = true

Unfortunately, this is only available on the QueryExpression or via FetchXml in a FetchExpression object. Be aware that this tip isn't necessarily as black and white as "always do this", as there are situations where it will not be feasible.

The upside of setting NoLock = true is that it won't request a record lock during the SQL read and therefore should neither block nor get blocked from making the call. This not only makes the call faster, but makes load balanced servers more efficient as they can read the same records without waiting on the data to be freed from a lock.

There is a downside to setting this which is the possibility of reading uncommitted data or even duplicate data. Depending on what you are reading you may or may not care about this, so you'll need to make a judgement accordingly.

For example, imagine I was reading a settings table to find out if I should display a field in red or blue. I happen to read some uncommitted data that changed the colour and was subsequently rolled back. This usually wouldn't be considered detrimental. In fact, this is probably a good example of when you should consider using NoLock.

On the other hand what if you were reading something more crucial? For example you read an uncommitted price of a stock on the market that in turn caused you to price an option incorrectly due to uncommitted data being rolled back. Probably an unrealistic scenario, but definitely something you would consider as a bigger issue!

The question you need to answer before applying this is - which is worse...
  • A deadlock
  • Inaccurate data
If inaccurate data isn't a huge problem then NoLock is going to offer a favourable solution.

3. OrganizationService.Retrieve locks the record!

This leads on from the previous point, if you perform an OrganizationService.Retrieve you'll end up locking the record on the database. If the retrieve doesn't need to lock then use a QueryExpression (or FetchExpression) to perform a NoLock and retrieve the record that way instead. This will return an entity collection which you'll just need the first record from, but if performance is king this shouldn't be a problem.

4. LINQ locks too!

Again, as with Retrieve, LINQ doesn't allow you to specify a NoLock. Which means you need to use a Query Expression (or FetchExpression) instead. This is particularly annoying as LINQ is such a beautiful way to access CRM, but such a pity you can't gain access to the QueryExpression it creates under the hood. (Well... you can... but not without performing some reflection to invoke a private method... let me know if you'd like to know how to do this)

5. Indexing

Finally, slightly beyond the realm of coding, but still something you could do - take a look at indexing in the database or lack thereof. If you have a DBA team these guys will be able to generate reports on missing indexes for you. On the other hand, if you know of fields that you are constantly searching / filtering then consider adding them to the quickfind views which will in turn create indexes in the background. Getting a DBA to add coverage is ultimately better, but quickfind is a quick (be it slightly dirty!) way to get a field indexed.

Take care though, indexes can deplete performance on other operations, such as inserts. So don't start adding indexes willy nilly all over the database! Something to be aware of.