An Unexpected Roadblock
I ran into a an unexpected roadblock the other day when refactoring code that controls JSON de-serialization. It forced me to read up on the dreaded topic of covariance and contravariance in C#. During code review, I was asked to change the technique I used to specify how JSON should be de-serialized into a root type that includes properties typed as interfaces. The conundrum is what classes should be instantiated for these properties?
I solved this problem using Json.NET JsonSerializerSettings.TypeNameHandling. This embeds type names into JSON text as properties named “$type”. When the JSON text is de-serialized, a type name (if present) indicates which class should be instantiated for the object it annotates. A simple one line solution. I was asked to use Json.NET’s JsonConverter class / attribute instead. This also is a fairly simple solution- and a perfectly reasonable request. The choice between these two techniques comes down to where you’d prefer to manage backwards-compatibility (1)(2).
Problems Caused by Unnecessary Interfaces
I ran into a covariance issue when writing a JsonConverter for a generic Dictionary class. While I got an interesting blog topic out of the ordeal, I was frustrated by the fact I would not have run into this roadblock if I wasn’t directed to define interfaces for all Record classes. That decision- unlike the TypeNameHandling versus JsonConverter decision- I view as architecturally unsound. I’ve blogged about the Record Pattern in a previous post. I see it as a data persistence mechanism, and therefore, an implementation detail that should be encapsulated within the domain class DLL. Public interfaces are used to describe class structure to external code. So I’d never write interfaces for Record classes because I’d never expose Record classes to external code. But I digress. The important points to remember are…
- The covariance issue occurs because of the Record interface.
- I’m not allowed to solve the problem by eliminating the Record interface.
Crash and Burn
Let me demonstrate the issue. I’ll use neither technique to specify class instances and we’ll crash and burn. First, let’s define some interfaces and classes for a nonsense domain.
I intend to serialize and de-serialize a ToolboxRecord class. So let’s write version 1 of that class.
Next, I’ll write a program that reads an input argument and instantiates a version of the ToolboxRecord class (we have only V1 now) but assigns it to a variable typed as an IToolboxRecord interface. This is simple polymorphism, no sweat. Then, serialize the record to JSON and write it to a file on the local disk. Finally, read JSON from the file and attempt (hint: something bad may happen here) to de-serialize it and assign it to the IToolboxRecord variable.
Notice I’ve commented out support for the Thingamajig property in the toolbox record and program. That’s because supporting JSON serialization of the Sprocket, Widget, and Thingamajig properties is increasingly difficult. Let’s not attempt to support the IDictionary<Orientation, IList<IThingamajigRecord>> Thingamajigs { get; set; } property now. We will eventually.
Run the program. It saves the following JSON to disk.
It fails to de-serialize the JSON due to the following exception.
PS C:\Users\Erik\...\Json IList Covariance> dotnet run -c release -- 1Exception Type = System.Exception Exception Message = Failed to de-serialize JSON to IToolboxRecord. Exception StackTrace = at ErikTheCoder.Sandbox.Covariance.Program.Run(IReadOnlyList`1 Arguments) in C:\Users\Erik\Documents\Visual Studio 2019\Projects\Sandbox\Json IList Covariance\Program.cs:line 53 at ErikTheCoder.Sandbox.Covariance.Program.Main(String[] Arguments) in C:\Users\Erik\Documents\ Visual Studio 2019\Projects\Sandbox\Json IList Covariance\Program.cs:line 21 Exception Type = Newtonsoft.Json.JsonSerializationException Exception Message = Could not create an instance of type ErikTheCoder.Sandbox.Covariance.ISprocketRecord Type is an interface or abstract class and cannot be instantiated. Path 'Sprockets[0].Foo', line 4, position 12. Exception StackTrace = at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject (JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
Json.NET complains it doesn’t know what class to instantiate for each item in the Sprockets list. The list items are typed as ISprocketRecord and in C# it’s not possible to instantiate an interface. Json.NET aborts de-serialization at the first error so the exception message is not an exhaustive list of problems with the JSON. If the Widgets property appeared first in the JSON the de-serializer would have complained it can’t instantiate an IWidgetRecord interface.
The TypeNameHandling Solution
Let’s fix this bug using the first technique I used, JsonSerializerSettings.TypeNameHandling. Version 2 of the ToolboxRecord class is identical to version 1. What differs are the serializer settings in Program.cs. Lines 6 and 7 below.
Run the program. It saves the following JSON to disk.
“ErikTheCoder.Sandbox.Covariance.SprocketRecord, ErikTheCoder.Sandbox.Covariance” means instantiate the SprocketRecord class from the ErikTheCoder.Sandbox.Covariance namespace in the ErikTheCoder.Sandbox.Covariance.dll. The program successfully reads JSON from the file and de-serializes it.
PS C:\Users\Erik\...\Json IList Covariance> dotnet run -c release -- 2 Successfully deserialized JSON from C:\Users\Erik\Documents\Temp\Toolbox.json to ErikTheCoder.Sandbox.Covariance.V2.ToolboxRecord in toolboxRecord variable. Toolbox record has 2 sprocket records. Toolbox record has 2 widget records.
The JsonConverter Solution
I was asked to use Json.NET’s JsonConverter class / attribute instead of its JsonSerializerSettings.TypeNameHandling setting. Our codebase already included a ListConverter. I followed the pattern and wrote a DictionaryConverter.
Then I applied the attribute to version 3 of the ToolboxRecord class.
Add a line to the Program.CreateToolboxRecord method for version 3 that’s identical to version 1. 3 => (new V3.ToolboxRecord(), new JsonSerializerSettings { Formatting = Formatting.Indented }). Run the program. It saves JSON to disk that is identical to the JSON produced by version 1. The program successfully reads JSON from the file and de-serializes it.
PS C:\Users\Erik\...\Json IList Covariance> dotnet run -c release -- 3 Successfully deserialized JSON from C:\Users\Erik\Documents\Temp\Toolbox.json to ErikTheCoder.Sandbox.Covariance.V3.ToolboxRecord in toolboxRecord variable. Toolbox record has 2 sprocket records. Toolbox record has 2 widget records.
A JsonConverter IList Covariance Problem
So far so good. We’ve refactored the serialization technique and the code functions correctly. However, we’ve only tested two of the three interface properties we must support. Now let’s add code to populate the Thingamajig dictionary in Program.PopulateToolboxRecord.
Uncomment the Thingamajig property and add the DictionaryConverter attribute to version 3 of the ToolboxRecord.
Attempt to compile the code.
PS C:\Users\Erik\...\Json IList Covariance> dotnet build -c release
Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
Determining projects to restore...
All projects are up-to-date for restore.
V3\ToolboxRecord.cs(17,104): error CS0311: The type 'System.Collections.Generic.List<ErikTheCoder.Sandbox.Covariance.
ThingamajigRecord>' cannot be used as type parameter 'TValue' in the generic type or method 'DictionaryConverter
<IKey, IValue, TKey, TValue>'. There is no implicit reference conversion from 'System.Collections.Generic.
List<ErikTheCoder.Sandbox.Covariance.ThingamajigRecord>' to 'System.Collections.Generic.IList<ErikTheCoder.
Sandbox.Covariance.IThingamajigRecord>'.
[C:\Users\Erik\Documents\Visual Studio 2019\Projects\Sandbox\Json IList Covariance\Json IList Covariance.csproj]
Build FAILED.
It fails to compile because there’s no implicit conversion from a list of Thingamajigs to an interface-typed list of interface-typed Thingamajigs. I attempted to solve the problem by requiring more generic type parameters and more constraints on those parameters. I wrote version 4 like this:
I named a generic type parameter “IIList” to avoid a name collision with the “IList” interface defined by Microsoft in the Base Class Library. V4 also failed to compile for the same reason as V3. There’s no implicit conversion from a list of Thingamajigs to an interface-typed list of interface-typed Thingamajigs.
PS C:\Users\Erik\...\Json IList Covariance> dotnet build -c release
Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
Determining projects to restore...
All projects are up-to-date for restore.
V4\ToolboxRecord.cs(17,131): error CS0311: The type 'System.Collections.Generic.List<ErikTheCoder.Sandbox.Covariance.
ThingamajigRecord>' cannot be used as type parameter 'TList' in the generic type or method 'DictionaryOfListsConverter
<IKey, IIList, IListValue, TKey, TList, TListValue>'. There is no implicit reference conversion from
'System.Collections.Generic.List<ErikTheCoder.Sandbox.Covariance.ThingamajigRecord>' to 'System.Collections.Generic.
IList<ErikTheCoder.Sandbox.Covariance.IThingamajigRecord>'.
[C:\Users\Erik\Documents\Visual Studio 2019\Projects\Sandbox\Json IList Covariance\Json IList Covariance.csproj]
Build FAILED.
The reason why V3 (with the Thingamajig property) and V4 fail to compile is because IList<T> is not covariant in .NET. What does that mean? It’s not easy to explain. At a high level, it means allowing an implicit type conversion would violate type safety guarantees of the .NET runtime. Eric Lippert, a former member of the C# compiler team, offers a brief explanation and numerous lengthier explanations. Jon Skeet, programmer extraordinaire with the highest reputation score on StackOverflow.com, demonstrates a type-safety violation that would be possible if IList<T> was covariant. In an answer to a question about polymorphism and covariance, he offers a succinct definition of covariance and contra-variance:
Covariance allows a “bigger” (less specific) type to be substituted in an API where the original type is only used in an “output” position (e.g. as a return value). Contravariance allows a “smaller” (more specific) type to be substituted in an API where the original type is only used in an “input” position. -Jon Skeet
The root cause of the C# compiler error is IList<T>’s indexer setter. It enables an update that would violate type safety. List interfaces that lack an indexer setter (such as IReadOnlyList<T>) are covariant (see Marc Gravell’s answer to a question about covariance and IList<T>). However, a read-only list isn’t a viable solution for my JSON de-serialization code (the real code, not this nonsense domain code) because client code needs to add or remove items to / from the list.
Solution
So how do we solve this problem? Actually, the developer who wrote the ListConverter class (in the codebase before I arrived) provided a hint. We must iterate over the dictionary and add each item, one by one, to the corresponding interface-typed dictionary. After all, the previous programmer didn’t simply cast the de-serialized list to an interface-typed list, like this:
This fails to compile for the same reason my DictionaryOfListsConverter class fails to compile.
Because the Thingamajig property introduces another collection (it maps Orientation to a list, not to a single object), its JsonConverter requires a second level of iteration. Let’s rewrite version 4 of the code and include a new converter, DictionaryOfListsConverter, that has two foreach loops, one for the dictionary keys and a second for each key’s list items.
Run the program. It successfully reads JSON from the file and de-serializes it.
PS C:\Users\Erik\...\Json IList Covariance> dotnet run -c release -- 4 Successfully deserialized JSON from C:\Users\Erik\Documents\Temp\Toolbox.json to ErikTheCoder.Sandbox.Covariance.V4.ToolboxRecord in toolboxRecord variable. Toolbox record has 2 sprocket records. Toolbox record has 2 widget records.
Conclusion and Notes
Hopefully this post helped demystify (if only slightly) the esoteric topic of covariance and contravariance as it relates to C# generics. Covariance and contravariance may be the province of compiler writers and class library authors, but occasionally it surfaces in line-of-business code. Understanding the concepts will help you avoid entanglements in your API designs.
I was uncertain what photo, diagram, or icon could illustrate the rather abstract concept of covariance and contravariance. I searched Google and didn’t find anything I liked. So I created an icon myself (seen immediately below the heading of this blog post at the top of the page). The arrows in the icon represent in and out generic type constraints. One arrow is entering into the interface and the other arrow is exiting out of the interface. Yeah… clearly I’m a more talented programmer than graphic designer.
You may review the full source code in the Json IList Covariance folder of my Sandbox project in GitHub.
(1) In the TypeNameHandling solution, the backwards-compatibility burden is located in the JSON text itself- which often is persisted to files or databases. Numerous files or database rows will contain namespaces and type names. Changing a namespace or type name in the source code will break de-serialization of the persisted JSON. This can be managed by writing a class that inherits from DefaultSerializationBinder and overrides the BindToType method. Do a simple string replacement of the old assembly and type name with the new new one, then return base.BindToType(newAssemblyName, newTypeName);
(2) In the JsonConverter solution, the backwards-compatibility burden is located in class property attributes. If de-serialization logic becomes complex and attribute properties must be added to the converter or sub-classed converters written, all record classes that rely on the converter must be reviewed and their attributes updated.
