Mapping JSON to objects with Fleece

In the last post I introduced Fleece and briefly explained how to use it to map objects to JSON. Sometimes I say “serialize” instead of “map”, but since the actual serialization is done by System.Json I think the right term to use here is “map” (as in mapping an object to a tree of JsonValues), or maybe "encoding" / "decoding".

Fleece can also do the opposite operation: map JSON to objects. There’s already an excellent F# library to deserialize JSON to typed objects: the JSON type provider from FSharp.Data (previously implemented in FSharpx), and so it’s impossible to avoid comparisons.

Some drawbacks of the JSON type provider

Whenever you need to deserialize JSON, I recommend you to try the JSON type provider first. When the conditions are right, nothing beats its simplicity.

But the conditions aren’t always right, and so the JSON type provider is sometimes not the best tool to use to deserialize JSON. Some of its drawbacks are:

1. Not total: throws exceptions when parsing fails. Exceptions hurt composability and your ability to reason about the code. This is mostly just an annoyance, as we can easily work around it with a small higher-order function:

let inline protect f x = 
    try
        Choice1Of2 (f x)
    with e -> Choice2Of2 e

(This is already part of FSharpx by the way).

2. Another annoyance is that the current implementation of the JSON type provider outputs erased types, so the types inferred from the JSON sample are only available in F#, not other languages. So you if you ever need to consume these types from C# or VB.NET you'll have to copy the implicit types explicitly and write the code to map them. This defeats the “no-code” benefit of using a type provider.

3. You’ll also usually want to write explicit types if you want to perform some additional validations. Think for example a NonEmptyList. In fact the type provider mechanism can’t generate records or discriminated unions, so if you want precise typing you have no choice but to write your types and then map them from the output of the type provider, again defeating the “no-code” benefit of using a type provider.

4. If the JSON input is “dynamic”, i.e. its structure is not exactly always the same, it varies depending on some request parameters, etc, then the type provider becomes useless because you can’t rely on a single sample (or a manageable number of samples) to infer the types. In this case you want to start working with the types, not with JSON samples. That’s why FSharp.Data also exposes and documents its underlying JSON parser/reader.

5. The parser generated by the type provider is monolithic, so you can’t “customize” a parser, or introduce validations while parsing/mapping, etc.

I don’t mean to make this post about criticizing the JSON type provider so I won’t go into detail about each of these points. As an exercise to illustrate these points try to use the JSON type provider to build a generic parser for the JSON output from Solr that can be consumed from any .NET language.

The concrete case that motivated me to write Fleece was the Edmunds API, which is very irregular (or “dynamic” depending on the point of view), plus I wanted specific types and needed to consume these API bindings from C#.

In a future post I might explore how to combine Fleece and the JSON type provider to take advantage of the benefits of each one where they are strong.

Fleece: the FromJSON typeclass

Back to Fleece: just as serialization is overloaded with the ToJSON fake typeclass, deserialization is built around a FromJSON typeclass.

The signature of the overloaded fromJSON function is:

fromJSON : JsonValue -> 'a ParseResult

where 'a is the type to decode (it must be overloaded in the FromJSON typeclass) and ParseResult a simple alias to Choice<'a, string>, i.e. you get either the decoded value or an error.

There’s also a convenience function parseJSON: string -> 'a ParseResult that takes a raw JSON string as input.

Let’s start with a simple example. We have this tree of people and their children:

let personJson = """
{
   "name": "John",
   "age": 44,
   "children": [{
       "name": "Katy",
       "age": 5,
       "children": []
   }, {
       "name": "Johnny",
       "age": 7,
       "children": []
   }]
}
"""

We can represent this with the following recursive type:

type Person = {
   Name: string 
   Age: int 
   Children: Person list 
}

Here’s one way to define FromJSON for the Person type:

type Person with 
   static member FromJSON (_: Person) = 
       function 
       | JObject o -> 
           let name = o .@ "name" 
           let age = o .@ "age" 
           let children = o .@ "children" 
           match name, age, children with 
           | Success name, Success age, Success children -> 
               Success { 
                   Person.Name = name 
                   Age = age 
                   Children = children 
               } 
           | x -> Failure (sprintf "Error parsing person: %A" x) 
       | x -> Failure (sprintf "Expected person, found %A" x)

Note the unused parameter of type Person: this is needed to make overloads unique and get the compiler to choose the right overloads.

Other than that, this is a function JsonValue -> Person ParseResult.

JObject is an active pattern identifying a JSON object (as opposed to a string, number, null, etc).

Success and Failure are simple aliases for the constructors Choice1Of2 and Choice2Of2 respectively, giving them more meaningful names. They’re also available as active patterns so we can use them in pattern matching.

The .@ operator tries to get a mapped value from a JSON object by key. That is, you can only call it for types that have a suitable FromJSON defined. Otherwise you get a compile-time error.

There’s also an operator .@? intended for optional keys in a JSON object, i.e. it returns Success None when the key isn't found, whereas .@ returns Failure "key xxx not found"

If you don’t like operators you can use the equivalent named functions jget / jgetopt.

That’s it, now we can parse JSON into Person:

let john : Person ParseResult = parseJSON personJson

Just as with serialization, deserialization in Fleece is total and ad-hoc polymorphic, and we get full compile-time checking. The same arguments about not breaking parametricity apply here.

Now, pattern matching each parsed value for Success/Failure doesn't sound like fun. Since these parsers return Choice<’value, ‘error> we can code monadically instead, so we can focus on the happy path, as Erik Meijer says. We can use the Choice.choose computation expression in FSharpx.Core, or the generic monad computation expression in FSharpPlus. Since Fleece already has a dependency on FSharpPlus, let’s use that:

type Person with 
   static member FromJSON (_: Person) = 
       function 
       | JObject o -> 
           monad { 
               let! name = o .@ "name" 
               let! age = o .@ "age" 
               let! children = o .@ "children" 
               return { 
                   Person.Name = name 
                   Age = age 
                   Children = children 
               } 
           } 
       | x -> Failure (sprintf "Expected person, found %A" x)

This reads much better. The ‘monad’ computation expression gets the compiler to infer the concrete type for the monad, in this case Choice<'a, 'e>.

We can write it even more compactly using applicative functors, though we need a curried constructor for Person:

type Person with 
   static member Create name age children = { Person.Name = name; Age = age; Children = children } 
   static member FromJSON (_: Person) = 
       function 
       | JObject o -> Person.Create <!> (o .@ "name") <*> (o .@ "age") <*> (o .@ "children") 
       | x -> Failure (sprintf "Expected person, found %A" x)

FSharpPlus already comes with overloaded applicative operators for the common applicatives (Choice, Option, etc).

We could go further and wrap that pattern match into a function, but let's just leave it at that.

What’s different about decoding JSON with Fleece is in the .@ operator. Just as with the .= operator does for encoding and the ToJSON typeclass, .@ only works for types that have a suitable FromJSON defined. Otherwise you get a compile-time error.

Not only that, but you also get decoder composition “for free”. Note how in the previous example o .@ "children" is inferring a Person list, which composes the decoder for 'a list with the very same decoder we’re defining for Person.
Fleece includes many decoders for common types, so if you want to decode, say, a Choice<(int * string) list, Choice<decimal option, string>> you don’t really need to do anything, and it’s all statically type-safe, pure and total, not breaking parametricity.

Roundtrips

When you need to both serialize and deserialize a type to JSON, it’s useful to make it roundtrip-safe, i.e. if you call toJSON and then fromJSON you get the original value again.

You can encode this property like this:

let inline roundtrip p = 
    let actual = p |> toJSON |> fromJSON
    actual = Success p

Use FsCheck to check this property, which will run it through a large number of instances for the type you want to check. Fleece does this for primitive types.

Also note the “inline” in this definition, which makes it possible to write generic code without having to specify any particular type. If you hover the mouse in Visual Studio over “roundtrip” it says val roundtrip: ('a -> bool) (requires member ToJSON and member FromJSON and equality), which means that the compiler is inferring the necessary constraints that the type must satisfy.

In the next post we’ll do some deeper analysis of the pros and cons of this typeclass-based approach to JSON encoding.