A HTTP content negotiation library in F#

I've been writing a HTTP content negotiation (server-driven) library in F# I called FsConneg (yeah, I've been pretty lazy lately about naming things).

First, here's a little introduction to the topic:

Content negotiation is briefly specified in section 12 of RFC2616, although the meat of it is really in the definitions of the Accept-* headers. There are four content characteristics that can be negotiated: media type, language, charset and encoding.

Encoding refers to content compression, and is usually handled at the web server level, for example IIS static/dynamic compression or mod_deflate in Apache.

Charset refers to UTF-8, ISO-8859-1, etc. The most interesting are media type and language which are the most commonly negotiated characteristics in user-level code. Language is self-explanatory, and media type negotiates whether the response should be XML, JSON, PDF, HTML, etc.

FsConneg is inspired by clj-conneg and has a similar interface. clj-conneg currently negotiates media types only, but FsConneg can negotiate any content characteristic. Like clj-conneg, FsConneg doesn't assume any particular web framework, it works with raw header values, and so it can be easily integrated into any web server or framework.

Let's say you have a server application that can respond with application/xml or application/json, but it prefers application/json:

let serves = ["application/json"; "application/xml"]

And you get a request from a user agent with an Accept header looking like this:

let accepts = "text/html, application/xml;q=0.8, */*;q=0.5"

Which means: Hey server, I prefer a text/html response, but if you can't do that I'll take application/xml, or as a last resort give me whatever media type you have.

Given these two constraints, the server wants to find out what media type it should use:

match bestMediaType serves accepts with
| Some (mediaType, q) -> printfn "Negotiated media type %s, now the server formats its response with this media type" mediaType
| _ -> failwithf "Couldn't negotiate an acceptable media type with client: %s" accepts

In this example, the negotiated media type is of course application/xml. In case of negotiation failure, the server should respond with status 406 Not Acceptable.

There are similar functions bestEncoding, bestCharset and bestLanguage to negotiate the other content characteristics.

At a lower level, you might want to use negotiate* functions. Instead of giving you a single best type, these give you a sorted list of acceptable types. For example, using the same serves and accepts as above:

> negotiateMediaType serves accepts

val it : (string * float) list =
  [("application/xml", 0.8); ("application/json", 0.5)]

Even though server-driven content negotiation was defined back in 1997, it hasn't been used a lot, and with good reason. Every party involved (user agent, server and proxies) has to implement negotiation semantics right, or Bad Things could happen, like the user agent asking for a page in English and getting it in French because some proxy didn't honor the Vary header.
Until a few years ago, Internet Explorer didn't handle the Vary header all too well, and some proxies had issues as well. Until version 9, Internet Explorer used to send a mess of an Accept header, and WebKit preferred application/xml over text/html, which doesn't make much sense for a browser. Here's a spreadsheet with the Accept header some browsers send. Also, we developers and the frameworks we use sometimes get details wrong. Pure server-driven language negotiation is most of the time insufficient and has to be complemented with agent-driven negotiation. Even Roy Fielding, who defined REST, says that "a server is rarely able to make effective use of preemptive negotiation".
As a server app developer, some of these issues are out of your control, yet affect how content gets served to your clients. Many people argue that content negotiation is broken, or overly complex, or an ivory tower concept, or just don't agree with it.

I think it still can work and has its time and place, in particular for APIs. But just like the rest of REST (no pun intended), content negotiation is not as simple as it might seem at first sight.

In the next post I'll describe some ways to do concrete content negotiation with this library and Figment.

The code for FsConneg is on github.