Showing posts with label http. Show all posts
Showing posts with label http. Show all posts

Wednesday, July 6, 2011

Content negotiation with Figment and FsConneg

A couple of posts ago I introduced FsConneg, a stand-alone HTTP content negotiation library written in F#. One of my goals in making it stand-alone is that it could be reused across projects, maybe eventually getting integrated into Frank or MonoRail3. Here I will show some potential integrations with Figment.

Let's start with a trivial function:

let connegAction _ = "hello world"

We want to bind this to an URL (i.e. GETting the URL would return "hello world" obviously), and negotiate the response media type with the user agent. The server will support XML and JSON.

We can build a dispatch table with the supported media types and corresponding actions:

let writers = [ 
                ["text/xml"; "application/xml"], Result.xml 
                ["application/json"], Result.json 
              ]

Just as a reminder, Result.xml and Result.json are of type 'a -> FAction, that is, they take some value and return an action where the value is serialized as XML or JSON respectively.

Wrapping actions

Now with FsConneg and this table, we write a generic function that wraps the action with content negotiation (this is all framework-level code):

let internal accepted (ctx: ControllerContext) = 
    ctx.Request.Headers.["Accept"]

let negotiateActionMediaType writers action = 
    let servedMedia = List.collect fst writers 
    let bestOf = accepted >> FsConneg.bestMediaType servedMedia >> Option.map fst 
    let findWriterFor mediaType = List.find (fst >> List.exists ((=)mediaType)) >> snd 
    fun ctx -> 
        let a = 
            match bestOf ctx with 
            | Some mediaType -> 
                let writer = writers |> findWriterFor mediaType 
                action >>= writer >>. vary "Accept" 
            | _ -> Result.notAcceptable 
        a ctx

Briefly, this function takes a table of acceptable media types and associated writers (just like the table we created above) and a "partial" action, and returns an action where the media type is negotiated with the user agent.

Armed with this function, let's bind the negotiated action to an URL:

get "conneg1" (negotiateActionMediaType writers connegAction)

For a second URL we'd also like to offer text/html. Here's a simple parameterized Wing Beats page:

let wbpage title =
    [e.Html [ 
        e.Head [ 
            e.Title [ &title ] 
        ] 
        e.Body [ 
            e.H1 [ &title ] 
        ] 
    ]]

We want to make this an action:

let html = wbpage >> wbview

I defined wbview in a previous article, it's not instrumental to this post. What's important is that html is a function string -> FAction so we can now add it to our dispatch table:

let conneg2writers = (["text/html"], html)::writers

and bind it to an URL:

get "conneg2" (negotiateActionMediaType conneg2writers connegAction)

Using routing

An entirely different approach is to use routing to select the appropriate 'writer' (or serializer, or formatter, whatever you want to call it)

let ifConneg3Get = ifMethodIsGet &&. ifPathIs "conneg3" 
action (ifConneg3Get &&. ifAcceptsAny ["application/xml"; "text/xml"]) (connegAction >>= Result.xml)

I think this snippet is pretty intuitive, even if you're not familiar with Figment or functional programming. I'll explain anyway:

ifMethodIsGet and ifPathIs are routing functions built into Figment. The &&. operator composes these routing functions as expected, i.e. the resulting routing function must satisfy both conditions. This is explained in more detail in my introduction to Figment.

The >>= operator is a monadic bind. The function

connegAction >>= Result.xml

is equivalent to:

result { 
    let! result = connegAction 
    return! Result.xml result }

or:

fun ctx -> 
    let result = connegAction ctx 
    Result.xml result ctx

Except the first one is evidently more concise. I explained this in more detail in my last post.

ifAcceptsAny uses FsConneg to determine if any of the media types in the list is acceptable to the client. Its definition is quite simple:

let ifAcceptsAny media = 
    fun (ctx: HttpContextBase, _) -> 
        let acceptable = FsConneg.negotiateMediaType media ctx.Request.Headers.["Accept"] 
        acceptable.Length > 0

Similarly, let's add JSON and HTML support:

action (ifConneg3Get &&. ifAcceptsAny ["application/json"]) (connegAction >>= Result.json) 
action (ifConneg3Get &&. ifAcceptsAny ["text/html"]) (connegAction >>= html)

We close by stating that all other media types are not acceptable:

action ifConneg3Get Result.notAcceptable

The HTTP RFC says it's ok to respond with some non-acceptable media type, so you could also use this to define a default media type instead of a "not acceptable".

The important thing to notice about this last example is that using routing like this doesn't yield proper content negotiation. If a user-agent requests with "Accept: application/json, application/xml;q=0.8" (i.e. prefers application/json), the code above will respond with application/xml, disregarding the client's preferences, simply because the action for application/xml was defined before application/json.

Many frameworks don't handle this properly. If you're planning to build a RESTful application I recommend testing the framework you'll use for this. For example, OpenRasta does the right thing, but Lift had issues until some months ago, and WCF Web API doesn't handle it correctly at the moment.

Using extensions

A common way to select a response media type is using extensions in the URL. Twitter used to do this, until they scrapped XML support altogether. MySpace still does it. Similarly, others use a query string parameter to select the media type, like last.fm.

This isn't really content negotiation as defined by HTTP, but some people do it in the name of simplicity, or to work around client issues. It could be considered as part of a client-driven negotiation, though.

At any rate, implementing extension-driven media types is quite easy. Similarly to the first example, we can build a dispatch table of extensions and then act on it:

let extensions = [
                    "xml", Result.xml
                    "json", Result.json
                    "html", html
                 ]
for ext,writer in extensions do
    let ifConneg4 = ifPathIsf "conneg4.%s" ext
    action (ifMethodIsGet &&. ifConneg4) (connegAction >>= writer)

Using extensions + conneg

Some prefer a compromise between conneg and extensions, by implementing an extensionless URL that supports content negotiation and then the same URL with extensions as a means to override conneg and work around possible client issues, or just selecting a media type without messing with headers.

For our example we'd want an URL /conneg5 that supports content negotiation, plus /conneg5.xml, /conneg5.json and /conneg5.html to force a particular media type.

As with the previous approaches, let's build a table:

let writers = [ 
                "xml", ["application/xml"; "text/xml"], Result.xml 
                "json", ["application/json"], Result.json 
                "html", ["text/html"], html 
              ]

Now let's map the extensions, just as in the last example:

let basePath = "conneg5"

for ext,_,writer in writers do 
    let ifBasePath = ifPathIsf "%s.%s" basePath ext 
    action (ifMethodIsGet &&. ifBasePath) (connegAction >>= writer)

Finally, the conneg'd URL:

let mediaTypes = List.map (fun (_,a,b) -> a,b) writers 
let ifBasePath = ifPathIs basePath 
action (ifMethodIsGet &&. ifBasePath) (negotiateActionMediaType mediaTypes connegAction)

If you do this for all your actions you could easily extract this to a reusable function.

Final words

As I said before, these are all potential integrations. I specifically want these libraries to be as non-opinionated as possible. But at the same time, I want to provide all the tools to let the developer easily create her own opinions/conventions to fit her project, using library calls and standard language constructs instead of having to learn framework extension points. For example, notice that the dispatch tables I explained are all regular lists of strings and functions, which are then mapped and iterated just like any other list. More on this in a future post.

All code posted here is part of the FigmentPlayground repository.

Tuesday, June 14, 2011

A HTTP content negotiation library in F#

I've been writing a HTTP content negotiation (server-driven) library in F# I called FsConneg (yeah, I've been pretty lazy lately about naming things).

First, here's a little introduction to the topic:

Content negotiation is briefly specified in section 12 of RFC2616, although the meat of it is really in the definitions of the Accept-* headers. There are four content characteristics that can be negotiated: media type, language, charset and encoding.

Encoding refers to content compression, and is usually handled at the web server level, for example IIS static/dynamic compression or mod_deflate in Apache.

Charset refers to UTF-8, ISO-8859-1, etc. The most interesting are media type and language which are the most commonly negotiated characteristics in user-level code. Language is self-explanatory, and media type negotiates whether the response should be XML, JSON, PDF, HTML, etc.

FsConneg is inspired by clj-conneg and has a similar interface. clj-conneg currently negotiates media types only, but FsConneg can negotiate any content characteristic. Like clj-conneg, FsConneg doesn't assume any particular web framework, it works with raw header values, and so it can be easily integrated into any web server or framework.

Let's say you have a server application that can respond with application/xml or application/json, but it prefers application/json:

let serves = ["application/json"; "application/xml"]

And you get a request from a user agent with an Accept header looking like this:

let accepts = "text/html, application/xml;q=0.8, */*;q=0.5"

Which means: Hey server, I prefer a text/html response, but if you can't do that I'll take application/xml, or as a last resort give me whatever media type you have.

Given these two constraints, the server wants to find out what media type it should use:

match bestMediaType serves accepts with
| Some (mediaType, q) -> printfn "Negotiated media type %s, now the server formats its response with this media type" mediaType
| _ -> failwithf "Couldn't negotiate an acceptable media type with client: %s" accepts

In this example, the negotiated media type is of course application/xml. In case of negotiation failure, the server should respond with status 406 Not Acceptable.

There are similar functions bestEncoding, bestCharset and bestLanguage to negotiate the other content characteristics.

At a lower level, you might want to use negotiate* functions. Instead of giving you a single best type, these give you a sorted list of acceptable types. For example, using the same serves and accepts as above:

> negotiateMediaType serves accepts

val it : (string * float) list =
  [("application/xml", 0.8); ("application/json", 0.5)]

Even though server-driven content negotiation was defined back in 1997, it hasn't been used a lot, and with good reason. Every party involved (user agent, server and proxies) has to implement negotiation semantics right, or Bad Things could happen, like the user agent asking for a page in English and getting it in French because some proxy didn't honor the Vary header.
Until a few years ago, Internet Explorer didn't handle the Vary header all too well, and some proxies had issues as well. Until version 9, Internet Explorer used to send a mess of an Accept header, and WebKit preferred application/xml over text/html, which doesn't make much sense for a browser. Here's a spreadsheet with the Accept header some browsers send. Also, we developers and the frameworks we use sometimes get details wrong. Pure server-driven language negotiation is most of the time insufficient and has to be complemented with agent-driven negotiation. Even Roy Fielding, who defined REST, says that "a server is rarely able to make effective use of preemptive negotiation".
As a server app developer, some of these issues are out of your control, yet affect how content gets served to your clients. Many people argue that content negotiation is broken, or overly complex, or an ivory tower concept, or just don't agree with it.

I think it still can work and has its time and place, in particular for APIs. But just like the rest of REST (no pun intended), content negotiation is not as simple as it might seem at first sight.

In the next post I'll describe some ways to do concrete content negotiation with this library and Figment.

The code for FsConneg is on github.