Enumerable.Skip vs Seq.skip

If you're an F# newbie like me^(*) you'll eventually try to use Seq.skip and Seq.take to implement pagination, just like you used Enumerable.Skip and Enumerable.Take (or a IQueryable implementation of them) in C#.

And more sooner than later you find out that they don't behave quite the same. If you haven't realized this yet, read on.

Load up fsi.

> let a = seq { 1..100 };;

An F# seq is a System.IEnumerable, it's just a convenient alias. In C# this would be expressed as:

var a = Enumerable.Range(1, 100);

Now let's paginate that. Assuming a page size of 40 elements, in C# we would do something like this:

var firstPage = a.Skip(0).Take(40).ToArray();
var lastPage = a.Skip(80).Take(40).ToArray();

Now in F# :

> let firstPage = a |> Seq.skip 0 |> Seq.take 40 |> Seq.toArray;;
> let lastPage = a |> Seq.skip 80 |> Seq.take 40 |> Seq.toArray;;

Uh-oh. The last expression throws a System.InvalidOperationException: The input sequence has an insufficient number of elements.

The thing is, Seq.skip and Seq.take are more strict and actually do bounds checking, whereas Enumerable.Skip and Enumerable.Take are more "tolerant" and may process and return fewer items than requested.

So how can we get Enumerable.Skip's behavior? The simplest option would be using it as in C#, e.g.:

> open System.Linq;;
> let lastPage = a.Skip(80).Take(40).ToArray();;

This works, but extension methods are generally not idiomatic in F#. We prefer function piping, currying and composition. So let's wrap them in curried forms, which is trivial:

> let eSkip n sequence = Enumerable.Skip(sequence, n);;
> let eTake n sequence = Enumerable.Take(sequence, n);;

And now we can operate as usual with pipes:

> let lastPage = a |> eSkip 80 |> eTake 40 |> Seq.toArray;;

By the way, F# already defines Seq.truncate which works like Enumerable.Take, so we can drop eTake and just write:

> let lastPage = a |> eSkip 80 |> Seq.truncate 40 |> Seq.toArray;;

---

(*) I've been learning F# and functional programming for about two years now, yet I still consider myself somewhat of a newbie about it... at least until I learn Haskell :-)

Figment: a web DSL for F#

As part of my master's thesis project, I'm writing Figment, an embedded DSL for web development for F#. In the spirit of similar web DSLs like Sinatra and Compojure, it aims to be simple, flexible, idiomatic.

It's still very experimental and likely to change but I'd like to show you what I have so far. So here's "Hello World" in Figment:

First a very basic Global.asax to set the entry point:

<%@ Application Inherits="BasicSampleApp.App" %>

and now the code itself:

namespace BasicSampleApp

open System.Web
open Figment.Routing
open Figment.Actions

type App() =
   inherit HttpApplication()
   member this.Application_Start() =
       get "hi" (content "<h1>Hello World!</h1>")

Run it, visit /hi and you get a big Hello World. Of course, everything but the last line is boring boilerplate, so let's focus on that last line. This is basically how it works: first, we have the action type:

type FAction = ControllerContext -> ActionResult

Yes, those are ASP.NET MVC2 classes. Figment is built on top of ASP.NET MVC2. Now, the get function takes a route and an action, and maps GET requests.

get : string -> FAction -> unit

and content is an action generator (or parameterized action), it creates an action that outputs a string as response.

content : string -> FAction

Similarly, there's a redirect action generator, so we can redirect /hello to /hi by saying:

get "hello" (redirect "hi")

Actions and Results

So far we've only seen action generators, now let's see proper actions with a variant of Hello World. We start with a simple function:

let greet firstName lastName age =   
 sprintf "Hello %s %s, you're %d years old" firstName lastName age

and now we bind it to the request and map it to a route:

let greet' (ctx: ControllerContext) =
   let req = ctx.HttpContext.Request
   greet req.["firstname"] req.["lastname"] (int req.["age"])
   |> sprintf "<p>%s</p>" |> Result.content
get "greetme" greet'

Visit /greetme?firstname=John&lastname=Doe&age=50 to see this in action.

Did you notice Result.content? It maps directly to ContentResult. Normally you don't have both Figment.Actions and Figment.Result open in the same file so usually you can skip writing "Result.".

We could have used Result.view (ViewResult) to send the information to a regular ASP.NET MVC view:

let greet2 (p: NameValueCollection) =
   greet p.["firstname"] p.["lastname"] (int p.["age"])
get "greetme2" (bindQuerystring greet2 >> Result.view "someview")

Note also how function composition make it easy to work at any level of abstraction (bindQuerystring is in Figment.Binding)

Filters

Filters are just functions with this signature:

type Filter = FAction -> FAction

With this simple abstraction we can implement authorization, caching, etc. For example, here's how to apply the equivalent of a RequireHttpsAttribute:

get "securegreet" (requireHttps greet')

requireHttps and others live in Figment.Filters.

Routing DSL

Sometimes you need flexibility when defining a route. For example, use a regular expression, or check for a mobile browser. Enter Figment.RoutingConstraints. A routing constraint is a function like this:

type RouteConstraint = HttpContextBase * RouteData -> bool

It returns true if it's a match, false if it's not. It's applied with the action router:

action : RouteConstraint -> FAction -> unit

A trivial route constraint:

let unconstrained (ctx: HttpContextBase, route: RouteData) = true
action unconstrained (content "Hello World")

would map that content to every URL/method. You might think that taking a single constraint is useless, but they can be combined with a few operators to create a small DSL:

let ifGetDsl = ifUrlMatches "^/dsl" &&. ifMethodIsGet

action
   (ifGetDsl &&. !. (ifUserAgentMatches "MSIE"))
   (content "You're NOT using Internet Explorer")

action ifGetDsl (content "You're using Internet Explorer")

Hopefully this last sample was self-explanatory!

Strongly-typed routing

A couple of blog posts ago I briefly mentioned using PrintfFormat manipulation to define strongly-typed routes. This is what I meant:

let nameAndAge firstname lastname age =
   sprintf "Hello %s %s, %d years old" firstname lastname age
   |> Result.content
getS "route/{firstname:%s}/{lastname:%s}/{age:%d}" nameAndAge

This actually routes and binds at the same time.

Conclusions

As I said, this is very much work in progress, and there's still a lot to do. I intend to make it fully open source when I finish writing my thesis. I'll have to analyze tons of web frameworks, in particular functional web frameworks, so hopefully I'll pick up some interesting stuff from Happstack, Snap, Haskell on a Horse, etc. In particular, I'm interested in implementing formlets, IMHO one of the coolest features of WebSharper.

Source code is here.

Compiling LaTeX without a local LaTeX compiler

 

I recently started working on my (long long overdue) master's thesis, and naturally I came to LaTeX for typesetting. As a LaTeX newbie*, I was pleased to find out that compiling LaTeX is just like compiling any program: there's a bunch of source code files (.tex) and resources (.bib, .sty, images) which you feed to preprocessors and compilers to yield a dvi/ps/pdf document. TeX is actually a turing-complete language, and LaTeX is a macro collection and a preprocessor for TeX.

There's even an online package repository!

And since writing LaTeX has so many similarities with writing code, even if you're just writing text and using existing LaTeX macros and not developing your own, some of the development practices can be applied here as well, like a build system, refactoring to keep the document manageable, version control, and continuous integration.

Continuous integration is nice not only because it asserts that the document is valid (yes, you can screw up TeX documents), but also you automatically get the resulting PDF as a build artifact. You could even hook a diff step in the process, to make reviews easier.

But finding a public continuous integration server is hard, not to mention one with a LaTeX distribution installed. Luckily, LaTeX compilation can be delegated to a remote server, thanks to CLSI (Common LaTeX Service Interface). So all the integration server has to do is forward a compilation request to the CLSI server when it detects changes in the VCS.

The arrow directions in the graphic indicate information flow, and the numbers order the actions.

How you implement this depends on what the integration server supports. For example, if it's Windows you can send a manually assembled compilation request (it's a simple XML) with a JScript script.

You can also use a CLSI server as a replacement for a local LaTeX installation. In that case you have to send the actual .tex file contents instead of just an URL, wrapped in a CLSI XML compilation request. I wrote a simple CLSI client in F# to do this, so a compilation script looks like this:

I'm using git and github for version control and ScribTeX's CLSI server, which is the only public CLSI server I have found so far. Kudos to ScribTeX for that! It's very fast (even faster than compiling locally!) and open source.

If you just want an online editor with integrated revision control and preview, check out ScribTeX's editor.

Also worth checking out is LaTeX Lab, a very cool open source, online LaTeX editor for Google Docs, by the same guy that developed CLSI (Bobby Soares).  It's currently on a preview release but it's very usable.

* Actually, I did use LaTeX several years ago, but I did so through LyX, which combines LaTeX with WYSIWYG, so I didn't really learn LaTeX.

Abusing PrintfFormat in F#

A cool little feature of OCaml and F# is their type-safe printf, e.g. you can write:

printfn "Hi, my name is %s and I'm %d years old" "John" 29

but:

printfn "I'm %d years old" "John"

won't even compile because "John" is a string and not an int as expected.

Under the covers, the expression printfn "I'm %d years old" returns a int -> unit function and that's how it type-checks the parameters. But how does it know what function signature it has to generate?

Looking at printfn's signature you can see it's Printf.TextWriterFormat<'T> -> 'T . TextWriterFormat<'T> is in turn a type alias for PrintfFormat<'T, TextWriter, unit, 'Result> or Format<'T,TextWriter,unit,'Result> . This Format type is kind of special, it seems the F# compiler has some magic that instantiates it implicitly from a string and runs the function generator that lets it type-check statically the *printf methods.

Now, we can't add custom formats due to this being built into the compiler, but we can process them however we want.
For example, how about executing SQL queries with type-safe parameters? Your first try might be something like this:

let sql = sprintf "select * from user where name = '%s'" "John" 
let results = executeSQL sql

However, this would be vulnerable to SQL injection attacks because it's not using proper SQL parameters. So let's build our own PrintfFormat processing function! We want to be able to write:

let results = runQuery "select * from user where id = %d and name = %s" 100 "John" // silly query, I know

The signature we need for this to work is:

runQuery: PrintfFormat<'a, _, _, IDataReader> -> 'a

Note that the first type parameter in PrintfFormat must be the same type as the return type, otherwise it won't type-check as we want. We can ignore the second and third type parameters for this example. The last type parameter, IDataReader is the return type after applying the required arguments.
The first type parameter, 'a, is really a function type that is responsible for processing the format arguments. So runQuery has to return the function that will process the format arguments.

Confused so far? Great, my job is done :-)
Just kidding! To illustrate, let's build a dummy implementation of runQuery that satisfies the test above:

let runQuery (query: PrintfFormat<'a, _, _, IDataReader>) : 'a = 
    let proc (a: int) (b: string) : IDataReader = 
        printfn "%d %s" a b 
        null 
    unbox proc 
let results = runQuery "select * from user where id = %d and name = %s" 100 "John"

This will compile correctly, and when run, it will print 100 John. results will be null so of course it's not really an usable IDataReader. But at least everything type-checks.

The problem is that this primitive implementation is not really generic. It breaks as soon as we change any of the format parameters. For example:

let results = runQuery "select top 5 * from usuario where name = %s" "John"

Compiles just fine, but when run will fail with an InvalidCastException. That's because in this last test, a string -> IDataReader format handler was expected, but we provided proc which is int -> string -> IDataReader.
In order to make this truly generic, we have to resort to reflection to dynamically build the format handler function. F# has some nice reflection helpers to handle F#-specific functions, records, etc.

let runQuery (query: PrintfFormat<'a, _, _, IDataReader>) : 'a = 
    let rec getFlattenedFunctionElements (functionType: Type) = 
        let domain, range = FSharpType.GetFunctionElements functionType 
        if not (FSharpType.IsFunction range) 
            then domain::[range] 
            else domain::getFlattenedFunctionElements(range) 
    let types = getFlattenedFunctionElements typeof<'a> 
    let rec proc (types: Type list) (a: obj) : obj = 
        match types with 
        | [x;_] -> 
            printfn "last! %A" a 
            box null 
        | x::y::z::xs -> 
            printfn "%A" a 
            let cont = proc (y::z::xs) 
            let ft = FSharpType.MakeFunctionType(y,z) 
            let cont = FSharpValue.MakeFunction(ft, cont) 
            box cont 
        | _ -> failwith "shouldn't happen" 
    let handler = proc types 
    unbox (FSharpValue.MakeFunction(typeof<'a>, handler))

This will print the format arguments, it's similar to the previous one, but this one is fully generic, it will handle any argument type and count.
Now all we have to do is abstract away the argument processing (I'll rename the function to PrintfFormatProc):

let PrintfFormatProc (worker: string * obj list -> 'd)  (query: PrintfFormat<'a, _, _, 'd>) : 'a = 
    if not (FSharpType.IsFunction typeof<'a>) then 
        unbox (worker (query.Value, [])) 
    else 
        let rec getFlattenedFunctionElements (functionType: Type) = 
            let domain, range = FSharpType.GetFunctionElements functionType 
            if not (FSharpType.IsFunction range) 
                then domain::[range] 
                else domain::getFlattenedFunctionElements(range) 
        let types = getFlattenedFunctionElements typeof<'a> 
        let rec proc (types: Type list) (values: obj list) (a: obj) : obj = 
            let values = a::values 
            match types with 
            | [x;_] -> 
                let result = worker (query.Value, List.rev values) 
                box result 
            | x::y::z::xs -> 
                let cont = proc (y::z::xs) values 
                let ft = FSharpType.MakeFunctionType(y,z) 
                let cont = FSharpValue.MakeFunction(ft, cont) 
                box cont 
            | _ -> failwith "shouldn't happen" 
        let handler = proc types [] 
        unbox (FSharpValue.MakeFunction(typeof<'a>, handler))

Note that I also added a special case to handle a no-parameter format. And now we write the function that actually processes the format arguments, by replacing %s, %d, etc to SQL parameters @p0, @p1, etc, and then running the query:

let connectionString = "data source=.;Integrated Security=true;Initial Catalog=SomeDatabase"
let sqlProcessor (sql: string, values: obj list) : IDataReader =
    let stripFormatting s =
        let i = ref -1
        let eval (rxMatch: Match) =
            incr i
            sprintf "@p%d" !i
        Regex.Replace(s, "%.", eval)
    let sql = stripFormatting sql
    let conn = new SqlConnection(connectionString)
    conn.Open()
    let cmd = conn.CreateCommand()
    cmd.CommandText <- sql
    let createParam i (p: obj) =
        let param = cmd.CreateParameter()
        param.ParameterName <- sprintf "@p%d" i
        param.Value <- p
        cmd.Parameters.Add param |> ignore
    values |> Seq.iteri createParam
    upcast cmd.ExecuteReader(CommandBehavior.CloseConnection) 

let runQuery a = PrintfFormatProc sqlProcessor a

let queryUser = runQuery "select top 5 * from user where id = %d and name = %s"

use results = queryUser 100 "John"
while results.Read() do
    printfn "%A" results.["id"] 

Pretty cool, huh? This trick can be used for other things, since PrintfFormatProc is fully reusable. For example, I'm currently using PrintfFormat manipulation to define type-safe routing in a web application (part of my master's thesis, I'll blog about it soon)

Full source code here

Hybrid lifestyles in Windsor

All Inversion of Control (IoC) containers offer some way to control the lifecycle or scope of the component instances. Singleton (one instance per container) and Transient (a new instance each time your request it from the container) are the two most common scopes.

Castle Windsor calls these scopes "lifestyles" (the term "lifecycle" is used for something else) and it comes with these built-in lifestyles (I'll just quote the documentation):

• Singleton (default): one instance per container
• Transient: a new instance each time your request it from the container
• PerThread: one instance per thread
• PerWebRequest: one instance per HttpContext (HTTP request)
• Pooled: instances will be pooled to avoid unnecessary constructions

More importantly, a custom lifestyle manager can be plugged-in, just by implementing the ILifestyleManager interface. Some examples of this are the WCF facility PerWcfSession and PerWcfOperation lifestyles and the PerHttpApplication lifestyle I implemented to inject dependencies to HttpModules.

There are other lifestyles that come in handy sometimes, especially in web applications. I created the Castle.Windsor.Lifestyles contrib project to host them. I have implemented these lifestyles so far:

• Abstract hybrid lifestyle
• Abstract hybrid PerWebRequest + X
• Hybrid PerWebRequest + Transient
• PerWebSession: one instance per HTTP session
• PerHttpApplication (mentioned above)

An hybrid lifestyle is one that actually blends two underlying lifestyles: a main lifestyle and a secondary lifestyle. The hybrid lifestyle first tries to use the main lifestyle; if it's unavailable for some reason, it uses the secondary lifestyle. This is commonly used with PerWebRequest as the main lifestyle: if the HTTP context is available, it's used as the scope for the component instance; otherwise the secondary lifestyle is used.

The PerWebSession has a couple of caveats:

• Components using this lifestyle have to be serializable if you're using a session-state mode other than InProc.
• Components using this lifestyle will not be properly released, since the session end event only fires when using the InProc session-state mode.

Usage:

1. Add a reference to Castle.Windsor.Lifestyles.dll
2. Use the appropriate lifestyle descriptor in your registration, e.g.:

   container.Register(Component.For<SomeService>()
                   .LifeStyle.HybridPerWebRequestTransient());