Thursday, March 31, 2011

SolrNet 0.3.1 released

SolrNet is a Solr client for .NET. I just released SolrNet 0.3.1.

This is a bugfix release. It's a drop-in replacement for 0.3.0, there are no breaking changes.

Here's the changelog:

  • Fixed fixed parsing of decimals with exponential notation
  • Fixed SolrQueryInList with empty strings
  • Fixed facet.missing=true
  • Added support for nullable Guid properties
  • Fixed date faceting for Solr 3.x by ignoring 'start' element
  • Fixed NullReferenceException with facet.missing=true
  • Null in range queries translate to *
  • Ignored LocalParams for facet field parameters, it generated an invalid query.

Thanks to everyone who reported these bugs and offered solutions!

Binaries are available on Google Code and NuGet.

Tuesday, March 29, 2011

Integrating Formlets with Wing Beats and Figment

I've blogged before about formlets, a nice abstraction of HTML forms. I started with a basic implementation, then showed validation. Now I'll present a non-toy implementation of formlets I called FsFormlets (really original name, I know). By "non-toy" I mean this implementation is not a proof of concept or just for didactic purposes, but is meant to be eventually production-quality.

I don't like to explain things in a vacuum, so I'll use a typical registration form to show how it works:


Even though FsFormlets can be used to generate HTML on its own, there's a much better tool for this in F#: Wing Beats. Wing Beats is an EDSL in F# to generate HTML, much like SharpDOM in C# or Blaze in Haskell, or Eliom's XHTML.M in OCaml (except that XHTML.M actually validates HTML statically). So I've added a module to integrate FsFormlets to Wing Beats. This integration is a separate assembly; FsFormlets is stand-alone (it only requires .NET 3.5 SP1 and the F# runtime). We'll use FsFormlets to express forms, and Wing Beats to express the rest of HTML. Also, we'll handle web requests with Figment, with the help of some more glue code to integrate it with FsFormlets and Wing Beats.


Let's start by defining a layout in Wing Beats:

let e = XhtmlElement()

let layout title body = 
        e.Html [ 
            e.Head [ 
                e.Title [ &title ] 
                e.Style [ 
                    &".error {color:red;}" 
                    &"body {font-family:Verdana,Geneva,sans-serif; line-height: 160%;}" 
            e.Body [ 
                yield e.H1 [ &title ] 
                yield! body 

No formlets so far, this is all pure HTML, expressed as a regular function. Now we'll build the form bottom-up.


FsFormlets already includes a reCaptcha formlet (I'll show its innards in a future post). We just have to configure it with a pair of public and private key (get it here) before using it:

let reCaptcha = reCaptcha {PublicKey = "your_public_key"; PrivateKey = "your_private_key"; MockedResult = None}

MockedResult lets you skip the actual validation web call and force a result, for testing purposes.

Date input

Now the date formlet, which is built from three inputs, plus labels and validation:

let f = e.Formlets

let dateFormlet : DateTime Formlet = 
    let baseFormlet = 
        yields t3 
        <*> (f.Text(maxlength = 2, attributes = ["type","number"; "min","1"; "max","12"; "required","required"; "size","3"]) |> f.WithLabel "Month: ") 
        <*> (f.Text(maxlength = 2, attributes = ["type","number"; "min","1"; "max","31"; "required","required"; "size","3"]) |> f.WithLabel "Day: ") 
        <*> (f.Text(maxlength = 4, attributes = ["type","number"; "min","1900"; "required","required"; "size","5"]) |> f.WithLabel "Year: ") 
    let isDate (month,day,year) = 
        let pad n (v: string) = v.PadLeft(n,'0') 
        let ymd = sprintf "%s%s%s" (pad 4 year) (pad 2 month) (pad 2 day) 
        DateTime.TryParseExact(ymd, "yyyyMMdd", CultureInfo.InvariantCulture, DateTimeStyles.None) |> fst 
    let dateValidator = err isDate (fun _ -> "Invalid date") 
    |> satisfies dateValidator 
    |> map (fun (month,day,year) -> DateTime(int year,int month,int day))

Here, baseFormlet is of type (string * string * string) Formlet, that is, it collects values in their raw form. This baseFormlet is then validated to make sure it's a date, and finally mapped to a DateTime. Note the use of the 'number' input type (a HTML5 input) and also HTML5 validation attributes (required, min, max)

Password input

The password formlet is next:

let doublePassword = 
    let compressedLength (s: string) = 
        use buffer = new MemoryStream() 
        use comp = new DeflateStream(buffer, CompressionMode.Compress) 
        use w = new StreamWriter(comp) 
    let isStrong s = compressedLength s >= 106L 
    let f = 
        yields t2 
        <*> (f.Password(required = true) |> f.WithLabel "Password: ") 
        <+ e.Br() 
        <*> (f.Password(required = true) |> f.WithLabel "Repeat password: ") 
    let areEqual (a,b) = a = b 
    |> satisfies (err areEqual (fun _ -> "Passwords don't match")) 
    |> map fst 
    |> satisfies (err isStrong (fun _ -> "Password too weak"))

There are two things we validate here for passwords: first, the two entered passwords must match; and second, it must be strong enough. To measure password strength I use the compression technique I described in my last post.

The <+ operator is used to add pure markup to the formlet, using Wing Beats.

Also note the 'required = true' parameter. In the date formlet, we just used required=required as an HTML attribute. Here we used the optional argument required=true, which validates the formlet both at the client (with a required=required HTML attribute) and the server (it has a satisfies clause). In the date formlet we don't really care to validate server-side if the user filled out each input, we just want to know if it's a valid date or not.

The optional parameter 'maxlength' does a similar thing: it outputs a maxlength attribute, and also checks the maximum length of he POSTed value. Sure all browsers implement maxlength properly, but anyone can easily craft a request (e.g. with cUrl, no need to program at all) to circumvent it. I see malicious bots doing things like this every day. This makes sure the data is really valid before you start processing it.

Putting it all together

Moving on, let's write the final formlet and store the posted information in a record:

type PersonalInfo = { 
    Name: string 
    Email: string 
    Password: string 
    DateOfBirth: DateTime 

let registrationFormlet ip = 
    yields (fun n e p d -> 
                { Name = n; Email = e; Password = p; DateOfBirth = d }) 
    <*> (f.Text(required = true) |> f.WithLabel "Name: ") 
    <+ e.Br() 
    <*> (f.Email(required = true) |> f.WithLabel "Email: ") 
    <+ e.Br() 
    <*> doublePassword 
    <+ e.Br() 
    <+ &"Date of birth: " <*> dateFormlet 
    <+ e.Br() 
    <+ &"Please read very carefully these terms and conditions before registering for this online program, blah blah blah" 
    <+ e.Br() 
    <* (f.Checkbox(false) |> satisfies (err id (fun _ -> "Please accept the terms and conditions")) |> f.WithLabel "I agree to the terms and conditions above") 
    <* reCaptcha ip

Some notes about this last snippet:

  • f.Email(required = true) generates an <input type="email" required="required"/> (again, HTML5 !), all server-validated.
  • Unlike previous formlets, this one is a function, because reCaptcha needs the client's IP address to validate.

If you've read my previous posts about formlets, you may be wondering why I'm using things like f.Text() and f.Checkbox() (which are members of an object) instead of the regular functions input and checkbox. Those functions are also present in FsFormlets and you may use them interchangeably with the object-style formlets, e.g. instead of f.CheckBox(false) you can write checkbox false []. The object-style formlets build on top of functional formlets, adding optional parameters for validation. They also integrate more seamlessly with Wing Beats.

And we're done with the form! Now let's build the actual page that contains it. Using the layout we wrote earlier:

let s = e.Shortcut

let registrationPage form = 
    layout "Registration" [ 
        s.FormPost "" [ 
            e.Fieldset [ 
                yield e.Legend [ &"Please fill the fields below" ] 
                yield e.Br() 
                yield s.Submit "Register!" 

This is also pure Wing Beats markup, I think it doesn't need much explanation except for the "yield!!+form" which indicates where to render the formlet (yes, the operator doesn't look very friendly, I'll probably change it). Note how easy it is to compose pieces of potentially reusable HTML with Wing Beats as they're just functions.

Handling requests with Figment

Now all we need to do is bind the formlet to a URL to render the page:

get "register" (fun _ -> registrationFormlet "" |> renderToXml |> registrationPage |> Result.wbview)

and handle the form POST:

post "register" (fun ctx -> 
    let env = EnvDict.fromFormAndFiles ctx.Request 
    match run (registrationFormlet ctx.IP) env with 
    | Success v -> Result.redirectf "thankyou?n=%s" v.Name 
    | Failure(errorForm,_) -> errorForm |> registrationPage |> Result.wbview)

Oh, I almost forgot the little "thank you" after-registration page:

get "thankyou" (fun ctx -> Result.contentf "Thank you for registering, %s" ctx.QueryString.["n"])

Now, this workflow we've just modeled is pretty common:

  1. Show form.
  2. User submits form.
  3. Validate form. If errors, show form again to user.
  4. Process form data.
  5. Redirect.

So it's worthy of abstraction. The only moving parts are the formlet, the page and what to do on successful validation, so instead of mapping get and post individually we can say:

formAction "register" { 
    Formlet = fun ctx -> registrationFormlet ctx.IP 
    Page = fun _ -> registrationPage 
    Success = fun _ v -> Result.redirectf "thankyou?n=%s" v.Name 

HTML5 ready

Implementation of HTML5 in browsers has exploded in 2010 (see the beautiful for reference). In particular, HTML5 forms are already implemented in Chrome 10, Opera and Firefox 4 (with caveats). Safari and IE haven't implemented it yet (at least not in versions 5.0.4 and 9 respectively), but WebKit already supports it so I guess Safari and other WebKit-based browsers will implement this in the short-term. So there's little reason not to use the new elements and attributes right now, as long as you also have server-side validation.

For example, here's how Chrome validates required fields when trying to submit:


Whereas in Safari 5.0.4/Windows the server-side validation kicks in:


If you want browser-side validation in Safari, IE, etc, you can easily use an unobtrusive polyfill. A thorough list of cross-browser HTML5 polyfills is available here.

For example, applying is as easy as:

let jsValidation = 
    e.Div [ 
        s.JavascriptFile "" 
        e.Script [ &"$('form').validator();" ] 

and putting it at the bottom of the Wing Beats layout.

All code posted here is part of the sample Figment app.

FsFormlets source code is here.

An overview of my OSS projects

Some time ago I was talking with Michael Maddox about OSS project visibility and long-term maintenance, and one of the things he pointed out is the importance of having a central place that lists all your OSS projects. Not only that, but also which ones are active at any moment. I fully agree with him.

A profile page at ohloh does just that. Ohloh sorts projects by last commit date, and it shows a nice commit activity graph for each project, so you can see exactly what anyone is working on at any moment (provided that he/she has registered the projects on ohloh, of course).

Github profiles also sort projects by last commit and show activity by project and by contributor.

Still, I think it's worth posting a summary every once in a while for reference and maybe even raise awareness about the projects, since most of them are little and quite specific. So allow me to toot my own horn for a bit and list my projects so far:

On the F# front, I've started a few projects. This is where I'm currently spending most of my OSS time.

  • Figment is a web framework inspired on Sinatra and Compojure. Original blog post here.
  • FsSql is a functional wrapper around ADO.NET. Blogged here and here. Recently released 0.1, available on GitHub and NuGet.
  • FsFormlets is an implementation of formlets based on the original papers. I wrote some articles about formlets recently, and I'll write some more in the near future.
  • CsFormlets is a layer on top of FsFormlets that makes it usable on C# and VB.NET. I still have to blog about this one.

I've also been a proud member of the Castle team since 2009. Although I haven't really contributed much code to the core projects (only the occasional patch), I've been more in a support role, answering questions from users on the mailing list and stackoverflow about just anything Castle-related, writing articles about it, helping migrating Castle to git.

I also wrote some Castle-related projects:

  • Quartz.NET integration for Windsor: as the name says, it's a Windsor facility that integrates the scheduler Quartz.NET. Blogged about it here and here.
  • Castle.Windsor.Lifestyles: additional lifestyles for Windsor, blogged about it here.
  • Castle.Fax: the first OSS project I wrote, back in 2007. It includes a facility to integrate Windows Fax Services, which can run on a client or a server. It also includes an IMAP monitor so you can send faxes via email, and a web manager to check the state of the fax server. I haven't touched this since 2009 and I no longer use it (and nobody else, it seems), so I guess it qualifies as a dead project.

As a contributor to other projects:

  • FAKE: I added support for Gallio and some other things that I needed to replace MSBuild in SolrNet.
  • GitSharp: I had a brief participation, helped automating the build and setting up continuous integration.

Most of my projects grew from scratching my own itches and are in production, except the F# projects which are more research-related. This doesn't mean they're not production-worthy, they just have a different origin and focus.