Fuchu: a functional test library for .NET

In my last two posts I showed how MbUnit supports first-class tests, and how you could use that to build a DSL in F# around it.

I explained how many concepts in typical xUnit frameworks can be more simply expressed when tests are first-class values, which is not the case for most .NET and Java test frameworks.

More concretely, test setup/teardown is a function over a test, and parameterized tests are... just data manipulation.

Since first-class tests greatly simplify things, why not dispense with the typical class-based, attribute-driven approach and build a test library around first-class tests? Well, Haskellers have been doing this for at least 10 years now, with HUnit.

HUnit organizes tests using this tree:

-- | The basic structure used to create an annotated tree of test cases. 
data Test 
    -- | A single, independent test case composed. 
    = TestCase Assertion 
    -- | A set of @Test@s sharing the same level in the hierarchy. 
    | TestList [Test] 
    -- | A name or description for a subtree of the @Test@s. 
    | TestLabel String Test 

Where Assertion is simply an alias for IO (). This is all you need to organize tests in suites and give them names.

We can trivially translate this to F# :

type TestCode = unit -> unit

type Test = 
    | TestCase of TestCode
    | TestList of Test seq
    | TestLabel of string * Test

Let's see an example:

let testA = 
    TestLabel ("testsuite A", TestList 
                         [
                            TestLabel ("test A", TestCase(fun _ -> Assert.AreEqual(4, 2+2)))
                            TestLabel ("test B", TestCase(fun _ -> Assert.AreEqual(8, 4+4)))
                         ])

It's quite verbose, but we can define the same DSL as I defined earlier for MbUnit tests, so this becomes:

let testA = 
    "testsuite A " =>> [
        "test A" => 
            fun _ -> Assert.AreEqual(4, 2+2)
        "test B" =>
            fun _ -> Assert.AreEqual(8, 4+4)
    ]

Actually, I first ported HUnit (including this DSL), then discovered that MbUnit has first-class tests and later wrote the DSL around MbUnit. Everything I described in those posts (setup/teardown as higher-order functions, parameterized tests as simple data manipulation, arbitrary nesting of test suites) applies here in the exact same way.

In fact, MbUnit's class hierarchy of Test/TestSuite/TestCase can be read as the following algebraic data type:

type Test =
| TestSuite of string * Test list
| TestCase of string * Action

which turns out to be very similar to the tree we translated from HUnit, only the names are embedded instead of being a separate case.

I called this HUnit port Fuchu (it doesn't mean anything), it's on github.

Assertions

Fuchu doesn't include any assertion functions, or at least not yet. (EDIT: assertions were added in 0.2.0) It only gives you tools to organize and run tests, but you're free to use NUnit, MbUnit, xUnit, NHamcrest, etc, or more F#-ish solutions like Unquote or FsUnit or NaturalSpec for assertions.

Tighter integration with FsCheck is planned. (EDIT: it was added in the first release of Fuchu)

Runner/Tooling

As with HUnit, the test assembly is the runner itself. That is, as opposed to having an external test runner as with most test frameworks, your test assembly is an executable (a console application). This is because it's more of a library instead of a framework. As a consequence, there is no need of installing any external tool to run tests (just hit CTRL-F5 in Visual Studio) or debug tests (just set your breakpoints and hit F5 in Visual Studio). Here's a clear signal of why this matters:

How does one run a #fsharp unit test project in #vs11?

— Community for F# (@c4fsharp) March 23, 2012

So how do you run tests with Fuchu? Given a test suite testA like the one defined above, you can run it like this:

[<EntryPoint>]
let main _ = run testA // or runParallel

But this is quite inconvenient, as it's common to split tests among different modules/files and this would mean having to list all tests somewhere, to feed them to the run function. HUnit works around this using Template Haskell, and OUnit (OCaml's port of HUnit) users generate the boilerplate code by parsing the tests' source code.

In .NET we can just decorate the tests with an attribute and then use reflection to fetch them:

[<Tests>]
let testA = 
    "2+2=4" => 
        fun _ -> Assert.AreEqual(4, 2+2)

[<Tests>]
let testB = 
    "2*3=6" => 
        fun _ -> Assert.AreEqual(7, 2*3)

[<EntryPoint>]
let main args = defaultMainThisAssembly args

This function defaultMainThisAssembly does exactly what it says on the tin. Notice that it also takes the command-line args, so if you call it with "/m" it will run the tests in parallel. (Curiously, you can't say let main = defaultMainThisAssembly, it won't be recognized as the entry point).

By the way, this is just an example, you wouldn't normally annotate every single test with the Tests attribute, only the top-level test group per module.

Run this code and you get an output like this:

2*3=6: Failed: 
  Expected: 7
  But was:  6

  G:\prg\Test.fs(15,1): SomeModule.testB@15.Invoke(Unit _arg1)
 (00:00:00.0017681)

2 tests run: 1 passed, 0 ignored, 1 failed, 0 errored (00:00:00.0058780)

If you run this within Visual Studio with F5 you can navigate to the failing assertion by clicking on the line that looks like a mini stack trace.

REPL it!

Since running tests is as easy as saying "run test", it's also convenient sometimes to do so from the F# REPL.

Pros:

• You can directly load the source code under tests in the REPL, which cuts down compilation times.
• Easy to cherry-pick one or a few tests to run instead of running all tests (with the provided Test.filter function)

Cons:

• Having to manually load all dependencies to the tests. It may be possible to work around this using a variant of this script by Gustavo Guerra.
• If you reference the assembly under test in the REPL, fsi.exe blocks the DLLs, so you have to reset the REPL session to recompile. But if you're testing F# code, you can work around this by loading source code instead of referencing the assembly.

You can see an example of running tests from the REPL here.

Other tools

Integrating with other tools is not simple. Most tools out there seem to assume that tests are organized in classes, and each test corresponds to a method or function. This also happens with MbUnit's StaticTestFactory: for example in ReSharper or TestDriven.Net you can't single out tests. Still, they can be made to run let-bound tests (which may be a test suite), so it should be possible to have some support within this limitation.

Also, no immediate support for any continuous test runner. I checked with Greg Young, he tells me that MightyMoose/AutoTest.NET can be configured to use an arbitrary executable (with limitations). Remco Mulder, of NCrunch, suggested wrapping the test runner in a test from a known test framework as a workaround. Maybe executing the tests after compilation (with a simple AfterBuild msbuild target) is enough. I haven't looked into this yet.

Coverage tools should have no problem, it makes no difference where the executable comes from.

Build tools should have no issues either; obviously FAKE is the more direct option, but I see no problems integrating this with other build tools.

C# support

I threw in a few wrapper functions to make this library usable in C# / VB.NET. Of course, it will never be as concise as F#, but still usable. I'm not going to fully explain this (it's just boring sugar) but you can see an example here.

NUnit/MbUnit test loading

Even though it may seem very different, Fuchu is still built on xUnit concepts. And since tests are first-class values, it's very easy to map tests written with other xUnit framework to Fuchu tests. For example, building Fuchu tests from NUnit test classes takes less than 100 LoC (it's already built into Fuchu)

This lets you use Fuchu as a runner for existing tests, and to write new tests. I'm planning to use this soon in SolrNet to replace Gallio (for example, Gallio doesn't work on Mono).

There is a limitation here: Fuchu can't express TestFixtureTearDowns. It can do TestFixtureSetups (and obviously SetUp/TearDown, as explained in previous posts), but not TestFixtureTearDowns (or at least not unless you treat that test suite separately). Give it a try and see for yourself :) . Is it a real downside? I don't think so (for example, TestFixtureTearDowns make parallelization harder), but it's something to be aware of. Also I haven't looked into test inheritance yet, but it should be pretty easy to support it.

Conclusions

Does .NET really need yet another test framework? Absolutely not. The current test frameworks are "good enough" and hugely popular. But since they don't treat tests as first-class values, extending them results in more and more complexity. Consider the lifecycle of a test in a typical unit testing framework. Inheritance and multiple marker attributes make it so complex that it reminds me of the ASP.NET page lifecycle. 

What I propose with Fuchu is a hopefully simpler, no-magic model. Remember KISS?

Code is here. Binaries on NuGet.

An F# DSL for MbUnit

In F# we typically organize tests much like in C# or VB.NET: writing functions marked with a [<Test>] attribute or similar. Actually there's a slight advantage in F#: you don't need to write a class marked as test fixture, you can directly write the tests as let-bound functions. Still, it's fundamentally the same model. (If you're into BDD there's also TickSpec as an alternative model).

Since it's the same model, you get the same issues I described in my last post, and then some: for example as Kurt explains, attributes in F# sometimes aren't treated exactly as in C#.

Also in my last post, I wrote about how MbUnit supports first-class tests as an alternative to attribute-defined tests. In F# we can take advantage of this and custom operators to build a very concise DSL to define tests.

First let's see a small test suite with setup/teardown, written with the classic attributes:

[<TestFixture>]
type ``MemoryStream tests``() =
    let mutable ms : MemoryStream = null

    [<SetUp>]
    member x.Setup() =
        ms <- new MemoryStream()

    [<TearDown>]
    member x.Teardown() =
        ms.Dispose()

    [<Test>]
    member x.``Can read``() =
        Assert.IsTrue ms.CanRead

    [<Test>]
    member x.``Can write``() =
        Assert.IsTrue ms.CanWrite

Looks simple enough, right? And yet, the mutable field is a smell, or at least an indicator that this isn't functional. Let's try to get rid of that mutable.

As a first step we'll rewrite this as first-class tests, that is, using [<StaticTestFactory>] as shown in my last post:

[<StaticTestFactory>]
let testFactory() =
    let suite = TestSuite("MemoryStream tests")
    let ms : MemoryStream ref = ref null
    suite.SetUp <- fun () -> ms := new MemoryStream()
    suite.TearDown <- fun () -> (!ms).Dispose()
    let tests = [
        TestCase("Can read", 
            fun () -> Assert.IsTrue (!ms).CanRead)
        TestCase("Can write", 
            fun () -> Assert.IsTrue (!ms).CanWrite)
    ]
    Seq.iter suite.Children.Add tests
    [suite]

Oh great, that's even uglier than what we started with! And we have replaced the mutable field with a ref cell, not much of an improvement. 
But bear with me, we have first-class tests now so there's a lot a room for improvement.

In order to keep refactoring this, we need to realize that the problem is that our test cases should be functions MemoryStream -> unit instead of unit -> unit. That way, they wouldn't have to depend on an external MemoryStream instance; instead the instance would be pushed somehow to the test. Let's write that:

let tests = [
    "Can read", (fun (ms: MemoryStream) -> Assert.IsTrue ms.CanRead)
    "Can write", (fun ms -> Assert.IsTrue ms.CanWrite)
]

Now we have this list of strings and MemoryStream -> unit functions. What we need now is to turn these functions into unit -> unit so we can ultimately build TestCases.

In other words, we need a function (MemoryStream -> unit) -> (unit -> unit). This function should create the MemoryStream, pass it to our test function, then dispose the MemoryStream. Hey, what do you know, turns out that's just what SetUp and TearDown do!

Still with me? It's much easier to see this in code:

let withMemoryStream f () = 
    use ms = new MemoryStream()
    f ms

Now we apply this to our list, building the TestCases and then the TestSuite:

[<StaticTestFactory>]
let testFactory() =
    let suite = TestSuite("MemoryStream tests")
    tests 
    |> Seq.map (fun (n,t) -> TestCase(n, Gallio.Common.Action(withMemoryStream t)))
    |> Seq.iter suite.Children.Add
    [suite]

We've eliminated all mutable references, and also replaced SetUp/TearDown with a simple higher-order function.

But we can do still better, in terms of readability. We can define a few custom operators to hide the TestSuite and TestCase constructors:

let inline (=>>) name tests =
    let suite = TestSuite(name)
    Seq.iter suite.Children.Add tests
    suite :> Test

let inline (=>) name (test: unit -> unit) =
    TestCase(name, Gallio.Common.Action test) :> Test

[<StaticTestFactory>]
let testFactory() =
    [
        "MemoryStream tests" =>> [
            "Can read" => 
                withMemoryStream(fun ms -> Assert.IsTrue ms.CanRead)
            "Can write" => 
                withMemoryStream(fun ms -> Assert.IsTrue ms.CanWrite)
        ]
    ]

And with a couple more operators we get rid of the duplicate call to withMemoryStream:

let inline (+>) f =
     Seq.map (fun (name, partialTest) ->
                    name => f partialTest)

let inline (==>) (name: string) test = name,test

[<StaticTestFactory>]
let testFactory() =
    [
        "MemoryStream tests" =>>
            withMemoryStream +> [
                "Can read" ==> 
                    fun ms -> Assert.IsTrue ms.CanRead
                "Can write" ==>
                    fun ms -> Assert.IsTrue ms.CanWrite
            ]
    ]

Conclusions

Confused about all those kinds of arrows? The good thing about first-class tests is that you can build them any way you want, no need to use these operators if you don't like them. That's also precisely one of its downsides: as there is no fixed idiom, it can get harder to read compared to attribute-based test definitions, where there is a single, well-defined way to do things.

In my last post I showed how first-class tests practically eliminate the concept of parameterized tests. In this post I showed how they eliminate the concept of setup/teardown, replacing them with a higher-order function, a more generic concept.

More generally, I'd say that whatever domain you're modeling (in this case, tests), there is much to gain if the core concepts are representable as first-class values. It should also be noted that different languages have very different notions of what language objects are first-class values. Some are more flexible than others, but that doesn't imply any superiority by itself. However it does mean that if you're not aware of this you'll probably misuse your language and end up with ever more complex workarounds to manipulate your domain objects as values. Nice APIs, conventions, configuration, etc, are all secondary and can be built much more easily on top of composable, first-class building blocks.

But I digress. In the next post I'll show a simple testing library built around tests as first-class values and more pros/cons about this approach.

First-class tests in MbUnit

Originally, xUnit style testing frameworks used inheritance to define tests. SUnit, the original xUnit framework, builds test cases by inheriting the TestCase class. NUnit 1.0 and JUnit derived from SUnit and also used inheritance. Fast-forward to today, unit testing frameworks in .NET and Java typically organize tests using attributes/annotations instead.

For a few years now, MbUnit has been able to define tests programmatically as an alternative, though it seems this feature isn't used much. Let's compare attributes vs programmatic tests with a simple example in C#:

Attributes

[TestFixture]
public class TestFixture {
    [Test]
    public void Test() {
        Assert.AreEqual(4, 2 + 2);
    }

    [Test]
    public void AnotherTest() {
        Assert.AreEqual(8, 4 + 4);
    }
}

Programmatic

public class TestFixture {
    [StaticTestFactory]
    public static IEnumerable<Test> Tests() {
        yield return new TestCase("Test", () => {
            Assert.AreEqual(4, 2 + 2);
        });

        yield return new TestCase("Another test", () => {
            Assert.AreEqual(8, 4 + 4);
        });
    }
}

At first blush, declaring tests programmatically is more verbose and complex. However, the real difference is that these tests are first-class values. It becomes more clear why this matters with an example of parameterized tests:

Attributes

[TestFixture]
public class TestFixture {
    [Test]
    [Factory("Parameters")]
    public void Parse(string input, DateTime expectedOutput) {
        var r = DateTime.ParseExact(input, "yyyy-MM-dd'T'HH:mm:ss.FFF'Z'", CultureInfo.InvariantCulture);
        Assert.AreEqual(expectedOutput, r);
    }

    IEnumerable<object[]> Parameters() {
        yield return new object[] { "1-01-01T00:00:00Z", new DateTime(1, 1, 1) };
        yield return new object[] { "2004-11-02T04:05:20Z", new DateTime(2004, 11, 2, 4, 5, 20) };
    }
}

Programmatic

public class TestFixture {
    [StaticTestFactory]
    public static IEnumerable<Test> Tests() {
        var parameters = new[] {
            new { input = "1-01-01T00:00:00Z", expectedOutput = new DateTime(1, 1, 1) },
            new { input = "2004-11-02T04:05:20Z", expectedOutput = new DateTime(2004, 11, 2, 4, 5, 20) },
        };

        return parameters.Select(p => new TestCase("Parse " + p.input, () => {
            var r = DateTime.ParseExact(p.input, "yyyy-MM-dd'T'HH:mm:ss.FFF'Z'", CultureInfo.InvariantCulture);
            Assert.AreEqual(p.expectedOutput, r);
        }));
    }
}

Programmatically, we just wrote the parameters and tests in a direct style. With attributes, not only we lost the types but also it's more complicated: you have to know  (or look up in the documentation) that you need a [Factory] attribute, that its string parameter indicates the method name that contains the test parameters, and the format for the parameters (e.g. can they be represented as a property? As a field? Can it be private? Static? Can it be a non-generic IEnumerable? An ArrayList[]?). Fortunately, MbUnit is quite flexible about it. Yet it doesn't handle an ArrayList[].

Something similar happens with JUnit and TestNG. Actually JUnit did have something close to first-class tests with its inheritance API.

With programmatic tests, you simply return a list of tests, there's no magic about it. It doesn't matter how they're built, they can be parameterized or not, all you have to know is [StaticTestFactory] public static IEnumerable<Test> Tests() . If they're parameterized, it doesn't matter what kind of parameters they are. Actually, the very concept of "parameterized tests" simply disappears.

With attributes, you may have tried to use [Row] first, only to have the compiler remind you that attribute parameter types are very limited and you can't have a DateTime. Or a function. Or even a decimal. The testing framework gets in the way. Attributes are just not the right tool to model this.

With programmatic tests, you are in control, not the testing framework. It becomes more of a library rather than a framework. Things are conceptually simpler.

What about SetUp and TearDown? Don't worry, MbUnit supports them directly as properties of TestSuite. However, as we'll see in the next post, they're not really necessary. We'll also see a few other pros/cons first-class tests have.

I'll leave you with this quote from the twitter-fake Alain de Botton:

I understand that annotations beat boilerplate XML config.But what's wrong with, you know, writing real code instead?

— PLT Alain de Botton (@PLTAlaindeB) March 13, 2012

Moroco: a minimal mocking library for C# / VB.NET

The more I learn about functional programming, the more I come to question many widely used and accepted practices in mainstream programming.

This time it's the turn of mocks and mocking libraries.

First, since there are so many different definitions for stubs, mocks, fakes, etc, here's my own definition of a mock: an entity (in an object-oriented language, usually an object) used to test an interaction between the entity under test and an external entity (again, in OO languages, these entities are objects).

So mocks are used for interaction-based testing which means testing for side-effects. The original paper on mock objects says this explicitly: "Test code should communicate its intent as simply and clearly as possible. This can be difficult if a test has to set up domain state or the domain code causes side effects."

Side effects are code smells, or more precisely, they should be few and isolated. Even the creators of mock object say that side effects make testing difficult! We should minimize the need for mocking. Code without side-effect doesn't need mocks. You still may need stubs or fakes, but those should be trivial to build.

Quoting Daniel Cazzulino (author of Moq): "The sole presence of a 'Verify' method on the mock is a smell to me, one that will slowly get you into testing the interactions as opposed to testing the observable state changes caused by a particular behavior.". Make those states immutable (i.e. a state change create a new state) and you're half-way to side-effect-free code.

Remember that discussion a few years ago where people said that Typemock was too powerful? The argument was that Typemock "doesn't force you to write testable code". What this really means is "it doesn't make you isolate side-effects". I'm thinking that all current mocking libraries are actually too powerful: instead of making you think how to write pure (side-effect free) code, it encourages you to just use a mock to replace your impure code with pure code in tests.

There's also the matter of library complexity. .NET mocking libraries are big and complex: Moq: 17000 LoC, NSubstitute: 12000 LoC, Rhino Mocks: 87000 LoC, FakeItEasy: 17000 LoC. Not counting the embedded runtime proxy library (usually Castle DynamicProxy).

Many have issues running mocks in parallel (1, 2, 3, 4). It's 2012 and most developers have at least a 4-core workstation, using a mock library that limits my ability to run tests in parallel is getting ridiculous.

In summary, I think mock libraries support an undesirable practice, are not worth their code and have to go, except for very specific scenarios. But I still have a lot of existing side-effecting code I have to test, I can't just wish it away. Refactoring to pure code is not trivial. So I decided that

I'm switching to manual mocking and using the mock count as a measure of code smell.

— Mauricio Scheffer (@mausch) December 19, 2011

To which I got an encouraging reply:

@mausch Lemme know if you need help achieving 0.

— Tony Morris (@dibblego) December 20, 2011

By the way I'm not the first or the only one that thinks that mocking libraries aren't worth it. Uncle Bob also prefers manual mocking (even in Java, which is much more verbose than any .NET language), though he mostly stresses the argument of simplicity.

In F# it's easy to do manual mocking thanks to object expressions (1, 2), so you don't have to actually create a new class for each mock. In C#/VB.NET we're not so lucky but we can get 80% there with a little boilerplate, making a "semi-manual", reusable mock class with settable Funcs to define behavior. Example:

interface ISomething {
    void DoSomething(int a);
}

class MockSomething: ISomething {
    public Action<int> doSomething;

    public void DoSomething(int a) {
        doSomething(a);
    }
}

class Test {
    void test() {
        var r = new List<int>();
        var s = new MockSomething {
            doSomething = r.Add
        };
        // etc
    }
}

This isn't anything new, people have been doing this for years. Downsides: doesn't play well with overloaded and generic methods, but still works.

Also, as Uncle Bob explains, manual mocks are prone to break whenever you change the mocked interface, but if you hit this often it could be revealing you that perhaps you should have used an abstract base class instead.

I added some code to track the call count of a Func and named the resulting library Moroco. So I wanted to get away from mocking libraries and ended up writing one, talk about hypocrisy! The difference between Moroco and other mocking libraries is that it's really minimal: less than 400 lines of pretty trivial code, fitting in a single file, with no dependencies. And I'm still against mocks: I get to count mocks to measure mock smell. And run tests in parallel.

I'm already using it in SolrNet, where I simply dropped Moroco's source code in the test project and replaced Rhino.Mocks in all tests. Here's the 'before vs after' of one of the tests:

Rhino.Mocks

[Test]
public void Extract() {
    var mocks = new MockRepository();
    var connection = mocks.StrictMock<ISolrConnection>();
    var extractResponseParser = mocks.StrictMock<ISolrExtractResponseParser>();
    var docSerializer = new SolrDocumentSerializer<TestDocumentWithoutUniqueKey>(new AttributesMappingManager(), new DefaultFieldSerializer());
    var parameters = new ExtractParameters(null, "1", "test.doc");
    With.Mocks(mocks)
        .Expecting(() => {
            Expect.On(connection)
                .Call(connection.PostStream("/update/extract", null, parameters.Content, new List<KeyValuePair<string, string>> {
                    new KeyValuePair<string, string>("literal.id", parameters.Id),
                    new KeyValuePair<string, string>("resource.name", parameters.ResourceName),
                }))
                .Repeat.Once()
                .Return(EmbeddedResource.GetEmbeddedString(GetType(), "Resources.responseWithExtractContent.xml"));
            Expect.On(extractResponseParser)
                .Call(extractResponseParser.Parse(null))
                .IgnoreArguments()
                .Return(new ExtractResponse(null));
        })
        .Verify(() => {
            var ops = new SolrBasicServer<TestDocumentWithoutUniqueKey>(connection, null, docSerializer, null, null, null, null, extractResponseParser);
            ops.Extract(parameters);
        });
}

Moroco

[Test]
public void Extract() {
    var parameters = new ExtractParameters(null, "1", "test.doc");
    var connection = new MSolrConnection();
    connection.postStream += (url, contentType, content, param) => {
        Assert.AreEqual("/update/extract", url);
        Assert.AreEqual(parameters.Content, content);
        var expectedParams = new[] {
            KV.Create("literal.id", parameters.Id),
            KV.Create("resource.name", parameters.ResourceName),
        };
        Assert.AreElementsEqualIgnoringOrder(expectedParams, param);
        return EmbeddedResource.GetEmbeddedString(GetType(), "Resources.responseWithExtractContent.xml");
    };
    var docSerializer = new SolrDocumentSerializer<TestDocumentWithoutUniqueKey>(new AttributesMappingManager(), new DefaultFieldSerializer());
    var extractResponseParser = new MSolrExtractResponseParser {
        parse = _ => new ExtractResponse(null)
    };
    var ops = new SolrBasicServer<TestDocumentWithoutUniqueKey>(connection, null, docSerializer, null, null, null, null, extractResponseParser);
    ops.Extract(parameters);
    Assert.AreEqual(1, connection.postStream.Calls);
}

Yes, I do realize this uses an old Rhino.Mocks API, but it's necessary to get thread-safety.

Conclusion

No, I don't expect anyone to use Moroco instead of Moq, Rhino.Mocks, etc. Yes, I know this means more code (though it's not as much as you might think), and I agree that .NET needs another mock library like I need a hole in my head. But I think we should think twice before using a mock and see if we can find a side-effect-free alternative for some piece of code.
Even when you do use mocks, don't just blindly reach for a mocking library. Consider the trade-offs.

Testing IIRF rules with MbUnit

If you're running IIS 6 like me, you know there aren't many options if you want extensionless URLs. I already had a custom URL rewriting engine in place with Windsor integration and stuff, but it couldn't handle extensionless URLs. The 404 solution seems kinda kludgy to me so after some pondering I decided to go with IIRF.

Ok, now let's see how do we test IIRF rules. IIRF is a regular Win32 DLL written in C, so there are no managed hook points that we can use from .Net. Fortunately, it comes with a testing tool called TestDriver, which is a standard .exe that takes a file with source and destination URLs and runs all source URLs against the rules, asserting that each result matches the destination.

All we need now is some glue code to integrate this to our regular tests so if any routes fail it also makes the whole build fail. I also want to see on my TeamCity build log exactly which route failed, and also be able to easily add new route tests. We can do all this with MbUnit's [Factory] and some stdout parsing. Here's the code:

[TestFixture]
public class IIRFTests {
    [Test]
    [Factory("IIRFTestFactory")]
    public void IIRFTest(string orig, string dest) {
        File.WriteAllText(@"SampleUrls.txt", string.Format("{0}\t{1}", orig, dest));
        var r = RunProcess(@"TestDriver.exe", @"-d .");
        if (r.ErrorLevel != 0) {
            var actual = Regex.Replace(r.Output.Replace("\r\n", " "), @".*actual\((.*)\).*", "$1");
            Assert.AreEqual(dest, actual);
        }
    }

    public IEnumerable<object[]> IIRFTestFactory() {
        yield return new object[] { "/questions/167586/visual-studio-database-project-designers", "/Question/Index.aspx?id=167586" };
        yield return new object[] { "/users/21239/mauricio-scheffer", "/User/Index.aspx?id=21239" };
    }

    public ProcessOutput RunProcess(string fileName, string arguments) {
        var p = Process.Start(new ProcessStartInfo(fileName, arguments) {
            CreateNoWindow = true,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false,
        });
        p.WaitForExit();
        return new ProcessOutput(p.ExitCode, p.StandardOutput.ReadToEnd() + p.StandardError.ReadToEnd());
    }

    public class ProcessOutput {
        public int ErrorLevel { get; private set; }
        public string Output { get; private set; }

        public ProcessOutput(int errorLevel, string output) {
            ErrorLevel = errorLevel;
            Output = output;
        }
    }
}

Testing private methods with C# 4.0

Here's another practical use for dynamic in C# 4: testing private methods. I won't discuss here whether this should or shouldn't be done, that's been argued to death for years. Instead, I'll show how it can be done easier with C# 4, without resorting to reflection (at least not directly) or stubs, while also reducing some of the friction associated with doing TDD in statically-typed languages like C#.

The answer is very simple: we just walk away from static typing using C# dynamic:

public class Service {
    private int Step1() {
        return 1;
    }
}

[TestClass]
public class TransparentObjectTests {
    [TestMethod]
    public void PrivateMethod() {
        dynamic s = new Service().AsTransparentObject();
        Assert.AreEqual(1, s.Step1());
    }
}

When going dynamic, it's kind of like turning off the compiler, so, for example, it won't bother you at compile-time if a method isn't present. Instead, you'll just get an exception when you run your test, which is more like the dynamic language way of testing things.

Here's the code that enables this. Just like my last post, it's just a spike, it's incomplete and I didn't test it properly, but it could serve as a base for a more complete implementation:

using System;
using System.Dynamic;
using System.Linq;
using System.Reflection;
using Microsoft.CSharp.RuntimeBinder;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace DynamicReflection {
    public class Service {
        private int Step1() {
            return 1;
        }

        protected int Step2() {
            return 2;
        }

        internal int Step3() {
            return 3;
        }

        private int Prop { get; set; }

        public int Execute(int a) {
            return Step1() + Step2() + Step3() + a;
        }

        private string ExecuteGeneric<T>(T a) {
            return a.ToString();
        }
    }

    [TestClass]
    public class TransparentObjectTests {
        [TestMethod]
        public void PublicMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(10, s.Execute(4));
        }

        [TestMethod]
        public void PrivateMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(1, s.Step1());
        }

        [TestMethod]
        public void ProtectedMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(2, s.Step2());
        }

        [TestMethod]
        public void InternalMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(3, s.Step3());
        }

        [TestMethod]
        public void PrivateProperty() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(0, s.Prop);
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentException))]
        public void GenericPrivateMethod_type_mismatch() {
            dynamic s = new Service().AsTransparentObject();
            s.ExecuteGeneric<int>("lalal"); // type param mismatch
        }

        [TestMethod]
        public void GenericPrivateMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual("1", s.ExecuteGeneric<int>(1));
        }
    }

    public static class TransparentObjectExtensions {
        public static dynamic AsTransparentObject<T>(this T o) {
            return new TransparentObject<T>(o);
        }
    }

    public class TransparentObject<T> : DynamicObject {
        private readonly T target;

        public TransparentObject(T target) {
            this.target = target;
        }

        public override bool TryGetMember(GetMemberBinder binder, out object result) {
            var members = typeof(T).GetMember(binder.Name, BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
            var member = members[0];
            if (member is PropertyInfo) {
                result = (member as PropertyInfo).GetValue(target, null);
                return true;
            }
            if (member is FieldInfo) {
                result = (member as FieldInfo).GetValue(target);
                return true;
            }
            result = null;
            return false;
        }

        public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) {
            var csBinder = binder as CSharpInvokeMemberBinder;
            var method = typeof(T).GetMethod(binder.Name, BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
            if (method == null)
                throw new ApplicationException(string.Format("Method '{0}' not found for type '{1}'", binder.Name, typeof(T)));
            if (csBinder.TypeArguments.Count > 0)
                method = method.MakeGenericMethod(csBinder.TypeArguments.ToArray());
            result = method.Invoke(target, args);
            return true;
        }
    }
}

UPDATE 5/17/2010: Igor Ostrovsky recently got the same idea outlined here and has written a proper implementation of this.

Google Chrome caches 301 redirects

Ever since the release of Chrome I've alternating between it and Firefox for casual browsing. At work, I mainly use Chrome for information seeking because of its sheer speed, but nothing beats Firefox + Firebug for web development and javascript debugging. Sometimes, however, I mix my browsers and run some non-javascript stuff on Chrome (of course I test javascript-heavy stuff on all main browsers).

So a couple of days ago I was checking a seemingly innocent piece of code on Chrome, something like:

Response.StatusCode = 301;
Response.RedirectLocation = newUrl;

and I messed up a querystring parameter on newUrl. I fixed it, ran it again, and it still pointed to the wrong URL. Hmm, was I using the right port (sometimes I run multiple branches of different ports)? Yes. So I set a breakpoint on Response.RedirectLocation, and it didn't fire. WTF?! After some more hair-pulling and cursing (spanish is a really wonderful language for cursing, you know, there's even a whole mathematical theory behind it⁽¹⁾) I ran it on Firefox and it worked as expected. And then it hit me: it is a permanent redirection after all, isn't it? So why not cache it? I ran it on every other browser I had (IE, Safari, Opera), they all behaved like Firefox, the only one caching the redirect was Chrome.

I couldn't believe it, so I wrote a simple "proof": an app that gives you a link, when you click the link it creates a cookie and redirects (301) to another page where it shows the content of the cookie and deletes it. Source code is here (nothing interesting) and you can run it here. Point your browser to http://localhost:12345/. Click the link. You'll see "Original=/first" as the response. Ok, now go back to root. Click the link again. Now, if your browser cached the redirect, you'll see "Original=", since the /first URL wasn't requested again. Go back to root, hit F5 and it clears the cache.

I don't mean to say this is wrong behavior, it's just that it's the first browser I see that implements 301 redirect caching, it's something to be aware of while developing and testing.

(1) For the non-Spanish-speaking readers: it's just a joke ;-)

Injectable file adapters

Someone must have done this, but I really couldn't find it. I'm talking about an IoC-friendly System.IO.File replacement. You know, unit tests shouldn't touch the file system, etc. So if your code writes a file and want to unit-test it, you pretty much have to mock the writing of the file. Except there's a known problem, System.IO.File is a static class, and those are not mockable. Even TypeMock can't mock System.IO.File since it's part of mscorlib. So the only solution left is to build an interface and a wrapper around File. So, here's the source, probably the most boring code I've ever written. I've also included a static locator that gets the IFile implementation using Ayende's IoC static accessor to the Windsor Container, so you can write code like this:

[Test]
public void Copy() {
    var mocks = new MockRepository();
    var container = mocks.CreateMock<IWindsorContainer>();
    var fileImpl = mocks.CreateMock<IFile>();
    IoC.Initialize(container);
    With.Mocks(mocks).Expecting(delegate {
        SetupResult.For(container.Resolve<IFile>()).Return(fileImpl);
        Expect.Call(() => fileImpl.Copy(null, null))
            .IgnoreArguments()
            .Repeat.Once();
    }).Verify(delegate {
        FileEx.Copy("source", "dest");                
    });
}

So you only have to replace your calls to System.IO.File to FileEx and that's it. You get the benefits of testability and extensibility (yes, sometimes you need to provide a different behavior for File.WriteAllText()) and you don't have to deal with interfaces, implementations, etc once you have set up the container. Personally, I prefer to make explicit the dependency for IFile in my components.

Like I said, I was very surprised that I couldn't find a working implementation of this... specially because there was a big debate a year ago about maintenability, YAGNI, dependency injection, coupling and more, and Anders Norås commented this solution on one of his own posts.

Well, I hope someone finds this useful.

Poor man's javascript testing

I'm sure someone else has done this, being so simple and old tech... But I couldn't find it anywhere, soo...

Let's say you have just finished writing a great, I mean really ground-breaking javascript library, and you call it lib.js:

Array.prototype.clear = function(){
  this.length=0;
}

And you want to test it. But how? Well, you can forget all about JsUnit, Script.aculo.us BDD, Crosscheck or the others, because now you can test your javascript with... cscript!! Yes, just write a little WSF that includes your great library:

<job>
    <script src="lib.js" language="jscript"/>
    <script language="jscript">
        var a = [1,2,3];
        a.clear();
        if (a.length != 0) {
            WScript.Echo("test failed");
            WScript.Quit(1);
        } else {
            WScript.Echo("test passed");
            WScript.Quit(0);
        }
    </script>
</job>

To execute the test, just invoke cscript on the WSF. Isn't it cool? Ok, ok, it's not cool at all, it only works for non-DOM javascript and only tests for internet explorer, BUT it's really simple and you get the potential to write the tests in another language ^[1], AND it can be easily integrated to a nant build with a <exec> task. In any case, it beats having no tests at all.

PS: Now seriously, go and check out the tools I mentioned above. GO GO!

[1] Doesn't really work most of the times. I tried ActivePython and ActiveRuby, but they don't see the functions exposed in jscript. Not to mention prototype modifications like the example above, since it doesn't make sense to other languages... VBScript seems to cooperate nicely but only for the most basic scenarios.