First-class tests in MbUnit

Originally, xUnit style testing frameworks used inheritance to define tests. SUnit, the original xUnit framework, builds test cases by inheriting the TestCase class. NUnit 1.0 and JUnit derived from SUnit and also used inheritance. Fast-forward to today, unit testing frameworks in .NET and Java typically organize tests using attributes/annotations instead.

For a few years now, MbUnit has been able to define tests programmatically as an alternative, though it seems this feature isn't used much. Let's compare attributes vs programmatic tests with a simple example in C#:

Attributes

[TestFixture]
public class TestFixture {
    [Test]
    public void Test() {
        Assert.AreEqual(4, 2 + 2);
    }

    [Test]
    public void AnotherTest() {
        Assert.AreEqual(8, 4 + 4);
    }
}

Programmatic

public class TestFixture {
    [StaticTestFactory]
    public static IEnumerable<Test> Tests() {
        yield return new TestCase("Test", () => {
            Assert.AreEqual(4, 2 + 2);
        });

        yield return new TestCase("Another test", () => {
            Assert.AreEqual(8, 4 + 4);
        });
    }
}

At first blush, declaring tests programmatically is more verbose and complex. However, the real difference is that these tests are first-class values. It becomes more clear why this matters with an example of parameterized tests:

Attributes

[TestFixture]
public class TestFixture {
    [Test]
    [Factory("Parameters")]
    public void Parse(string input, DateTime expectedOutput) {
        var r = DateTime.ParseExact(input, "yyyy-MM-dd'T'HH:mm:ss.FFF'Z'", CultureInfo.InvariantCulture);
        Assert.AreEqual(expectedOutput, r);
    }

    IEnumerable<object[]> Parameters() {
        yield return new object[] { "1-01-01T00:00:00Z", new DateTime(1, 1, 1) };
        yield return new object[] { "2004-11-02T04:05:20Z", new DateTime(2004, 11, 2, 4, 5, 20) };
    }
}

Programmatic

public class TestFixture {
    [StaticTestFactory]
    public static IEnumerable<Test> Tests() {
        var parameters = new[] {
            new { input = "1-01-01T00:00:00Z", expectedOutput = new DateTime(1, 1, 1) },
            new { input = "2004-11-02T04:05:20Z", expectedOutput = new DateTime(2004, 11, 2, 4, 5, 20) },
        };

        return parameters.Select(p => new TestCase("Parse " + p.input, () => {
            var r = DateTime.ParseExact(p.input, "yyyy-MM-dd'T'HH:mm:ss.FFF'Z'", CultureInfo.InvariantCulture);
            Assert.AreEqual(p.expectedOutput, r);
        }));
    }
}

Programmatically, we just wrote the parameters and tests in a direct style. With attributes, not only we lost the types but also it's more complicated: you have to know  (or look up in the documentation) that you need a [Factory] attribute, that its string parameter indicates the method name that contains the test parameters, and the format for the parameters (e.g. can they be represented as a property? As a field? Can it be private? Static? Can it be a non-generic IEnumerable? An ArrayList[]?). Fortunately, MbUnit is quite flexible about it. Yet it doesn't handle an ArrayList[].

Something similar happens with JUnit and TestNG. Actually JUnit did have something close to first-class tests with its inheritance API.

With programmatic tests, you simply return a list of tests, there's no magic about it. It doesn't matter how they're built, they can be parameterized or not, all you have to know is [StaticTestFactory] public static IEnumerable<Test> Tests() . If they're parameterized, it doesn't matter what kind of parameters they are. Actually, the very concept of "parameterized tests" simply disappears.

With attributes, you may have tried to use [Row] first, only to have the compiler remind you that attribute parameter types are very limited and you can't have a DateTime. Or a function. Or even a decimal. The testing framework gets in the way. Attributes are just not the right tool to model this.

With programmatic tests, you are in control, not the testing framework. It becomes more of a library rather than a framework. Things are conceptually simpler.

What about SetUp and TearDown? Don't worry, MbUnit supports them directly as properties of TestSuite. However, as we'll see in the next post, they're not really necessary. We'll also see a few other pros/cons first-class tests have.

I'll leave you with this quote from the twitter-fake Alain de Botton:

I understand that annotations beat boilerplate XML config.But what's wrong with, you know, writing real code instead?

— PLT Alain de Botton (@PLTAlaindeB) March 13, 2012