Test Execution Order

In a perfect world, the order of execution of your tests shouldn’t matter. If each test is isolated and makes no unfair assumptions about the state under test, they’ll behave the same no matter the order. In reality, we’re all humans working on tough problems. Sometimes we mess up, sometimes we make tradeoffs of a little shared state in exchange for practical test speeds, and sometimes the code under test is brittle due to shared state like static fields.

Test execution order can reveal real problems. If your tests are brittle and run in the same order each time, you might not notice for a while. If your tests are brittle and run in randomized order, you might experience the pain earlier, giving you some pressure to improve the situtation right away. xUnit once helped me catch a serious bug early, because it runs the tests within a test class in random order. If you’re not aware that your test runner shuffles its tests, though, you may be extra-confused during the debugging experience.

I see three possibilities:

1. You don’t care what order your tests run in. Whatever order the test runner finds them in is good enough. This order is actually undefined by the reflection API, but tends to be the order the tests are declared.
2. You deliberately want to shuffle the tests in order to catch issues earlier, and you accept that it’s up to you to keep that in mind when tests start failing inconsistently.
3. You have an extremely unusual situation in which you want to order the tests in a test class by some other rule.

I recently added support for test execution order in the Fixie test framework. By default, no order is enforced, meaning you get the undefined (but not very surprising) order provided by the reflection API. If you want to opt into random shuffling or apply your own rule for ordering tests, you can do so in your testing conventions. To randomize the order of test cases within a test class, declare that they should be shuffled:

public class CustomConvention : Convention
{
    public CustomConvention()
    {
        Classes
          .NameEndsWith("Tests");

        Cases
          .Where(method => method.IsVoid());

        ClassExecution
          .CreateInstancePerTestClass()
          .ShuffleCases();
    }
}

To specify your own sorting rule, call SortCases instead of ShuffleCases, providing a rule for ordering any two test cases:

public class CustomConvention : Convention
{
    public CustomConvention()
    {
        Classes
          .NameEndsWith("Tests");

        Cases
          .Where(method => method.IsVoid());

        ClassExecution
          .CreateInstancePerTestClass()
          .SortCases((caseA, caseB) => String.Compare(caseA.Name, caseB.Name, StringComparison.Ordinal));
    }
}

When Fixie tests itself, it needs to say things like “Run all of the tests in this sample test class and then assert on the results and the output across the whole test class.” It’s just easier to write these assertions in a small amount of code if I can assume a reliable order of execution, so I sort them by name. Rather than sorting, most users would likely do nothing or call ShuffleCases, but SortCases is there if you really, really want it.