The tricky cases in testing - exceptions

This article is a revised version. It was previously published at refactoring-legacy-code.net published.

This is a series of articles. You can find the other articles here:

• The tricky cases in testing - GUI
• The tricky cases in testing - Resources
• The tricky cases in testing - Events
• The tricky cases in testing - Visibility

Exception testing with NUnit and MSTest

Exception testing is always a challenge for developers. There is often a lack of a clear strategy. And yet it is actually quite simple...

In the context of automated tests Exceptions into one of the following two categories:

• A method itself throws an exception if it detects an exception state.
• During the execution of a method, an exception may occur to which the method reacts.

All other cases of exceptions are not relevant for automated tests. In particular, the most common case where an exception is triggered but not handled further is irrelevant for testing. Such cases end up in the application's global exception handler. If the code to be tested does not handle an exception, there is nothing to test.

An exception is expected

If an exception can occur in the regular program flow, there should be an automated test for this case. Testing exceptions here means checking whether an exception occurs as expected. On the one hand, the test serves to ensure that the code also triggers an exception as expected in the future. In addition, such a test is used for documentation purposes. The test then expresses the circumstances under which the exception is expected, so that developers who read the test or the associated implementation later will be less surprised.

The following example shows how NUnit can be used to check whether an exception is triggered.

[TestFixture]
public class ExceptionTests
{
    [Test]
    public void Exact_expectation_about_an_exception() {
        Assert.Throws(
            () => Sut.DoSomethingThatThrows());
    }

    [Test]
    public void Any_Exception() {
        Assert.Catch(
            () => Sut.DoSomethingThatThrows());
    }
}

public static class Sut
{
    public static void DoSomethingThatThrows() {
        throw new NotImplementedException("Bumsdi!");
        //throw new NullReferenceException();
    }
}

The first test expects that a NotImplementedException is thrown. If any other exception is thrown, even if it is a derivative of the expected exception type, the test fails. The second test, on the other hand, expects an exception type that is derived from Exception is derived. In the example, both tests are green.

ExpectedException attribute

There is a second variant for both NUnit 2.x and MSTest to express that an exception has been thrown. By annotating the test with the ExpectedException attribute, the test is only considered successful if the named exception is triggered within the test method.

[TestClass]
public class ExceptionTests
{
    [TestMethod,
     ExpectedException(typeof(NotImplementedException), AllowDerivedTypes = false)]
    public void Exact_expectation_about_an_exception() {
        Sut.DoSomethingThatThrows();
    }
    [TestMethod,
     ExpectedException(typeof(Exception), AllowDerivedTypes = true)]
    public void Any_Exception() {
        Sut.DoSomethingThatThrows();
    }
}

The (small) disadvantage: the exception can be somewhere occur within the test. The NUnit syntax shown above is much more specific. You should therefore always ensure that the test method is written in a concise and focused manner so that the test is not inadvertently green even though the exception is triggered at a different location than expected.

A note on MSTest I can't help myself here. With Assert.AreEqual the expected value is the first parameter. With StringAssert.StartsWith he is in second place at the back. That is inconsistent and makes it difficult to work fluently with MSTest. Switch to NUnit!

Test details of an exception

Sometimes it is important to check whether an exception is thrown with the expected details. In particular, the property Message comes into question here. The following test checks whether the Message of an exception corresponds to the string "Bumsdi"".

[Test]
public void Properties_of_exception() {
    var exception = Assert.Catch(
        () => Sut.DoSomethingThatThrows());
    Assert.That(exception.Message, Is.EqualTo("Bumsdi!"));
}

Alternatively, you can use Does.StartWith be checked whether the Message begins with the string "Bum".

Assert.That(exception.Message, Does.StartWith("Bum"));

With MSTest, the test looks like this:

[TestMethod]
public void Properties_of_exception() {
    try {
        Sut.DoSomethingThatThrows();
    }
    catch (Exception exception) {
        Assert.AreEqual("Bumsdi!", exception.Message);
        StringAssert.StartsWith(exception.Message, "Bum");
    }
}

As I said, switch away from MSTest...

Throw an exception

If exceptions are reacted to in the implementation, this code must of course also be tested automatically. Testing exceptions now means throwing an exception in the test. The method behaves differently when an exception occurs. However, this means that the exception to which the code to be tested should react must be thrown in the test. With the appropriate tools, this is not a problem, as the following example shows. First of all, you can see a class that is defined in its FormatContent method either the method used by ReadContent read string or, in the case of an exception, returns a different string supplies.

public class SUT
{
    public string FormatContent() {
        string content;
        try {
            content = ReadContent();
        }
        catch (Exception) {
            return "An exception...";
        }
        return content;
    }

    public string ReadContent() {
        return "Some content...";
    }
}

The so-called Happy Day scenario is easy to test. This is a test for the situation where everything runs smoothly.

[Test]
public void Happy_day() {
    Assert.That(new SUT().FormatContent(), Is.EqualTo("Some content..."));
}

The case becomes more interesting if we now want to check what the result is when the ReadContent method throws an exception. I use the mock framework for this test TypeMock isolator.

[TestFixture, Isolated]
public class ExceptionsTriggerTests
{
   [Test]
    public void Eine_Exception_ausziehen() {
        var sut = new SUT();
        Isolate.WhenCalled(() => sut.ReadContent()).WillThrow(new Exception());
        Assert.That(sut.FormatContent(), Is.EqualTo("An exception..."));
    }
}

First, I create an instance of the System Under Test (SUT). Subsequently I point out TypeMock isolator with the method Isolate.WhenCalled when calling the ReadContent method to throw an exception. Finally, I check whether the call to the FormatContent method now returns the expected string.

Curse or blessing?

The use of such powerful mock frameworks is a recurring topic of discussion among developers. Since TypeMock isolator and other products of this type are realized with the help of the Profiler API, they can be used to replace virtually everything with dummies. This makes it possible to automatically test code that does not handle dependencies properly. But hey, this blog is about the question of how you can make your Refactor legacy code into clean code step by step can. Powerful tools that can also be used to put legacy code under test are welcome 🙂 Testing exceptions is no longer witchcraft with TypeMock Isolator.

Of course, this does not mean that Clean Code Developer principles and practices can go overboard if your tools are powerful enough.

Dealing with exceptions

Exceptions are intended for exceptional cases that should not occur in the regular program flow but do occur because something goes wrong outside our area of responsibility. When reading a file, an exception can suddenly occur out of the blue because the network has just gone down. As a developer, however, I can foresee this eventuality. Disruptions can occur when dealing with external resources.

However, if I try to select any string into a int this is not a case for an exception. Because it should be clear to me that this cannot always succeed. As soon as I start parsing with the int.TryParse function, everything is fine. Only when using int.parse an exception can be triggered. In the regular program flow, I should therefore stop the parsing of string to int better by means of int.TryParse because then it is immediately clear that there can be two cases.

The int.parse Call into a try-catch is not a good idea. A try-catch gives the reader the impression that something unforeseen can happen. That a string not in every case in a int can be converted but is predictable. Foreseeable cases should not be linked in the program flow with try-catch be solved. But as is so often the case: exceptions (pun intended) prove the rule. Sometimes the code is easier to read if a method reports errors by means of an exception. However, you should always check whether a method reports its errors by exception or whether, as in the case of TryParse there is a variant that works without exceptions.

Conclusion

Exceptions are to be used for exceptional situations that were not foreseen by the developer. Furthermore, for cases in which something can go wrong during the operation. This usually has to do with accessing resources. A database may initially respond and then suddenly stop responding. It is completely normal for an exception to be triggered. However, this behavior is also predictable by the developer. Consequently, automated tests should check whether everything runs as it should in the event of an exception.