Announcing NSubstitute

Tonight at Sydney ALT.NET Anthony Egerton announced NSubstitute, his mocking framework for .NET 3.5+. I've been contributing some code to NSubstitute and I like to blog, so I thought I'd take it upon myself to do the written version of the announcement. :)

This all started a few months ago during some pair programming where we started discussing how we would ideally like our mocking code to read. Anthony came up with some neat syntax ideas and we decided to start working on it as a hobby project in our spare time, the aim being to play around with a simple, succinct syntax for mocking, as well as to try out some different testing techiques and to learn a bit about DynamicProxy.

We liked the syntax so much we've started using it on work projects, and so we thought we'd put it out there in case anyone else was interested or has some ideas on how to improve it. It's still a little rough around the edges (the exception message formatting needs lots of work, and it doesn't support Mono or Silverlight yet), but it's pretty usable in its current state and should be enough to give people an idea of what we're aiming for.

NSubstitute examples

Say we have an ICalculator interface:

public interface ICalculator {
  int Add(int a, int b);
  string Mode { get; set; }
  event Action PoweringUp;
}

We can ask NSubstitute to create a substitute instance for this type. We could ask for a stub, mock, fake, spy, test double etc., but why bother when we just want to substitute an instance we have some control over?

var calculator = Substitute.For<ICalculator>();

Now we can tell our substitute to return a value for a call:

calculator.Add(1, 2).Returns(3);
Assert.That(calculator.Add(1, 2), Is.EqualTo(3));

We can check that our substitute received a call, and did not receive others:

calculator.Add(1, 2);
calculator.Received().Add(1, 2);     
calculator.DidNotReceive().Add(5, 7);

If our Received() assertion fails, NSubstitute tries to give us some help as to what the problem might be:

NSubstitute.Exceptions.CallNotReceivedException : Expected to receive call:
    Add(1, 2)
Actually received:
    Add(4, 5)
    Add(3, 4)

We can also work with properties using the Returns syntax we use for methods, or just stick with plain old property setters (for read/write properties):

calculator.Mode.Returns("DEC");
Assert.That(calculator.Mode, Is.EqualTo("DEC"));

calculator.Mode = "HEX";
Assert.That(calculator.Mode, Is.EqualTo("HEX"));

NSubstitute supports argument matching for setting return values and asserting a call was received:

calculator.Add(10, -5);
calculator.Received().Add(10, Arg.Any<int>());
calculator.Received().Add(10, Arg.Is<int>(x => x < 0));

We can use argument matching as well as passing a function to Returns() to get some more behaviour out of our substitute (possibly too much, but that's your call):

calculator
  .Add(Arg.Any<int>(), Arg.Any<int>())
  .Returns(x => (int)x[0] + (int)x[1]);
Assert.That(calculator.Add(5, 10), Is.EqualTo(15));

Returns() can also be called with multiple arguments to set up a sequence of return values.

calculator.Mode.Returns("HEX", "DEC", "BIN");
Assert.That(calculator.Mode, Is.EqualTo("HEX")); 
Assert.That(calculator.Mode, Is.EqualTo("DEC")); 
Assert.That(calculator.Mode, Is.EqualTo("BIN")); 

Finally, we can raise events on our substitutes (unfortunately C# dramatically restricts the extent to which this syntax can be cleaned up):

bool eventWasRaised = false;
calculator.PoweringUp += () => eventWasRaised = true;

calculator.PoweringUp += Raise.Action();
Assert.That(eventWasRaised);

How can I try this out?

Easy, just head on over to nsubstitute.github.com and download the binary. Then add the NSubstitute.dll as a project reference, add using NSubstitute; to your CS file and start substituting. Or grab the source from the project site on GitHub and start tinkering.

If you try it out I'd love to hear how you go!

Some quick acknowledgements

It doesn't seem right to plug this and not give kudos to some of the people that made it possible, both directly and indirectly.

First, Anthony deserves the credit for the awesome syntax ideas in NSub. I take full credit for all the implementation bugs, as well as for some of the ridiculous abstractions in the code (including a really dodgy, mini-IoC container for wiring up routes and handlers. Yuck. :)). Troy Hunt did a great job of knocking out a logo for us. Xerxes and Shannon also helped out with some code and suggestions respectively. Krzysztof also provided some much needed help over email and twitter on how to dynamically create delegates using .NET 3.5 expressions.

There are also a few people and projects that NSubstitute owes its existence to. Ayende and the awesome Rhino Mocks project really started us mocking in the first place, and did a great job making mocking viable on the .NET platform. Also thanks to Daniel Cazzulino and moq for bringing Arrange Act Assert-style mocking to .NET (to the world?). NSubstitute relies entirely on AAA mocking for its syntax, intentionally providing no support for strict mocks or record/replay. And finally to Castle Project's DynamicProxy which we rely on (like almost everyone else) to generate our proxies.

 

Happy substituting! :)

TDD, not UTDD

When I first learned TDD I was taught that the first step in the process is to write a failing test, not specifically a failing unit test. After all, it is Test Driven Development, not Unit Test Driven Development. I even read books that were apparently written using a TDD-style approach, without a unit test in sight*. This didn't stop me from focussing almost entirely on using unit tests for TDD.

Fast forward a few years and I'm now finding a lots of benefits in other forms of testing for TDD, to complement the traditional unit test.

* In case you're wondering how this works: write the goal for an unwritten section, write the section, verify it meets the goal, edit. Red, green, refactor. :)

Accepting acceptance tests

Acceptance testing is a practice that seems very easy to get wrong in ways that cause a lot of friction, resulting in it being ignored or given only cursory treatment by developers. This is unfortunate, as I see acceptance testing as essential for effective TDD.

In Steve Freeman's and Nat Pryce's Growing Object Oriented Software, Guided by Tests, the first failing test they write is an acceptance test for the feature they are working on. They then drill down into unit tests so they can take small steps to incrementally build an implementation that passes the acceptance criteria. This was how I was initially introduced to TDD from reading about Extreme Programming (XP): an outer TDD cycle with acceptance tests, and an inner cycle with unit tests that had several iterations to get the acceptance test to pass.

The key to effective acceptance tests (at least for me, YMMV) is making sure they exercise a specific feature of the system from top to bottom, using as much of the real system as possible. It should clearly specify the behaviour of that feature -- once it passes you should be fairly confident that the customer's requirements for that feature have been met.

The main benefit I've found from acceptance testing is that the feedback from these tests help produce an architecture that is flexible, maintainable, and scriptable by virtue of being testable at such a high level. They also help me focus on exactly what I need to get this feature done, which in turn helps guide where I should start applying unit tests to drive the more specific elements of my design.

These benefits, using tests to define and design, are fairly universal to TDD regardless of which type of tests are used. In the case of acceptance tests, the large scope of the tests provide feedback on the larger aspects of the design.

I've also found acceptance tests to be invaluable when I've had to make radical design changes (e.g. when I've stuffed up somewhere), letting me cull over-specified unit tests and make sweeping changes while still having enough coverage to be confident the software works.

If you're doing TDD but not using acceptance tests, or have tried acceptance testing before but haven't been able to make it work for you, I'd really recommend giving it another shot. Don't worry about them being customer-writable (or even customer-readable for now, provided you can explain what is being tested), don't worry about what tool you use, just get them working. You're architecture will thank you for it. :)

Don't mock types you don't own -- integration test them!

Recently I was test driving some code that uses Castle DynamicProxy. I mocked out the Castle interface and checked my subject under test interacted with that library in a way that I thought was correct. The problem here is I do not own the Castle type, and you should not mock types you don't own.

Mocking types you don't own gives you very little in the way of ensuring correctness, and is potentially misleading in terms of the design guidance it provides. The problem is that you are testing based on your assumption of how the type works, not how it actually works. Sure, you're testing that your code correctly calls the method you told it to, but what about testing it calls the correct method? If the type changes in a later version, or if it's behaviour is slightly different than you expect under different conditions or arguments, then your tests can pass but your software fails. A misleading test like this can be more harmful than having no test.

Aside: The same criticism can be levelled at mocking in general. The difference is that you have tests defining how your own types work, and have the ability to easily change the types if they do not function as required.

Another drawback, especially if you are working with libraries or frameworks that are not designed in a particularly test-friendly way (to put it diplomatically**), is that you may end up starting to push the behaviour of those libraries into those mocks in order for your class under test to interact with them in a meaningful way. Once you start simulating behaviour in your mocks you are doing it wrong -- you are well on your way to brittle, over-specified, complicated tests.

** I'm not talking about Castle here, it's awesome. I will remind readers I have worked with SharePoint before... :)

Of course, if you are avoiding mocking types you don't own, this implies you need to use the real types, which means we are in the realms of integration testing. For my Castle-calling code, I ended up unit testing down to my own class that needed to use Castle to achieve something, then writing integration tests with real Castle objects to ensure that my class actually did use Castle correctly. This ended up being much more valuable to me, and much more flexible. It was more valuable because my tests actually told me my class was using the library correctly and was getting the results my system required, rather than just calling the method I thought was needed. It was more flexible because I had not over-specified every interaction with the third-party library, and so could easily and independently vary both my code and how my code interacted with that library.

I've had a habit of avoiding integration tests as I always assumed they had too wider scope and were too slow to be useful. Now I look forward to hitting a case I can easily cover with integration tests, as it means I've reached the bottom of my software's abstractions and can just test a concrete piece that actually does some real work by interacting with its environment.

Note: A few words of caution about integration tests. I wouldn't recommend switching to integration tests until you are at the very bottom layers of abstraction. Test-drive down until you reach a single class that is an effective facade for some behaviour from a third-party library, then use integration tests to make sure it works as required. Integration tests can also slow down your test suite if they end up hitting the file system, database etc., in which case you should make sure you are able to run them separately from your unit tests and only run them when needed (such as prior to checkin or when you change something related).

Conclusion

I still rely very heavily on unit tests when test driving software, but I feel it is really important to know when to use other forms of testing with TDD (and without TDD for that matter). Acceptance tests are a great way to kick off a TDD cycle from the top down, while integration tests are invaluable once you reach the bottom and need to write the code that interacts with the rest of the world. Then there's unit testing for everything in between.

Finally, of course, there's manual, exploratory testing. This probably won't feature too much in your standard TDD cycle, but is so important for checking your software actually works that it didn't feel right not to mention it. :)