In an earlier post, we saw how to use dotCover in simple scenarios, whereby we have a single unit test project for which we want to obtain coverage reports on. There are times however, when this is not the case; that is, we might have multiple tests projects. It is very common (and recommended practice) to separate Unit test projects from Integration test projects, allowing us to run the former more frequently during development and the latter during an Automated Continuous Integration setup. In these cases, running the dotCover analyse command, as shown in the previous post might not be ideal.

Fine tuning the coverage steps

For the scenarios described, whereas we have multiple test projects, dotCover allows us to fine-tune the coverage process by offering us a series of commands that breakdown the coverage, merging and reporting into individual steps. Suppose we have the following Project Layout

The AppCode is our application code that we want to run coverage for, and TestProject1 and TestProject2 are specific the test projects that cover AppCode. These can be Unit Test or Integration Test projects. The process we need to follow is shown in the figure below:

We run coverage on each of the tests projects by using the cover (c) command using the following configuration file (corresponds to testProject1.xml. testProject2.xml is similar)

[Command to execute: dotcover c testProject1.xml]

Executing this command generates a snapshot file with coverage data, the Coverage Results Descriptor (by default, unless specified in the configuration file using the TempDir element, the snapshots are saved in the User temporary folder). These snapshots are not the final XML report that we are after but intermediate information that dotCover uses to generate reports from. There is a correspondence between snapshots and processes, such that each process generates a single snapshot. In our case, since we are calling a unit testing framework to launch a single test project, there will only be one snapshot. However there are cases, for instance when running coverage report on an application, whereby the application can launch another process  itself. In this case, multiple snapshots will be generated.

When we run this command, once for each Project, we end up with two snapshot files. We now need to merge the snapshots in order to generate the XML report. This is accomplished using the merge (m) command

[Command to execute: dotcover m merge.xml]

which takes the output of the two cover commands (output1.xml and output2.xml) and generates a single snapshot file. The only thing remaining is to generate an XML coverage report based off of this data, using the report command

[Command to execute: dotcover r report.xml]

providing us with the XML report containing coverage information (percentage of code covered, statements covered), which like in the previous post we can apply an XSLT to format into HTML or JSON.

Other Coverage Commands

dotCover provides two more commands:

• list
• delete

The list command simply obtains a list of all snapshot files given the coverage descriptors, that is, the output of the different cover commands. The configuration file lists these descriptors:

The delete command deletes the list of given snapshots:

Summary

As we can see, dotCover allows us to fine-tune the coverage process by letting us control the different steps. One question that might arise is why couldn’t we have done the same by running the analyse command twice, once for each Test Project. If we were to do that, we would get two independent reports, whereas here the results are all merged into a single report. The analyse command is a great fit for simple scenarios that only have a single project. Once we move onto more complex setups, having a finer control over the coverage process has its benefits.

In this short screencast you can see the basics of dotCover and how to get up and running in a matter of minutes.

dotCover allows us to run coverage analysis on our code. However, there are times when we do not want to perform an analysis on certain areas. This could be our test assemblies, certain third-party assemblies or even specific parts of our own project. Since the analysis has an impact on the overall statistics and potentially can take longer, it is often interesting for us to filter certain assemblies or classes out.

The figure below shows the coverage report of the default MVC project (yes, yet another MVC example, but it ships in the box and probably the ONLY template with VS that comes with tests). In this case we’re using xUnit for tests, which is mostly to demonstrate that dotCover works with other frameworks, not only MSTest (personally I’ve moved on to MSpec, which dotCover also supports).

If we focus on the Coverage results we can see that there are some areas in grey, which indicate that the corresponding PDB files were missing so coverage could not take place. We can also see other assemblies that have been included in the results yet might not be of interest, such as the test assemblies.

In order to filter these out, we can use the Coverage Filters, which can be accessed via the dotCover menu in the IDE

By default, everything is set to be covered, as shown by the Everything entry in the Allow Filters tab (which most likely will be renamed to Included). Examining this entry (clicking on Edit) we can see that it is composed of three values:

• Module Mask indicates the project name
• Class Mask indicates the class name
• Function Mask indicates the method name

(All entries support wildcards as displayed by the * that appears)

We can now uses these patterns to exclude projects, assemblies, namespaces, classes and methods from out tests.

Excluding entire project

Click on Deny Filters tab (most likely will be renamed to Excluded) and click on Add, entering the following information:

Clicking OK  and re-running the tests, we now see that the entire test assembly has been excluded

Excluding an entire Namespace

We can exclude entire namespaces by setting it in the Class Mask as shown below:

this will exclude all the classes inside the MvcApplication16.Models namespace. If we just wanted a specific class, we add it after Models instead of *.

Excluding a method

We can exclude a method, similar to how we’ve done it with the previous cases:

Adding a series of filters for test and auxiliary assemblies using the previous steps, we can end up with a cleaner and more accurate coverage report:

produced by adding the following filters:

Notice that by using the *.Tests (or alternatively as I call it *.Specifications), we can automatically exclude all our test projects from code coverage.

Beta Disclosure

Note 1: Currently these settings are Global, that is, they apply to all projects. Most likely we’ll be changing it so that they are Solution and Global scoped. This way, common libraries can be excluded for all solutions, and specific projects and/or namespaces can be solution based.

Note 2: We’re also looking at adding functionality to make it easier to define these exclusions (a la Right-Click on Folder, add to Excluded list)

As always, we’re still in beta and feedback is more than welcome!

One of the new features dotCover has added is the ability to find tests that cover a certain piece of code. Something remotely similar has been available in ReSharper, although it has been kind of an archaic solution (i.e. Find Usages on Method calls, locate Test assemblies in Result window).

dotCover makes this easier by providing quick access to this information and extends it in functionality.

Below we can see some tests from an MVC application. Let’s run Code Coverage on it using dotCover first.

Now let’s switch over to the Source Code and select some random source code, in this case the MembershipService.ChangePassword line in the ChangePassword action:

In order to see the tests that cover this line of code, we can either press the default key combination of Ctrl+Alt+K or select the option Show Covering Tests from the dotCover:

dotCover will then display a small window showing all the different tests that cover that line of code.

At this point, we can run the selected tests or add them to the existing ReSharper Test Runner session. This allows us to easily jump from specific sections of code to the corresponding tests and execute them instantly.

One minor note: The default key mapping conflicts with KeePass, but you can easily re-assign it via Visual Studio Tools | Options | Keyboard, or do as I did and change KeePass to Ctrl+Alt+P (P as in Password…makes more sense).

I’ve been giving talks for quite a while now on best practices, unit testing and other things that aim to make our lives as developers easier. Having myself suffered and lived in a World where nothing was put through automated testing, QA was usually delegated to our customers, and having faced the problems that I and other members of my team encountered when something went wrong on deployment, I’ve truly come to appreciate the value of how certain practices improve my code (and my life). In fact it makes my work more enjoyable. It’s fun. I’ve been on both sides of the fence and I’m on the greener side now. It’s like when you quit smoking (I actually did that too, It’s been 6 years and I’ve not looked back. You can too!).

We loved the songs

I’m now on those greener pastures, running and skipping, with the wind blowing through my hair, freezing my bold head, and singing how unit tests are great and following certain design principles are wonderful. People hear my sing away and they really enjoy it. They love my entire repertoire. They enjoy the dark ballads where I pour my heart out about the pain I suffered during those early days, because they identify themselves with those problems. They enjoy the nice energetic melodies where I talk about how to ease the pain. They see it can ease their pain too. The show ends on a high!

Then it’s over!

This is great and all, but it’s not for me

I was at two of these shows last week. The first one was 4Developers and one of my talks was a short-talk on unit testing with JavaScript. For some reason they had put me on the PHP track,  so I was standing in front of an audience of PHP developers, something I know absolutely nothing about. But it was OK. I was there to talk about JavaScript and Unit Testing.

As usual, to set the context, I asked a series of questions regarding unit tests:

“How many of you know what a Unit Test is” – 80% put up their hand.

“How many of you do Unit Testing?” – 10% put up their hand.

“How many of you think Unit Testing is valuable?” – 80% put up their hand

(rough figures obviously)

Two days before that, I was giving a talk at the Architecture Forum in Madrid, on design principles. I asked a similar question. I got worse results. Primarily because at 4Developers there were roughly 30 people in the room. At the Architecture Forum, there were around 200.

WTF?!

It sounds absurd doesn’t it? Think about it.

“A is good. I like A. I should do A. But I won’t do A”.

People identify themselves with the issues we talk about. They see the value writing automated tests and complying with design principles adds. They get excited, yet they don’t do it. Why?

I usually ask this question too, and the typical responses are:

1. Tight Schedules.

2. Decrease in Productivity. Drag and Drop is more productive

3. Customers want deliverables. Tests are not deliverables

and my all time favorite

4. “I’m paid to write code, not tests”.

See, developers live in the real world with tight schedules. They have to deliver to the customer. Their boss and customers are waiting. They don’t have time to do all this theoretical nonsense. In an ideal situation, with all the time to waste, of course they would do all we talk about. But that’s not the reality. They have a product to deliver and this kills their productivity.

What we have here, is a failure to communicate…

For those in the choir on those green pastures, it is crazy. We realize that there is in fact no decrease in productivity. We understand that by taking time to write maintainable code we are making our lives easier in the long run. We appreciate the fact that unit tests allow us to find a higher number of bugs before deployment. We realize that a unit test is not only about making sure our code works now, but that it will also work in the future, and if something breaks, we’ll know about it before we ship it to the customer. We won’t jeopardize the customer’s productivity by stopping his activities because of our failures. Last but not least we also know that unit tests can in fact define our specifications.

Do we fail to communicate properly? Is the Grand Finale missing?

Don’t look at me…

As I mentioned initially, I also didn’t unit test or write maintainable code. So what made me change? Was it a blog post or conference?

Well, I had suffered the issues so I knew I needed to find a better way to do things, and gradually with a lot of time and effort I started to improve how I did things. But it wasn’t an overnight process. And it isn’t over. I’m still at it.

The point however is that I also live in the real world. I’ve had my fair share of bosses and PM’s that thought writing a CRM was dropping a component on a form and hooking up some properties. I’ve had tight schedules and deadlines to meet, but I didn’t throw in the towel with some excuse.

See, all those reasons above are excuses. It’s up to you, as an individual, as a member of a team, to improve what you do. You job involves talking to customers, to project managers, to your boss, to fellow team members and making them understand. Your job doesn’t start and end with writing code. You’ve read that blog post or gone to that talk and seen the value it has to you as a developer. You need to communicate that value to those you deal with.

Your boss doesn’t pay you to write a unit test, of course he doesn’t. But I’m betting he didn’t pay you to write two IF statements instead of one Switch statement either. He doesn’t pay you to comply with the Single Responsibility Principle or care whether you broke the Law of Demeter.

See, what he and your customers do care about is how what you’re doing adds value for them, and it’s your job to not only deliver, but make sure they see that.

Take a look at this code:

   1: public class ClassToTest {

   2:  

   3:     public void MethodToTest()

   4:     {

   5:         

   6:         IDAL dal = new DAL();

   7:  

   8:         int discount = dal.GetDiscount();

   9:  

  10:         if (discount > 10)

  11:         {

  12:             // Do something

  13:         }

  14:  

  15:     }

  16: }

The tight coupling to DAL just stands out like a sore thumb. One consequence of this tight coupling is that it's hard to test if we don't have an underlying database setup with correct values. Forget about testing this while taking a shower, unless you have a habit of taking your SQL Server box to the bath tub with you.

Take a look at the following approach to making this code testable without having a database:

   1: public class ClassToTest {

   2:  

   3:     public void MethodToTest()

   4:     {

   5:         

   6:         DAL dal = new DAL();

   7:         

   8:         #if testing_in_shower

   9:         int discount = 3;

  10:         #else 

  11:         int discount = dal.GetDiscount();

  12:         #endif

  13:         if (discount > 10)

  14:         {

  15:             // Do something

  16:         }

  17:  

  18:     }

  19: }

Are you still here? Didn't you just fall off of your chair, not knowing whether to laugh or cry in anguish? It's just plain wrong isn't it? Code Smell just takes on a whole new dimension (and consistency?). Something like this shouldn't be done for many reasons. It makes our code testable but it's wrong.

Introducing code for testing purposes

The reason I bring this up now is because in the past 2 weeks I've seen a few references and examples of introducing code just for the sake of testability. Two weeks ago I was skimming over an article that was explaining unit tests. The class the author was trying to test had a dependency that was causing issues during his tests. The solution offered was to introduce a new constructor and inject the dependency in. In our case, the ClassToTest would like:

   1: public class ClassToTest {

   2:  

   3:     public ClassToTest()

   4:     {

   5:         

   6:         // Initialize some stuff

   7:  

   8:     }

   9:  

  10:     public ClassToTest(IDAL dal)

  11:     {

  12:         // Initialize some stuff

  13:         _dal = dal;

  14:     }

  15:  

  16: ....

Now that's great. The problem is that the author left both constructors in there, emphasizing that one is for testing purposes and the other for production code. Why that's wrong I'll get to in a moment.

The other day, Scott Hansleman made a blog post on testing some code that had a dependency on the current principal. His solution was to inject IPrincipal into various methods of the Controller, where Controller is in the context of an ASP.NET MVC Controller. Personally I don't agree with the solution but that's another subject (and another post on a different approach I have in the backlog for the same scenario). However, injecting the dependency seemed valid enough. Later on he updated the post with the following:

UPDATE: Phil had an interesting idea. He said, why not make method overloads, one for testing and one for without. I can see how this might be controversial, but it's very pragmatic.

1. [Authorize] 
2. public ActionResult Edit(int id) 
3. { 
4. return Edit(id, User); //This one uses HttpContext
5. } 

	[Authorize]
	public ActionResult Edit(int id)
	{
	return Edit(id, User); //This one uses HttpContext
	}
	

You'd use this one as before at runtime, and call the overload that takes the IPrincipal explicitly for testing.

The same thing has happened as in the case of an overloaded constructor. Code has been introduced into the base just for the sake of testability. In the previous case it was a new constructor, and in this case, a new method.

Why is this wrong?

Introducing code just so that you can test it is wrong for several reasons, including an increase in noise, decrease in readability and more importantly, permitting the possibility of production code that will not be tested. It might seem improbable at first, seeing that the overloaded versions always call the base ones, but as your code base grows, by allowing this habit, you can promote it to other structures such as conditionals, switch statements, etc, and not limit it only to methods or constructors. Then you'll start initializing values that only make sense in tests and gradually your code starts to smell more and more. To top it off, you can throw in some compiler conditional that rips those parts of the code out from the production version.

Code should also serve as documentation. The cleaner you make your code, the less noise and redundancy you add, the easier it is for others to understand and maintain.

Conclusion

Being pragmatic is good, but don't do it at any cost. Try and not introduce code into your code base that is there exclusively for the purpose of testability (and no, dependency injection is not only for testability). Anything that has to do with testing, keep it where it should be, in the test assemblies. Having overloaded methods or constructors, mocks or stubs in your code base is just as bad as having compiler conditionals in your code for testability. If you run into issue where your code is hard to test, step back and re-think your design. Maybe the problem you're facing now that's impeding your tests is actually a can of worms waiting to be opened. And the sooner you open it, the better. That's one of the great advantages of Test Driven Development.

Testing for exceptions in unit tests can be tricky. Most frameworks use the ExpectedException attribute to denote that the test will pass when a specific exception is raised. As an example, let's look at the following test:

   1: [TestMethod]

   2: [ExpectedException(typeof(AuthenticationException))]

   3: public void Authenticate_With_Invalid_Credentials_Throws_AuthenticationException{

   4:  

   5:     AuthenticationServices services = new AuthenticationServices();

   6:     

   7:     services.Authenticate("user", "wrong");

   8:     

   9: }

In many cases this works. If you are throwing a custom exception, then the chances of the same exception occurring in your code in the wrong place are minimal. However, things change when you are testing for system exceptions. Imagine in the previous example if we were to throw a SecurityException instead of AuthenticationException. The .NET framework could throw a security exception itself due to some specific reason. As far as the test is concerned, it passes because it doesn't care who or where the exception is thrown. It just cares that it's happened.

An alternative approach to this would be to wrap the specific call in a try..catch block and not use the ExpectedException attribute.

The guys that designed xUnit understood the shortcomings of testing exceptions and took a much cleaner approach. In xUnit, the previous test would be written like so:

   1: [Fact]

   2: public void Authenticate_With_Invalid_Credentials_Throws_AuthenticationException()

   3: {

   4:     AuthenticationServices services = new AuthenticationServices();

   5:  

   6:     Exception ex = Assert.Throws<AuthenticationException>(() => services.Authenticate("user", "wrong"));

   7:     

   8:     Assert.Equal("Authentication Failed", ex.Message);

   9: }

As you can see, there is no ExpectedException on the test (called a Fact in xUnit). Instead, the Assert.Throws construct is used. This is a generic method that takes a type parameter the type of exception we want to check for. As parameter we pass a delegate or lambda expression with the actual call that will throw the exception.

The obvious advantage to this is that if any other part of our system were to throw the same type of exception, our test would fails, as it should. So instead of creating a custom exception we were to use SecurityException and on creation of AuthenticationServices the framework would throw a security exception, our test would fail.

Another advantage of Assert.Throws is that it allows you to examine the returned exception object, so you can run further assertions on it (like that on line 8).

Note: While I was doing my Unit Testing session at Devreach this week, someone asked me how to test for two exceptions within the same call in Assert.Throws. You don't. Each test should check for only one exception. Remember, a unit test only tests one thing, one situation. If your code is throwing two different exceptions, it's can't be doing it under the same conditions.