In an earlier post, we saw how to use dotCover in simple scenarios, whereby we have a single unit test project for which we want to obtain coverage reports on. There are times however, when this is not the case; that is, we might have multiple tests projects. It is very common (and recommended practice) to separate Unit test projects from Integration test projects, allowing us to run the former more frequently during development and the latter during an Automated Continuous Integration setup. In these cases, running the dotCover analyse command, as shown in the previous post might not be ideal.

Fine tuning the coverage steps

For the scenarios described, whereas we have multiple test projects, dotCover allows us to fine-tune the coverage process by offering us a series of commands that breakdown the coverage, merging and reporting into individual steps. Suppose we have the following Project Layout

The AppCode is our application code that we want to run coverage for, and TestProject1 and TestProject2 are specific the test projects that cover AppCode. These can be Unit Test or Integration Test projects. The process we need to follow is shown in the figure below:

We run coverage on each of the tests projects by using the cover (c) command using the following configuration file (corresponds to testProject1.xml. testProject2.xml is similar)

[Command to execute: dotcover c testProject1.xml]

Executing this command generates a snapshot file with coverage data, the Coverage Results Descriptor (by default, unless specified in the configuration file using the TempDir element, the snapshots are saved in the User temporary folder). These snapshots are not the final XML report that we are after but intermediate information that dotCover uses to generate reports from. There is a correspondence between snapshots and processes, such that each process generates a single snapshot. In our case, since we are calling a unit testing framework to launch a single test project, there will only be one snapshot. However there are cases, for instance when running coverage report on an application, whereby the application can launch another process  itself. In this case, multiple snapshots will be generated.

When we run this command, once for each Project, we end up with two snapshot files. We now need to merge the snapshots in order to generate the XML report. This is accomplished using the merge (m) command

[Command to execute: dotcover m merge.xml]

which takes the output of the two cover commands (output1.xml and output2.xml) and generates a single snapshot file. The only thing remaining is to generate an XML coverage report based off of this data, using the report command

[Command to execute: dotcover r report.xml]

providing us with the XML report containing coverage information (percentage of code covered, statements covered), which like in the previous post we can apply an XSLT to format into HTML or JSON.

Other Coverage Commands

dotCover provides two more commands:

• list
• delete

The list command simply obtains a list of all snapshot files given the coverage descriptors, that is, the output of the different cover commands. The configuration file lists these descriptors:

The delete command deletes the list of given snapshots:

Summary

As we can see, dotCover allows us to fine-tune the coverage process by letting us control the different steps. One question that might arise is why couldn’t we have done the same by running the analyse command twice, once for each Test Project. If we were to do that, we would get two independent reports, whereas here the results are all merged into a single report. The analyse command is a great fit for simple scenarios that only have a single project. Once we move onto more complex setups, having a finer control over the coverage process has its benefits.

In this short screencast you can see the basics of dotCover and how to get up and running in a matter of minutes.

dotCover allows us to run coverage analysis on our code. However, there are times when we do not want to perform an analysis on certain areas. This could be our test assemblies, certain third-party assemblies or even specific parts of our own project. Since the analysis has an impact on the overall statistics and potentially can take longer, it is often interesting for us to filter certain assemblies or classes out.

The figure below shows the coverage report of the default MVC project (yes, yet another MVC example, but it ships in the box and probably the ONLY template with VS that comes with tests). In this case we’re using xUnit for tests, which is mostly to demonstrate that dotCover works with other frameworks, not only MSTest (personally I’ve moved on to MSpec, which dotCover also supports).

If we focus on the Coverage results we can see that there are some areas in grey, which indicate that the corresponding PDB files were missing so coverage could not take place. We can also see other assemblies that have been included in the results yet might not be of interest, such as the test assemblies.

In order to filter these out, we can use the Coverage Filters, which can be accessed via the dotCover menu in the IDE

By default, everything is set to be covered, as shown by the Everything entry in the Allow Filters tab (which most likely will be renamed to Included). Examining this entry (clicking on Edit) we can see that it is composed of three values:

• Module Mask indicates the project name
• Class Mask indicates the class name
• Function Mask indicates the method name

(All entries support wildcards as displayed by the * that appears)

We can now uses these patterns to exclude projects, assemblies, namespaces, classes and methods from out tests.

Excluding entire project

Click on Deny Filters tab (most likely will be renamed to Excluded) and click on Add, entering the following information:

Clicking OK  and re-running the tests, we now see that the entire test assembly has been excluded

Excluding an entire Namespace

We can exclude entire namespaces by setting it in the Class Mask as shown below:

this will exclude all the classes inside the MvcApplication16.Models namespace. If we just wanted a specific class, we add it after Models instead of *.

Excluding a method

We can exclude a method, similar to how we’ve done it with the previous cases:

Adding a series of filters for test and auxiliary assemblies using the previous steps, we can end up with a cleaner and more accurate coverage report:

produced by adding the following filters:

Notice that by using the *.Tests (or alternatively as I call it *.Specifications), we can automatically exclude all our test projects from code coverage.

Beta Disclosure

Note 1: Currently these settings are Global, that is, they apply to all projects. Most likely we’ll be changing it so that they are Solution and Global scoped. This way, common libraries can be excluded for all solutions, and specific projects and/or namespaces can be solution based.

Note 2: We’re also looking at adding functionality to make it easier to define these exclusions (a la Right-Click on Folder, add to Excluded list)

As always, we’re still in beta and feedback is more than welcome!

One of the new features dotCover has added is the ability to find tests that cover a certain piece of code. Something remotely similar has been available in ReSharper, although it has been kind of an archaic solution (i.e. Find Usages on Method calls, locate Test assemblies in Result window).

dotCover makes this easier by providing quick access to this information and extends it in functionality.

Below we can see some tests from an MVC application. Let’s run Code Coverage on it using dotCover first.

Now let’s switch over to the Source Code and select some random source code, in this case the MembershipService.ChangePassword line in the ChangePassword action:

In order to see the tests that cover this line of code, we can either press the default key combination of Ctrl+Alt+K or select the option Show Covering Tests from the dotCover:

dotCover will then display a small window showing all the different tests that cover that line of code.

At this point, we can run the selected tests or add them to the existing ReSharper Test Runner session. This allows us to easily jump from specific sections of code to the corresponding tests and execute them instantly.

One minor note: The default key mapping conflicts with KeePass, but you can easily re-assign it via Visual Studio Tools | Options | Keyboard, or do as I did and change KeePass to Ctrl+Alt+P (P as in Password…makes more sense).

Last week we announced a new product by JetBrains for .NET developers: dotCover, a code coverage tool. We are aiming to open the EAP this week and some of our Academy Members have already been playing with it since last week. In this post I’m going to provide some more info on how it works.

Code Coverage 101

Code Coverage is a means by which you measure the percentage of your code that is covered by tests. Technically,the higher the percentage, the less chances of finding hidden bugs when it’s too late. You could therefore assume that 100% coverage is ideal, and you’d most likely be wrong. Having your entire code base covered by tests doesn’t necessarily mean that you are safe. Code Coverage analyzes what parts of your code are covered by tests. It doesn’t tell you whether the code you’ve written is correct. Additionally having a higher percentage often gives a false sense of security, that the code is now bug-free and doesn’t require any other type of testing.

Does this mean therefore, that Code Coverage is useless? Quite the contrary. Code Coverage tools normally provide, at a minimum, two pieces of relevant information: the percentage of code covered and the actual lines of code covered. However, what is important is not so much the code that is covered, but what is not covered. If I have a code base with 70% code coverage, what is more valuable to me than knowing that 30% of my code is not tested, is finding out what the code is. This allows for me to find potential issues that would otherwise go unnoticed.

Testing code for coverage with dotCover

Disclaimer: At the time of writing this blog post, dotCover is still pre EAP so not everything might work as expected

It should somehow be apparent that coverage tools need to work in conjunction with testing frameworks and unit tests. I’m going to base this post on the sample MVC application that ships with ASP.NET MVC 2, the one that comes with some default controllers, not the blank one, you know, the one that doesn’t even bother asking us whether we want a unit test project or not. The project ships with two controllers, HomeController and AccountController along with some corresponding tests.

Similar to dotTrace, dotCover works nicely with ReSharper. When you install dotCover, you see several options appear in the ReSharper menu options as well as the Test Runner window. First of all, under ReSharper | Unit Tests menu option, there is a new option to run coverage reports, either for specific tests or for the entire solution.

Similarly, the Test Runner Window offers us the same options. We get a new icon to run tests under coverage, as well as a new tab that will display the output of the coverage report.

Based on the information we want to obtain, we can run coverage reports on the entire solution or only certain tests. Running on the entire solution will give us a measurement of how much our entire code base is covered. In order to do so, we can select Cover All Tests from Solution. You can assign whatever keyboard shortcut you want to this action by selecting the ReSharper_UnitTest_CoverSolution command, as shown below

Executing the command will run the unit tests and once completed will then proceed to run a coverage report.

We can also run coverage on individual tests or classes, by selecting them and choosing to Cover with dotCover

Let’s try this on the AccountController tests. When we run the coverage, the unit tests will execute inside the ReSharper test runner and once completed, it will then proceed to run coverage reports. To view the results, we can click on the dotCover tab

The window groups coverage reports based on the option selected in the dropdown list. We’re interested in the amount of coverage of our main assembly, not that of the tests (in the future we’ll able to filter test assemblies out). By expanding the node that references our assembly, we get more insight into the actual coverage. Notice that HomeController is at 0% coverage. This doesn’t mean that there are no tests, but that in this specific test run, none of the tests have been executed. This is because we chose to run only the AccountController tests.

dotCover provides us with two measurements: percentage of code covered and number of statements. As such, if we have 83 statements in AccountController out of which 54 are covered with tests, that should indicate 65% coverage. Let’s dig a little deeper to see what is not covered.

We can see that out of the eight actions, four have complete coverage and two have partial coverage. One of these with partial coverage is ChangePassword. There are two overloaded actions for it and it’s a common pattern in ASP.NET MVC applications; one corresponding to a GET request and another to a POST. The second one in the coverage report corresponds to the latter, and shows only 53% coverage. To see in more detail why this is, we can select the Highlight Code icon on the test runner panel

This will highlight the paths of the code that are covered/not covered by tests. Next step is to select the ChangePassword action by double-clicking on it

The lines colored in Cyan mean that they are covered by tests, whereas those in Lilac are not in the execution path of any of the executed tests. In the case of ChangePassword we can see that the test only covers the situation where the model state is valid. This allows us to easily identify non-happy paths of our application and make sure that when relevant, the correct tests are in place for it.

What’s coming…

dotCover is still in it’s infancy and we’ve got a lot planned in terms of features and functionality. Quite a few people have asked whether it will support running coverage reports from a console, which is important for continuous integration. dotCover will support running from a console with some basic functionality, and we have plans to include specific features for CI processes.

If you’re not already following the dotCover Twitter account, make sure you do as we’ll release information and updates as things move forward.