Setting up coverage reports on TFS with OpenCover

Code coverage is a metric which indicates the percentage of volume of your source code covered by your tests. It is
certainly a good idea to have code coverage reports generated as part of Continuous Integration – it allows you to keep track of quality of your tests or even set requirements for your builds to have a certain coverage.

Code coverage in Visual Studio is only available in the Enterprise edition. Fortunately, thanks to OpenCover you can still generate coverage reports even if you don’t have access to the Enterprise license.

In this article I will show you how to configure a Build Definition on Team Foundation Server 2015/2017 to use OpenCover to produce code coverage reports.

UPDATE: The full script is available here.

Preparations

We are going to put some files on TFS. We will need:

  • RunOpenCover.ps1 – PowerShell script that will run OpenCover – we are going to write it in a moment
  • vsts-task-lib – a PowerShell script library which provides some helpful util functions
  • OpenCover executable
  • OpenCoverToCoberturaConverter – a tool to convert the report to a format understandable by Visual Studio
  • (optional) ReportGenerator – a tool do generate HTML reports

The last three items are available as NuGet packages. I suggest organizing all these files into the following directory structure:

BuildTools
* Packages
  * OpenCover.4.6.519 - the contents of OpenCover package goes here
  * OpenCoverToCoberturaConverter.0.2.6.0 - the contents of OpenCoverToCoberturaConverter package goes here
  * ReportGenerator.2.5.6 - the contents of ReportGenerator package goes here
* Scripts
  * vsts-task-lib - the contents of vsts-task-lib / powershell / VstsTaskSdk goes here
  * RunOpenCover.ps1 - the script that we are going to write

Once done, check it in to your TFS instance.

I’ve put the BuildTools directory on the top level of the repository. Next, I’ve added a mapping to my Build Definition in order to make that directory available during the build.

Create the PowerShell script

Let’s now write the PowerShell script. The script is going to perform a couple of steps:

  • We would like our script to use a file pattern to scan for test assemblies in the same way that the “native” Visual Studio Tests task does. For that, we can use Find-Files cmdlet available in vsts-task-lib.
  • Next, we run OpenCover and use the list of paths with test assemblies as parameters.
  • Next, we need to convert the results file produced by OpenCover to Cobertura – a file format which TFS can understand.
  • Finally, we can use the same results file to produce an HTML, human-readable report.

The script will take a couple of parameters as input:

Param(
    [string]$sourcesDirectory, #the root of your project
    [string]$testAssembly, #the file pattern describing test assemblies to look for
    [string]$testFiltercriteria="", #test filter criteria (as in Run Visual Studio Tests task)
    [string]$openCoverFilters="+[*]*" #OpenCover-specific filters
)

Next, let’s run the Find-Files utility to search against the pattern defined in $testAssembly. This code is copied from the original Run Visual Studio Tests task source code.

# load helper functions from the vsts-task-lib library
. $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1
# resolve test assembly files (copied from VSTest.ps1)
$testAssemblyFiles = @()
# check for solution pattern
if ($testAssembly.Contains("*") -Or $testAssembly.Contains("?"))
{
    Write-Host "Pattern found in solution parameter. Calling Find-Files."
    Write-Host "Calling Find-Files with pattern: $testAssembly"    
    $testAssemblyFiles = Find-Files -LegacyPattern $testAssembly -LiteralDirectory $sourcesDirectory
    Write-Host "Found files: $testAssemblyFiles"
}
else
{
    Write-Host "No Pattern found in solution parameter."
    $testAssembly = $testAssembly.Replace(';;', "`0") # Barrowed from Legacy File Handler
    foreach ($assembly in $testAssembly.Split(";"))
    {
        $testAssemblyFiles += ,($assembly.Replace("`0",";"))
    }
}
# build test assembly files string for vstest
$testFilesString = ""
foreach ($file in $testAssemblyFiles) {
    $testFilesString = $testFilesString + " ""$file"""
}

We can finally run OpenCover. The command to do this is pretty complicated. OpenCover supports different test runners (VSTest being only one of them) so we need to specify the path to VSTest as one of the arguments. The path below (%VS140COMNTOOLS%\..\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe ) is valid for Visual Studio 2015 installation.

Another important argument is -mergebyhash . It forces OpenCover to treat assemblies with the same hash as one. I’ve spent a few hours figuring out why my coverage score is so low. It turned out that OpenCover analyzed few copies of the same assembly.

Start-Process "$PSScriptRoot\..\Packages\OpenCover.4.6.519\OpenCover.Console.exe" -wait -NoNewWindow -ArgumentList "-register:user -filter:""$OpenCoverFilters"" -target:""%VS140COMNTOOLS%\..\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"" -targetargs:""$testFilesString /TestCaseFilter:$testFiltercriteria /logger:trx"" -output:OpenCover.xml -mergebyhash" -WorkingDirectory $PSScriptRoot

Next, let’s convert the results generated by OpenCover to Cobertura format.

Start-Process "$PSScriptRoot\..\Packages\OpenCoverToCoberturaConverter.0.2.6.0\tools\OpenCoverToCoberturaConverter.exe" -Wait -NoNewWindow -ArgumentList "-input:""$PSScriptRoot\OpenCover.xml"" -output:""$PSScriptRoot\Cobertura.xml"" -sources:""$sourcesDirectory"""

Finally, we will generate a HTML report based on the results from OpenCover.

Start-Process "$PSScriptRoot\..\Packages\ReportGenerator.2.5.6\tools\ReportGenerator.exe" -Wait -NoNewWindow -ArgumentList "-reports:""$PSScriptRoot\OpenCover.xml"" -targetdir:""$PSScriptRoot\CoverageReport"""

And that’s it.

Configure the Build Definition

We will need to add three build steps to our Build Definition. If you have a Visual Studio Tests task in it, remove it – you will no longer need it.

  • PowerShell task – set the Script Path to point to RunOpenCover.ps1 and specify the Arguments:
-sourcesDirectory "$(Build.SourcesDirectory)" -testAssembly "**\*.Tests.dll;-:**\obj\**" -testFiltercriteria "TestCategory!=INTEGRATION"
  • Publish Test Results task – configure it as on the image below; as a by-product of generating coverage reports, we produce test results – we need to tell TFS where to find them

  • Publish Code Coverage Results task – configure it as on the image below; thanks to this task the results will be visible  on the build summary page

And that’s it! Run the build definition and enjoy your code coverage results. You can find the on the build summary page. The HTML report is available as one of the build artifacts.

20 thoughts on “Setting up coverage reports on TFS with OpenCover

  1. Hi,

    Do you see an issue creating this into a normal TFS / VSTS extension on the marketplace that can be used with any project? Is there something a blocking / missing feature stopping you(or me) from making this into a extension?

    Just asking before I set out to make it into one… 🙂

  2. hi, i am converting open cover xml to cobertura using below given code but cobertura xml file generated does not show any code coverage which is there originally

    Start-Process “Path To OpenCoverToCobertura\OpenCoverToCoberturaConverter.exe” -Wait -ArgumentList “-input:”” Path to open cover \Coverage.xml”” -output:””output path \Cobertura.xml”” ”

    the open cover xml is generated using sql cover tool for finding code coverage on database.

    Can you please help me out ?

  3. Hi Miłosz. Great post!

    I’m struggling in getting this to work, when my TFS executes the step that runs the RunOpenCover.ps1 script it fails saying:

    Find-Files : The ‘Find-Files’ command was found in the module ‘Microsoft.TeamFoundation.DistributedTask.Task.Common’, but the module could not be loaded. For more information, run ‘Import-Module Microsoft.TeamFoundation.DistributedTask.Task.Common’.

    I’ve followed the same directory structure that you suggested, so I’ve the script LegacyFindFunctions.ps1 (which is where I’ve found the function Find-Files) in: scripts\vsts-task-lib\LegacyFindFunctions.ps1.

    On the other hand, I’ve the script RunOpenCover.ps1 in: scripts\RunOpenCover.ps1.

    Any thoughts?

    Thanks!
    Walter

  4. Hi Walter,

    It seems that I forgot to mention that you should load the LegacyFindFunctions.ps1 file. This command should do the trick:

    `. $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1`

    Let me know if it helped.

  5. Hello! Thank you for the post!

    I am trying to follow step-by-step, but I have a problem.

    When the PowerShell task runs the RunOpenCover.ps1, I have an error:

    “The term ‘Find-Files’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again”.

    I use the same directory structure (the directory scripts contains the lib and RunOpenCover.ps1)

  6. Hi Nathalie,

    It seems that I forgot to mention that you should load the LegacyFindFunctions.ps1 file. This command should do the trick:

    . $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1

    Let me know if it helped.

    1. Thank you!

      But now I have another error:

      ##[error]Start-Process : This command cannot be run due to the error: The system cannot find the file specified.

      I just add the line on the top of script:
      Start-Process “$PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1”

      1. Nathalie, you actually need to use the dot (.). It will make PowerShell include code from the script you are point it to. Use the command exactly as below:

        `. $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1`

  7. Is it possible to provide a working example for newbies? Right now, each .ps1 file I import asks for another. Importing all of them does not solve the matter either. I need this and I am not very familiar with PowerShell.

      1. Hi,
        Thanks a lot, I will be waiting for that. Also, part of the problem is perhaps that I don’t know how to get vsts-task-lib in vsts using NuGet. Should I just download it from the repo and put it next in the folder where the script exists (that is what I am doing right now)?

        Thanks again.

      2. Oh and as for the error message, the advice you gave to Nathalie did not quite work. From what I understood, it imported `LegacyFindFunctions.ps1` so that Find-Files function would work. However, this resulted in another error:

        `##[error]Trace-EnteringInvocation : The term ‘Trace-EnteringInvocation’ is not recognized as the name of a cmdlet, function,
        script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is
        correct and try again.`

        Trace-EnteringInvocation was no defined, and so I had to import `TraceFunctions.ps` as well to resolve the error, and so on, each file kept complaining about a function that’s not defined, until I ended up importing all `.ps1` files and yet there was still another function that could not be found.

        I also had these errors (although the VSTS build pipeline shows that NuGet got the corresponding packages):

        `2017-10-05T13:23:20.0871036Z ##[error]Start-Process : This command cannot be run due to the error: The system cannot find the file specified.
        At C:\agent\_work\2\s\scripts\RunOpenCover.ps1:35 char:1
        + Start-Process “$PSScriptRoot\..\packages\OpenCover.4.6.519\OpenCover. …
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : InvalidOperation: (:) [Start-Process], InvalidOperationException
        + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand

        2017-10-05T13:23:20.0871036Z Report does not exist: C:\agent\_work\2\s\scripts\OpenCover.xml
        2017-10-05T13:23:21.0877421Z ##[error]Start-Process : This command cannot be run due to the error: The system cannot find the file specified.
        At C:\agent\_work\2\s\scripts\RunOpenCover.ps1:39 char:1
        + Start-Process “$PSScriptRoot\..\packages\ReportGenerator.2.5.6\tools\ …
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : InvalidOperation: (:) [Start-Process], InvalidOperationException
        + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand
        `

        1. Hi Alex. I’ve put the link to the full script on the top of the page. Apparently I’ve missed some more `vsts-task-lib` includes. The library itself should be downloaded, unpacked and placed next to the script (as I indicated in the suggested file structure). The `packages` folder has to be in the right place too.

  8. Hi,

    Thanks for the great article, it looks like my best bet in having coverage on TFS online. I’ve followed your instructions and used your linked RunOpenCover.ps1 file, but upon running on TFS, I receive the following error:

    2017-11-03T00:07:15.9771155Z Calling Find-Files with pattern: **\*.Tests.dll;-:**\obj\**
    2017-11-03T00:07:16.2721130Z ##[error]Unable to find type [VstsTaskSdk.FS.FindFlags].
    At D:\a\1\s\BuildTools\Scripts\vsts-task-lib\LongPathFunctions.ps1:49 char:9
    + [VstsTaskSdk.FS.FindFlags]$Flags = [VstsTaskSdk.FS.FindFlags] …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (VstsTaskSdk.FS.FindFlags:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

    So my question, what version or tag of VSTS did you take to have it working ? It looks like it changed to be behaving that way. Maybe you could upload a zip containing the version you got it working on ?

    Thanks again !
    Bruno

Leave a Reply