Setting up coverage reports on TFS with OpenCover

Code coverage is a metric which indicates the percentage of volume of your source code covered by your tests. It is
certainly a good idea to have code coverage reports generated as part of Continuous Integration – it allows you to keep track of quality of your tests or even set requirements for your builds to have a certain coverage.

Code coverage in Visual Studio is only available in the Enterprise edition. Fortunately, thanks to OpenCover you can still generate coverage reports even if you don’t have access to the Enterprise license.

In this article I will show you how to configure a Build Definition on Team Foundation Server 2015/2017 to use OpenCover to produce code coverage reports.

UPDATE: The full script is available here.

UPDATE 2: Christian Klutz has created a VSTS task based on this article. You may want to check it out. Unfortunately, I won’t be able to offer any help on the topic since I haven’t been using TFS for some time.

Preparations

We are going to put some files on TFS. We will need:

  • RunOpenCover.ps1 – PowerShell script that will run OpenCover – we are going to write it in a moment
  • vsts-task-lib – a PowerShell script library which provides some helpful util functions
  • OpenCover executable
  • OpenCoverToCoberturaConverter – a tool to convert the report to a format understandable by Visual Studio
  • (optional) ReportGenerator – a tool do generate HTML reports

The last three items are available as NuGet packages. I suggest organizing all these files into the following directory structure:

BuildTools
* Packages
  * OpenCover.4.6.519 - the contents of OpenCover package goes here
  * OpenCoverToCoberturaConverter.0.2.6.0 - the contents of OpenCoverToCoberturaConverter package goes here
  * ReportGenerator.2.5.6 - the contents of ReportGenerator package goes here
* Scripts
  * vsts-task-lib - the contents of vsts-task-lib / powershell / VstsTaskSdk goes here
  * RunOpenCover.ps1 - the script that we are going to write

Once done, check it in to your TFS instance.

I’ve put the BuildTools directory on the top level of the repository. Next, I’ve added a mapping to my Build Definition in order to make that directory available during the build.

Create the PowerShell script

Let’s now write the PowerShell script. The script is going to perform a couple of steps:

  • We would like our script to use a file pattern to scan for test assemblies in the same way that the “native” Visual Studio Tests task does. For that, we can use Find-Files cmdlet available in vsts-task-lib.
  • Next, we run OpenCover and use the list of paths with test assemblies as parameters.
  • Next, we need to convert the results file produced by OpenCover to Cobertura – a file format which TFS can understand.
  • Finally, we can use the same results file to produce an HTML, human-readable report.

The script will take a couple of parameters as input:

Param(
    [string]$sourcesDirectory, #the root of your project
    [string]$testAssembly, #the file pattern describing test assemblies to look for
    [string]$testFiltercriteria="", #test filter criteria (as in Run Visual Studio Tests task)
    [string]$openCoverFilters="+[*]*" #OpenCover-specific filters
)

Next, let’s run the Find-Files utility to search against the pattern defined in $testAssembly. This code is copied from the original Run Visual Studio Tests task source code.

# load helper functions from the vsts-task-lib library
. $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1
# resolve test assembly files (copied from VSTest.ps1)
$testAssemblyFiles = @()
# check for solution pattern
if ($testAssembly.Contains("*") -Or $testAssembly.Contains("?"))
{
    Write-Host "Pattern found in solution parameter. Calling Find-Files."
    Write-Host "Calling Find-Files with pattern: $testAssembly"    
    $testAssemblyFiles = Find-Files -LegacyPattern $testAssembly -LiteralDirectory $sourcesDirectory
    Write-Host "Found files: $testAssemblyFiles"
}
else
{
    Write-Host "No Pattern found in solution parameter."
    $testAssembly = $testAssembly.Replace(';;', "`0") # Barrowed from Legacy File Handler
    foreach ($assembly in $testAssembly.Split(";"))
    {
        $testAssemblyFiles += ,($assembly.Replace("`0",";"))
    }
}
# build test assembly files string for vstest
$testFilesString = ""
foreach ($file in $testAssemblyFiles) {
    $testFilesString = $testFilesString + " ""$file"""
}

We can finally run OpenCover. The command to do this is pretty complicated. OpenCover supports different test runners (VSTest being only one of them) so we need to specify the path to VSTest as one of the arguments. The path below (%VS140COMNTOOLS%..\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe ) is valid for Visual Studio 2015 installation.

Another important argument is -mergebyhash . It forces OpenCover to treat assemblies with the same hash as one. I’ve spent a few hours figuring out why my coverage score is so low. It turned out that OpenCover analyzed few copies of the same assembly.

Start-Process "$PSScriptRoot\..\Packages\OpenCover.4.6.519\OpenCover.Console.exe" -wait -NoNewWindow -ArgumentList "-register:user -filter:""$OpenCoverFilters"" -target:""%VS140COMNTOOLS%\..\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"" -targetargs:""$testFilesString /TestCaseFilter:$testFiltercriteria /logger:trx"" -output:OpenCover.xml -mergebyhash" -WorkingDirectory $PSScriptRoot

Next, let’s convert the results generated by OpenCover to Cobertura format.

Start-Process "$PSScriptRoot\..\Packages\OpenCoverToCoberturaConverter.0.2.6.0\tools\OpenCoverToCoberturaConverter.exe" -Wait -NoNewWindow -ArgumentList "-input:""$PSScriptRoot\OpenCover.xml"" -output:""$PSScriptRoot\Cobertura.xml"" -sources:""$sourcesDirectory"""

Finally, we will generate a HTML report based on the results from OpenCover.

Start-Process "$PSScriptRoot\..\Packages\ReportGenerator.2.5.6\tools\ReportGenerator.exe" -Wait -NoNewWindow -ArgumentList "-reports:""$PSScriptRoot\OpenCover.xml"" -targetdir:""$PSScriptRoot\CoverageReport"""

And that’s it.

Configure the Build Definition

We will need to add three build steps to our Build Definition. If you have a Visual Studio Tests task in it, remove it – you will no longer need it.

  • PowerShell task – set the Script Path to point to RunOpenCover.ps1 and specify the Arguments:
-sourcesDirectory "$(Build.SourcesDirectory)" -testAssembly "**\*.Tests.dll;-:**\obj\**" -testFiltercriteria "TestCategory!=INTEGRATION"
  • Publish Test Results task – configure it as on the image below; as a by-product of generating coverage reports, we produce test results – we need to tell TFS where to find them

  • Publish Code Coverage Results task – configure it as on the image below; thanks to this task the results will be visible  on the build summary page

And that’s it! Run the build definition and enjoy your code coverage results. You can find the on the build summary page. The HTML report is available as one of the build artifacts.

31 thoughts on “Setting up coverage reports on TFS with OpenCover

  1. Hi,

    Do you see an issue creating this into a normal TFS / VSTS extension on the marketplace that can be used with any project? Is there something a blocking / missing feature stopping you(or me) from making this into a extension?

    Just asking before I set out to make it into one… 🙂

  2. hi, i am converting open cover xml to cobertura using below given code but cobertura xml file generated does not show any code coverage which is there originally

    Start-Process “Path To OpenCoverToCobertura\OpenCoverToCoberturaConverter.exe” -Wait -ArgumentList “-input:”” Path to open cover \Coverage.xml”” -output:””output path \Cobertura.xml”” ”

    the open cover xml is generated using sql cover tool for finding code coverage on database.

    Can you please help me out ?

  3. Hi Miłosz. Great post!

    I’m struggling in getting this to work, when my TFS executes the step that runs the RunOpenCover.ps1 script it fails saying:

    Find-Files : The ‘Find-Files’ command was found in the module ‘Microsoft.TeamFoundation.DistributedTask.Task.Common’, but the module could not be loaded. For more information, run ‘Import-Module Microsoft.TeamFoundation.DistributedTask.Task.Common’.

    I’ve followed the same directory structure that you suggested, so I’ve the script LegacyFindFunctions.ps1 (which is where I’ve found the function Find-Files) in: scripts\vsts-task-lib\LegacyFindFunctions.ps1.

    On the other hand, I’ve the script RunOpenCover.ps1 in: scripts\RunOpenCover.ps1.

    Any thoughts?

    Thanks!
    Walter

  4. Hi Walter,

    It seems that I forgot to mention that you should load the LegacyFindFunctions.ps1 file. This command should do the trick:

    `. $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1`

    Let me know if it helped.

  5. Hello! Thank you for the post!

    I am trying to follow step-by-step, but I have a problem.

    When the PowerShell task runs the RunOpenCover.ps1, I have an error:

    “The term ‘Find-Files’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again”.

    I use the same directory structure (the directory scripts contains the lib and RunOpenCover.ps1)

  6. Hi Nathalie,

    It seems that I forgot to mention that you should load the LegacyFindFunctions.ps1 file. This command should do the trick:

    . $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1

    Let me know if it helped.

    1. Thank you!

      But now I have another error:

      ##[error]Start-Process : This command cannot be run due to the error: The system cannot find the file specified.

      I just add the line on the top of script:
      Start-Process “$PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1”

      1. Nathalie, you actually need to use the dot (.). It will make PowerShell include code from the script you are point it to. Use the command exactly as below:

        `. $PSScriptRoot\vsts-task-lib\LegacyFindFunctions.ps1`

  7. Is it possible to provide a working example for newbies? Right now, each .ps1 file I import asks for another. Importing all of them does not solve the matter either. I need this and I am not very familiar with PowerShell.

      1. Hi,
        Thanks a lot, I will be waiting for that. Also, part of the problem is perhaps that I don’t know how to get vsts-task-lib in vsts using NuGet. Should I just download it from the repo and put it next in the folder where the script exists (that is what I am doing right now)?

        Thanks again.

      2. Oh and as for the error message, the advice you gave to Nathalie did not quite work. From what I understood, it imported `LegacyFindFunctions.ps1` so that Find-Files function would work. However, this resulted in another error:

        `##[error]Trace-EnteringInvocation : The term ‘Trace-EnteringInvocation’ is not recognized as the name of a cmdlet, function,
        script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is
        correct and try again.`

        Trace-EnteringInvocation was no defined, and so I had to import `TraceFunctions.ps` as well to resolve the error, and so on, each file kept complaining about a function that’s not defined, until I ended up importing all `.ps1` files and yet there was still another function that could not be found.

        I also had these errors (although the VSTS build pipeline shows that NuGet got the corresponding packages):

        `2017-10-05T13:23:20.0871036Z ##[error]Start-Process : This command cannot be run due to the error: The system cannot find the file specified.
        At C:\agent\_work\2\s\scripts\RunOpenCover.ps1:35 char:1
        + Start-Process “$PSScriptRoot\..\packages\OpenCover.4.6.519\OpenCover. …
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : InvalidOperation: (:) [Start-Process], InvalidOperationException
        + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand

        2017-10-05T13:23:20.0871036Z Report does not exist: C:\agent\_work\2\s\scripts\OpenCover.xml
        2017-10-05T13:23:21.0877421Z ##[error]Start-Process : This command cannot be run due to the error: The system cannot find the file specified.
        At C:\agent\_work\2\s\scripts\RunOpenCover.ps1:39 char:1
        + Start-Process “$PSScriptRoot\..\packages\ReportGenerator.2.5.6\tools\ …
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : InvalidOperation: (:) [Start-Process], InvalidOperationException
        + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand
        `

        1. Hi Alex. I’ve put the link to the full script on the top of the page. Apparently I’ve missed some more `vsts-task-lib` includes. The library itself should be downloaded, unpacked and placed next to the script (as I indicated in the suggested file structure). The `packages` folder has to be in the right place too.

  8. Hi,

    Thanks for the great article, it looks like my best bet in having coverage on TFS online. I’ve followed your instructions and used your linked RunOpenCover.ps1 file, but upon running on TFS, I receive the following error:

    2017-11-03T00:07:15.9771155Z Calling Find-Files with pattern: **\*.Tests.dll;-:**\obj\**
    2017-11-03T00:07:16.2721130Z ##[error]Unable to find type [VstsTaskSdk.FS.FindFlags].
    At D:\a\1\s\BuildTools\Scripts\vsts-task-lib\LongPathFunctions.ps1:49 char:9
    + [VstsTaskSdk.FS.FindFlags]$Flags = [VstsTaskSdk.FS.FindFlags] …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (VstsTaskSdk.FS.FindFlags:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

    So my question, what version or tag of VSTS did you take to have it working ? It looks like it changed to be behaving that way. Maybe you could upload a zip containing the version you got it working on ?

    Thanks again !
    Bruno

      1. @Ryan, @theraven403 If you are still struggling, I managed to figure this out.
        The vsts-task-lib libraries have changes, so what was formerly a powershell script has been pulled out into a DLL. I needed to add the following to the script, so that Powershell can use the DLL:

        Add-Type -Path “$PSScriptRoot\vsts-task-lib\VsTsTaskSdk.dll”

        I added that immediately below the param definitions at the top of the file.

        Hope this helps

  9. Hi! Thank you very much for this article!

    We have a very strange issue with the Code Coverage report so maybe you have some experience with this. After integrating everything listed here, we are finally able to see the Code Coverage info on the Build summary page in VSTS. We are also able to download the Code Coverage results and it works great. However, the Code Coverage tab on the Build summary looks like it does show the right page from the report, but without any css. We are seeing a very basic HTML, no styles, no colors, no nothing, so it’s not very useful 🙁

    Do you have any idea what might be the problem?

    Thanks,
    Jelena

  10. Hi Miłosz,
    Thank you for this great solution, using your article and some useful comments we were able to get it to work on TFS2018.
    Like Jelena we get the code coverage info on the Build summary page, however in TFS there is an OpenCover tab. When you click it, some column headers are shown but the actual coverage data is missing.
    Any ideas how to fix this?
    Thank you,
    Willem

  11. Hi Milosz!

    I love your work! however, i am encountering the error of the “test source files were specified”. Would like to seek your advice on this matter. Log as below

    ——————
    PS C:> . ‘C:agent_work6sScriptsRunOpenCover.ps1’ -sourcesDirectory “C:agent_work6s” -testAssembly “***test*.
    dll !**obj**” -testFiltercriteria “TestCategory!=INTEGRATION”
    start—–(echoing out the variables passed in)
    C:agent_work6s ([string]$sourcesDirectory)
    ***test*.dll !**obj** ([string]$testAssembly)
    TestCategory!=INTEGRATION ([string]$testFiltercriteria=””)
    +[*]* ([string]$openCoverFilters=”+[*]*”)
    end—–
    Pattern found in solution parameter. Calling Find-Files.

    Calling Find-Files with pattern: ***test*.dll !**obj**
    Found files:
    RunOpenCover.ps1 : Removing old testresults
    RunOpenCover.ps1 : Running OpenCover using vstest.console.exe
    Executing: C:Program Files (x86)Microsoft Visual Studio2017CommunityCommon7IDECommonExtensionsMicrosoftTestWindowvstest.console.exe
    Microsoft (R) Test Execution Command Line Tool Version 15.7.2
    Copyright (c) Microsoft Corporation. All rights reserved.

    No test source files were specified.
    Committing…
    No results, this could be for a number of reasons. The most common reasons are:
    1) missing PDBs for the assemblies that match the filter please review the
    output file and refer to the Usage guide (Usage.rtf) about filters.
    2) the profiler may not be registered correctly, please refer to the Usage
    guide and the -register switch.
    RunOpenCover.ps1 : Converting Code Coverage result to Cobertura format
    ———-

    My File structure is as below
    * “Scripts” folder
    ** RunOpenCover.ps1
    ** “vsts-task-lib” folder
    * “WebGoat” folder (.net web application)
    * “WebGoatUnitTest” folder
    * “Packages” folder

      1. No worries Milosz. I found the fix to the issue. It seems that the opncover porifler is not register properly so I added a script to register it and it’s working properly now.

        Thank you again for sharing this solution. It’s a big help for me!

Leave a Reply

Your email address will not be published.