The MSTest plugin can convert TRX format test reports into JUnit XML format so it can be recorded by Hudson.

General Usage

To enable the MSTest plugin features you need to set up your build to run MSTest.exe (or VSTest.Console.exe) first. Example:

"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:MyTestProject\LocalTestRun.testrunconfig /testcontainer:MyTestProject\bin\Release\MyTestProject.dll /resultsfile:TestResults\testResults.trx

and then specify the path to the MSTest TRX file, TestResults/testResults.trx in the above example.

When using VSTest.Console.exe, you can't specify the path to the resulting TRX file. Thus, you can specify a pattern like TestResults/*/*.trx (/ or \, either will work).

If you use the vstestrunner-plugin, the full path to the TRX file is exposed by an environment variable. So, you can also use an environment variable, like $VSTEST_RESULT_TRX or ${VSTEST_RESULT_TRX}.

Test Reporting Options

Test authors can control how each test appears in Hudson/Jenkins by writing specially formatted text to stdout (the console) at the beginning of the test method. The following options are available:

• Append text to the test name (useful for identifying individual data-driven test instances)
• Omit a test from Hudson/Jenkins (useful for some data-driven test scenarios)
• Change the test's display name from the name of the test method to an alternate name (useful when reporting results to non-developers)

In all cases, the specially formatted text must be the first text to appear in the individual test's stdout.

Appending to a test's display name (naming each data-driven test instance)

You may want a test to appear in Jenkins with additional text appended to the usual name of the test method. This is particularly useful for data-driven tests, allowing you to give each individual test case an identifying suffix. To change a test's display name, write the following line as the first text written to stdout during the test's execution:

test-instance-name:{test instance name}:

Replace "{test instance name}" with whatever text you want to have appended to the test name in test reports. For example, if a test method is named MyTestMethod, and the first line of a particular execution of that test method is test-instance-name:Test Case A:, that instance of the test will be displayed in Jenkins as MyTestMethod.Test Case A. Colons are used to delimit the test instance name, so the instance name itself must not contain a colon.

This option is particularly useful if you store the test case name as part of each data row in the data source. You can then refer to the name in the test itself:

Console.WriteLine(String.Format("test-instance-name:{0}:", data_row("TestCaseName")))

You could also use the values of the input parameters themselves as the test instance name:

Console.WriteLine(String.Format("test-instance-name:{0}, {1}:", data_row("param1"), data_row("param2")))

Note that if you don't specify a test-instance-name, then data driven tests are automatically given a suffix based on the data row ID (such as "row 3"), similar to what is displayed in Visual Studio.

Be aware that TRX files generated by vstest.console.exe differ slightly from those generated by mstest, and these differences affect how data-driven test names appear in test reports. Specifically, data-driven tests appear with their data row number, even if you append a test instance name. For example, an instance of the test MyTestMethod with the instance name Test Case A will be displayed as MyTestMethod.Test Case A if the TRX file was generated by mstest, and as MyTestMethod (Data Row 0).Test Case A if the TRX file was generated by vstest.console.exe.

Omitting a test from the test report

You may want a test to be removed from the test report. This is useful in some data-driven test scenarios. For example, consider a set of input parameters in a data source, each of which will be provided to multiple tests. Some inputs may not apply to certain tests. The testing framework will run each test against every available data row, even if the input doesn't apply to a particular test. Removing irrelevant test cases from the test report is cleaner and gives more accurate statistics. The syntax to omit a test is similar to other reporting options:

omit-from-jenkins:true

Like the other reporting options, this must be the first line of text written to stdout during the execution of the test you want to filter from the test report.

Example

Here is an example of how this capability might be used. Consider a data source with the following data:

TestCaseName	Param1	ExpectedResult1	ExpectedResult2
Case A	"Input1"	3.1	1.7
Case B	"Input2"	0.9	1.0
Case C	"Input3"	4.2

In this example, Test1 applies to all 3 data rows, with the expected result specified as ExpectedResult1. Test2, however, only applies to the first 2 data rows; the test does not apply to Case C, as indicated by the blank ExpectedResult2 for that data row. The Test2 test method will need to recognize which test cases to ignore, and write reporting instructions accordingly:

     [DataSource("blah blah blah")]
     [TestMethod()]
     public void Test2()
     {
        if (testContext.DataRow("ExpectedResult2") == null) {
            Console.WriteLine("omit-from-jenkins:true");
            return;
        }

        Console.WriteLine(String.Format("test-instance-name:{0}:", testContext.DataRow("TestCaseName")));
        // do test logic
        ...
     }  
     

Renaming a test

You may want a test to appear in Jenkins with a different name than the name of the test method, possibly to provide a more readable name than the name of the test method. To change a test's display name, write the following line as the first text written to stdout during the test's execution:

test-alternate-name:{alternate name}:

Replace "{alternate name}" with whatever text you want to use as the name of the test in Jenkins reports. For example, test-alternate-name:My Test: will be displayed in Jenkins as My Test, regardless of the name of the test's test method. Colons are used to delimit the alternate test name, so the alternate name itself must not contain a colon.

The test-alternate-name option may not be combined with any other reporting option. If you want to rename a test and append a suffix, include the suffix in the name supplied to the test-alternate-name directive.