Friday, April 13, 2012

Assigning Build to Test Plan in MTM

Introduction
So at this point you have your automation running using Microsoft Unit Testing Framework, you have set up your test environments, and you have test cases tied to your automation. So last how do we run the automated tests using MTM? This last step is you need to create a build, using the Microsoft Build Server. Then assign the test plan to this build.

Prerequisites
  • TFS Installed
  • MTM Installed
  • Familiar with Source Control and have a solution running
  • Have Microsoft Build Server installed and running.
Step 1 Add your solution to source control
1a. Go To Team Explorer, then click on Source Control. Click on the icon on the top menu.
1b. Select the your solution folder, and the location that you want to store it in source control. Then click Map.


1c. Check in all of your solution into source control. Right click on the solution in Solution Explorer, and
click on Check in.

Step 2 Create a new build
2a Go to Team Explorer, and left click on builds and click on New Build Definition
2b Go through and fill out each page of the build definition, making sure to associate the build to your project in source control under the Workspace section.
2c Save Build Definition

Step 3 Queue new Build
3a. Right click on the build definition under Builds in the Team Explorer and click Queue new build.

Step 4 Associate new build to Test Plan
4a. In MTM go to Testing Center - Plan and click Properties, Under the properties there is a builds section. Select the build you just added, and save the properties.


Step 5 Run automated test cases
With your test cases associated to the automation, and a build associated to the test plan, you are now ready to run the tests. Go to Testing Center - Test, select the automated test case and click on run. Within a minute it should start running one of your set up test agents.




Thursday, April 12, 2012

Download MormonOrgTest Source Code

Attached is my project source code for Webdriver/Microsoft Unit Testing Software

Using MTM as a load balancer to run test automation



Introduction

So now that you have created your 1,000 automated test cases, now what? Running a 1,000 test cases on one machine, is going to take a day or two. One of the great benefits of Microsoft Test Mamangement System (MTM) is that you can create a lab environment to run your tests on.

Prerequisites

I am assuming that you have a TFS server up and running, and are using MTM for running and tracking test cases.

Overview of TFS Lab Environment Set Up

Microsoft Test Manager (MTM) communicates to the Team Foundation Server (TFS), to get test cases, automation builds, and a list of Test Controllers. When running automated tests, MTM sends the tests to TFS. The Test Controller is on a timer to check the TFS Server for tests needed to be run, it then picks up these tests and waits for each of the test agents to check in, once each of the test agents check in, it delivers a portion of the tests to the test agent, then each test agent reports back the results of the tests it ran, including any files relating to the testing. After the test controller has collected the results of all the tests, it then delivers them to the TFS Server, where the results can be viewed using MTM.



Step 1 Install Test Controller

You are going to want to install a Test Controller on one of your lab machines. Note that the Test Controller can exist on the same machine as a Test Agent.

You download the controller and agent installs here:

When installing the controller you will need a domain user that has access to the TFS server, and has admin rights of the machine that the controller is installed on.

1a. Login as the Domain User that you have created to run the controller. (must be admin)
1b. Download the controller, you may need something to convert ISO to a physical file, so you can launch the setup.
1c. Install the Test Controller
1d. Configure Test Controller should launch after install.
1e. Use specific domain account to login to TFS and other Test Agents. Put in the user credentials of the domain user created in step 1a.
1e. Check "Register with Team Project Collection" and click on Browser add your TFS server location, and select your project.
1f. Click Apply Settings.



Step 2 Install Test Agents

For each lab machine, install the Test Agent on the machine. You will want to add the new domain user to each machine, login as that user on each machine, and install and configure the agent on each machine. To save time, you can copy the files that you expanded from the ISO image in step 1 to each machine, then run the set up from each machine.

1a. Login as the Domain User that you created. (must be admin rights)
1b. Download the Test Agent, copy files from machine in step 1.
1c. Install the Test Agent
1d. Configure Test Agent should launch after install.
1e. Use specific domain account you used to login to machine as.
1e. Check "Register with Test Controller" and put the name of the machine that is on the same domain, that contains the Test Controller
1f. Click Apply Settings.

Step 3 Add New Physical Environment

3a. Open Microsoft Test Management
3b. Navigate to Lab Center -Lab - Environments, Click on New, then Physical Environment.




3c. Fill out the name and description of the new environment, and set the location of the new controller you just added in step 1.


3d. Next select Machines (on the left menu), This page is set up into two planes, on the right is the available machines from the selected controller, and on the left is the machines added to the environment. Select each of the machines that you want to add to the environment. (that will be used for your automation), Set them all to type of Desktop Client. Then click finish.

Step 4 Associate Environment to Test Plan

4a. Navigate to the Testing Center - Plan - Properties, set the Test Environment to the environment you just created. Now any automated test ran in this test suite will run on the Test lab you just created.




Microsoft Unit Testing Framework - Using Test Context to determine test result

When creating your test automation, and you need to run a procedure when the test did not pass. Use the TestContext.CurrentTestOutcome property. This property will determine the result of the test in the Microsoft Unit Testing Framework.

In this following example I wanted to capture a screen shot only if the result had failed.



[TestCleanup]
public void CleanUpTest()
{
if(TestContext.CurrentTestOutcome!=UnitTestOutcome.Passed)
{
Tool.GetScreenShot();
Tool.SaveScreenShot();

}
 
}


The TestContext.CurrentTestOutcome property will reference a UnitTestOutcome enumerator. In the above example, if the test hadn't passed, then on clean up I will take a screen shot, and then save all screen shots to a result file. .

Saturday, April 7, 2012

Microsoft Unit Testing Framework - Using Test Context as a Static Variable

Using Test Context as a static variable
One of the problems I ran into when trying to capture a screen shot. I wanted to capture a screen shot only when an error happened. I wanted the screen shot to be called as part of Test Clean Up or Class Clean Up, I also wanted to add a screen shot right when I logged in, to make sure the screen looked normal at the beginning of the class.

If you are using your initialize and clean up attributes correctly, you probably created a class file that contains all of the logic for starting and ending a test. In this class file if you add the Test Context Initialize sequence:
private TestContext m_testContext;javascript:void(0)
        public TestContext TestContext
        {
            get { return m_testContext; }
            set { m_testContext = value; }
        }

Then add this to the Assembly Initialize:

Screenshot ss = Driver.GetScreenshot()
            String fileNames = Environment.GetEnvironmentVariable("TEMP") + "\\TempFile" + testName + DateTime.Now.ToString("MMddHHmmss") + ".Png");
            ss.SaveAsFile(fileNames, ImageFormat.Png);
            TestContext.AddResultFile();


The Problem
You will get an error on TestContext.AddResultFile(): "An object reference required for the non-static field, method, or property"

Which poses as a big bummer. So does this mean I can't use Test Context for uploading a file on Assembly initialize? No.

Solution
So here's what I did.
First create a tool class if you haven't already that keeps all of the supporting framework methods for your automation. This class should be a static class. In this static class create a two methods 1. Get Screen Shot, 2. Save Screen Shot. The first method will capture the screen shots, and save the file names in a static collection. The second will then use Text Context to go through and upload each of the files contained in the collection and add to the result file. The first method I will call every time I need a screen shot, the second method I will call only on Clean Up attribute at the end to add the files to the result file.


Step 1: Add Static Test Context to Base File
If you have a file that you use for all of your static variables like Driver, Website address, or db connection. Add these lines to the file.

private static TestContext testContextInstance;

public static TestContext TestContext
        {
            get
            {
                return testContextInstance;
            }
            set
            {
                testContextInstance = value;
            }
        }


This will create a static accessible Test Context.

Step 2: Initialize Test Context
On [AssemblyInitialize] include TestContext testContext in the call, then assign testContext to the Static Test Context in the class.

for example:
[AssemblyInitialize]
        public static void SetUp(TestContext testContext)
        {
            TestContext = testContext;
        }


Step 3: Create Method in Tool

The following code will create a screen shot using the Web Driver Framework, save the file in a TEMP directory, and Store the full file name in a static string collection.

First: Create a static collection for the filename:

private static List Filenames = new List();


Second: Create a method to capture screen shot:

public static void GetScreenShot()
{
string testName = Base.TestContext.TestName;
Screenshot ss = Base.Driver.GetScreenshot();
Filenames.Add( Environment.GetEnvironmentVariable("TEMP")+"\\TempFile"+ testName + DateTime.Now.ToString("MMddHHmmss") + ".Png");
ss.SaveAsFile(Filenames[(Filenames.Count-1)], ImageFormat.Png);
}


(Note you will need references: System.Drawing.Imaging; OpenQA.Selenium; System.Collection.Generic; System.Collection.ObjectModel; )

(Additional Note: Notice that we are accessing the static TestContext in the base class to the get the TestName, and include it as part of the file name. )


Step 4: Add Screen Shots to Result File
Now we have to create one more tool method that will add the file to the result set. This method will go through each of the full file names in the static file name collection, and save each file, then delete the temp file.

public static void SaveScreenShot()
        {
 foreach(string filename in Filenames)
            {
                Base.TestContext.AddResultFile(filename);
                File.Delete(filename);
            }
            Filenames.Clear();
         }


Step 5: Call methods

In the TestCleanup add the following:
[TestCleanup]
        public void CleanUp()
        {
      
            Tool.SaveScreenShot();
           
            
        }


Next Add the following to any place you want to grab a screen shot
For Example here I am getting a screen shot before I sign out of a test.

[TestCleanup]
        public void CleanUp()
        { 
            Tool.GetScreenShot();
            Tool.SaveScreenShot();
            
            _nav.SignOut();
            
        }

Tuesday, March 27, 2012

Capturing Screen Shot and Uploading to Result File in Microsoft Unit Testing Framework (MUTF)

One of the great benefits to Microsoft Unit Test Framework, is it allows you to attach files to the be delivered with the result file. This could be a log file, a system file, or in this example a screen shot.

To create the screen shot I am using the tool built directly into WebDriver.

The code is:

Screenshot ss = Base.Driver.GetScreenshot();
ss.SaveAsFile("TempFileLocation", ImageFormat.Png);

This is the code I used, notice I added the GetEnvironmentVariable to grab the Temp directory of the computer, and I added the Date and Time to the file name to create a unique instance of the file. You will want to keep the filename in a variable so you can reuse later on. (

Screenshot ss = Base.Driver.GetScreenshot();
String Filename = Environment.GetEnvironmentVariable("TEMP")+"\\TempFile"+ testName + DateTime.Now.ToString("MMddHHmmss") + ".Png");
ss.SaveAsFile(Filenames[(Filenames.Count-1)], ImageFormat.Png);


To include the file in the result set you will need to instantiate TestContext, then call:

TestContext.AddResultFile("locationAndNameOfFile");

In the example above I would use the code

TestContext.AddResultFile(Filename);

This will add the file to the result set.

Monday, March 26, 2012

Unit Testing with Test Cases in MTM



In Microsoft Test Manager, you can store test cases that contain steps, and parameters for testing. One of the great advantages to using Microsoft Unit Testing Framework, is you can have a test look at the parameters of a test case, and use those parameters to run automated tests. This way all of the critical information is stored between the test case and the automation.

First start by creating a Test Case in Microsoft Test Manager. In the test case outline the simple steps that your automation will be following. Example:



Notice in the above image I have identified each of the steps in my test. I have also identified the parameters I am going to use for my testing, and the validating parameters needed to assert if the test passes or fails. The parameters are preceded with an @ symbol in the steps section, and for each parameter identified in the steps section, there is a correlating column in the parameter section of the test case. In this section I have put in all of the different variations that I want to test for. In this example I am testing a search function which has 3 different filters Age, Gender, Ethnicity, and also a search text box. I am testing each of the different fields by running through a series of tests.

Now that the test case has all of the parameters, and scenarios identified we now create the automation needed to run the scenarios. When creating the automation you will need to have the following.

1. Make sure your test class is using the using the reference Microsoft.VisualStudio.TestTools.UnitTesting.

2. Instantiate an instance of TestContext. Example:
private TestContext m_testContext;     
public TestContext TestContext
{
get { return m_testContext; }
set { m_testContext = value; }
}


2. On or with your [TestMethod] add a DataSource() attribute. Template:

[TestMethod]
[DataSource( "Microsoft.VisualStudio.TestTools.DataSource.TestCase", ";", "", Microsoft.VisualStudio.TestTools.UnitTesting.DataAccessMethod.Sequential)]

Example:

[TestMethod]
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase",
"http://icstfs.blogspot.org:8080/tfs/blogspot;Quality Management","77300",
Microsoft.VisualStudio.TestTools.UnitTesting.DataAccessMethod.Sequential)]


3. Call the parameters from the test case into your method using TestContext.DataRow[].ToString();

For example:

  public void SearchPerson()
{
string gender = TestContext.DataRow["Gender"].ToString();
string age = TestContext.DataRow["Age"].ToString();
string ethnicity = TestContext.DataRow["Ethnicity"].ToString();
string searchText = TestContext.DataRow["SearchText"].ToString();
string validPerson = TestContext.DataRow["ValidPerson"].ToString();
string invalidPerson = TestContext.DataRow["InvalidPerson"].ToString();
string validDescription = TestContext.DataRow["ValidDescription"].ToString();
if(!_meetMormonFilter.IsMeetMormonsVisible())
{
_menu.GoToHome(); //If Meet Mormon wigit isn't visible go to home page }
}
 _meetMormonFilter.FindPeople(gender, age, ethnicity, searchText); //Find a person using the wigit
_peopleSearchResults.ViewByListClick(); //View the search to list to see names and descriptions

Assert.IsTrue(_peopleSearchResults.ListFinderHasPerson(validPerson),"Person: "+validPerson+" Not found in grid"); //Check person name is in grid
Assert.IsFalse(_peopleSearchResults.ListFinderHasPerson(invalidPerson),"Person: "+invalidPerson+" Is found in grid and should not be"); //Check a invalid person is not in grid

string description = _peopleSearchResults.GetProfilDescription(validPerson); //Get Description associated with Valid User
Assert.IsTrue(description.Contains(validDescription), "Description for Person: " + validPerson + " did not contain phrase: "+validDescription);//Check that description contains the text stored in Valid Description
_peopleSearchResults.ListFinderSelectProfile(validPerson); //Go to the User Profile
}

Finally with the automation linked to the test case, visual studio will run the test once for each row of parameters that you have. After the test is done running you will see one test that has failed or passed in the Test Result. However if you double click on the test result, you will be able to see all of the tests ran.