Thursday, January 17, 2013

View Into the Automation Lab

The fun of having technical skills on a test team is that you can fix problems that bother you. One  bothersome thing that irked us is babysitting automated test execution.  Starting the execution is easy  (the build system does it), but managing the run to completion (rerunning failed tests) eats time.

Here are details about our situation:
  • Our automation environment has a limited number of test machines (11-ish to be exact), these machines are locked away in a room because the desktop cannot be locked to run the automated tests, and they are shared by teams across three continents (Europe, North and South America).  
  • Today we have a team of testers that monitor the automated regression run and make sure that all of the tests have run.  When tests are identified for rerun this team will then take a group of tests and manually kick them off.
  • Once the tests are running, someone on the team then monitors this set to make sure the tests complete and then kicks off another set of reruns as they are identified.
  • If anyone from the other offices wants to kick off a test on those machines, he has to coordinate it with the regression team.  There is no good visibility.
  • The end result here is a lot of time spent watching tests run and potentially creating large gaps in machine usage.

Cool Tech to Solve Problems

“Why not build a web based application that can be accessed by anybody in the company needing to know a current status of their tests?”  Right away the ideas started to fly around and the solution, while all relatively easy to accomplish, all of a sudden became further reaching than just automatically rerunning tests.  It would also be nice to have some sort of a dashboard that shows the status of the test machines.  Oh, and how about the ability to change which machine a test is running on.  While we’re at it, since certain projects may have a higher priority let’s add the ability to reorder tests in the queue so those can run first....the ideas were now flowing.

The first step of this solution was to build a function to monitor the test machines.  The rerun app will need to know this.  It would also be cool for people to know this too, so we built a web based dashboard that shows the current status of each machine.  It’s very simple, but provides a huge amount of information that is helpful when evaluating test runs.

This part is taken care of through Ruby and some SQL queries against the QualityCenter database.  Here's how QualityCenter database tells if a machine is available or not:

2:   FROM td.RUN
3:  WHERE RN_STATUS <> 'Not Completed' 
4:    AND RN_HOST IN ('<your machine name here>') 
7:          FROM td.RUN
8:         GROUP BY RN_HOST)

With the dashboard in place, the lion’s share of the work is left up to backend code. The rest is display, database updates and other code-type stuff to give it some appeal and not look like a site that a bunch of QA folks came up with. Afterall, we are a technical test team!

1 comment:

  1. Nice blog.
    However, we went about such problem in entirely different manner.
    Instead of writing tools that "query machines" we have one centralized service that:
    [1] assigns machines for particular tests on request from test framework and soley on required capabilities advertised by both machine and test itself and maintains this state; test's are no longer bound to specific test machine, just to capability required to run. Added value is that it helps to weed out tests that are portable only with machine that was used for their development
    [2] this state information is both logged in db and displayed 1. live on selected page on company wiki 2. will be displayed, basing on the information from the service, in tiny tool on the system tray.

    The added advantage is that nobody is actually concerned about where exactly the test is launched - it's all taken care by our test framework along with this "test machine dispatcher service". There's no possiblity of conflict and using one machine for more than one test at the same time - this is ensured by test framework itself, just like configuring machine for particular tests.

    Results are automatically fed back to test management system and artifact repository. Also, machine allocation for particular test plan execution takes place in various modes, depending on test plan type and scenario processing mode (each scenario may be launched on defined list of test environments or browsers, data sets or combination of thereof).
    Basically - develop test, configure test plan, click play, wait until done.