View Into the Automation Lab

The fun of having technical skills on a test team is that you can fix problems that bother you. One  bothersome thing that irked us is babysitting automated test execution.  Starting the execution is easy  (the build system does it), but managing the run to completion (rerunning failed tests) eats time.

Here are details about our situation:

• Our automation environment has a limited number of test machines (11-ish to be exact), these machines are locked away in a room because the desktop cannot be locked to run the automated tests, and they are shared by teams across three continents (Europe, North and South America).  
• Today we have a team of testers that monitor the automated regression run and make sure that all of the tests have run.  When tests are identified for rerun this team will then take a group of tests and manually kick them off.
• Once the tests are running, someone on the team then monitors this set to make sure the tests complete and then kicks off another set of reruns as they are identified.
• If anyone from the other offices wants to kick off a test on those machines, he has to coordinate it with the regression team.  There is no good visibility.
• The end result here is a lot of time spent watching tests run and potentially creating large gaps in machine usage.

Cool Tech to Solve Problems

“Why not build a web based application that can be accessed by anybody in the company needing to know a current status of their tests?”  Right away the ideas started to fly around and the solution, while all relatively easy to accomplish, all of a sudden became further reaching than just automatically rerunning tests.  It would also be nice to have some sort of a dashboard that shows the status of the test machines.  Oh, and how about the ability to change which machine a test is running on.  While we’re at it, since certain projects may have a higher priority let’s add the ability to reorder tests in the queue so those can run first....the ideas were now flowing.

The first step of this solution was to build a function to monitor the test machines.  The rerun app will need to know this.  It would also be cool for people to know this too, so we built a web based dashboard that shows the current status of each machine.  It’s very simple, but provides a huge amount of information that is helpful when evaluating test runs.

This part is taken care of through Ruby and some SQL queries against the QualityCenter database.  Here's how QualityCenter database tells if a machine is available or not:

1: SELECT RN_HOST
2:   FROM td.RUN
3:  WHERE RN_STATUS <> 'Not Completed' 
4:    AND RN_HOST IN ('<your machine name here>') 
5:    AND RN_RUN_ID IN
6:       (SELECT MAX(RN_RUN_ID)
7:          FROM td.RUN
8:         GROUP BY RN_HOST)

With the dashboard in place, the lion’s share of the work is left up to backend code. The rest is display, database updates and other code-type stuff to give it some appeal and not look like a site that a bunch of QA folks came up with. Afterall, we are a technical test team!