Free Your QTP Test Results!

While building our web site to speed our analysis of our daily regression tests, we needed to access result data quickly.  Our failed error messages and snapshots of crashes were trapped inside QTP result logs.  As anyone who has looked into these logs knows, they were not designed for casual deciphering.

Rather than fight the HP logs, file structures, compression methods, and database, we went around it.  We started collecting our own copy of key information -- free and easy for us to use.  We are able to build this site with crash snapshots and error messages:

We developed shadow methods for collecting information.  When a test step fails or when something crashes, QTP writes information to its log, and we write a copy of the same information to our database or file server.

To support this, we created a database table that stores error messages and is keyed off cycle and run ids and wrote a ShadowFailMessage method for our QTP function library:

To make this work, we added a reference to the ShadowFailMessage subroutine after each fail (it would have been more elegant to have opened the Reporter class and added the additional functionality, but this is VBScript, not Ruby L).  This takes the error message content and writes it to a database table along with other information needed to associate the message with the right test.  The pattern looks like this:

Reporter.ReportEvent micFail, title_of_error, “Expected: “ & expected_value & “ – Actual: “ & actual_value

ShadowFailMessage title_of_error, actual_value, expected_value

Here is the code for the subroutine we referenced:

Sub ShadowFailMessage(nameOfDataValue, actual, expected)
    actual = Replace(actual,"'","''")'escape single quotes that will break the SQL query
    expected = Replace(expected,"'","''")
    On Error Resume Next
    cycle_id = QCUtil.CurrentTestSet.ID
    run_id = QCUtil.CurrentRun.ID
    If expected ="" Then
        fail_message =     Left((nameOfDataValue & ": " & "Error message: " & actual), 500)
    Else
        fail_message =     Left((nameOfDataValue & ": " & "Expected: " & expected & "; " & "Actual: " & actual), 500)
    End If
    'Write to the cfc_SNAPSHOTS table. set sn_type to 1 to indicate error message
    strSQL = "INSERT INTO test_snapshots (sn_cycle_id, sn_run_id, sn_snapshot_url, sn_type) " &_
              "VALUES (" & cycle_id &"," & run_id & ",'" & fail_message & "', 1)"
    objRecordSet = GetRecordSet(strSQL, "QC")
    On Error GoTo 0
End Sub

We also have Shadow subroutines for crash snapshots and actual/expected QTP datatables (for verifying reports and tables).

We no longer complain about the limitations of our existing tools – we own the solutions to our own problems.  In our case, we used open-course tools like Ruby and Sinatra libraries to develop our own applications for using our test results.  I have even started to like HP's problems.  They have become excuses for some fun coding -- too bad we are paying for the problems though.

What Have We Been Up To?

It has been a while since we updated this blog.  Fortunately for us, the reason we have not been writing is not because we are out of ideas, but because we have been implementing new ideas!

One problem that plagued us is how long it took us to analyze our BPT/QTP test results.  HP’s QC does some things well, but it is not exactly nimble. 

·         You have to use Internet Explorer (I die a little inside when I use IE, and there is no way that other developers will use IE),
·         Updating QC results in QC is sooo slow  (in this case, I do not die a little – instead, I want to kill a little.  Why does it take so long to update things in QC and why does it hang my browser while it makes updates?)
·         It takes too long to find the important information in the QTP result viewer
·         Licensing requirements to view results through QC

Anyway, it is clear that we cannot dramatically increase the speed of our daily work by using the QC tools as they come from HP.

We developed our own tool for analyzing BPT/QTP results.  In this view, we can see almost everything we need to understand why a test failed – at one quick glance, maybe even from my iPhone – and we can complete our analysis (without any of the HP “slows”).

There are a couple cool stories/technical solutions related to this tool:

·         Super-fast updating of QC results – even setting tests to rerun
·         Creation of shadow results – pictures and fail messages
·         Super support of analysis of tabular results
·         Prioritization of results
·         Browser plug in for QTP results viewer
·         How good/reliable is this test
·         Easily sharing test results with others on the development team

Expect detailed postings on each of these topics (for real, this time).