Monday, October 14, 2013

Making Use of Your Fail Pics' Newfound Freedom

The benefit of letting your fail pics go is that it provides special flexibility in observing and analyzing test failures as they occur. As mentioned in the "extra bonus benefit" section of our last post, we have been having some trouble with our test execution lately. Between an unsettling number of illegitimate test failures and mysterious blank snapshots, we have found ourselves in need of a tool to help troubleshoot our test execution.

Real Time? How About Node.js...

We wanted a tool that would allow us to monitor snapshots as they occur in real time by listening to our snapshots directory and pushing newly added photos to a web client. At first, Node.js seemed like the perfect tool for the job. In fact, with a combination of Node.js, Express, Socket.IO and a directory watcher module, we had a simple implementation up and running on my local machine in little time. Unfortunately, we soon discovered that, while it worked well for watching directories on my local machine, the Node.js event based file watcher does not work with network drives. As a result, we turned back to our good friend Ruby.

Back to Ruby...

Not only would we be able to watch the network drive with Ruby, it has the added benefit of being the lingua franca of our testing team. This means that all members of the testing team can feel more comfortable reading and contributing to the project's code base in the future. The application was built using Ruby, Sinatra, and a directory watcher gem, making the server side code simple enough to place in a single server.rb file:

On the client side, we long poll the server for new events, passing in the time stamp of our latest result. As you can see, the server then hands over all events with a time stamp later than the one we passed in. The client takes that response and adds each event to the page. Now, we are able to visit the "Quality Center Snapshot Feed" web page and watch the fail pics roll in.

Future Plans

The current implementation works, but leaves plenty of room for refactoring and improvement. In the future, we plan to utilize web sockets via Event Machine, rather than long polling. In addition, we would like the application to associate each snapshot with the test machine it occurred on and which team owns the test.

Wednesday, October 2, 2013

Let My Fail Pics Go

In Free Your QTP TestResults, we shared how we create a shadow set of test results that we can write to own database and can use however we want. We are freed of the shackles of the QTP test results viewer.  We didn’t stop there.  We also freed the snapshots that we take when QTP crashes or hits some wall of pain.

Before using the shadow data strategy, we had QTP take a picture that we could see in the result log.  Did something like this:

setting("SnapshotReportMode") = 0 'take a picture
Some test object.Exist(0)
setting("SnapshotReportMode") = 1 ‘turn off the pic taking

As I said, this did the job, but it also trapped the results in the result viewer.  We did some investigation on how to access the snapshots from Quality Center. There may be an easy way, but we didn’t find it.  What we found would have required us to uncompress the image file before using it and it is hard to find anyway.

To set pics of our failures free, we implemented ShadowSnaphot:

Sub ShadowSnapshot()
    On Error Resume Next
    cycle_id = QCUtil.CurrentTestSet.ID
    run_id = QCUtil.CurrentRun.ID
    strUrl = "\\somefileserver\Testing\QC\snapshots\" & CreateTimeStamp & "_" & run_id & ".png"
    Desktop.CaptureBitmap strUrl 'Write the image to file server
    'publish the url to the result viewer
    Reporter.ReportEvent micDone, "View Desktop Snaphot""See the desktop here: " & strUrl
    'Write to our database 
    strSQL = "INSERT INTO cfc.test_snapshots (sn_cycle_id, sn_run_id, sn_snapshot_url, sn_type) " &_
              "VALUES (" & cycle_id &"," & run_id & ",'" & strUrl & "', 0)"
    objRecordSet = GetRecordSet(strSQL, "QC")
    On Error GoTo 0
End Sub

We save the snapshot to a file server and we write the information we need to find our snapshot to a database table. We use ShadowSnapshot method before we exit out of a test and when the recovery scenario is kicked off.

This has been working out great.  Our results are saved as .png files which is lightweight and easy to use.  For example, we can easily view our images in web apps that we build to make our lives easier.  Personal happiness with Shadow snapshots.

Extra bonus benefit:  We are troubleshooting some problems with our test execution.  Since we are using ShahowSnapshot on most test fails, and the method adds a pic file to our file server, we can write a utility to troll for new pics in the directory and display them in real time.  Ruby project. Will share more with this soon.