Tables in your Gherkin Scenarios

When writing gherkin, we describe the expected behavior in business terms.  Sometimes, this is text; other times, mixing small tables or lists helps to describe the behavior better.  In the following feature file, there are examples of three types of gherkin tables:

Each of these tables has different implementation steps. Each is simple to use, but it is easy to use the wrong method by mistake (this happened to me)

Simple List of Items

In the first example, Check for pets, there is a simple list of items.  In the Ruby/Cucumber implementation, you use the ‘raw’ method to handle the list of items.  Use raw when you do not have a column header.

Table with Headers

In the second scenario, Check for valuables, there is a table of items with a header. In this example, use the hashes method. Hashes returns an array of hashes.  With this collection of hashes, you can then retrieve your values normally.

List of Items with Side Header

In the final scenario, Check denominations of money in safe, there is an example of named list of items with the column on the left side. With this type of table, you use the rows_hash method.  This returns your data as a hash. In this example, you can see label (which is a Ruby symbol) follows the format of :dollar_bills_[denomination].  This is because Ruby symbols cannot begin with numbers. This impacts the readability a little but not too much.

Scenario outlines with examples are a way to data drive scenarios using tables.  I didn’t include an example of them.  They are generally well understood and there are more examples of them than there are of the other tables.

Making Use of Your Fail Pics' Newfound Freedom

The benefit of letting your fail pics go is that it provides special flexibility in observing and analyzing test failures as they occur. As mentioned in the "extra bonus benefit" section of our last post, we have been having some trouble with our test execution lately. Between an unsettling number of illegitimate test failures and mysterious blank snapshots, we have found ourselves in need of a tool to help troubleshoot our test execution.

Real Time? How About Node.js...

We wanted a tool that would allow us to monitor snapshots as they occur in real time by listening to our snapshots directory and pushing newly added photos to a web client. At first, Node.js seemed like the perfect tool for the job. In fact, with a combination of Node.js, Express, Socket.IO and a directory watcher module, we had a simple implementation up and running on my local machine in little time. Unfortunately, we soon discovered that, while it worked well for watching directories on my local machine, the Node.js event based file watcher does not work with network drives. As a result, we turned back to our good friend Ruby.

Back to Ruby...

Not only would we be able to watch the network drive with Ruby, it has the added benefit of being the lingua franca of our testing team. This means that all members of the testing team can feel more comfortable reading and contributing to the project's code base in the future. The application was built using Ruby, Sinatra, and a directory watcher gem, making the server side code simple enough to place in a single server.rb file:

On the client side, we long poll the server for new events, passing in the time stamp of our latest result. As you can see, the server then hands over all events with a time stamp later than the one we passed in. The client takes that response and adds each event to the page. Now, we are able to visit the "Quality Center Snapshot Feed" web page and watch the fail pics roll in.

Future Plans

The current implementation works, but leaves plenty of room for refactoring and improvement. In the future, we plan to utilize web sockets via Event Machine, rather than long polling. In addition, we would like the application to associate each snapshot with the test machine it occurred on and which team owns the test.

Free Your QTP Test Results!

While building our web site to speed our analysis of our daily regression tests, we needed to access result data quickly.  Our failed error messages and snapshots of crashes were trapped inside QTP result logs.  As anyone who has looked into these logs knows, they were not designed for casual deciphering.

Rather than fight the HP logs, file structures, compression methods, and database, we went around it.  We started collecting our own copy of key information -- free and easy for us to use.  We are able to build this site with crash snapshots and error messages:

We developed shadow methods for collecting information.  When a test step fails or when something crashes, QTP writes information to its log, and we write a copy of the same information to our database or file server.

To support this, we created a database table that stores error messages and is keyed off cycle and run ids and wrote a ShadowFailMessage method for our QTP function library:

To make this work, we added a reference to the ShadowFailMessage subroutine after each fail (it would have been more elegant to have opened the Reporter class and added the additional functionality, but this is VBScript, not Ruby L).  This takes the error message content and writes it to a database table along with other information needed to associate the message with the right test.  The pattern looks like this:

Reporter.ReportEvent micFail, title_of_error, “Expected: “ & expected_value & “ – Actual: “ & actual_value

ShadowFailMessage title_of_error, actual_value, expected_value

Here is the code for the subroutine we referenced:

Sub ShadowFailMessage(nameOfDataValue, actual, expected)
    actual = Replace(actual,"'","''")'escape single quotes that will break the SQL query
    expected = Replace(expected,"'","''")
    On Error Resume Next
    cycle_id = QCUtil.CurrentTestSet.ID
    run_id = QCUtil.CurrentRun.ID
    If expected ="" Then
        fail_message =     Left((nameOfDataValue & ": " & "Error message: " & actual), 500)
    Else
        fail_message =     Left((nameOfDataValue & ": " & "Expected: " & expected & "; " & "Actual: " & actual), 500)
    End If
    'Write to the cfc_SNAPSHOTS table. set sn_type to 1 to indicate error message
    strSQL = "INSERT INTO test_snapshots (sn_cycle_id, sn_run_id, sn_snapshot_url, sn_type) " &_
              "VALUES (" & cycle_id &"," & run_id & ",'" & fail_message & "', 1)"
    objRecordSet = GetRecordSet(strSQL, "QC")
    On Error GoTo 0
End Sub

We also have Shadow subroutines for crash snapshots and actual/expected QTP datatables (for verifying reports and tables).

We no longer complain about the limitations of our existing tools – we own the solutions to our own problems.  In our case, we used open-course tools like Ruby and Sinatra libraries to develop our own applications for using our test results.  I have even started to like HP's problems.  They have become excuses for some fun coding -- too bad we are paying for the problems though.

Importance of a Common Programming Language on Test Teams

How can a technical team work together on techy stuff if they don't understand the same language?  On our team, the native languages are English, Polish, Spanish, and possibly Jive (<-- funny Airplane reference).  Fortunately, everyone speaks English (even though the native English speakers  don't speak Polish, Spanish, or Jive -- other than enough Polish to get me slapped).  We are able to work together in English on business facing testing issues.  To work as a technical test team, we need a technical lingua franca.

A lingua franca is a common language shared by people from different backgrounds so they can understand each other and they can work together.  At home, different languages are used; at work (or at market or international diplomacy), an agreed upon, common language is used.  The common language is no better than the others, but it enables people to communicate better.

Programming teams are typically formed with a built-in lingua franca.   If the team is developing web apps in Java, then all the programmers hopefully share a working knowledge of Java.   At home or on side projects, these same developers may switch a language that is more comfortable for them, such as Python, F#, or Zimbu.  Even though these programmers may speak Zimbu at home, they can still communicate with other programmers at work in Java.

Test teams don't often have the same advantage.  Testers often range from non-technical to super technical.  And because there is no industry standard for testers (not advocating this), it is more challenging for test teams to work together on technical solutions.  For example, members of our test team  have experience with VB (VBA/VB Script), Ruby, Python, Java, C#, and a few other languages.

We are literate, but if we are not literate in the same language, what good is it?  We can build things for ourselves, but we cannot leverage ourselves and our influence.  We cannot grow greater than one.   If we each build tools and testing applications in VB, Ruby, Python, Java, and C#, we cannot take advantage of each other's work, understand each other's work, develop significant tools, or maintain that work.  With a common programming language, we can become more powerful team.

There are a few considerations in picking a common language. It might make sense to form around the common development language in your shop (C#, in our case), go with the most common language on your team (probably VB) , or select from open-source languages commonly used in the test space, such as Ruby or Python.  Anyway you go, pick a language, and go with it.

We selected Ruby as our lingua franca.  When we started, none of us had any experience with it.  Part of our shared experience was learning to use it.  We worked together before and after work and at lunch to learn the basics of the syntax and understand the personality of the language.

As a team, we work together, challenging and motivating each other.  This common language is helping to pull us together as a team.  We work together on skill-building challenges and building applications to help us with our jobs.  Simple tools are becoming more ambitious as we grow skills and gain confidence.

A side benefit of our experience with Ruby as our shared language is that it helped many of us on the test team better understand technology discussions related to our application development.  For instance, when our .net development team talks about MVC frameworks, ASP files, and reconciling NuGet references, we can translate to our language to understand:  Rails type framework, .erb or .haml files, and RubyGems.  We can understand others better, and we now have a vocabulary talking about these things.

How to Write a Testing Framework

Sometimes, test frameworks seem to be a dime a dozen.  There are many commercial and homegrown frameworks.  All have the same general goal – make automated testing easier.  This is an awesome goal.  When done well, it enables testers to focus on high-value work and lets the framework take care of low-value stuff (important but low value for people to do).

Over the course of the several posts, I will share the story of a services testing framework that several members of my test team and I wrote (with some technical and design help from friends and well wishers).  We started with a few goals and principles and we are driving toward an open-source release of the tool.

To give you a sneak peak, here is the current start page of Cham (there is more than a start page -- we have  a working system with high levels of abstraction and fast, scalable performance). 

Start page of Cham

From this view, you can see some of the things that are important to us when we think about frameworks.

• Separate the technical stuff from the tests.  This is the difference between the templates section and the build tests section.  Hide technical stuff where you can and build awesome tests using business words (also known as people words).  Test built using people words are easier to share, to discuss, and to maintain.
• Focus on making everything understandable by a wide audience.
• Results are easy to share with people who care.  In Cham, we do this a couple ways.  One is by simply making it easy to pass specific results to people (our pages are RESTful and you can simply give someone a URL). Another way (not seen here) is by making the tests work with the developer workflow.  For example, programmers can use Cham to test services before code is checked in.
• Make it easily accessible.  You can’t see this here, but everything is done with open-source code (go Ruby) and libraries so there are no licensing issues limiting access.
• Make it look nice – this gives users more confidence in the tool.  And it makes you feel better about yourself.  We can all use that.