Thursday, May 2, 2013

Our Current GUI Framework – Love and Hate

I have started a series of blog posts on how to write a test framework.  I am excited to share our story (and soon share tools that we developed).  Before getting too far into that, I would like to share some details on a significant influence on how we see things:  the dreaded bad relationship.

I have a passionate love/hate relationship with our current GUI test framework.  We use HP’s Business Process Testing (BPT) framework which uses HP’s QuickTest Pro (QTP) as the execution engine.

It doesn’t take a Googling genius to find complaints about HP tools – I have written some of those complaints.  I also know that many of the complaints are bogus.  I am sure that many are written by people who have never learned how to use the tools well and have never worked with a talented team to implement them  (sometime, I will write a proper defense of QTP) .

From my perspective, HP tools are the best thing out there for GUI testing.  And I can’t wait to move far away from them (Sorry HP)

What makes BPT so great?
  • Great abstraction.  All test frameworks have some level of abstraction. BPT has several. From lowest to highest: low-level function libraries, high-level function libraries, object repository (a great xref between the properties of a GUI object and a logical name), ability to use keywords, functions, and custom scripts), business components to package all the bits into something with business user meaning, and finally the test where components and test data are assembled into tests.  As I look back over the list, I know that it is probably not so meaningful to those not on my team, but I think BPT adds better abstraction than other frameworks I have seen.
  • Super easy to read tests.  If you understand our business, you will understand our tests.
  • Clear separation between automation development and test building. Perhaps not unique to BPT, but it does it well.  Depending on your needs, you can build automation without understanding or caring about the use case, build tests without any automation knowledge, or do it all.
  • Ideal for scaling and growing teams. Several years ago, our team was small and we used regular QTP (scripted tests with shared actions – regular stuff).  It was hard to scale beyond a couple automators – we couldn’t agree on anything.  BPT’s structure and our implementation enabled us to grow from 5 to over 20 and one office to four offices in different countries. BTW, the implementation  was no thanks to HP.  Much of what we do is not standard HP implementation (because there is no standard HP implementation!)
Where is it suckish? (i.e., what are design features for our own frameworks that our experience with HP  have influenced)
  • Sucks for integration with development teams. Between licensing issues and the QC interface, this is not a tool for the full dev team.  No one outside of the test team wants anything to do with the HP tools.  For us, HP tools interfere with our goals of sharing ownership of tests.
  • Execution is slow as hell. We run our full suite every night and it takes every bit of it. As an advocate of BPT, it embarrasses me to no end when someone new watches one of these dinosaur runs. As we move to new tools (and look for ways to enhance our current tools), the new goal is to regression test our apps fully in under 20 minutes (we have to trim about 9 hours from current).  HP tools are not going to help us there.  Our new approaches are and will be fully scalable and capable of cloud busts.
  • Support is a nightmare. There was a period after a new HP release that I was sure I worked for HP as a tester.  The release was a nightmare, every issue I reported was a homework assignment for me, and I was helpless to solve my own problems. No more test tools that I cannot fix myself.
  • “A good used Toyota” – this is what you could buy for the price of a single floating QPT license.The core parts of a good framework (for web apps) are open sourced and there for the taking. Free out of pocket. I know that time is not free, but it is much more satisfying to spend time writing your own tools rather than burning time with a closed system that you can't directly influence.
Love and hate.  Not a perfect relationship, but we learned a lot from it and it’s time to move on.  Our HP experience has and will and will guide us as we move on with services testing and GUI testing frameworks.