Test Management is Wrong

Test Management is wrong. There. I said it.

I can't believe it took me this long to notice the obvious. If you are doing Software Development in any fashion, and are worried about how to manage your testing to develop a "quality" product, stop it.

Let's be clear about what I mean here. If you consider any software development life cycle (SDLC) process, you will find activities like the following arranged in some fashion or other:

• Requirements gathering, specification
• Software design
• Implementation and Integration
• Testing (or Validation)
• Deployment
• Lather, Rinse, Repeat (i.e. Maintain, Enhance, Fix, Mangle, Spin, and so on)

These activities aren't Waterfall or Agile or anything else, they are just activities. HOW you choose to do them will reflect your SDLC. I don't care about that right now. The part I'm picking on is the Testing bit near the middle, regardless of whether you do them in an agile, Waterfall, or some other way.

In particular, I am picking on the fallacy or myth that a good Test Management plan/process is what you need to develop and release a high Quality product.

In almost 25 years of working in the Software/IT industry, I have never worked at a company or heard of any company that had a Test Management process so solid that it lead to high quality, I don't believe there ever will be either.

Here's an analogy. Let's say I want to make some Rice Krispies treats. The ingredients are: butter, marshmallows and rice krispies. You heat 'em up, mix 'em up, flatten 'em out, cool, slice and eat. That's it.

Ya, but you know what? I have this really excellent Marshmallow Plan that will produce the Bestest treats anyone has ever tasted.

How does that sound to you? Interested to learn more about the plan? Be truthful.

If you said "Yes", I don't know if I can help you. If you said "No" there may be hope for you yet.

Why is it that Software companies are attracted to the idea that Testing is somehow equated with Quality? Testing is an activity that you may choose to do or not do and still have a great quality product. The Agile Manifesto doesn't mention testing anywhere and yet the Manifesto's signatories, thought leaders, and practitioners produce great quality stuff for people. How does that work?

Why is it only the Testing phase/activities/part that equates with Quality? Why aren't companies and managers everywhere promoting super awesome Requirements Management processes and plans at Requirements Conferences to help deliver high Quality? What about Design Management? No, no, wait, I got it. We need Deployment Management. That's it! I've solved it. Aha!

No, these are all stupid suggestions. And you know it too.

Is it because Testing/Verification activities are part of the other 'phases' or steps in software development, so surely we should be able to manage all that, right? Well, actually, no. That's the wrong approach.

Test Management systems manage and measure some of the testing activities performed on a project. Completing your testing activities tells me nothing about the overall "Quality" of the product. By definition, it is an incomplete part of the picture.

By putting your faith in Test Management plans or systems you are effectively saying "I don't know how to measure what you want (i.e. Quality), so I will measure what I can do (i.e. Test)."

Does this mean it's pointless to track the testing you've done? NO! I am not saying that. If you do something that you believe contributes to the value of the project, then please track what you are doing so that others can see what you have done. Preferably, make your progress visible in some way.

What we need to focus on is what the customer needs. Ask yourself: What problem are they trying to solve? What are we trying to deliver that is of value to them?

What we need is Value Management. There are successful (i.e. "quality") products built and released to customers that never had any independent testers on the development team. Cool. These teams get it.

I have a friend and colleague who started up a company and within hours had sold a product that didn't even have a single line of code written. Way cool! He gets it. Did he worry about Test Management? Bah! He didn't even need to pay a programmer to sell something of value to a customer.

I have mentioned it before, if you are doing any kind of testing on a software development project, you can likely place it somewhere within the Agile Testing Quadrants. Heck, you don't even have to be on an Agile team or project to see that your testing fits somewhere on the chart.

So, if you somehow create a super Test Management Plan or system that tracks all of these activities for a given project, then would you have a Quality product? That would be really cool, by the way, but no, not necessarily. And don't fool yourself into thinking that it will.

The problem is one of relationships. Human relationships.

The definition of Quality I like to use comes from Jerry Weinberg and it is based on your understanding of relationships between people. The definition of "software" that I use when teaching/training/coaching is that software reflects your understanding of the relationships between people and the systems they interact with to meet particular needs. The definition of Testing that I like to work with is the intelligent use of models and heuristics to explore the relationships between the people designing a system and those who intend to use it.

In short, if you want to understand what provides "quality" or value to your customers, look to how you are managing the human relationships. Forget Test Management. You might have better luck implementing a CRM system in your development departments as a predictor of Quality.

Testers: if you want to do a great job, focus on the testing activities that explore the systems from the perspectives of the people who matter. Never hide or bury your work (i.e. in documents, spreadsheets, test management systems, etc.). Make it visible - use dashboards, mind maps or other visual mediums because they assist with team collaboration and understanding. Keep detailed records archived somewhere if you need them, but don't worry about "managing" to those fiddly bits as an indicator of "quality" - because they're not.

Forget about the Test Plans though. Disregard Test Management systems. AVOID providing any metrics on testing coverage as an indicator of Quality... especially if your development team isn't producing equally misleading metrics about code complexity, requirements reviews, and other esoteric development activities.

Anyone (ALM Vendors take note) who sells a "Quality Management" system based upon managing test cases is lying to themselves and to you. They are doing it wrong. Don't be a victim.

You can't kludge a development process enough to make a Quality or Test Management system produce meaningful, valid indicators of value to your customers and stakeholders. It just doesn't make sense.

Approach your development teams from a human relationship perspective. Focus on how people collaborate and work together, and high quality products will emerge as a by-product. Manage the relationships with your customers, and deliver working software frequently to help them see what you can do and how you learn from past experiences. This isn't easy. It's definitely worth it though.

So. You want to deliver Quality? Test Management isn't the answer. Test Management tells you about a specific task's management. It's about as useful as Marshmallow Management in making Rice Krispies treats. That is, it may have a place in the big picture, but it's certainly not the right way to look at the problem.

Read More

Shuhari

When I work with teams to help them learn something new, I try to pay attention to a few things. Firstly, I pay attention to how people are learning, and secondly how I am teaching.

When I used to teach Physics and Chemistry in high school, one validation of 'success' often came from how the students left the classroom. Generally, teenagers often came into one of those classes the same way (at least at the start of the year): I don't want to be here, this isn't important to me, I'm not going to learn anything useful.

Okay. Gauntlet down. Let's begin.

I knew I had a good class when students left the room smiling and/or talking about the ideas covered in the lesson. The real learning happens when they talk about the subjects among themselves. We need time to absorb ideas and make them a part of us. Talking about them with others is a good first step in the learning process.

Putting ideas into practice is an important next step in the learning process. Practice makes the knowledge concrete, more permanent. Through practice we also begin to understand the limits of success under different conditions. In a classroom, practice might happen through assigned questions/exercises, experiments or projects of some sort.

Once someone groks an idea through practice, we can engage in the next level of discussion -- what next? Applicability, adaptation, extending the ideas, and so on. How can we be more successful? To paraphrase Newton, how can we stand on the shoulders of giants?

There are many learning models and ideas that apply for both learning and teaching material, and I don't mean to fill this space with them.

In recent years, my teaching (coaching, consulting) has focussed on Agile and Testing (rather than Physics and Chemistry). When I am approached for help or advice on Agile, Scrum, Exploratory Testing, or something else, I often think about the term Shuhari. (In English, pronounced: shoe - ha - ree)

Shuhari is a Japanese term from martial arts that describes the learning path to mastery. It roughly translates to "first learn, then detach, and finally transcend." From the Wikipedia page, here's the breakdown:

• shu -- "protect", "obey" - traditional wisdom - learning fundamentals, techniques, heuristics, proverbs
• ha -- "detach", "digress" - breaking with tradition - detachment from the illusions of self
• ri -- "leave", "separate" - transcendence - all moves are natural, becoming one with spirit alone without clinging to forms

Shuhari reminds me that rules and rituals are in place for beginners and that we learn to go beyond them as we mature in a particular discipline.

When I teach people about Scrum or Exploratory Testing, I often see people want to start improvising or adjusting practices right from the beginning. When you do that, you jump to "ha" but without the solid foundation or appreciation of "shu". In a martial arts class, the sensei (instructor) might smack you on the head for doing something like that. (If you're lucky.)

As we explore new ways of doing things, it's important to start at the beginning and practice the forms as described. Become comfortable with the practices. Become bored with them. Make them a part of your muscle memory so that you don't have to consciously think about them anymore. Keep practising.

*Then* one day you may ask "how about if we change this [step] a little? What do you think?" That is an excellent question. A question that drives an experiment. An experiment that drives learning and helps us to enter "ha".

Timing is the difference. Asking to vary a practice at the beginning doesn't help. Asking after you understand it makes sense.

There are different concepts and models to describe the paths to mastery. What do you think of this one?

I have three other models floating around in my head on Mastery and plan to cover them in future posts. I offered to talk about "Mastery" at the Test Coach Camp last year but there wasn't enough interest at the time. I wonder who the target audience is for this topic. People sometimes fall into careers. I actively sought mine, so these models meant something to me at the time. I reflect upon the value that each one offered and look to new insights still waiting for me.

When I think of Shuhari, I think "Practice before Change." And that reminds me of the old joke: How do you get to Carnegie Hall?

Read More

The Human Side of Living

As I go through life I keep noticing stories, ideas and insights into humanity and I sometimes wonder if we are meant to discover these lessons slowly or if there isn't a quicker way to learn them.

Take for example, in high school we had a really weird Religion teacher who was very Zen or meta or something, and no one got him. I mean he would use examples like "take an extension cord and plug it into itself and there you go." Huh? None of us got it. And then there would be times when he would repeatedly say things like "attack the point not the person" and that was a phrase I understood.

From him, I learned that sometimes we can meet real jerks that we can learn interesting things from. Learn to separate your feelings about what you hear and understand from the messenger. It's hard sometimes, but you can get good at this.


Later, I read a story about a samurai warrior. It's short, so here it is:

There was once a samurai who wanted to learn the difference between heaven and hell. He sought until he found a master from whom he thought he could learn. He stood before the Master and asked him what was the difference between heaven and hell. The Master took the samurai’s sword and, turning it to the flat of the blade, struck the samurai on the head. The samurai was surprised at this but chose to ignore it. He thought that the Master had failed to understand his question. He once again asked the Master about the difference between heaven and hell. Again the Master struck the samurai on the head. The samurai staggered back and puzzled over this. He approached with his question for a third time and, before he could utter a word, the Master struck him a third time. The samurai was now so enraged at this behaviour that he grabbed his sword from the Master, raised it over his head and was prepared to bring it down on the Master’s head when the Master raised one finger and the samurai paused.
"That is hell," said the Master.
The samurai was instantly so overcome by the courage of this frail old man - to have risked his life for the sake of a stranger’s question - that he fell to his knees and bowed before the Master.
"That is heaven," said the Master.

This story keeps popping into my head every now and then. What is good and evil? Is it a matter of perspective? Is it a matter of time? How is it related to compassion? Do we need to judge people/situations, or should we learn to see the good and bad in all things? We can choose what we want to make from a situation. We don't always understand the motives of others, so which stance do you initially take - heaven or hell?

On the compassion thread, I learned about HH the (14th) Dalai Lama (of Tibet) when I was in university. It wasn't part of a course, I don't remember what it was. It might have been a movie. He's a really interesting guy and has done some cool things. I pondered his thoughts on compassion and felt that he really has good insights into the human condition so some of those ideas stuck with me. (HHDL is on Twitter by the way.)

When I left school and started working full time, I discovered Jerry Weinberg. Jerry published many technical books up to that point, and started the Amplifying Your Effectiveness (AYE) conference and Problem-Solving Leadership (PSL) workshop. The workshop and conference are based upon applying the work of Virginia Satir, a family therapist, to the workplace. I find some of the models very insightful.

Skipping over many other little opportunities and lessons, I find myself thinking about a recently-published book called "The Human Side of Agile" by a colleague Gil Broza. It's a good book. I like it. It sums up a lot of lessons I learned over the years, and includes new ones I didn't know about. The title really sticks with me though.

When I am at work, I focus on doing things to help others. Help the customers get high quality software of value. Help the team members to learn, grow and become more confident in their abilities. I show patience and temper difficult situations with humour. That's my style. When the going get's tough, I get silly. Sometimes, though, I hear Jerry W's words ringing in my ears "Change your organisation or change your organisation."

I am an agent of change. I am here to help you establish a new norm, a new status quo, one that is better than you were before. I work with people to help them adapt into their new roles, and I often come across people who neither want my help nor anything to do with change.

I can understand when people are afraid of uncertainty or the unknown and I am patient enough to work with them to try and build congruence (Satir) and focus on the point not the person (high school teacher). Then there are times when certain people can very intentionally do malicious things to undermine and attack you through a show of power or superiority. I'm too old for this crap.

From Jerry, I know it is time to change my organisation when this becomes a pattern, because it is my life and I choose how I want to live and enjoy it. I don't want to be miserable at work and then bring that negativity home with me to my family that I love so much.

It's my life. I want to be happy and helping others makes me happy. I'm weird that way. I understand that not everyone gets that. I'm not here to inflict compassion and other zen mumbo-jumbo on you. I really like and appreciate the Lean and Agile values. The focus is on *people* working together to make great things that make your customers happy.

After 25 years of working in the IT sector, I can tell you that I agree with Jerry when he said that "all problems are people problems." (especially the technical ones.) When I truly came to understand that, I discovered that people are at the heart of the answer to "what is Quality?" After almost 20 years in Testing, I also discovered that test techniques are really models to test the interactions between people working on the projects. This is a bit of unique perspective and I haven't heard anyone else describe it that way, but that's how I see it and teach it.

People working with people to make other people happy. Lots of other people actually. There's nothing non-human about software development. It's all about the human side of things. And yet. Schools don't teach this. Some people choose to act in inhuman ways. How do you deal with that? Heaven or Hell?

It's your life, your choice. Change your organisation or change your organisation.

Thank you to all my teachers, past, present and future. There is still more for me to learn.

Read More

Testing is a Medium

In a few days I will be giving a presentation to the local Agile/Lean Peer 2 Peer group here in town. The group has a web site - Waterloo Agile Lean, and the announcement is also on the Communitech events page.

I noticed the posted talk descriptions are shorter than what I wrote.  The Waterloo Agile Lean page has this description:

"This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it’s done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

The Communitech page has this description:

"Exploratory Testing is the explosive sound check that helps us see things from many directions all at once. It takes skill and practice to do well. The reward is a higher-quality, lower-risk solution that brings teams a richer understanding of the development project.
This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it's done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

This is what I submitted:

"Testing is the medium in which solutions are developed. The value of our delivered solutions depend upon how well we understand and utilize that medium. We can fly straight like an arrow or explode outwards like a spherical sound wave.
Traditional automated TDD "checks" help us fly straight in the direction we choose. Is it the right direction though? How do we know? Are you sure?
Exploratory Testing is the explosive sound check that helps us see things from many directions all at once. It takes skill and practice to do well. The reward is a higher-quality, lower-risk solution that brings teams a richer understanding of the development project.
This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it's done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

This blog post is *not* about the differences in session descriptions. (In fairness, I really should learn to keep it to one paragraph. I hope to get better at writing session descriptions - it'll come with practice.) Reading the last one first, I can't help think that a bit of context is missing from the two posted descriptions for the final "blown away" statement. That is, that phrase comes from the sound wave analogy and not from some arrogant expectations I have for my presentation abilities. If I had known the descriptions would be shortened, I would have at least changed that sentence.

This blog post is about the idea that I try to convey in the first sentence that is unfortunately missing from both posted session descriptions: "Testing is the medium in which solutions are developed."

Software Development is a creative process. It is the intersection of people, skills, tools and experimentation to solve a people-problem with technology. As Jerry Weinberg once said: "all problems are people-problems." Therefore all developed products or services are "people-solutions."

Perhaps the main thought with "Testing is a medium" is that you cannot (successfully) solve any problem without trying to understand what the problem is in the first place. e.g.: Who is it a problem for? Where? When? How? -- these are all *Testing* questions. When you deliver a solution, you check it with something like: "how does this meet your needs?" -- again, another question.

Software Development begins and ends with questions. Somewhere in the middle of the creative development process are more testing questions. Lots of different kinds of questions, tests and checks depending on the people, skills and risks involved in developing each particular solution. One could say that Software cannot be developed without Testing.

Have you tried? Have you ever been on a project where you didn't first ask the customer what the problem was? You didn't check to see if what you are building is working towards that design? Or you didn't ask the customer afterwards if what you delivered meets their needs? How did that work for you?

If one cannot hope to hit the target without checking many different things, why is Testing often given so little attention or recognition on development projects? Too few individuals try to develop expertise in the Testing field to elevate their contributions to the development effort.

Anyone may ask a question. That doesn't make you an "expert" in asking questions. My 10-year-old son uses scissors to cut things, but that doesn't mean I want to let him cut my hair! Everyone I meet feels they know what Testing is. Okay, then why do so many projects fail?

So, what does it take to become really good at Testing? It appears to be somewhat important for the success of most software projects. Maybe it's time to take a good look at Software Development through the medium of Testing. How might you look at things differently then? What skills or knowledge would you want to learn more about? Who do you think should be involved?

I won't be talking about any of this stuff on Tuesday though. After all, it's not in the session description. =)

Read More

Quality Agile Metrics

I was asked recently what metrics I would collect to assess how well an agile team is improving. I paused for a moment to scan through 12 years of research, discussion, memories and experiences with Metrics on various teams, projects and companies - mostly failed experiments. My answer to the question was to state that I presently only acknowledge one Metric as being meaningful: Customer Satisfaction.

We discussed the topic further and I elaborated some more on my experiences. Regarding specific "quality" metrics, I explained that things like counting Test Cases and bug fix rates are meaningless. I also referred to the book "Implementing Lean Software Development" by Mary and Tom Poppendieck (which I highly recommend BTW) which warns against "local optimizations" because they will eventually sabotage optimization of the whole system. In other words, if I put a metric in place to try and optimize the Testing function, it doesn't mean the whole [agile] development team's efficiency will improve.

It needs to be a whole team approach to quality and value. Specific measurements and metrics often lead to gaming of the system and focus on improving the metrics rather than putting the focus on delivering quality and value.  If the [whole] team is measured on the customer satisfaction, then that is what they will focus on. I have long since stopped measuring individual performance on a team.

I haven't stopped thinking about this question though, so I put this question out on Twitter this morning:

Aside from Customer Satisfaction, are there any other Quality metrics you'd recommend in an #agile environment?

Here are the responses I received:

1. Churn or team turnover. (Real case: Product delivered on time, customer happy, whole team left.)
2. Escaped defects, inbound support calls/emails
3. Value created - Reference: "Lean Startup" by Eric Ries (I'm reading it right now)
4. Number of contributors to each story - not because exact count is meaningful but because it encourages collaboration & review.
5. Profitability of the project is one. Are the goals (whatever they may be) of the project met?
6. Lines of code changed during regression. That can expose some severe problems.
7. Production Defects in 15 days after 'Go Live'.
8. Cost of rework due to requirement changes.
9. Code churn as a measure of "quality" (e.g. System Defect Density) - Research, Code example, Discussion, and Sample Stats.

Hm, almost a Top 10 list. I am happy with the responses here and think many of them may be worth exploring further to see what insights they provide. Cautionary Note with all Metrics: Beware the impact they have on the team. If behaviour or performance starts to change in a negative way, STOP immediately!

One of the things I talked about in the conversation was the importance of Retrospectives to allow the team to own their improvement activities. If the team uses these opportunities, their improvements should be observable over several iterations. I think a Happiness Index reading during Retrospectives might be an interesting indicator of the overall effectiveness of the improvement strategies employed.

What do you think? Anything else I should consider?

Read More

Testers, Learn about Agile (and Lean)

Let me tell you about something called Dramatic Irony. You see it in movies, television shows, plays and in many other places. It happens when you (as the audience or observer) see or understand something that the main characters don't. Often times this is funny, sometimes it's not. Personally, I am one of those that likes to laugh when I see this happen.

On my learning/education quest over a decade ago, I took many different positions and roles within various IT organisations so that I could learn different aspects of Quality. I went through various phases, and the one I am least proud of was the "Quality champion." This wasn't a job title so much as a belief that (mis-)guided my actions. The role/part/perspective came mainly from believing what my employer(s) told me at the time - namely that "the QA/Test team was responsible for quality."

If you have worked in Software Development for a while, and perhaps for a larger organisation, you have likely seen someone who believes they are a Quality Champion. They don't want to see any (known) bugs go out; they check up on everyone in the team to see that they have done their reviews or had someone else inspect their work before passing it onto the next person/team; they join committees to create, document, maintain or present processes that will increase the quality of the delivered products/solutions; and so on.

Ah, the poor misguided fools.  Bless their hearts.


The first problem is in the company creating a scapegoat culture that puts the responsibility/blame of poor quality on the group of individuals who are least likely to help change the quality late in a development cycle - especially when the (test) team is under-informed, under-funded, under-staffed, under ridiculous time constraints, unappreciated and/or uneducated.

Quality is everyone's job. And I mean everyone. It starts with the president, moves through every person in the organisation and even includes the customers and users of your product/solution.

Returning to the naive tester who doesn't know or understand this, they do their part to motivate, inspire, nudge and, to some extent, manage the individuals affecting the quality of the released products. The effect of this is easy to predict. The nail that sticks out gets hammered down.

As it happened to me, I have seen it happen to other testers - they become disheartened, give up and withdraw back into their own work routine, ignoring everyone else and just focussing on their part.

I didn't give up entirely though - I can be persistent. I kept my eyes and ears open and looked in different places for ideas to help me understand how the organisation/system and "quality" fit together.

At the start of the 21st century, I stumbled upon something called the Agile Manifesto. I can't say that I completely understood it at the time but I was certainly excited about it. I mentioned it to my manager and he said it was a passing fad and that in 5 years no one would ever remember it. I felt that he was wrong and trusted my instincts on this one.

Over the next several years I learned about Agile and the different implementations. It all seemed very programmer/developer-centric to me as none of the models, articles, books or people ever seemed to talk about the testers. There was certainly a lot about devs taking responsibility for the quality of their work and incorporating testing practices into their regular routines. Hey, I'm all over that! Rah, rah, rah, sis-boom-bah, yaaaay Agile! :-D

Now, a decade later, I understand the Agile movement in a deeper, richer way. It is part of how I think and solve problems. It is part of how I encourage people to work together and to focus on the things that matter when delivering value to the customers. Just as I once described myself as someone who eats, sleeps and breathes 'Testing', I believe I would say the same thing about 'Agile'.

Here's the kick: the Agile movement is an ENTIRE COMMUNITY OF QUALITY CHAMPIONS! (oops, the caps lock got stuck for a moment there.)

That's right, listen up all of you testers out there who think you are alone and that no one is listening to your cries of "there must be a better way." There are people out there - Agile Coaches and Consultants - who are working to do just that. They find, create and teach better ways for development teams to work together to raise the quality/value of the delivered solutions.

If you don't know anything about Agile, start now. Read up on it, talk to others, attend a course, find online webinars, go to a conference - anything! Just get out there are start learning about Agile now! The same applies for Lean Software Development - learn about it.

Please keep in mind that there is a big difference between going through the motions of agile practices and actually being/thinking "agile". The agile mindset is more important to me than any particular set of practices.

This is especially important to keep in mind if someone tells you that testers have no place in Agile teams. Those people are what I like to call "wrong." (Get off my lawn.)

Testers can bring valuable insights to the agile software development process if the team works together and embraces the strengths of each team member. You, dear testers, must be open to change and adapt your role to work in new ways.

Warning, Warning, Danger, Danger: if someone tells you that you are "doing agile" and you (as a tester) should keep updating your manual regression test cases and test plans, please tell them to kindly "get off my lawn" for me. Thanks.

I still see this happen. I go into a client's office and look at how the software development team members are working. I see the testers off on their own, testing things in isolation and complaining about how no one seems to care about Quality because they only see and test the product at the end, just before it goes out the door.

Ooo, irony. I see something happening around you (within the community, industry, and sometimes within your own company) that you don't see.

Testers: if you feel like you are in this "Quality champion" role, be the hero and talk to an Agile Coach. Get one to come in and do an assessment. You aren't alone. You can help make a difference. Ask for the right help.

Be prepared to change yourself - to learn, to adapt to new ways of working with others, and to deliver a whole new level of quality and value that you didn't think was possible. Be a model team member and let the Agile coach help guide the rest of the team. Quality isn't your job - it's everyone's.

Read More

Quality Center Must Die

It is not a matter of "if" -- it is a matter of "when" HP's Quality Center software will die.  And you, my dear readers will help make that happen.

"How?" you may ask? Simple. There are two things you should do: (1) think, and (2) don't put up with crap that gets in the way of delivering value to the customer and interacting intelligently with other human beings.

But I am getting ahead of myself. Let's rewind the story a bit...

Several months ago I was hired by my client to help train one of the test teams on agile and exploratory testing methods. The department has followed a mostly Waterfall development model until now and wants to move in the Agile direction. (A smart choice for them, if you ask me.) Why am I still there after all this time? That's a good question.

After attending the Problem Solving Leadership course last year, and after attending a few AYE conferences, I changed my instructional style to be more the kind of consultant that empowers the client with whatever they need to help themselves learn and grow.  It's a bit of a slower pace, but the results are more positive and long-lasting.

I am a part of a "pilot" agile/scrum team and am working closely with one of the testers (I will call him "Patient Zero") to coach him on good testing practices to complement the agile development processes. I have done this several times now at different clients, so this is nothing new to me. One of the unexpected surprises that cropped up this time was that this development team is not an end-to-end delivery team, so when they are "done" their work, the code moves into a Waterfall Release process and it all kind of falls apart. There are still some kinks to be solved here and I am happy to see some really bright, caring people trying to solve these problems. So that's okay.


Patient Zero and I are part of a larger test team, and the rest of the test team all work on Waterfall-style projects, use Waterfall-compatible tools, and they generally don't get how we work. :) Unfortunately, one of the tools mandated for our team's use is HP's Quality Center (HPQC).  I hadn't seen that tool in about a decade and it looked very similar to how I last remember it.

To my agile coach/practitioner friends I should clarify that at no time during our sprint development work does anyone ever touch HPQC! However, once the code is deployed/falls into the Waterfall Release process, regression test cases are created in HPQC and it is used for defect tracking. It is mandated, and so shall it be done. I can live with that. It's just a tool at this point and the impact to our ability to deliver a good solution is eliminated by the fact that we don't touch it until after we are "done". (Communication and collaboration FTW!)

Two days ago.

Our whole test team took part in a 2-day HPQC training workshop on something HP calls "Business Process Testing" or BPT. Being naturally curious to learn something new, I wanted to know what BPT was and how it fits into the bigger testing picture. Here we go.

We were given a handout with some "test scenarios" to be used for training. The test scenarios fell into this pattern:

• Scenario name/title
• Requirement description
• Test Situation (I am staring at this right now and I still don't know what this means)
• Role (kind of system user this requirement/situation applies to)
• Steps

That's okay information. The "Steps" are what you might typically expect to see if you have been testing for a while. Here is an example for working with a sample web app:

1. Login to the system with (a certain type of user)
2. Navigate to some module in the app
3. Click "Create" button from the tool-bar
4. Enter mandatory field values and save
5. Search for the information you created in the previous step
6. Logout of the system

And there were 6 of these scenarios.

I read through these tests and then I tried to follow them using the system. I quickly encountered a half-dozen bugs - some with the system, some with the test scenarios/cases, and some were open questions that I would follow-up with the Product Owner for requirement clarification.

But, woah-woah-woah-hey.. wait a minute. We only need to worry about *these* documented test scenarios! I struggled hard to keep my mouth shut about the value of time and the many different kinds of tests I would happily engage in at this point if I could leave the tool alone. But, I left that to my "inner voice" and we were now one hour into the first day's training.

At this point, we were given an overview of the HPQC modules and told to (1) enter the Requirements, (2) create BPT test scenarios, and (3) "componentize" the test scenarios into groupings of related steps. This last part required some explanation since this was new to me, and once we got it, we set to work on the task.  Since we could see (1) and (2) on the handout sheets in front of us, we went straight to work on part (3). That was kind of fun working in small group of 4 looking at these scenarios and trying to come up with solutions.

And then someone went and spoiled the fun.  We were "told" that we weren't supposed to do part (3) until after we had entered the information for (1) and (2) into the tool.  I was all like "really?" and then a team lead came by and repeated the exact same thing.  This may be summed up as: Enter data into the tool first, and think later.

I was kind of shocked with this comment and attitude and in retrospect it kind of foreshadowed the rest of the training experience -- i.e. Tool and process first; think later; maybe.  Okay. I'll play along and see where this goes.

During this exercise, I began to grasp this idea of "components" as HP uses them. I think they are like Page Objects - chunks of code (or in this case, test steps) that perform a certain function, promote reuse, and reduce duplication. Although it was never described in this way, I believe the HP "BPT" module is a proprietary Do-It-Yourself DSL. Aha! I have experience with those. I get that stuff.

So, once I got it, I started to explain the concept of YAGNI to my group team members.  That is, let's not overthink or over-engineer these components. Let's build/write them based on the needs of the requirements in front of us. We will modify the components in the future as new requirements appear.  This idea was well received and we quickly came to an efficient solution for our small group.

When we took up the exercise, I found I had to explain the YAGNI concept to the instructor/trainer (and rest of the class/team) as he proposed that we try to abstract out these components to allow for compatibility with other features and system elements. What a waste of time! We cannot know what we will need beyond our immediate needs so that kind of abstraction is a pointless exercise that leads to more headaches than you need.

Eventually, I started to get that the point of writing the test scenarios using BPT was that these components form a basic vocabulary that may then be automated at some point in the future -- yes, BPT integrates with HP's QTP automation tools.  Now, I'm all in favour of consistency, clarity, reuse, and automating tests that humans should never have to do more than once (and if there is value in re-running the test), so I struggled to understand why it was never explained to us this way.  As long as I kept the DSL/automation model in my head, I understood what we were doing and can see the potential benefits of it.

I saw many testers struggle with the exercises and models we were presented with. End of day one.

Day two began with a quick recap and then we were introduced to the bureaucracy that is Waterfall and HPQC. That is, there are review processes and workflows to cover each requirement, BPT and component. Welcome to Wasteland. (I mean the Lean Development concept of "waste" here, although other interpretations of "wasteland" may be just as valid.) Easily a third of the 2-day training was spent on the processes surrounding the management and review of the various HPQC objects. sigh.

We then moved onto the concept of "parameters" for the components we created yesterday.  Okay, I get method parameters when I am scripting with Ruby, so this was no sweat.  Given the length of time I spent parametrizing my components compared to everyone else in the class, I think I may be one of the few who really got it.

I learned a few more things about the main instructor. One was that he didn't know how to identify web page elements using commonly available browser tools. Umm, aren't these tools supposed to interact with web pages? This never came up before? You have never wondered how to find the name/id for an element on a web page? really?

The other was that he had a really bad sense of humour.  He made a reference to a "QA" joke that I shall not repeat here.  Needless to say that I found it insulting, offensive, it made my skin crawl and my blood boil.  Many unpleasant feelings and ideas arose in me and it took all my willpower and strength not to react to the blatant stupidity of insulting the profession of the students in your class and the market that the HPQC tool represents.

The final blow for the day came to me when we tried to "execute" these BPT test scenarios using the HPQC tool. THEN I discover that these "parameters" can have 2 kinds of values - fixed/hard-coded and run-time. Anyone who has done intelligent automation knows NOT to hard-code values in their scripts. Data-driven is way better, and as an exploratory tester, I may not know what value I will choose until moments before.

Here's where the HPQC tool gets stupid..er.  Regardless which parameter type you choose, they may ONLY be defined BEFORE the test execution! You cannot leave empty parameters so that when you get to a particular step, the tester can decide what value to input and feed it back into the tool.

What does this mean?  As a tester, let's say you want to create a new user profile in an app of some kind. The test parameters for things like Name, Email, Address, Country, and so on, must all be defined and set before the test is executed. You cannot decide while you are actually testing!

Why does this matter to me? It matters because it means that the humans who execute these BPT tests manually have no choice in what values they may input. Testing techniques like Equivalence Classes and BVA that help guide our choices to pursue interesting paths are completely cut-off! It turns out that HPQC treats humans worse than the automated counterparts. In discussion with an automation "expert" at the end of the class I learned that at least you can code some variability into the QTP automation scripts.  This is not possible with the same test scenarios executed manually by human beings.

So. Much. Wrong.

So after my first ever QC training session, here are some of my take-aways:

• It took me 2 days to "script" 6 test scenarios in this tool and they were rotten test cases to begin with! I suspect that outside of the training environment, it will actually take longer to complete since you won't have a "reviewer" sitting next to you waiting for you to finish your piece.
• And they weren't even automated! Who knows how long it would take to tweak the "components" to make them work with a particular automation strategy.
• HP QC will never be a useful tool for any agile or rapid development efforts
• (HP best practice) Put the tool, data and review processes first, before you think. Maybe always instead of it too.
• BPT is a DSL framework for test scripting
• BPT component parameters cannot be customised during test execution. They may only be set/defined before you start testing. => No thinking allowed while testing.
• The instructor didn't appear to be knowledgeable on anything outside of the tool itself. This includes how we might actually want to use the tool. No, no, no. And I quote: "Testers must change how they work to use the tool in the way it was designed."
• As long as there are people pushing these kinds of horrible tools that suck the life and intelligence out of people, inject mountains of wasteful activities that provide no value to the customers or end users, and continue to create barriers between testers and their developer counterparts, I will always have job security in helping organisations recover from these cancers.

At the end of the day, there were several people interested in my idea of randomising tests. The automation expert in the class insisted that it couldn't be done with automation, so I called up my "Unscripted Automation" presentation slides that I gave a few years ago. He said that what I proposed was not a "best practice" and that everyone, the whole industry, was using the tools in the way that he described how they should be used.

My response was to simply say "they are all wrong."

Read More