Test Management is Wrong

Test Management is wrong. There. I said it.

I can't believe it took me this long to notice the obvious. If you are doing Software Development in any fashion, and are worried about how to manage your testing to develop a "quality" product, stop it.

Let's be clear about what I mean here. If you consider any software development life cycle (SDLC) process, you will find activities like the following arranged in some fashion or other:

• Requirements gathering, specification
• Software design
• Implementation and Integration
• Testing (or Validation)
• Deployment
• Lather, Rinse, Repeat (i.e. Maintain, Enhance, Fix, Mangle, Spin, and so on)

These activities aren't Waterfall or Agile or anything else, they are just activities. HOW you choose to do them will reflect your SDLC. I don't care about that right now. The part I'm picking on is the Testing bit near the middle, regardless of whether you do them in an agile, Waterfall, or some other way.

In particular, I am picking on the fallacy or myth that a good Test Management plan/process is what you need to develop and release a high Quality product.

In almost 25 years of working in the Software/IT industry, I have never worked at a company or heard of any company that had a Test Management process so solid that it lead to high quality, I don't believe there ever will be either.

Here's an analogy. Let's say I want to make some Rice Krispies treats. The ingredients are: butter, marshmallows and rice krispies. You heat 'em up, mix 'em up, flatten 'em out, cool, slice and eat. That's it.

Ya, but you know what? I have this really excellent Marshmallow Plan that will produce the Bestest treats anyone has ever tasted.

How does that sound to you? Interested to learn more about the plan? Be truthful.

If you said "Yes", I don't know if I can help you. If you said "No" there may be hope for you yet.

Why is it that Software companies are attracted to the idea that Testing is somehow equated with Quality? Testing is an activity that you may choose to do or not do and still have a great quality product. The Agile Manifesto doesn't mention testing anywhere and yet the Manifesto's signatories, thought leaders, and practitioners produce great quality stuff for people. How does that work?

Why is it only the Testing phase/activities/part that equates with Quality? Why aren't companies and managers everywhere promoting super awesome Requirements Management processes and plans at Requirements Conferences to help deliver high Quality? What about Design Management? No, no, wait, I got it. We need Deployment Management. That's it! I've solved it. Aha!

No, these are all stupid suggestions. And you know it too.

Is it because Testing/Verification activities are part of the other 'phases' or steps in software development, so surely we should be able to manage all that, right? Well, actually, no. That's the wrong approach.

Test Management systems manage and measure some of the testing activities performed on a project. Completing your testing activities tells me nothing about the overall "Quality" of the product. By definition, it is an incomplete part of the picture.

By putting your faith in Test Management plans or systems you are effectively saying "I don't know how to measure what you want (i.e. Quality), so I will measure what I can do (i.e. Test)."

Does this mean it's pointless to track the testing you've done? NO! I am not saying that. If you do something that you believe contributes to the value of the project, then please track what you are doing so that others can see what you have done. Preferably, make your progress visible in some way.

What we need to focus on is what the customer needs. Ask yourself: What problem are they trying to solve? What are we trying to deliver that is of value to them?

What we need is Value Management. There are successful (i.e. "quality") products built and released to customers that never had any independent testers on the development team. Cool. These teams get it.

I have a friend and colleague who started up a company and within hours had sold a product that didn't even have a single line of code written. Way cool! He gets it. Did he worry about Test Management? Bah! He didn't even need to pay a programmer to sell something of value to a customer.

I have mentioned it before, if you are doing any kind of testing on a software development project, you can likely place it somewhere within the Agile Testing Quadrants. Heck, you don't even have to be on an Agile team or project to see that your testing fits somewhere on the chart.

So, if you somehow create a super Test Management Plan or system that tracks all of these activities for a given project, then would you have a Quality product? That would be really cool, by the way, but no, not necessarily. And don't fool yourself into thinking that it will.

The problem is one of relationships. Human relationships.

The definition of Quality I like to use comes from Jerry Weinberg and it is based on your understanding of relationships between people. The definition of "software" that I use when teaching/training/coaching is that software reflects your understanding of the relationships between people and the systems they interact with to meet particular needs. The definition of Testing that I like to work with is the intelligent use of models and heuristics to explore the relationships between the people designing a system and those who intend to use it.

In short, if you want to understand what provides "quality" or value to your customers, look to how you are managing the human relationships. Forget Test Management. You might have better luck implementing a CRM system in your development departments as a predictor of Quality.

Testers: if you want to do a great job, focus on the testing activities that explore the systems from the perspectives of the people who matter. Never hide or bury your work (i.e. in documents, spreadsheets, test management systems, etc.). Make it visible - use dashboards, mind maps or other visual mediums because they assist with team collaboration and understanding. Keep detailed records archived somewhere if you need them, but don't worry about "managing" to those fiddly bits as an indicator of "quality" - because they're not.

Forget about the Test Plans though. Disregard Test Management systems. AVOID providing any metrics on testing coverage as an indicator of Quality... especially if your development team isn't producing equally misleading metrics about code complexity, requirements reviews, and other esoteric development activities.

Anyone (ALM Vendors take note) who sells a "Quality Management" system based upon managing test cases is lying to themselves and to you. They are doing it wrong. Don't be a victim.

You can't kludge a development process enough to make a Quality or Test Management system produce meaningful, valid indicators of value to your customers and stakeholders. It just doesn't make sense.

Approach your development teams from a human relationship perspective. Focus on how people collaborate and work together, and high quality products will emerge as a by-product. Manage the relationships with your customers, and deliver working software frequently to help them see what you can do and how you learn from past experiences. This isn't easy. It's definitely worth it though.

So. You want to deliver Quality? Test Management isn't the answer. Test Management tells you about a specific task's management. It's about as useful as Marshmallow Management in making Rice Krispies treats. That is, it may have a place in the big picture, but it's certainly not the right way to look at the problem.

Read More

Testing is a Medium

In a few days I will be giving a presentation to the local Agile/Lean Peer 2 Peer group here in town. The group has a web site - Waterloo Agile Lean, and the announcement is also on the Communitech events page.

I noticed the posted talk descriptions are shorter than what I wrote.  The Waterloo Agile Lean page has this description:

"This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it’s done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

The Communitech page has this description:

"Exploratory Testing is the explosive sound check that helps us see things from many directions all at once. It takes skill and practice to do well. The reward is a higher-quality, lower-risk solution that brings teams a richer understanding of the development project.
This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it's done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

This is what I submitted:

"Testing is the medium in which solutions are developed. The value of our delivered solutions depend upon how well we understand and utilize that medium. We can fly straight like an arrow or explode outwards like a spherical sound wave.
Traditional automated TDD "checks" help us fly straight in the direction we choose. Is it the right direction though? How do we know? Are you sure?
Exploratory Testing is the explosive sound check that helps us see things from many directions all at once. It takes skill and practice to do well. The reward is a higher-quality, lower-risk solution that brings teams a richer understanding of the development project.
This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it's done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

This blog post is *not* about the differences in session descriptions. (In fairness, I really should learn to keep it to one paragraph. I hope to get better at writing session descriptions - it'll come with practice.) Reading the last one first, I can't help think that a bit of context is missing from the two posted descriptions for the final "blown away" statement. That is, that phrase comes from the sound wave analogy and not from some arrogant expectations I have for my presentation abilities. If I had known the descriptions would be shortened, I would have at least changed that sentence.

This blog post is about the idea that I try to convey in the first sentence that is unfortunately missing from both posted session descriptions: "Testing is the medium in which solutions are developed."

Software Development is a creative process. It is the intersection of people, skills, tools and experimentation to solve a people-problem with technology. As Jerry Weinberg once said: "all problems are people-problems." Therefore all developed products or services are "people-solutions."

Perhaps the main thought with "Testing is a medium" is that you cannot (successfully) solve any problem without trying to understand what the problem is in the first place. e.g.: Who is it a problem for? Where? When? How? -- these are all *Testing* questions. When you deliver a solution, you check it with something like: "how does this meet your needs?" -- again, another question.

Software Development begins and ends with questions. Somewhere in the middle of the creative development process are more testing questions. Lots of different kinds of questions, tests and checks depending on the people, skills and risks involved in developing each particular solution. One could say that Software cannot be developed without Testing.

Have you tried? Have you ever been on a project where you didn't first ask the customer what the problem was? You didn't check to see if what you are building is working towards that design? Or you didn't ask the customer afterwards if what you delivered meets their needs? How did that work for you?

If one cannot hope to hit the target without checking many different things, why is Testing often given so little attention or recognition on development projects? Too few individuals try to develop expertise in the Testing field to elevate their contributions to the development effort.

Anyone may ask a question. That doesn't make you an "expert" in asking questions. My 10-year-old son uses scissors to cut things, but that doesn't mean I want to let him cut my hair! Everyone I meet feels they know what Testing is. Okay, then why do so many projects fail?

So, what does it take to become really good at Testing? It appears to be somewhat important for the success of most software projects. Maybe it's time to take a good look at Software Development through the medium of Testing. How might you look at things differently then? What skills or knowledge would you want to learn more about? Who do you think should be involved?

I won't be talking about any of this stuff on Tuesday though. After all, it's not in the session description. =)

Read More

Testers, Learn about Agile (and Lean)

Let me tell you about something called Dramatic Irony. You see it in movies, television shows, plays and in many other places. It happens when you (as the audience or observer) see or understand something that the main characters don't. Often times this is funny, sometimes it's not. Personally, I am one of those that likes to laugh when I see this happen.

On my learning/education quest over a decade ago, I took many different positions and roles within various IT organisations so that I could learn different aspects of Quality. I went through various phases, and the one I am least proud of was the "Quality champion." This wasn't a job title so much as a belief that (mis-)guided my actions. The role/part/perspective came mainly from believing what my employer(s) told me at the time - namely that "the QA/Test team was responsible for quality."

If you have worked in Software Development for a while, and perhaps for a larger organisation, you have likely seen someone who believes they are a Quality Champion. They don't want to see any (known) bugs go out; they check up on everyone in the team to see that they have done their reviews or had someone else inspect their work before passing it onto the next person/team; they join committees to create, document, maintain or present processes that will increase the quality of the delivered products/solutions; and so on.

Ah, the poor misguided fools.  Bless their hearts.


The first problem is in the company creating a scapegoat culture that puts the responsibility/blame of poor quality on the group of individuals who are least likely to help change the quality late in a development cycle - especially when the (test) team is under-informed, under-funded, under-staffed, under ridiculous time constraints, unappreciated and/or uneducated.

Quality is everyone's job. And I mean everyone. It starts with the president, moves through every person in the organisation and even includes the customers and users of your product/solution.

Returning to the naive tester who doesn't know or understand this, they do their part to motivate, inspire, nudge and, to some extent, manage the individuals affecting the quality of the released products. The effect of this is easy to predict. The nail that sticks out gets hammered down.

As it happened to me, I have seen it happen to other testers - they become disheartened, give up and withdraw back into their own work routine, ignoring everyone else and just focussing on their part.

I didn't give up entirely though - I can be persistent. I kept my eyes and ears open and looked in different places for ideas to help me understand how the organisation/system and "quality" fit together.

At the start of the 21st century, I stumbled upon something called the Agile Manifesto. I can't say that I completely understood it at the time but I was certainly excited about it. I mentioned it to my manager and he said it was a passing fad and that in 5 years no one would ever remember it. I felt that he was wrong and trusted my instincts on this one.

Over the next several years I learned about Agile and the different implementations. It all seemed very programmer/developer-centric to me as none of the models, articles, books or people ever seemed to talk about the testers. There was certainly a lot about devs taking responsibility for the quality of their work and incorporating testing practices into their regular routines. Hey, I'm all over that! Rah, rah, rah, sis-boom-bah, yaaaay Agile! :-D

Now, a decade later, I understand the Agile movement in a deeper, richer way. It is part of how I think and solve problems. It is part of how I encourage people to work together and to focus on the things that matter when delivering value to the customers. Just as I once described myself as someone who eats, sleeps and breathes 'Testing', I believe I would say the same thing about 'Agile'.

Here's the kick: the Agile movement is an ENTIRE COMMUNITY OF QUALITY CHAMPIONS! (oops, the caps lock got stuck for a moment there.)

That's right, listen up all of you testers out there who think you are alone and that no one is listening to your cries of "there must be a better way." There are people out there - Agile Coaches and Consultants - who are working to do just that. They find, create and teach better ways for development teams to work together to raise the quality/value of the delivered solutions.

If you don't know anything about Agile, start now. Read up on it, talk to others, attend a course, find online webinars, go to a conference - anything! Just get out there are start learning about Agile now! The same applies for Lean Software Development - learn about it.

Please keep in mind that there is a big difference between going through the motions of agile practices and actually being/thinking "agile". The agile mindset is more important to me than any particular set of practices.

This is especially important to keep in mind if someone tells you that testers have no place in Agile teams. Those people are what I like to call "wrong." (Get off my lawn.)

Testers can bring valuable insights to the agile software development process if the team works together and embraces the strengths of each team member. You, dear testers, must be open to change and adapt your role to work in new ways.

Warning, Warning, Danger, Danger: if someone tells you that you are "doing agile" and you (as a tester) should keep updating your manual regression test cases and test plans, please tell them to kindly "get off my lawn" for me. Thanks.

I still see this happen. I go into a client's office and look at how the software development team members are working. I see the testers off on their own, testing things in isolation and complaining about how no one seems to care about Quality because they only see and test the product at the end, just before it goes out the door.

Ooo, irony. I see something happening around you (within the community, industry, and sometimes within your own company) that you don't see.

Testers: if you feel like you are in this "Quality champion" role, be the hero and talk to an Agile Coach. Get one to come in and do an assessment. You aren't alone. You can help make a difference. Ask for the right help.

Be prepared to change yourself - to learn, to adapt to new ways of working with others, and to deliver a whole new level of quality and value that you didn't think was possible. Be a model team member and let the Agile coach help guide the rest of the team. Quality isn't your job - it's everyone's.

Read More

Quality Center Must Die

It is not a matter of "if" -- it is a matter of "when" HP's Quality Center software will die.  And you, my dear readers will help make that happen.

"How?" you may ask? Simple. There are two things you should do: (1) think, and (2) don't put up with crap that gets in the way of delivering value to the customer and interacting intelligently with other human beings.

But I am getting ahead of myself. Let's rewind the story a bit...

Several months ago I was hired by my client to help train one of the test teams on agile and exploratory testing methods. The department has followed a mostly Waterfall development model until now and wants to move in the Agile direction. (A smart choice for them, if you ask me.) Why am I still there after all this time? That's a good question.

After attending the Problem Solving Leadership course last year, and after attending a few AYE conferences, I changed my instructional style to be more the kind of consultant that empowers the client with whatever they need to help themselves learn and grow.  It's a bit of a slower pace, but the results are more positive and long-lasting.

I am a part of a "pilot" agile/scrum team and am working closely with one of the testers (I will call him "Patient Zero") to coach him on good testing practices to complement the agile development processes. I have done this several times now at different clients, so this is nothing new to me. One of the unexpected surprises that cropped up this time was that this development team is not an end-to-end delivery team, so when they are "done" their work, the code moves into a Waterfall Release process and it all kind of falls apart. There are still some kinks to be solved here and I am happy to see some really bright, caring people trying to solve these problems. So that's okay.


Patient Zero and I are part of a larger test team, and the rest of the test team all work on Waterfall-style projects, use Waterfall-compatible tools, and they generally don't get how we work. :) Unfortunately, one of the tools mandated for our team's use is HP's Quality Center (HPQC).  I hadn't seen that tool in about a decade and it looked very similar to how I last remember it.

To my agile coach/practitioner friends I should clarify that at no time during our sprint development work does anyone ever touch HPQC! However, once the code is deployed/falls into the Waterfall Release process, regression test cases are created in HPQC and it is used for defect tracking. It is mandated, and so shall it be done. I can live with that. It's just a tool at this point and the impact to our ability to deliver a good solution is eliminated by the fact that we don't touch it until after we are "done". (Communication and collaboration FTW!)

Two days ago.

Our whole test team took part in a 2-day HPQC training workshop on something HP calls "Business Process Testing" or BPT. Being naturally curious to learn something new, I wanted to know what BPT was and how it fits into the bigger testing picture. Here we go.

We were given a handout with some "test scenarios" to be used for training. The test scenarios fell into this pattern:

• Scenario name/title
• Requirement description
• Test Situation (I am staring at this right now and I still don't know what this means)
• Role (kind of system user this requirement/situation applies to)
• Steps

That's okay information. The "Steps" are what you might typically expect to see if you have been testing for a while. Here is an example for working with a sample web app:

1. Login to the system with (a certain type of user)
2. Navigate to some module in the app
3. Click "Create" button from the tool-bar
4. Enter mandatory field values and save
5. Search for the information you created in the previous step
6. Logout of the system

And there were 6 of these scenarios.

I read through these tests and then I tried to follow them using the system. I quickly encountered a half-dozen bugs - some with the system, some with the test scenarios/cases, and some were open questions that I would follow-up with the Product Owner for requirement clarification.

But, woah-woah-woah-hey.. wait a minute. We only need to worry about *these* documented test scenarios! I struggled hard to keep my mouth shut about the value of time and the many different kinds of tests I would happily engage in at this point if I could leave the tool alone. But, I left that to my "inner voice" and we were now one hour into the first day's training.

At this point, we were given an overview of the HPQC modules and told to (1) enter the Requirements, (2) create BPT test scenarios, and (3) "componentize" the test scenarios into groupings of related steps. This last part required some explanation since this was new to me, and once we got it, we set to work on the task.  Since we could see (1) and (2) on the handout sheets in front of us, we went straight to work on part (3). That was kind of fun working in small group of 4 looking at these scenarios and trying to come up with solutions.

And then someone went and spoiled the fun.  We were "told" that we weren't supposed to do part (3) until after we had entered the information for (1) and (2) into the tool.  I was all like "really?" and then a team lead came by and repeated the exact same thing.  This may be summed up as: Enter data into the tool first, and think later.

I was kind of shocked with this comment and attitude and in retrospect it kind of foreshadowed the rest of the training experience -- i.e. Tool and process first; think later; maybe.  Okay. I'll play along and see where this goes.

During this exercise, I began to grasp this idea of "components" as HP uses them. I think they are like Page Objects - chunks of code (or in this case, test steps) that perform a certain function, promote reuse, and reduce duplication. Although it was never described in this way, I believe the HP "BPT" module is a proprietary Do-It-Yourself DSL. Aha! I have experience with those. I get that stuff.

So, once I got it, I started to explain the concept of YAGNI to my group team members.  That is, let's not overthink or over-engineer these components. Let's build/write them based on the needs of the requirements in front of us. We will modify the components in the future as new requirements appear.  This idea was well received and we quickly came to an efficient solution for our small group.

When we took up the exercise, I found I had to explain the YAGNI concept to the instructor/trainer (and rest of the class/team) as he proposed that we try to abstract out these components to allow for compatibility with other features and system elements. What a waste of time! We cannot know what we will need beyond our immediate needs so that kind of abstraction is a pointless exercise that leads to more headaches than you need.

Eventually, I started to get that the point of writing the test scenarios using BPT was that these components form a basic vocabulary that may then be automated at some point in the future -- yes, BPT integrates with HP's QTP automation tools.  Now, I'm all in favour of consistency, clarity, reuse, and automating tests that humans should never have to do more than once (and if there is value in re-running the test), so I struggled to understand why it was never explained to us this way.  As long as I kept the DSL/automation model in my head, I understood what we were doing and can see the potential benefits of it.

I saw many testers struggle with the exercises and models we were presented with. End of day one.

Day two began with a quick recap and then we were introduced to the bureaucracy that is Waterfall and HPQC. That is, there are review processes and workflows to cover each requirement, BPT and component. Welcome to Wasteland. (I mean the Lean Development concept of "waste" here, although other interpretations of "wasteland" may be just as valid.) Easily a third of the 2-day training was spent on the processes surrounding the management and review of the various HPQC objects. sigh.

We then moved onto the concept of "parameters" for the components we created yesterday.  Okay, I get method parameters when I am scripting with Ruby, so this was no sweat.  Given the length of time I spent parametrizing my components compared to everyone else in the class, I think I may be one of the few who really got it.

I learned a few more things about the main instructor. One was that he didn't know how to identify web page elements using commonly available browser tools. Umm, aren't these tools supposed to interact with web pages? This never came up before? You have never wondered how to find the name/id for an element on a web page? really?

The other was that he had a really bad sense of humour.  He made a reference to a "QA" joke that I shall not repeat here.  Needless to say that I found it insulting, offensive, it made my skin crawl and my blood boil.  Many unpleasant feelings and ideas arose in me and it took all my willpower and strength not to react to the blatant stupidity of insulting the profession of the students in your class and the market that the HPQC tool represents.

The final blow for the day came to me when we tried to "execute" these BPT test scenarios using the HPQC tool. THEN I discover that these "parameters" can have 2 kinds of values - fixed/hard-coded and run-time. Anyone who has done intelligent automation knows NOT to hard-code values in their scripts. Data-driven is way better, and as an exploratory tester, I may not know what value I will choose until moments before.

Here's where the HPQC tool gets stupid..er.  Regardless which parameter type you choose, they may ONLY be defined BEFORE the test execution! You cannot leave empty parameters so that when you get to a particular step, the tester can decide what value to input and feed it back into the tool.

What does this mean?  As a tester, let's say you want to create a new user profile in an app of some kind. The test parameters for things like Name, Email, Address, Country, and so on, must all be defined and set before the test is executed. You cannot decide while you are actually testing!

Why does this matter to me? It matters because it means that the humans who execute these BPT tests manually have no choice in what values they may input. Testing techniques like Equivalence Classes and BVA that help guide our choices to pursue interesting paths are completely cut-off! It turns out that HPQC treats humans worse than the automated counterparts. In discussion with an automation "expert" at the end of the class I learned that at least you can code some variability into the QTP automation scripts.  This is not possible with the same test scenarios executed manually by human beings.

So. Much. Wrong.

So after my first ever QC training session, here are some of my take-aways:

• It took me 2 days to "script" 6 test scenarios in this tool and they were rotten test cases to begin with! I suspect that outside of the training environment, it will actually take longer to complete since you won't have a "reviewer" sitting next to you waiting for you to finish your piece.
• And they weren't even automated! Who knows how long it would take to tweak the "components" to make them work with a particular automation strategy.
• HP QC will never be a useful tool for any agile or rapid development efforts
• (HP best practice) Put the tool, data and review processes first, before you think. Maybe always instead of it too.
• BPT is a DSL framework for test scripting
• BPT component parameters cannot be customised during test execution. They may only be set/defined before you start testing. => No thinking allowed while testing.
• The instructor didn't appear to be knowledgeable on anything outside of the tool itself. This includes how we might actually want to use the tool. No, no, no. And I quote: "Testers must change how they work to use the tool in the way it was designed."
• As long as there are people pushing these kinds of horrible tools that suck the life and intelligence out of people, inject mountains of wasteful activities that provide no value to the customers or end users, and continue to create barriers between testers and their developer counterparts, I will always have job security in helping organisations recover from these cancers.

At the end of the day, there were several people interested in my idea of randomising tests. The automation expert in the class insisted that it couldn't be done with automation, so I called up my "Unscripted Automation" presentation slides that I gave a few years ago. He said that what I proposed was not a "best practice" and that everyone, the whole industry, was using the tools in the way that he described how they should be used.

My response was to simply say "they are all wrong."

Read More

Thoughts on the StarEast 2011 conference

I first attended StarEast in 1999. I remember the day-long tutorial I attended (by Rick Craig), and two track sessions - one by Cem Kaner on hiring testers, and one by James Whittaker on "Exploiting a Broken Design Process." I know I attended other sessions but I don't have active memories of them any more. I do remember the experience of attending the conference - one of surprise and excitement. Surprise at seeing so many other people in the testing community with similar questions and problems as myself, and excitement at the speakers with lots of great information and advice to give.

Fast forward to 2011 - I returned to StarEast, this time as a speaker. I suppose I didn't need to wait 12 years to return as a speaker. I didn't intentionally ignore the conference. I think I've been busy with other things and it just didn't come up - until last Fall when I received an invitation in my inbox to submit a proposal. I'm really glad I went.

Some things were familiar - the beautiful hotel, the Florida sunshine, the amazingly fresh orange juice, and the basic conference format. One thing that was different for me this time around was the number of people/speakers that I new who were also speaking at the conference. After having attended and spoken at several other conferences over the years, I guess I have gotten to know many of the popular speakers.

I was happy to see many more people speaking that I have never heard about before.  That tells me that the community is still growing after all this time and that there are still many more people sharing their knowledge to help enlighten future generations of testing leaders. That's awesome!


I was particularly surprised at the calibre of the Keynote speakers. I was genuinely inspired by every Keynote that I attended (I think I only missed one). (You can find the 2011 program online with details of the speakers and talks but I don't believe they were video recorded.) The Keynote presentations I attended were by: Andy Kaufman, Naomi Karten, and Julie Gardiner. The majority of my #StarEast tweets were from these keynotes.  Unfortunately, I missed the Keynote by Gojko Adzic (I heard it was good too), and I'm not including the "lightning" talks here although I did attend those and some were very good.

I was excited to hear that Lisa Crispin and Janet Gregory, authors of Agile Testing, were attending the conference and am happy to have had the opportunity to meet them and speak with them! =)  Lisa even attended my track session and wrote up a nice summary about it on the Software Testing Club site.

I won't really say much about my session here. It went okay I guess. I haven't received the session feedback evaluations yet, but I can say that I was really happy when several people came up to me afterwards in the hallway to thank me for my talk. They said that they really enjoyed it. My favourite comment came from Nawwar, my track chair -- i.e. the person who introduces the speaker during the session. He went from asking me "who are you?" just before the session to "Wow. That was the best session I've seen during this whole conference!"  Nice. Thanks. =)

I had fun giving the talk - "Real-time Test Design for Exploratory Testing". Test Design is a topic that I am really passionate about and I can talk about for hours. Okay, days. ;-)  I think the main thing that I wasn't terribly crazy about my talk was the format. When I talk about Test Design, it is usually when I am teaching it. So I found it hard to just talk about it and not have an interactive session with the attendees to give them a chance to practice it. Don't get me wrong - I had some interaction in my session, but not the sit-in-front-of-a-computer-and-try-things-out kind of way.  If I give this talk again, I will try to find some way to make it more interactive. (I may need more glow-in-the-dark straws.)

Back to the conference.  I was pleasantly surprised to bump into a former employee - a tester on a team that I managed about a decade ago. Wow! Still in Testing after all this time. AND attending a conference too! Double rainbows! We've hooked another one! ;-)

I've worked with many testers over the last decade and I hear from so few of them. I am always happy to meet them again and know that they are still doing and learning about Testing. I am happy to know when I help testers get better at what they do - I get ecstatic when I find out that they continue to see Testing as a Profession and participate in conferences and networking events!

StarEast had a few more surprises for me. At the Vendor Expo I bumped into Rick Craig - the tutorial speaker from when I first attended StarEast in '99. I introduced myself and told him how I remembered him. He was both pleased and suddenly felt older. Ha ha. We had a good chat.

Continuing my stroll through the Expo, the vendors were vendors. I still think some of the tools are the wrong ones for testers (like, totally and completely without value) but I did see some interesting ones that might have potential in certain circumstances.

As an independent consultant, most vendors weren't interested in me. I don't represent some big company with deep pockets to shell out on their bloatware documentation tracking systems. Of all the vendors, only one stopped to talk to me and really understand what it was that I had difficulty with their tools. He was fascinated by my knowledge of Agile practices, how testers fit on Scrum teams, and why these tools don't help. He asked me for my card and said that he was interested in contacting me to see if I can help provide some feedback for a future generation of tools that might work better. I gave him my card. I'll be curious to see if he contacts me. =)  I love to help - just ask!

My final pleasant surprise, and ultimately the best reason for attending the StarEast conference, came between sessions and during the meal breaks. I made an effort to sit with people I didn't know and engage them in conversations. Mostly I was curious to know what they did and what brought them to the conference. I was blown away by the passion that many of these attendees have for Testing. Wow. Some described to me the problems they face at work, the inequalities from other development team members (both in status and in salary), and the barriers preventing them from doing more. And yet, here they were at the conference to "learn from the best", to find ways to provide more value, make life better for them, their teams and their organisations.

They were dripping with hope and enthusiasm and I must say that I was overwhelmed with joy on more than one occasion following a conversation like this. If I had nothing else nice to say about the StarEast conference and received no other benefit from attending, witnessing the passion of the attendees was enough to fill my heart with passion and drive to get out there and do more, teach more!

I have been exposed to a lot of negativity over the last decade in several of the companies I have worked at. It wears you down after a while. You begin to lose faith in yourself and in the [testing/quality/value] mission. Attending StarEast changed that for me and recharged my spirit in a way that I totally wasn't expecting.

I was asked if I planned to speak at StarEast again next year. I don't know. I hadn't planned on it. I might not in 2012. We'll see about the following year or maybe I will speak at one of the other SQE conferences - like Better Software, Agile Development Practices or maybe even StarWest (since I haven't been to the Pacific coast yet).

There is so much to do in this area, in my "local" part of the world. It seems I am gaining recognition everywhere but here and I need to do something about that. StarEast helped recharge me and I hope I enlightened or entertained some attendees of my track session on Test Design and Exploratory Testing. Time will tell if I had a lasting impression on anyone. I hope I did. That wasn't why I went, though, and I got so much more out of it than just the opportunity to share some of what I have learned over the years.

I'm glad I went.

Read More

Testing & Programming = Oil & Water

I was watching a science program just now and it occurred to me that Testing is very much science. And then I wondered about Programming.

I started in IT over 22 years ago doing programming.  For me, the process of programming broke down to  three parts: figuring out the algorithm to solve the problem, implementing/coding the solution, and cleaning up the code (for whatever reason - e.g. maintainability, usability of UI, etc.).  It gets more complicated than that of course, but I think that about sums it up the major activities as I saw them. (SIDE NOTE: I didn't write those to mirror TDD's Red-Green-Refactor, but it does align nicely that way.)

When I think back on my experiences in programming, I don't see a lot of overlap with my experiences in Science (~ 8 years studying, researching and doing Physics & Environmental Science + teaching Science on top of that).  Science is about answering questions.  The Scientific Method provides a framework for asking and answering questions.  Programming isn't about that.  Building software isn't about that.  I'm having difficulty at the moment trying to see how testing and programming go together.

It occurs to me that schools and universities don't have any courses that teach students how to build software.  It also occurs to me that schools and universities provide students with the opportunities to learn and develop the skills required to build software well.  The schools just don't know they're doing that and consequently the students don't get that opportunity intentionally.

I'm not talking about learning to program.  That's trivial.  Building software isn't about programming.


Building software starts with an idea - an idea that someone will pay money for.  School courses to watch for here include - Economics, Entrepreneurship.

Building software requires people to work together. School courses that may apply - Business, Math/Finances, Psychology.

Building software requires people to work under constraints. "Project Management" is not really taught in schools, but there are many courses available to the public. I found this really painful to learn by doing. A whole world of insights opened up when I took my first PjM course. Reinventing the wheel really is dumb here.  I believe this should be formally taught in schools - in High School actually (the earlier, the better).

Building software requires people to solve difficult problems creatively.

This one is interesting.  I think there are many opportunities for people to learn this skill in school.  I know that we definitely covered this in Science.  I also know that Engineering programs teach students how to do this.  There are many more faculties and programs that this would apply to and they all have one thing in common - there's some formula or method for solving problems for some purpose.

The thing is, that purpose is different in each case.  See, here's where my mind is doing flip-flops.

In Science, when we solve a problem, the outcome is usually more information.  This information feeds back into the original idea to help us check the validity of our initial premise/hypothesis.  Sure, there are moments of free-form exploratory investigation, but I don't believe that happens a lot.  There are an infinite number of paths that any experiment may go in if you don't have a particular question in mind when you start, so unless you are trying to intentionally waste time and money, you will start with some question in mind before you start your experiment or investigation.

In Engineering, solving a problem is different in that the outcome is usually something real, something tangible, some application or system that fills a need.

The purpose of Science is not to build things but to answer questions.  The purpose of Engineering is to build things - and to do so safely, ethically, within desired parameters for intended purpose, and so on.

So where does that leave us in building software?  *Building* software is definitely an Engineering task... and then some. I am temporarily over-simplifying the process of building software to focus only on the requirements gathering, design, coding and deployment phases.

"Testing" comes from and is an integral part of Science, so how does it fit in with these software engineering/development phases of requirements gathering, design, coding and deployment?  Well, it doesn't.  It has nothing to do with them.  And yet, it has everything to do with them.

That is, from one perspective, at no time do you ever need to test anything to get through any of those phases.

That last statement, while true, kind of goes against everything I ever learned in school.  Whenever I did math problems, I always checked that I got the right answers against the solutions in the back of the textbooks.  Why did I do this? Why did I care?

I did it so that I could tell myself that *how* I solved the problem was correct - it got me the same answer that the textbook and my teacher cared about.  I discovered there were exceptions, of course.  That is, there were times when I got the correct answer but my method was wrong in some way.  Dumb luck does play a role in life and that was when I first discovered the evil twins named Type I and Type II errors.  (Side Note: I wonder if that's where Dr. Seuss got his idea for Thing 1 and Thing 2?  Hmm..)

So, the process of checking answers with the "approved" solutions, and handing in assignments for grading by teachers is a feedback mechanism to tell me that I've learned how to solve certain kinds of problems in ways that provide the desired results.  Let's assume for a moment that's a good thing.

Getting back to building software, you can go through requirements gathering, design, coding and deployment without ever once checking that you are producing the desired solution or results.  In the end, this is a monumental waste of time and money, and is completely incongruous with the initial premise that you are building a product/service/solution that someone will pay money for.  That's bad economics.  That's psychotic.

So how do we fix this?  What's the problem here?

Well, one problem is that we now have a question, we have doubt at the end of each of these phases.  We have a desire to learn if the end result of each stage and for the whole process is meeting [our/someone's] expectations.  The answer at the back of the textbook here will be provided by the people who are choosing to pay you and not your competitor for what you produce/release/ship.

Hey! Wait a minute!  Science helps us answers question!  Testing is a small part of that process.  The bigger process starts with a question that stems from some research or exploration of the initial area of interest.  This hypothesis is the part we really care about.  The "how do we go about gathering enough information to answer this question" part is something different and we should get people who know how to do this to either (a) do it for us, or (b) help us do it for ourselves. Then there is the analysis of that data or observations in context of the initial hypothesis.

But this is a different layer now!  We're adding a layer of "science" on top of "engineering".  That's weird.  That's like trying to mix oil and water together.  That is, they don't mix.  If you shake them up together you only end up with some cloudy mess that eventually will separate out again.

So what does this mean for building software? We need people who are skilled at engineering solutions, and we need people who are skilled at identifying and answering questions about the solutions being engineered.  I believe these are two, very separate skill sets required to be successful.

However, my experience in the Software/IT industry over the last 20 years has been that only one of those skill sets has really been identified as important or relevant -- that of the programmer or engineer in building the solutions.

This is a problem.  There's a huge knowledge gap here.

Schools don't teach you how to "test" in the context of software development.  Every single Testing "certification" agency I have met to date misses the mark. They don't teach you the correct skills. They "teach" you superficial documentation skills that produce information *like* the kind of information that is required.  That would be like me handing out Plumbing certificates to anyone who successfully completes the Mario & Luigi video games because these characters are plumbers in the games.  That's just not right.  Likewise, there is no actual "science" performed by teaching people to create scientific-like reports. (Although it happens sometimes that testers learn testing skills accidentally if they pay attention to what they're really doing.)

So, where are we here? Building software requires layers of intelligent, creative effort and problem-solving abilities.  These layers are complementary and require different skill sets.  Just like you wouldn't hire an accountant to deploy your systems, I don't think it's wise to hire a programmer to provide valuable testing insights into the development process.  It's the wrong skill set.

Oil and water, or oil and vinegar - the analogy holds for me. Testing is something completely different from Programming and building software.  It's a layer on top to help you know that what you are doing is on target for what your paying customers are expecting.  Some might call that value.

Read More

Software Testing "Popcorn" button

I made myself some microwave popcorn for a snack just now.  Placed the popcorn bag in the microwave, pressed the 'popcorn' button and then 'start'. Someone next to me said: "There's a popcorn button?" Um, yes, there is.  In fact, there has been a 'popcorn' button on every microwave oven I've ever seen.

I explained to my colleague that the recommended time on the bag (in this case it was 2 min 30 sec) doesn't work on every oven.  Different ovens have different power output and so the actual cook time may vary.  If I go with the default time, it might burn or be under-done and leave too many unpopped kernels in the bag.  You could figure out the correct time in a few ways.


Method 1: Math
Start by taking a look at the power output of the oven. Full-size ovens deliver 1,000 - 1,600 watts of power, and mid-size ovens yield 800 - 1,000 watts. Higher wattage heats food more quickly.  If it's not explicitly written on the bag, assume the default popcorn popping time is for a 1,000 watt oven.

Use a time/power ratio along the lines of: ( t_your_micro_oven / P_your_micro_oven ) = ( 150 sec / 1000 W)

I converted the 2:30 recommended time on the bag to 150 seconds for convenience.  The power of your microwave oven should be written on the back somewhere, probably close to the plug.  This leaves your popping time as the only unknown variable in the equation, so it should be straightforward to solve.

That might work.

Method 2: Brute Force/Iterative
Put the bag in the oven and follow the instructions exactly.  Make no changes.  When the time is complete, take the bag out and put the contents in a bowl.

If you are happy with the result, congratulations! You are done.  If not, we will need to change the cooking time for the next bag.

On a piece of paper (or in a document on your computer if you prefer), make some observations to capture things like: time settings used, quality of the popcorn output, taste, number of unpopped kernels, and so on.  Keep a note next to the microwave oven or on the container where you keep more popcorn bags -- a note to reference these 'test results' so that you can have something for comparison the next time.

Basically, you know where I'm going with this.  Pop the first bag with the default recommendations and keep varying the subsequent popping times until you are happy with the result.  This will require "n" bags to iterate through until you are happy with the end result.

This should eventually produce high-quality results.  It may take you some time and will be costly to do it this way as you may have to go through many bags until you get it just right.

Method 3: Popcorn button

Put the bag in the oven, press the 'popcorn' button, serve and enjoy.

I'm not certain how this works.  I shall call it "magic" for now.  I can speculate that perhaps the microwave oven manufacturer has a team of engineers dedicated to the "Perfect Popcorn Production" using their equipment and are responsible for programming the correct power and time settings into their ovens.

Maybe they use Method 1 above.  Maybe they use a combination of methods 1 and 2 above.  Maybe it's something else.

The point here is that someone has already done the thinking for me so that I can focus on the quality of experience of using their oven.

Wow.  I have so many comparisons and analogies back to Software Testing running through my head right now.

One that jumps to mind is development & testing terminology/jargon.

• When is a test not a test? When it is a check. When it is an inspection. When it is a question. and so on.
• Are your [agile] development sprints 2 weeks? No? Why not? That's the default value, so it should work for you, right?

What if software organisations were responsible for their own 'popcorn' buttons? That is, where the 'popcorn button' is a way to communicate the methods and models used within their organisations that produce the desired 'quality' result.

As a consultant, one of the first things I do is watch and listen to the development and project team members. I need to understand their terminology and way of doing things.  That helps set a reference frame for me.  If I want to make an improvement or change somewhere, I need to have an understanding of how to do that on their terms, not terms according to some industry standard or certification terminology dictionary that may not apply.

What would be the point?  Well, I think it would be handy if we could abstract out some of the desired practices from the terminology and implementation details that are specific and custom to every organisation, team or project.

Now that I think about it.  Maybe, we as consultants are the "Popcorn Programming Facilitators."  That is, the company wants to implement an automated regression testing activity.  What does that look like? How can that work for us? What kind of output do we get?

Some days we identify the required popcorn buttons (e.g. Regression Testing, Bug Tracking System, Produce Status/Progress Reports). Some days we help program them.

What do you think?

Read More

Using MS Outlook to support SBTM

Okay, to recap, Session-Based Test Management (SBTM) is a test framework to help you manage and measure your Exploratory Testing (ET) effort. There are 4 basic elements that make this work: (1) Charter or mission (the purpose that drives the current testing effort), (2) Time-boxed periods (the 'sessions'), (3) Reviewable result, and (4) Debrief. There are many different ways that you might implement or apply these elements in your team or testing projects.

Let's take a look at tracking the testing effort from strictly a Project Management perspective. Years ago, when I first became a test manager, I was introduced to the idea of the 60% 'productive' work day as a factor to consider when estimating effort applied to project schedules. That is, in a typical 8-hour workday you don't really get 8 complete, full hours of work from someone. I don't believe it's mentally possible to get that. The brain needs a break, as does the body, and there are many natural distractions in the workplace (meetings, email, breaks, support calls, stability of the code or environments, and so on), so the reality is that the number of productive working hours for each employee is actually something less than the total number of hours they're physically present in the workplace.

That 'productivity' factor changes with each person, their role and responsibilities, the number of projects in the queue, and so on. Applying some statistical averaging to my past experiences, I find that 60% seems about right for a tester dedicated to a single project. I have worked with some teams that have been more productive and some much less.

So what does this look like? If we consider an 8-hour day, 60% is 4.8 hours. I'm going to toss in an extra 15 minute break or distraction and say that it works out to about 4.5 hours of productive work from a focussed employee in a typical 8-hour day. Again, it depends on the person and the tasks that they're performing, so this is just an averaging factor.


4.5 hours is an interesting number. In the SBTM framework, the "Time box" length for the test session revolves around a normalised value. The original SBTM presentation ("How to Measure Ad Hoc Testing") suggests that 1 "Normal" session = 90 minutes (+/- 15 minutes). Your "normal" time box session may be 90 minutes, it may be less. Customise it to suit your team's needs and fit.

I'll use 90 minutes for now because it's the default recommended value to have a focussed test effort to get some solid testing done. 90 minutes. That's 1.5 hours. So 3 * Normal = 4.5 hours.

Therefore, from a project management perspective, 3 Normal sessions is an average tester's target productivity level for completed test sessions in an average 8 hour day.

It doesn't seem like a lot, three 90-minute sessions, but you'd be surprised at the number of distractions that happen throughout the day and development sprints/cycles that can prevent a tester from completing 3 sessions in a day.

Recently, I've been working with a tester on an Agile team who has been facing a high distraction level. As a result of all the interruptions for quick checks, reviews, design meetings, and so on, he has been struggling to complete 3 sessions in a day.

I should mention that I *AM NOT* asking him or anyone else on my team to complete 3 sessions in a day. I am not concerned with the numbers or the math of tracking the sessions completed, especially because the development team environment is very integrated and highly collaborative.

What I am concerned about is trying to find ways to block off his time so that he can get some solid testing done, uninterrupted. One of the powerful aspects of the time-boxed session is to reduce or minimise the distractions so that your brain can maintain the focus on the testing problems at hand. The more the distractions and interruptions, the more likely that something important or interesting will be missed.

Enter MS Outlook. When I look at my Outlook calendar it is riddled with meeting requests throughout the week. I have recurring meetings, impromptu meetings, lunch and learns, scheduled debriefs, sprint planning, retrospectives, demos and so on. I'm the Test Manager/Lead, so that's expected. When I look at my tester's calendar, it is fairly blank/open, save for the daily stand-ups and a few other weekly team meetings. It seems counter-intuitive that someone with an open calendar, so much available 'free' time, should be unable to block off time-boxed periods dedicated to testing specific features and charters.

So I've started to schedule his test sessions in MS Outlook by scheduling Appointments. At the start of the day, we'll sit down and come up with 3 important charters that we'd like to cover in the day. We block off one session/appointment in the morning and two in the afternoon. The "Subject" is a summary of the charter and priority. If the session involves another Dev or team member, they can be included in the appointment, which is now a 'meeting request'.

Outlook is a convenient tool since it (a) is available on everyone's desktop, (b) has built-in reminders, and (c) allows the tester to *see* all the free time available between scheduled test sessions. It's a way for him to try and regain control of the *uninterrupted* test sessions.

SBTM isn't intended to be a time-tracking tool. It's a productivity-enhancing tool. By keeping focussed on the charter, you give your mind time to think about the important things -- to learn about the system and the software, to apply the appropriate tools and techniques, to make the best observations you can in the time you have available. Performing good Exploratory Testing is a delicate and complex thought process.

I see my job as doing whatever I can to help provide the best environment possible for good testing to succeed. I never would have thought that Outlook would factor into that. Who knew?

Read More

Test-Driven Development isn't new

I used TDD as an analogy to a tester today to explain how logging bugs in a bug tracking system drives the development. A bug report represents a failing test (when you verify that it's really a bug that is) according to some stakeholder need/want.

In Test-Driven Development, the programmer writes/automates the test first that represents the user story that the customer/user wants. The test fails. The programmer then writes enough code required to pass the test and then moves on. (refactoring code along the way, etc..)

It's much the same with regular system testing (i.e. in the absence of agile/TDD practices) where a tester identifies and logs a bug in the bug tracking system. One difference is that these bug reports/tests aren't always automated. (Okay, I've never seen anyone automate these bug reports/tests before but I like to believe that some companies/dev teams out there actually do do this.) That doesn't change the fact that a bug report is the failing test. Even if it's a manual test, it drives the development change and then the bug report is checked/retested to see that the fix works as expected.

Bug regression testing, then, is a requirement for good testing and system/software development, not an option.

So, while the agile practices of TDD and others may seem new, I see this one as a retelling of a common tester-programmer practice. If anything, I see TDD as an opportunity to tighten/shorten/quicken the loop between testing feedback and development. With practice, TDD helps programmers develop the skills and habits they need to create code and systems with confidence -- to know that as the system grows, the specific needs of the customers are being met every step along the way. No one gets left behind.

How can we, as testers, help? If your programmers don't practice TDD or automate tests, start investigating ways that you can do this. Investigate Open Source scripting languages. Engage your programmers in discussions of testability of the interfaces. There are many articles and presentations on the internet on the topics of test/check automation, frameworks and Domain Specific Languages (DSL).

Start reading. Participate in discussions (in real life and online). Start developing scripting skills (I recommend Ruby, of course, especially to the tester newbie). If you don't feel confident with your programming skills, help hire someone onto your test team that can help all the testers advance their skills, knowledge, and productivity in that area.

Be the Quality Advocate by putting your words into practice. You want your programmers to start practicing TDD? Show them how you can do it. You are already doing it - scripting/automating the checks that demonstrate a bug failure is just the next step.

Start by automating a single bug failure. Take it from there.

Read More