SBTM is not ET

There's a subtle but important distinction that I'd like to talk about. Session-Based Testing is *not* Exploratory Testing. Please stop using those terms interchangeably because they're not.

Exploratory Testing (ET) is a testing approach that puts the emphasis on real-time learning, test design and test execution, as opposed to a more "scripted" approach that puts the emphasis on the separation of these activities - separated in time, space, and usually with copious amounts of documented artifacts.

When I first started in I.T. over 20 years ago, any testing I did as part of my programming contracts were exploratory in nature. I didn't call it 'ET' at the time and I certainly didn't approach it with the same discipline and formality that I do today. Back then, Programming was my main focus and testing was just something I did as required along the way. Ten years later (or about 12 years ago depending on your perspective), I took a workshop class on "Test Case Design" with Ross Collard. That was an amazing class that opened my eyes to a whole new world of analysis and problem solving that I didn't know before. Cool!

After that workshop, I had plenty of opportunities to practice what I learned, try new techniques and tools, and explore additional testing ideas thrown out onto the just-budding software testing mailing lists. One of the things we discussed in Ross' class was the role of "ad hoc" or informal testing. I don't have access to the data, but some study-or-other at the time (90's sometime?) showed that ad hoc testing failed to produce the same amount of testing coverage that formal test design analysis would.

Okay, I buy that. To paraphrase: guessing ideas off the top of your head consistently produced less coverage than having some structured analytical approaches/techniques/heuristics/models at your disposal. Okay. I don't need a formal study to tell me that.


So what's different with Exploratory Testing? Well, when I first learned about ET at the turn of this century, it instantly clicked with me. Rather than the "guessing" attitude normally associated with "ad hoc" testing, ET clearly defined the testing approach in a way that made you think. You learn something; you design something; you test and observe something; repeat. Note that nowhere in there does it say "take a wild-ass guess and call it good, complete or even 'good enough' testing coverage."

Before I was introduced to ET, I had spent several years practising and training other testers on test design techniques. That helped me fill in the "test design" step of ET. That step is the weakest link with most of the testers I have met and spoken with over the years who have tried and given up on (i.e. failed with) ET. You can't really fake your way through test design. That's why I make it an important part of my hiring/interviewing process (you can read the article online).

So, what *don't* I like about ET? There's just one thing really. The ET approach formalised the learning, test design and test execution aspects of testing, but not the interpersonal communication aspect of it.

The 'scripted' (waterfall) approach to testing relies on the documenting (and maintenance) of hundreds or thousands of test cases, each with their own set of pre-conditions, steps, expected results, and so on. While the value of these documented test cases may be questionable, one thing going for it is that you can share these test ideas with other people quite easily. (They're documented; pass it on.)

In ET, not so much. If the important parts of testing takes place in your head as you process all of the inputs and information, and compare them with explicit and implicit requirements and expectations, in order to assess the quality of the A/SUT, then when/how do you share those test ideas with other people (testers, developers, business analysts, etc.)? Well, you don't. Or rather, ET alone doesn't give you any advice for communicating test ideas or testing coverage with others.

Enter "Session-Based Test Management" (SBTM) or just '"Session-Based Testing".

Aha! After a year or two of using ET, I instantly found the merit in SBTM. SBTM provides the framework that you can wrap around an ET approach. It is a way that you can manage the testing effort. It has four main elements: develop specific charters, time-box an uninterrupted work session, create a reviewable result, and review/debrief the session afterwards.

Here's the catch: it is *not* a testing approach! It is a test management framework. Actually, when I teach/describe it to others, I sometimes refer to it as "Session-Based Task Management."

I have taught SBTM to programmers as a way to help them manage their time and reduce the number of interruptions during a work day. I have also successfully implemented SBTM in a waterfall organisation where very little ET was ever performed.

Yes, you read that correctly. I have even wrapped SBTM around a *scripted* testing approach.

Eek! Egad! Gadzooks! Isn't that blasphemy?

Well, actually, no.

You see, I have found that SBTM is an incredibly powerful tool for a test manager. It gives you insights into aspects of testing that you might never have without it.

The four main SBTM elements provide a solid foundation to managing your work, and can be transferred to activities other than just ET. For example: programming, writing, organising/cleaning your basement, any consulting work, and so on.

The original SBTM framework included some Perl scripts that I have long since stopped using. The original archive included a session sheet template, but like any template you can modify it and tailor it to your needs. (If in doubt, just ask James Bach for his thoughts on Test Plan templates! :)) That's the main reason I rewrote the SBTM scripts in Ruby - so that I could customise the session sheets to the needs of the projects I worked on. So, for one project I added a section to the session sheet, and for another project I completely removed the TBS metrics; my Ruby scripts are flexible and can handle the changes easily. (ASIDE: I haven't made this customisable script publicly available on my site yet. Send me a note if you are interested in trying it.)

In fact, if you follow the intent of SBTM, you don't need to use the session sheet template or scripts at all - as long as you have some agreed-upon reviewable result that you can later debrief. In this way, I have heard of some test teams that have implemented SBTM using Wiki's, and others that have integrated old Test Case Management systems into the process. Sounds cool and innovative to me!

So, what's my gripe? In the last several weeks, I have read several times that Exploratory Testing includes time-boxed, chartered sessions with reviewable results. Umm, no, I'm pretty sure you're confusing the framework with the approach, the wrapper with the content, the book format with the story.

If you have implemented SBTM on a project, I can make no assumptions about what testing approach you are following. Likewise, if you include ET in your overall testing strategy, I won't assume you are using SBTM to manage that effort.

If you want to talk about ET or SBTM, please try to describe them in the correct context. It will make it less confusing for beginners and other interested parties. Granted, together you have a very powerful combination. But Superman is still super in a different suit. =)

Read More

Time - Bane or Innovation Catalyst?

Time. What time is it? How much time do we have? When do you want/need it? What's the deadline? I need more time!

If we had all the time in the world for software development, would the delivered results really be of better quality?

A co-worker at a past employer wrote the following when someone sent an email submission for a fun, internal contest the day after the deadline:

The contest ended a long time ago. Trying to submit something now is like submitting your late university assignment.
One of my profs told me:
"I don't care if you have something that's better than all the works of Shakespeare. If you can't get it in before the deadline it's worth nothing to me."

Ha, ha. It was intended as a funny remark at the time but there's some truth in there too.

So, if someone submits an assignment "on time" but of lesser value/quality than they might produce if they had more time, would they still continue to work on their opus or would they give it up to move onto the next project? Do we (as a collective group of intelligent human beings) lose out by putting Time ahead of Quality?


The traditional "Project Management Triangle" puts the emphasis on: functionality/scope, cost and schedule. An experienced consultant can tell the employer: pick/fix any two and we can estimate the third.

I noticed years ago that "quality" isn't in this "triangle". As a novice, I took "scope" and "quality" to be part of the same point. Clearly I was mistaken. When people are focussed on delivering something, on time, at a fixed cost, everyone interprets "quality" in different ways.

I think the Agile manifesto/movement is an interesting response to the "traditional" (a.k.a. Waterfall) approach to software development. It takes the same 3 constraints (of scope, cost and time) and changes up the order of activities to integrate quality into the deliverable products. This is done by embedding customer involvement (via collaboration, user stories, automated acceptance tests) and rapid delivery releases to allow for quicker feedback into the design and implementation. For example, in a traditional/waterfall project, it may take anywhere from 6-18 months to find out your interface/implementation fails to meet the needs of the customers. Or, using agile methods, it might take anywhere from 2-14 days. Your choice.

So what about software testing?

In every waterfall project I have worked on, development always delivered software late into the "test" phase. This meant less time to provide feedback, because the release deadline was fixed. Time is my bane here. I've got less of it and need more of it! ... or do I?

If I stick to a waterfall approach to testing - i.e. develop & document test plans, test strategies, test cases, execute the tests, log the results and communicate the summaries - then, no, time is not my friend here.

But is it a requirement to do testing this way? Whose requirement? How much does their opinion really matter?

I watched my son play a game recently and describe the "glitches" (his lingo, not mine) to his younger brother so that he could try and work around them. I'm pretty sure my boys don't care whether the software team used waterfall or agile methods, or how well their test cases and processes were documented. They found bugs in their game, are annoyed by them, and figured out ways to work around them. Sometimes they just give up on a game altogether.

Personally, I'd say that the customer doesn't really care about how you do your testing - as long as the end result has good enough quality that doesn't interfere with their intended use of the software or system.

Here's a secret: Nobody cares.

Some lawyers may pretend to care when they are paid to do so, but the reality is that I don't know of a single tester who has ever been charged with manslaughter for failure to document critical test cases that may have caught the bugs that resulted in loss of life.

The FDA doesn't care. Their lawyers tell them that they should care about documented tests and results, so they impose regulations. But the FDA doesn't really care about your documented test cases or test processes. What they really care about is that a minimum standard of due diligence has been performed to demonstrate that a particular product will not harm anyone. That's it in a nutshell. You may not even need testers to achieve that level of quality either.

I could go on, but I think I made the point - nobody cares how you do your testing as long as the collective development effort produces a quality product. You remember "quality" - it's that thing that project managers leave off their project management triangle.

So, if we disregard the premise that testing needs to happen in a "waterfall" fashion, what's left? Well, what do we know? We know that (1) we don't have a lot of time, and (2) we have a lot of features to cover. Oh, and it's also very likely that (3) you have a limited number of resources and people - most likely less than what you'd probably like. (Hey, if we're screwed on the 'time' factor, why not get screwed on the 'cost' factor too, right? ;))

So where does that leave us? Time to innovate! Time to become agile! Talk to your customers; collaborate with your developers and business analysts/product managers; learn the software and functionality as you design and execute the tests because there really isn't time to do those things separately.

Risk-based testing (RBT) works on the premise that there might be something bad/undesirable that could happen, so why don't we start by looking in those places first. RBT is also an appropriate response to the statistical impossibility of complete testing coverage for any useful software program with more than 2 lines of code. That is, if it will take an infinite amount of time to test something, how about if we narrow it down to just some of the areas that we think might be risky in some way (i.e. popular, critical, complex, and so on).

What else can you do? You have a lot of features to cover in a short amount of time. Well, start by ditching all the test documentation requirements and focus on: what is necessary to establish a minimum level of understanding of what's going on.

Do you really need all those documented steps for every test case? No, you don't. Unintelligent automated systems and robots need step-by-step instructions, humans don't. And most humans don't follow the steps consistently either, so just let that one go. Instead, describe the scope of the testing you want to do using checklists and decision tables. The important things need to be discussed in person to ensure clarity of requirements and information, but everything else should be fine with using point form.

Worried about how you will capture the test results if you are denied the Pass/Fail test status column? Work it out! Figure out a solution that fits your project's (and organisation's) needs. There are a number of far more useful alternatives out there - e.g. application logging, screen captures, note taking, and so on.

If you don't have enough time to complete a project using the same approach you've used in the past, it's time to try something new. Time to think up of new solutions, new processes, and identify/create new tools to help you reach those goals.

The end goal is a high quality product.. or maybe just "good enough" quality depending on your situation. The end goal is not to produce sparkling, publishable test documentation. (If it is, consider changing your title from "tester" to "test biographer")

Don't lose sight of what's important. What will you do with the time you've been given? How will you choose to react to the situation?

Read More

What I learned about Testing from a crazy ex-girlfriend

I was reminiscing with a tester colleague today about how our mothers used to mess with our stuff when we were younger and how it really got on our nerves.

Picture the scene: you have a desk in your room that's plastered with papers and stuff everywhere. And you know precisely where everything is. It's your mess after all.

Enter the mom. She looks around, maybe she's come in to drop off some laundry or to complain about the state of your room or whatever. You aren't around. She starts to tidy. She tidies the papers on your desk and arranges your action figures/books/pencils/Lego/rubber band collection/whatever into a neat arrangement of some kind.

You return. "Ahhhh! Where's my stuff?!?! You changed the order! I can't find anything now! Don't touch my stuff!!"

Your mom, now hurt because she was "only trying to help," vows to never touch your stuff again unless someone's life depends on it. Maybe. We'll see next week.


We chuckled over the memories, but the connection my colleague made was how that ability to memorise tiny details and the placement of certain pieces of information in a messy desk was perhaps already the mark of a good tester. When you look at a computer screen, you take in all the details and it becomes a new mess of our own design that we track in our heads. We notice when details are moved or changed. If we think there might be something different, if we have a hunch, we can use tools to help us verify it or we can check with an oracle of some kind. It's that ability to make a connection, develop a hunch and act on it, that makes a good tester.

So, you're probably wondering "where does the crazy ex-girlfriend come into the story here?" Good question. No, I haven't forgotten. This is the spot.

In university I had .. er, how would I describe it now.. a short-lived relationship with a girl who was definitely the outgoing/extrovert type. One morning, I met her before classes started and didn't meet up with her again until after lunch when we both had a free spot in our schedules. When I saw her after lunch, I did a double-take but I wasn't sure what I was noticing. Something was different but I couldn't put my finger on it.

Then it hit me! Her earrings were different. She had two piercings in each ear and in the morning she had studs and stars (in that order) and in the afternoon they were reversed - stars and studs.

Being the attentive boyfriend, I asked her if her earrings had changed their places or if I was just losing my mind. She looked at me and didn't say anything for a minute. Then she said 'yes', she was bored in one of her morning classes and decided to switch the order. Then she got mad at me. She was upset that I had noticed because she didn't want anyone to notice that she had changed them.

Umm, really? That includes me? So, I don't get any points for noticing you and paying attention? Okay, I don't get this relationship stuff. I think our relationship lasted perhaps another 48 hours. Oh well. C'est la vie.

In retrospect, that was another example where somewhere in my head I had made the connection - something was different even if I didn't know what. Then I began the process of methodically going through the list of possibilities and checking them off one by one - was it her hair? her eyes? makeup? lipstick? top? something she was carrying? a scent? a mark? necklace? wait, did I check the ears? hey, there are 2 thingies there - could they have changed? Spider sense tingling.. better consult the oracle and check if we have a match. Bingo!

Success in finding the difference! Failure at love. =( You win some, you lose some.

I didn't know it at the time but the best relationship was still to come! =) And on that note, sometimes you notice the little things, and sometimes someone has to hit you over the head with a frying pan to tell you to open your eyes and see what's right in front of you!

Ah, but love makes you blind, doesn't it? ;)

Read More

What skill does Exploratory Testing require?

I've just been challenged with a sobering reality.

I've heard the term "Exploratory Testing" used many times over the last few years by developers and testers at various gatherings. I've practiced it myself for over 6 years in various black-box system testing efforts. When training new testers on my team, I provided them with foundational concepts in context, risk, scientific method, test techniques and communication. Then over the course of several weeks, I reviewed their test sessions and provided feedback during debrief sessions to improve their understanding and application of the various testing skills required to be efficient and effective.

People have told me that I have really high standards, and perhaps I do. To me, testing is a passion and fun, and quality is an ideal achieved through effective communication and interactions with all the stakeholders on a project.

But that's all besides the point. If the question is "what is Exploratory Testing and how do you do it?" then my standards and expectations from team members are irrelevant.

ET is simply an approach to testing software where the tests are not predefined and the focus is on iterative learning, test design and execution (to paraphrase a simplified definition).

How someone learns, how someone designs tests, how someone executes those tests - these things are not defined by any standard; they are applied differently by different people. ET can be performed by anyone. There aren't any requirements for how well or thoroughly someone should perform it.

To quote from the animated movie "Ratatouille": "Anyone can cook. But I realize, only now do I truly understand what he meant. Not everyone can become a great artist, but a great artist can come from anywhere."

So, when I hear the term ET thrown around, I have about as much understanding of how they're testing as I do from a development shop that uses the term "Agile". That is, I don't know anything about what it means to them, how they're applying it, how effective it is, or how it compares to my standards/expectations.

I've been reading articles and research lately comparing ET and Test Case-driven Testing (TCT) approaches, and it never ceases to amaze me how stats and research may be twisted to support everyone's beliefs about which is better than the other.

Developers and Product Managers who have worked with me understand the quality of the information and feedback that my testing style provides. They have said that it is a whole new level of testing feedback they've never seen before. It makes me feel good to hear that - that I'm providing a valuable service.

But when I read one of these comparison articles, I have to assume that the ET applied in the research studies aren't at the same level that I apply it. I have to accept that. I may not like it, but that's the reality. To me, the same research applied would likely show that Agile and Waterfall aren't really all that different in terms of produced output. Sigh.

Am I missing something?

Read More

It's My Fault

Many years ago when I was in university, a friend approached me one day to ask if I'd be interested in going skydiving with her. She said she wanted to go but she wanted the company. I went. It was on my list of things "to try at least once before I die" so why not. =)

There was a full day of training for newbies - which included hands on (wearing the jumpsuits, learning the equipment, how the chutes are packed, jumping off picnic tables, etc.), videos, in-class instruction and discussion, and ended with an exam. The written exam was the last thing before everyone suited up for the plane ride up.

I was a bit surprised when 2 of the instructors pulled me aside after the exam to go over my test results. They wanted to discuss my answer to the final question. I think it was something like: "If something goes wrong, whose fault is it?"

I had run out of room trying to fill in a suitable answer. I had wondered why they didn't give much space to write. My answer was something along the lines of: "Well, if the wind blows me off course and I float into power lines and die it's nobody's fault. Or if I land in some marsh and get eaten by alligators I don't blame the alligators." and so on .. until I ran out of room at the end of the page. (There are no alligators in this part of Canada, by the way.. unless I land in a zoo.)

Looking back at my response now, I was doing what a good tester might do and thought of how many different ways something might "go wrong". But I missed the point. My instructors patiently kept rephrasing the question to see if they could get a different answer from me.

One of them blurted out: "Paul, is someone holding a gun to your head and asking you to jump out of the plane?" To which I replied "no." "Okay, so who is making you do this?" "No one," I replied. Wrong answer again. I still didn't get what they were trying to say.

Finally they explained to me that *I* was the only one making myself do anything here. If something goes wrong, it's simply my fault. Then they told me to cross out what I had written and write in big letters: "IT'S MY FAULT."

They explained that the whole exam didn't matter - none of the answers mattered except for this one question. If I didn't answer this question correctly then I wouldn't be allowed to jump. It was like signing the waiver.

I wrote what I needed to. I went up in a perfectly good plane and then I jumped out. It was an amazing experience and we all had the same silly grins on our faces when we met up with each other on the ground again.

I learned an important lesson that day. I didn't realise that I deflected responsibility for my own actions and decisions. It wasn't intentional, it was just how I thought about things in such abstract ways. When there are billions of different possible events that may occur from any given moment, why would I even consider taking responsibility for one of those outcomes if things go bad?

Well, it's easy. You make the choice yourself. Assuming no one is holding a gun to your head or threatening to harm your loved ones, then the choice is yours. When you make a choice you own the responsibility for the outcome. It's your fault if something goes wrong. It's your fault if something goes right.

Working in the software industry all these years, I have witnessed many times when people don't take responsibility for the decisions they make and how they choose to act - at all levels within the organisation. I have seen employees who whine about not getting the raise or praise that they think they deserve but they don't put in the care/attention/effort required. I have seen managers look for scapegoats when projects/things don't go as planned. I have witnessed the irrational, childish backtracking of senior management who refuse to admit that they ever did anything wrong or made a wrong decision.

Why is it so hard for people to take responsibility for the choices they make?

We make choices every day. Everyone does. Sometimes we have help making them, sometimes we don't. Some are big choices, others not so much.

What about the choices people make when they are at work? How they act? Or rather, how they choose to act towards others?

No one's putting a gun to your head and telling you to "test!" (Well, there was that one scene in the movie Swordfish that was quite entertaining, but that's the movies for you.)

Testing is not easy. Software Development is not easy. If you think it is, it's likely because you don't understand the problem.

As testers, we should aim to provide as much required information about the product/system/service as possible so that the stakeholders can make informed, timely choices. That is, we help others make big/important choices - hopefully good choices. Good information too late doesn't help either. (Note: what the stakeholders choose to do with that information is a different matter. Sometimes all we can do is just provide the information. How they use it is up to them. Good choices are not automatic, even when you do have the information you need in time.)

If the choice you make turns out not to be the right choice (i.e. you get an undesired outcome), then admit it, learn from it, and try not to make the same mistake the next time.

That's the rub - that process right there. If there is no admission of responsibility for the choice you make, do you learn from it? I don't think so. When I see someone deflect responsibility or look for a scapegoat, they aren't learning anything from the situation and will likely make the same mistakes the next time.

I have seen good leaders admit mistakes/poor choices to their departments. Admit that they've learned from it and will try something new or different to try and get a better outcome. That's the kind of leader I like working with/for. Someone who learns. Someone who grows. Someone willing to admit shortcomings and knows how to leverage the strengths others to help make better choices in the future.

No one is omniscient, so why hide your mistakes? There's some risk in every choice you make. That's life. Don't whine about it or blame others if things don't go your way. Take responsibility, learn from it and try not to make the same mistake again.

When was the last time you said "it's my fault"?

Read More

Reflections on AYE

I had the privilege to attend the Amplifying Your Effectiveness (AYE) conference this year. Finally! I've mentioned that conference in several of my presentations and talks over the years, so I was pleased to finally be able to make it out to Phoenix, AZ, this year for the event.

There isn't much I'm going to say about the conference at this time. Browse the conference web site to get an idea of the kinds of sessions and discussions that happen there. Reading about it doesn't do it justice.

Everyone I know who has attended an AYE conference in the past has told me how wonderful it was and how much I would enjoy it. They were right. Even though I was told to expect it, and I hoped it would live up to those expectations, I felt a kind of relief and happiness in knowing that I wasn't disappointed.

In my experience, I've noticed that testers tend to start out only interested in developing their technical skills (e.g. programming/scripting language, automation tools, databases, etc.) - if they show any interest at all in professional development related to their jobs. If you take your career and profession seriously, there will come a time when you realise that the technical skills aren't as important as communication and people skills.

Why does learning happen in that order? Does it make sense? Build People skills upon/after your Technical skills? Shouldn't we start with a good base in communication, understanding and relationship-building, and then work to develop technical skills and expertise afterwards?

Should we focus more effort on teaching teenagers in High School how to understand and communicate effectively with each other to prepare them for developing good working relationships in adulthood? Why is it that the High School/teenage experience tends to do the opposite?

I've seen my children play nicely with other kids, even strange/unknown children in the playground quite nicely. So when do adults forget how to be nice to each other? To play nicely or fairly with others? When do they forget how to show respect and trust, and act with integrity and honesty towards others?

At AYE, those values were apparent. I saw kindness, respect, trust and honesty in abundance. It was overwhelming at times. I wasn't expecting that. I felt a sense of instant community at the conference.

Learning happened. Sharing happened. Discussions and conferring happened. It was fun.

It was everything that I hoped how a group of adults would act. I wish that was a more common occurrence. I wonder what we could accomplish if more people acted that way.

Read More

The Price of Clarity

I went to a "Startup Drinks" event tonight and met some interesting entrepreneurs with lots of great ideas. One thing I discovered (about myself) is that I'm going to need to get on Twitter in the near future. ;-)

One person mentioned a tweet that he posted yesterday from the "Monday Morning Memo" about advertising. The tweet was about the topic "Why Most Ads Don't Work" (by Roy Williams, author of the book "The Wizard of Ads") and is summed up in the nine secret words: "The Risk of Insult is the Price of Clarity."

This was an interesting phrase. I instantly thought about the feedback that we, as testers, give about the products we test. You know what I'm talking about.. the Ugly Baby Syndrome. Sometimes we have to be the bearer of bad news. Hopefully, we can find a nice way to say it, but ultimately, I believe that the risk of insult is the price of clarity.

Thinking about my motto "ubi dubium, ibi opportunitas" (where there is doubt, there is opportunity), we often exploit the vague, unclear areas of products and applications.. because there will usually be lots of bugs there. The bugs we report are about more than just little typos and minor UI issues, though. If we do our jobs well, bug reports can be the catalyst to help bring clarity to features/requirements/implementations and consensus among the stakeholders about the app/system under test.

When I test, I report every bug. It's all information. I can't judge when something will be worthwhile (to report) and when something won't. You won't recognise that your baby is ugly because it has thousands of little things wrong with it if you don't report the thousands of little things. It's not often that you'll test a system where you find a "nose" in the completely wrong place, or an "ear" where an "eye" should be (to continue the 'baby' analogy). Those moments are easy - you *have* to risk the insult or risk completely failing to meet the customers' (and other stakeholders') needs.

If you test, and report bugs, in a way so as to "not offend" (i.e. by watering down the message, not testing certain features too hard, or choosing not to report certain bugs) are you really providing a helpful service?

What are you willing to risk to ensure clarity on your projects? What do you think?

Read More