What is Exploratory Testing?

What is Exploratory Testing (ET)? I am asked this every once in a while and I hear a wide range of ideas as to what it is. This is one of those topics where Wikipedia doesn't really help much.

For some, ET is just "good" testing and the reason we say "exploratory" is to distinguish it from bad testing practices. Unfortunately, bad, lazy, haphazard, thoughtless, incomplete, and incompetent testing is quite popular. I won't go into the reasons or supporting evidence for this disgraceful blight on the Software Development industry at this time. Suffice it to say, I don't want to be mixed in with that lot either, so I am happy to describe what I do as something different - something that is far more successful and rewarding when done well.

Okay, so if ET = [good] testing, what is testing then? According to Cem Kaner, "software testing is a technical investigation conducted to provide stakeholders with information about the quality of the product or service under test." This definition took me a while to absorb but the more I thought about it the more I found it to be a pretty good definition.

If you ask Elisabeth Hendrickson, she would say that "a test is an experiment designed to reveal information or answer a specific question about the software or system." See, now I really like this definition! I studied Science in university and I love the way this definition reminds me of the Scientific Method. The more I learn about testing software, the more I find similarities with doing good Science. (By the way, if you want to learn more about how to do good testing, I highly recommend you read up on the Scientific Method. So much goodness in there!)

So, is that all there is to it? Testing = Science, blah blah blah, and we're done? Um, well, no, not really. ET has its own Wikipedia page after all!


I dislike the first line description of ET on the Wikipedia page. I dislike it because it is incomplete. It says that ET is "concisely described as simultaneous learning, test design and test execution." ... AND?!? And then what?! This definition is kind of missing what happens after you do the execution part. That's really important.

Elisabeth Hendrickson offers a better description (IMHO): "Exploratory Testing is simultaneously learning about the system while designing and executing tests, using feedback from the last test to inform the next."

I like this because it closes the loop between the purpose of the test and what you do with the results. In this case, when you learn something from the test you intentionally performed, you use that to decide what you will do next. It is kind of like playing the game of 20 questions. If you play the game poorly, you ask specific questions about what you think the answer is - e.g. "is it a turnip? No. Is it a bicycle? No. Is it a ...?" There's a slight, random chance you may guess it right, but that's really unlikely. If you play the game well, each question you ask helps you narrow down the possibilities until you can make a good guess with a high probability of getting it right.

I often use this diagram to explain the relationship between exploratory testing and test cases:

When we first look at a new feature or system, we don't know very much. We design experiments (or tests) to help us learn more about it. Initially, it is like the game of 20 questions, where we try many things and look at the system in different ways to try and discover what is important to someone who matters. That is, we explore the system for qualities and risks that we believe the customers, users, or other stakeholders may care about.

Test cases are different. When you have learned something about the feature (ET session complete), you may choose to document or automate important or representative paths (i.e. test cases) through the software for future reference (e.g. "regression testing"). You don't learn anything new from these test cases, so we sometimes refer to them as scripted "checks". We may use and reuse specific test cases for many different purposes - e.g. regression testing, performance testing user profiles, sanity checks, and so on.

At some point in history, the (1) intent or purpose and (2) test design behind the testing activities were lost and some idiot propagated the idea that test cases are the important part of the testing activity. This "bad practice" has doomed most of the technological world for a few generations now.

Let me be clear about this: test cases are not important. To anyone who knows how to test, we can create and choose new representative paths at any time, and, often times, the variations between the chosen paths through a system helps us uncover new risks and potential problems. Testing requires thinking. Checking, or blindly executing test cases, does not. If executing test cases doesn't require thinking, you may as well program a computer to run them because humans are famously bad at precisely following instructions.

An important difference between exploratory testing and scripted testing is that scripted testing blinds you to everything else going on in the system while exploratory testing aims to help you see more. To use a literary example, author Paulo Coelho posted a short story on "the secret of happiness" that illustrates this point. (NOTE: please read that story before continuing here - it'll only take a few minutes. I'll wait.)

I don't know if that is the secret to happiness, but I do know that in the first run through the palace, the young man was so focussed on the task that he missed everything else - this is exactly like scripted testing. The second time, the young man took in everything but forgot about the spoon - this is random or haphazard testing. Many people think this is exploratory testing but it is NOT! Exploratory testing would be how the wise man described the secret of happiness - complete your task and take in your surroundings.

It sounds hard, doesn't it? You know what, it IS hard. Good testing requires thoughtful effort and practice. If good testing was as easy as we are often led to believe then we wouldn't have all the software problems we have today, now would we?

Okay, so if doing good testing, exploratory testing, is hard, who can do it? Good question.

From one perspective, many people do this kind of testing naturally. BUT WAIT! Many people do this style of testing naturally, the same way many people can solve rate and calculus problems intuitively in their heads whenever you try to catch a ball. The mathematics behind the motion of a ball through the air (gravitational, kinetic and frictional forces) coupled with your movement relative to the ball in order to catch it is really quite complex. Not many people would say they understand or can do the math, but most people can catch the ball. So, part of your brain knows how to do the math even if your brain doesn't tell you how it does it.

It's the same with testing. There is a method to the madness. When someone goes looking for information, it is usually in response to some question in their head. Either someone asked them the question, or they thought it up based upon some related thought. That question drives you to poke, look, observe and evaluate what you learn to answer that question. That is testing. It has important elements: the question, intentional test design, observations, and analysis of results.

Some people are good at all of these elements, some are good at some of these elements, and some suck at all of them. To the latter group of individuals I say: please step away from the keyboard, and avoid management roles. Please.

There is an interesting side note related to Agile Software Development. Practitioners and coaches of agile methods may be familiar with the Agile Testing Quadrants. You will see that "Exploratory Testing" appears in quadrant 3, so what's that all about?

Funny you should ask. It is a bit misleading.

You may think that ET in Q3 means that it is something that is only done to critique the product with some business-facing tests. Not so. Exploratory testing will be performed in any and every quadrant as long as the person doing the testing is thinking, intentionally designing their tests, and learning from the results. Last time I checked, that happens in all the quadrants.

For example, when a programmer is creating unit tests to drive the development (Q1), they are thinking about the feature and design and making choices about what to automate. There is a lot of learning going on in this process and I would very much consider this discovery process as "exploratory". However, when the unit tests are coded and running automatically with every build, these are now "checks" and no more learning is taking place. So, executing these checks that were created in an exploratory way is no longer an exploratory testing activity. Get it?

Same thing with functional tests (Q2). You start off learning and exploring but once you decide upon and document a specific set of test cases, these test cases are no longer exploratory.

Quadrant 3 is an interesting place. It is the catch-all space for the million other tests that the system users and stakeholders may be interested in. The problem here is that complete testing is impossible and there is an infinite number of perspectives one may use to examine a particular system. The human brain is uniquely qualified to process a lot of different factors really quickly, integrating and adapting to new information, and eliminating and ignoring aspects that are not a priority to the stakeholders.

Computers cannot do this. Not even close. That's why the bubble in the corner of the matrix says "Manual" - because our brains are the most efficient tools to perform this kind of testing! Of course, we make use of tools and automation to help us gather information when appropriate; we just can't let ourselves fall into the trap of thinking that computers can do this for us.

So, while exploratory testing is a means to an end in the other agile testing quadrants, it is the primary approach in this particular quadrant (Q3). Got it?

So, if you fumble your way through the other three quadrants on your agile project and you are wondering why your quality still sucks, you may need to take a serious look at finding an awesome tester with some mad exploratory testing skills. Sorry to say that this is not widely taught in schools yet, so we are still something of a rare breed.

Does this help clear a few things about Exploratory Testing? Please let me know. Cheers!

Read More

The Future(s) of Software Testing

This topic keeps coming up in various discussions so here are some thoughts on what I think a future may hold for us one day.  What does it mean to predict the future? Does it mean it is inevitable? If it is something appealing, maybe it is something we can work towards.

What does it mean to talk about the Future of Software Testing without talking about "Quality" (whatever that is)? I believe that Testing is a means to helping you attain the desired quality but that on its own it is not an indicator of what the delivered quality will be. I think it is fair to speculate on the practice of Testing in isolation of the idea of Quality. Just like it is fair to speculate on the kind of vehicles you use for travel without saying it is a clear indicator of the quality of the journey.

When it comes to predicting the future, how far ahead do we look? It used to bother me that H.G. Wells chose to go 800,000 years in the future in his famous work, The Time Machine. Some authors go 100 years or even just 5 or 10 to tell their tale. I will not be writing about what will happen in 5 years time and I really hope it doesn't take 100 years. I don't know how long some of these events will take to manifest. I do have some idea of 'markers' that we may see along the way though.

When it comes to Software Testing, two thoughts jump immediately to mind: 1) Software Testing is an inseparable part of the creative Software/Solution Development process; and 2) there will be many different possible futures depending on how you do testing today.  Put another way, there is no one right way to solve a problem, and creating software is a complex problem-solving activity performed in shifting organisations of human interaction so there are many ways to do it. In my opinion, the technical 'fiddly bits' like code, tests and user documentation pale in comparison to the difficulty of those two primary challenges: (1) solving the real problem, and (2) working with others to get the job done.

When we ask what is the future of software testing, we are really asking what is the future of software development. So what will Testing look like in the future? Well, what does Testing look like today? What does Development look like today?


Scenario 1: No (internal formal) Testing
For some, testing is only done by the end users. This may be a single individual, a small group of people or a larger population depending on the application. For example, if you have a highly-specialised application, the best testers may be the domain experts themselves (e.g. researchers, scientists, and so on). Aside from some quick checks made during development, no formal testing is performed prior to release of the software to the end user(s) for evaluation.

In some cases, non-critical software applications that people can live without if they fail are sometimes also released without any special internal testing phases. I have found that this often happens when a company is in a new niche market and they have no competitors yet. These applications may be prototypes or ways to try to figure out user/market value. If the apps stop working, people can continue on without them.

I don't see the future of this kind of software development changing much. No (internal) Testers are affected in the present or future of this kind of development shop. I believe with the advances in development technologies, we will see the quality of deliverables improve over time although the activity of determining suitability or fitness for purpose will always remain. That is, specialised software will always require the approval of the expert customers. They are the testers here.

Scenario 2: Programmer-driven-testing
I believe that "testers" are "developers" by definition, so rather than saying "developer-testing" (which would be confusing) I will say the code jockeys are the ones who own and perform the testing here. This is different from the scenario above in that formal test frameworks are in place and testing happens prior to release to the customer or users.  There are no separate testers or test teams within the company to take a second look at the system to make sure it "looks right" (big air quotes here). Many Agile Development shops operate this way, especially smaller ones.

Again, as there are no separate testers to speak of here, the future here won't seem very surprising. I expect things will pretty much look the same as they do today - only the tools will become more sophisticated. (more about that below, in the next scenario)

Scenario 3: Functional System Testing
Unfortunately, this makes up a large part of the software development companies out there with QA/Test teams - the "traditional" software testing. It depends on the context of course, however my experience has been that this activity is largely a waste of time and money and is performed more for show and to satisfy lawyers (i.e. copious test paperwork as "evidence" of due diligence) than it is to actually raise software quality.

Anyone today whose job is to create mountainous test documentation that slows the creative development process, creates division and mistrust between project team members, and only serves to check that what has already been built has been "built as specified" is completely wasting everyone's time and money.  I believe the term here is WOMBAT - waste of money, brains and time.

The future here is easy to predict. This tester role will completely disappear in the future as it is "wasteful" (in "Lean" development terms) and provides no additional value to the development process or solution quality. Sorry kiddos - adapt or die.

What will trigger this future? Simple. Responsible development practices and smarter development tools. When you take a look at the value that these test cases (i.e. usually very narrow-focussed functional "checks") provide, they are generally standard, straightforward things like:

1. Is the feature element (button, field, widget, text, whatever) that you said is supposed to be there actually there?

• Test-Driven Development (TDD) is a development practice (available today!) that completely eliminates the need to have separate testers do this kind of thing manually. There is simply no good reason for this testing activity to continue in a manual fashion into the future. It shouldn't even be done manually today!
• (TDD-style) Automated functional "checks" directly linked to the code are way more efficient to maintain. They also facilitate good quality deliverables that don't degrade unexpectedly over time.

2. Do input fields allow unexpected inputs?

• I believe that (future) advanced development tools (programming languages, compilers, etc.) will automatically include model-based testing (MBT) subroutines that will automatically scan for such trivial aspects of developed code.
• There is really nothing magical in this kind of testing activity. Yes, it finds a lot of really good bugs. The only perceived "magic" here comes from inexperienced programmers who don't know how to develop better solutions.

Generally speaking, I don't know when programmers stopped taking responsibility for such basic testing and checking activities as part of their coding tasks. If you go back to the 1960's and 70's, there were no separate testers - programmers did it all. This is easy stuff. I really believe that if programmers had continued to "own" this kind of testing, it would have been part of the development tools by now.

We took one step forward in the 60's and 70's and then two steps backwards in the 90's and 2000's. It's no wonder that the "agile movement" is trying to move programmers back in the right direction. When this quality/value ownership in development becomes more widespread, I will be happy to report that we will have achieved 1960's-level of development craftsmanship. Again. sigh.

Scenario 4: Business Analysts, Personas and Suitability Testing
Testing in some companies happens with BA's or other specialists who act on the customer's behalf to check that the application or system developed (SUT) meets the expected fitness for purpose. That is, does the SUT meet the business needs (rules, SOP's, statutes, industry standards, and so on) of the customers or users? These types of internal testers generally have some kind of industry or domain experience or knowledge.

In testing jargon, this is more "validation" kind of testing or "did we build the right thing?" Sometimes this is done with the help of "typical user profiles" called "personas." In the absence of formal requirements or tests, one can ask the question "what would user X do in this situation?" This kind of testing has more of a basis in business and psychology and doesn't presently lend itself to automation very well.

I believe that in the future, these kinds of tests will be automated as well, using intelligent systems and algorithms that can calculate the percentage probability of a developed solution falling within the desired user parameters.

I believe that the Eureqa system provides us with a glimpse of what intelligent computer systems can do today. With advancements in hardware technology, I believe it will be practical for humans one day to interact with computers via natural-language voice control and have the computers do these kinds of checks for us. The "comparing system" or "oracle" will group business rules together into a model, run through the SUT and heuristically compare the data output with the desired business model. At this point it is simple mathematics to let you know that the SUT meets approximately 72% of the desired model, include confidence limits, and tell you which rules the SUT fails to meet.

We're not there yet. If this describes your job, I'm pretty sure your job is safe for the next 5-10 years (if you are good at what you do). Development of this kind of "oracle" really depends on a lot of things, including how quickly we get through scenario 3 above.

Scenario 5: Life-Critical Systems
This is more than simply a variation of scenario 4 above. These kinds of systems are things like medical devices, nuclear/energy management systems, aerospace and deep-sea technologies, and so on. Basically, any system that if it fails someone will very likely die.

When people's lives are on the line, I believe the responsible action will be to always have people involved in the evaluation and assessment of the SUT.

Yes, I believe that advanced development tools and technologies such as what I mentioned in scenarios 3 & 4 will greatly improve the foundational quality of all systems developed - including life-critical systems. However, the role of the responsible development organisation here will be to have good, smart, skilled people own the field trials in a way that determines "fitness for use" at a level well beyond what I have already described above.

If my word processor is unavailable, I may get annoyed but I will find another way to write a message. If a pacemaker stops running after two weeks of use, you can be sure that the patient involved cares about this problem in a really big way. As will his or her family.

Scenario 6: Black-Box System Testing, Para-Functional Testing, Exploratory Testing
This is a superset of scenarios 3 & 4 and complementary to scenario 5.  In this role, the tester takes a look at more than just the functionality of the system and is trained to ask questions about the SUT and user expectations that sometimes challenges the current design of the developed solution.

A good Exploratory Tester identifies assumptions and asks open questions about them. For example, questions about the user experience, flow of data, security and privacy, internationalisation, reliability and many other facets that are often skipped or ignored in "traditional" software test teams.

I do this style of testing, and I know many good people who also do it around the world. Unfortunately, I also know that we represent a small percentage of all the test teams out there.

I believe that this testing role fills an important niche in the software development "creative problem-solving" activity that is currently lacking in many companies. That is, we go back to the important question: are we solving the right problem for everyone who matters?

I see two possible futures here. First, if people are still involved in the development process, I believe that this testing role will be split in two parts. The hands-on testing part will be automated using the advanced development tools I already described above. We can simply add new MBT subroutines to the tools to account for new personas, perspectives and potential problem types. The creative, investigative people-interaction part will still need a skilled person to go around, talk to people and ask the right questions. This will lead to creating the right tests for the tools to help us answer.

In the second possible future, if people are not involved in the software development process, this whole testing activity will be automated in a fashion similar to what I described in scenario 4 above.

Scenario 7: Specialised tests - e.g. Performance, Usability
This is a variation of scenario 6 above.  A specialised test is one that answers a specific class of questions. My experience in Performance Testing tells me that how I do this kind of testing is different from other kinds of testing. At the end of the day, there are a set of rules and models that apply to do this kind of testing properly. We will be able to program or teach these rules to an "oracle" system at some point in the future.

Human research and expertise will go into the problem-solving models that we will program into these computer systems.  Depending on the complexity of the development project, the oracle system may be able to ask the appropriate questions and execute the tests without further prompting. In some cases, I expect the initial questions will come from a person and the computer system will be able to perform the test and report the results as required.


To sum up, I see the future of software development as being much more plug-and-play than anything we have today. Software testing activities will be largely automated with the possibility that humans may still be involved in asking important questions that lead to suitability or fitness for purpose. In time, I think that learning computer systems will be able to anticipate those kinds of questions and they will free us up to do different, more interesting creative work.

Don't worry, testers. Your jobs are safe as long as the programmers' jobs are safe - provided you are contributing value to the development activities. I believe our "developer" roles will disappear in close tandem. Software development and engineering is still a relatively young field/industry. We still have a lot of growing up to do.

Read More

Radiating Testing Information - Part 1

This topic is one that I have been asked about many times over the years and I am long overdue for a detailed discussion of it. Back in 2006 I presented an Experience Report at the STiFS workshop in New York titled "Low-Tech Testing Dashboard Revisited." The content of that presentation will be in Part 2. To quote "The Do-Re-Mi Song" from the movie The Sound of Music, "Let's start at the very beginning, a very good place to start."

I attended the StarEast conference in 1999 and there was a talk by James Bach titled "A Low Tech Testing Dashboard." This presentation clicked with me as I was managing several test teams at the time and it addressed a problem that I felt was important. I have used this communication tool many times ever since. If you are not familiar with it, I suggest you read through the PDF slides on the Satisfice web site before you continue. Go ahead. I'll wait.

In this review I will cover some of the who, what, where, when, how and why of the Low Tech Testing Dashboard (LTTD) through examples from past projects I have worked on. I expect your context is different, so my hope is that these examples may help you think about how you might apply this communication tool on your project.


Who are you? Whom is this for?

I have used this tool as a single tester on a development team, and as a team lead/manager managing multiple concurrent teams and projects - using one dashboard for each project.

The first time I used this dashboard, I was one of two testers working on a web application. We were both doing unscripted, risk-based Exploratory Testing and I needed a way to try and understand high-level testing progress since we had no other testing documents to use for reference. That is, it's hard to really understand what a tester means when they say they are "done" testing a feature and you have no way of really knowing if that means the same thing to both of you. Adopting the terminology and scales outlined in the LTTD presentation helped bring us together on the same page in describing progress.

The primary audience for this information was our test team. We stood around the board daily and used it to identify priorities and plans for the day. The secondary audience was the development and project team. Our team was located next to the kitchen so the Testing Dashboard was in a highly-visible location. As soon as we identified our first "BLOCKED" item on the board, the dashboard became a tool for the development team to identify the immediate priorities to unblock our testing. We explained what 'Blocked' meant the first time it happened and then Dev automatically resolved these issues every time after that without additional prompting. Cool!

I have used this board in Waterfall-type projects with a lot of scripted tests and documentation, and on Agile projects too. It is helpful for daily Scrum (stand-up) meetings as it can help you remember what you worked on, what you plan to do, and it identifies any blocking issues or risks that we need help with.

What is the LTTD?

The name has two parts: (1) Low Tech, and (2) Testing Dashboard.  They are both important. One describes how you communicate, and the other indicates what it is.

In Agile parlance, I have heard this dashboard referred to as an "information radiator." From the Agile Advice blog, "an information radiator is a large display of critical team information that is continuously updated and located in a spot where the team can see it constantly." One may argue that the entire Testing function/role/job purpose is to radiate (good) information. I agree with that perspective but will save that topic for discussion another day. Here I will focus on the outward presentation of particular information that communicates important project status and quality details to the project team.

The Testing Dashboard communicates different aspects or dimensions of the testing effort in a tabular format for everyone on the project team to see. The Low Tech part reminds you this information can and should be communicated using simple, readily available tools in a highly-visible area - such as a dry-erase board. You could make it high tech but you don't need to.  In fact, in my experience, for a single co-located development team (i.e. where the whole team is in the same room), anything more high tech than a dry-erase board is likely a waste of time.

The Testing Dashboard is novel because it communicates (i) Test Effort, (ii) Test Coverage, (iii) Quality Assessment, and (iv) current risks for each Product Area under test, all in one convenient location. It represents a snapshot of these dimensions for a given moment in time or for a particular build.  The use of smiley emoticons for the Quality Assessment is particularly effective. I have seen it happen several times where a developer will draw little devil horns on a red unhappy face. This is good because it shows their interest in interacting with the board and it is an expression of their feelings - in a humourous way. Fun interaction like this is always a good thing in my opinion.

Prior to using this dashboard, I was used to being asked for Test Coverage estimates (in terms of percentage tests complete and tests remaining) during Project Status Update meetings. This was likely a measure of the number of Test Plans, Test Cases, or Requirements covered using a Traceability Matrix of some kind. Those percentages always left me somewhat uneasy as I knew that precise measurements and estimates of such things don't reflect the reality of the remaining work to do.

What I like about the LTTD "Test Coverage" scale is that it was like going from Digital to Analog (i.e. a Physics/Electronics analogy). Suddenly I have this dial that I can use to more accurately describe the coverage! This is a good thing. In my opinion, precision in calculating remaining work to do is a waste of time. If humans are involved in the process, especially good testers, then your precise estimates will almost always be wrong. (NB: there is an important difference between accuracy and precision. It's great to have both but if I have to choose I'd rather be more accurate than precise.)

Where do you put the LTTD?

As previously mentioned, an information radiator like this works best when it is in a highly-visible area. In one company, I placed the LTTD on a whiteboard next to the kitchen, which also happened to be on the way to the development area, so everyone in the company saw it sooner or later. In another company, we placed it on a rolling dry-erase board and positioned it at the entrance to the development area (see photo at right). Again, pretty hard to miss.

I've had mixed success with making the Dashboard high tech. At one company, I put a copy of the Testing Dashboard on the wiki. We used Atlassian Confluence at the time and it worked very well. Whenever we updated the dry-erase board during our regular morning stand-up meetings, I would take a few minutes to update the wiki version too. I started receiving comments from team leads in other departments thanking me for the information because they could now get testing updates without having to come up to our floor and see the board directly.

The nice thing about the wiki version was that we could create links to our test strategy pages, bug reports, and other helpful online information. It really does make a handy interactive Testing cover page or executive summary for each project!

This is a good time to mention another tester I know who has implemented a high tech version of the dashboard. Marlena Compton wrote about her experiences with the LTTD and posted it on her blog - see background and CAST 2010 presentation. I like what she's done with it.

Another time at a different company, I tried putting the Testing Dashboard on a wiki to help facilitate the communication for a test team that was distributed geographically. It didn't work very well. Okay, it didn't work at all. Unfortunately, the test team didn't use the wiki for testing purposes. I believe communication in general was a problem on this particular project, so the effort was probably doomed from the start. I haven't ruled out a wiki-based Testing Dashboard as a helpful tool for distributed teams, so I plan to try again when the opportunity presents itself.

When do you update the dashboard?

I create a fresh Testing Dashboard as soon as a project is kicked off. We add testing areas and features to the board as we hear about them and it is a good way to keep track of what is coming and when.

The original presentation suggests updates 2-5 times per week depending on your needs. That sounds about right. One time we were so involved in our testing that we only updated it once a week. This lasted for several weeks. We noticed that the dashboard was failing as a useful, timely communication tool so we returned to more frequent updates. I recommend nothing less than 2 updates per week.

If you are making noticeable testing progress then I would expect to update the board daily or maybe every other day. I find that it is a good place for the test team to gather around each morning and discuss what everyone is working on for the day. We can update it based on what we learned from the previous day, and it helps us to stay on top of the current priorities and risks as they come up.

If you are doing a dedicated Exploratory Testing (ET) effort (i.e. as opposed to ET accidentally happening between scripted test cases), then a tool like this may be useful in helping to communicate testing progress.  It's possible that you may already have other tools set up for tracking and communicating progress of your scripted or automated tests. I find that this dashboard communicates more than just progress through a count of tests, so I would still recommend it as a way to supplement your regular testing updates.

How do you build a Testing Dashboard?

Remember that the key here is to build something simple, easy to maintain, and gets the message across clearly. As in Real Estate, the 3 most important things are: location, location, location. Once you have that, then what? Well, it's really up to you, the tools you have available, and your imagination.

In the beginning, I found that drawing straight lines on white/dry-erase boards was a challenge. In the absence of a metre stick, I would grab whatever was handy and worked - e.g. the metal edge strip off a cubicle wall or even a pizza box. I found that the width of a whiteboard eraser is just right for horizontal line spacing. Then it's just a matter of filling in the table using the appropriate marker colours.

One annoyance was that every time we updated the board we would always have to re-draw the lines. By the end of the project the dashboard would look pretty ugly. The answer came at another company when I learnt that a developer had brought in painter's tape to make their agile board.


In particular, he used Painter's Mate Green, 1/4 in x 60 yds or 6 mm x 55 m. It looks like this photo at right:

The 6 mm width is just about the same width as a board marker and the best part is that you can erase over it and the lines are always there! The green is also more visible than standard yellow masking tape on a whiteboard. I have seen blue painter's tape too, but I could never find one thin enough to use. Check your local hardware store to see what you have available.

So the tools I used to make the last few Testing Dashboards are: (1) 6 mm painters tape, (2) a post-it note (for spacing), and (3) colour markers.

I start by marking the width of the whiteboard eraser on a post-it note, and then use the post-it note as a guide for making the lines across the board (see photos below). Follow the general table outline as shown in the presentation slides and you will get the general idea.



A finished board looked like this a few weeks later:

I highly recommend using different-coloured markers when you create and update the dashboard. It makes the different meanings really stand out - i.e. it enhances how you communicate the message. For example, when you write "BLOCKED" in big red letters and have a red unhappy smiley face next to some show-stopper bug report number, people notice.

Why use the LTTD? Why not?

A good reason to use a tool like this is that it helps you and your test team to communicate important information to the project team without them having to ask you every time. In one sense, it does some work for you. For example, project managers can check the dashboard if they want an update and they don't have to interrupt you while you are testing. I don't know many tools that add time to my day. I am happy to say that this is one of them.

If your role as a tester is to provide valuable information, then radiate information! Let everyone know how things are going about things that are important to them before they even ask! Is there other information that people care about? Add it to the board!

The dashboard displays important info at a high level - a level that everyone on the project should understand. I have seen it happen many times when someone asks a tester how things are going and the tester gets lost in the details of the latest problem or issue that he or she encountered. The Testing Dashboard helps pull the tester up out of the details to put such events in a different perspective. Is that latest issue a blocking issue, or just another bug? Is it a show-stopper, or something that you require additional assistance from someone else on the team to help you investigate? What impact does this issue have on the overall quality of this feature or this release? How far along are we in testing this feature or all the features at this point?

The answers to these questions are always immediately visible on the dashboard. While we are fascinated by the puzzles and challenges of the testing problems we face every day, it's nice to have a dashboard around to remind us how to communicate at a level that our customers need.

So why wouldn't you want to use a board like this?

This question is partly rhetorical. I can think of at least one good reason why you might not want to.

For example, if the security policies of your organisation prevent you from leaving information like this up in a visible location. While making information visible is kind of the whole point of this dashboard, if your organisation won't allow it then you may wish to investigate other options - maybe even high-tech solutions. You have a team of smart people working with you. Get together and brainstorm a solution that works for you.

There may be other conditions or times when this may not be the right tool for you. "Because it hasn't been done before" is not a good reason to avoid it. If you are looking to grow, if you are looking for new ways to provide more value in innovative, simple and effective ways, you should give this a try. What will your Testing Dashboard look like? What will it say?

Radiate!

---

Here ends Part 1, an introduction to the Low-Tech Testing Dashboard. In Part 2, I will delve deeper into some of the aspects of the dashboard - how we modified it in different projects, and how we integrated it with Session-Based Test Management.

Before I post Part 2, I am interested to hear what you think. Have you tried using a Testing Dashboard like this before? Did it work for you? Are you thinking about trying it? Any other concerns that I haven't addressed yet? Please leave a comment and let me know what you think.

Cheers! Paul.

Read More