The Human Side of Living

As I go through life I keep noticing stories, ideas and insights into humanity and I sometimes wonder if we are meant to discover these lessons slowly or if there isn't a quicker way to learn them.

Take for example, in high school we had a really weird Religion teacher who was very Zen or meta or something, and no one got him. I mean he would use examples like "take an extension cord and plug it into itself and there you go." Huh? None of us got it. And then there would be times when he would repeatedly say things like "attack the point not the person" and that was a phrase I understood.

From him, I learned that sometimes we can meet real jerks that we can learn interesting things from. Learn to separate your feelings about what you hear and understand from the messenger. It's hard sometimes, but you can get good at this.


Later, I read a story about a samurai warrior. It's short, so here it is:

There was once a samurai who wanted to learn the difference between heaven and hell. He sought until he found a master from whom he thought he could learn. He stood before the Master and asked him what was the difference between heaven and hell. The Master took the samurai’s sword and, turning it to the flat of the blade, struck the samurai on the head. The samurai was surprised at this but chose to ignore it. He thought that the Master had failed to understand his question. He once again asked the Master about the difference between heaven and hell. Again the Master struck the samurai on the head. The samurai staggered back and puzzled over this. He approached with his question for a third time and, before he could utter a word, the Master struck him a third time. The samurai was now so enraged at this behaviour that he grabbed his sword from the Master, raised it over his head and was prepared to bring it down on the Master’s head when the Master raised one finger and the samurai paused.
"That is hell," said the Master.
The samurai was instantly so overcome by the courage of this frail old man - to have risked his life for the sake of a stranger’s question - that he fell to his knees and bowed before the Master.
"That is heaven," said the Master.

This story keeps popping into my head every now and then. What is good and evil? Is it a matter of perspective? Is it a matter of time? How is it related to compassion? Do we need to judge people/situations, or should we learn to see the good and bad in all things? We can choose what we want to make from a situation. We don't always understand the motives of others, so which stance do you initially take - heaven or hell?

On the compassion thread, I learned about HH the (14th) Dalai Lama (of Tibet) when I was in university. It wasn't part of a course, I don't remember what it was. It might have been a movie. He's a really interesting guy and has done some cool things. I pondered his thoughts on compassion and felt that he really has good insights into the human condition so some of those ideas stuck with me. (HHDL is on Twitter by the way.)

When I left school and started working full time, I discovered Jerry Weinberg. Jerry published many technical books up to that point, and started the Amplifying Your Effectiveness (AYE) conference and Problem-Solving Leadership (PSL) workshop. The workshop and conference are based upon applying the work of Virginia Satir, a family therapist, to the workplace. I find some of the models very insightful.

Skipping over many other little opportunities and lessons, I find myself thinking about a recently-published book called "The Human Side of Agile" by a colleague Gil Broza. It's a good book. I like it. It sums up a lot of lessons I learned over the years, and includes new ones I didn't know about. The title really sticks with me though.

When I am at work, I focus on doing things to help others. Help the customers get high quality software of value. Help the team members to learn, grow and become more confident in their abilities. I show patience and temper difficult situations with humour. That's my style. When the going get's tough, I get silly. Sometimes, though, I hear Jerry W's words ringing in my ears "Change your organisation or change your organisation."

I am an agent of change. I am here to help you establish a new norm, a new status quo, one that is better than you were before. I work with people to help them adapt into their new roles, and I often come across people who neither want my help nor anything to do with change.

I can understand when people are afraid of uncertainty or the unknown and I am patient enough to work with them to try and build congruence (Satir) and focus on the point not the person (high school teacher). Then there are times when certain people can very intentionally do malicious things to undermine and attack you through a show of power or superiority. I'm too old for this crap.

From Jerry, I know it is time to change my organisation when this becomes a pattern, because it is my life and I choose how I want to live and enjoy it. I don't want to be miserable at work and then bring that negativity home with me to my family that I love so much.

It's my life. I want to be happy and helping others makes me happy. I'm weird that way. I understand that not everyone gets that. I'm not here to inflict compassion and other zen mumbo-jumbo on you. I really like and appreciate the Lean and Agile values. The focus is on *people* working together to make great things that make your customers happy.

After 25 years of working in the IT sector, I can tell you that I agree with Jerry when he said that "all problems are people problems." (especially the technical ones.) When I truly came to understand that, I discovered that people are at the heart of the answer to "what is Quality?" After almost 20 years in Testing, I also discovered that test techniques are really models to test the interactions between people working on the projects. This is a bit of unique perspective and I haven't heard anyone else describe it that way, but that's how I see it and teach it.

People working with people to make other people happy. Lots of other people actually. There's nothing non-human about software development. It's all about the human side of things. And yet. Schools don't teach this. Some people choose to act in inhuman ways. How do you deal with that? Heaven or Hell?

It's your life, your choice. Change your organisation or change your organisation.

Thank you to all my teachers, past, present and future. There is still more for me to learn.

Read More

Sharing thoughts

I've been asked a few times over the past few months why I haven't blogged in a while. Funny that. I have been writing more in this past year than I think I have my whole life. It turns out that the blog posts have fallen off the list of thought-sharing media for a short while. So here's a brief note to let you know why.

About a year ago, I started keeping a daily journal as a consultant. I got the idea spark from my friend Pradeep in India when he posted an annual report a few years ago that listed a brief note for each day that year. I looked upon that report with awe and thought that might be a cool way for me to track where some of my time and days go.

The daily journal is for personal reflection and I don't share it with anyone. It's pretty boring really. Disparate facts and ideas mostly. The challenge for me was in re-developing the habit of writing a little something every day.

When I was in high school I used to write every day. Again, I never shared those thoughts with anyone. But unlike my current journal, those entries were chock full of entertaining and emotional snippets of teenage life. I suppose if I hadn't lost those journals/books, I might have enough material to create a few teen book series. ;-) As an aside, several of my high school teachers thought I would go into English and continue writing in university. There was more than one surprised look when I said I wanted to go into Science.

I stopped writing personal journal entries in University and I discovered the internets - Usenet in particular. I started writing socially on topics of interest in the alt.* user forums. This was the early 90's, so the internet was still young then. The idea of Usenet forums wasn't new to me. In fact, I was attracted to them because they reminded me of the BBS forums I used to participate on during the 80's. (Ah, stories for another day.)

After graduating and joining the workforce full-time, I discovered and joined some email discussion forums in the late 90's. Around 2000, I dived into some Yahoo groups followed by some Google groups a few years later.

Around that time I started up this Blogger account to keep track of some thoughts as I began to take an active interest in my career in Software Testing. My blog posts were infrequent as I had lost the habit of writing daily. So many things go on in my life and the work-focus of this particular blog has kept me from writing more. Now if it were a general blog on random thoughts, I would likely have written so much more.

Several years ago I jumped onto Twitter. Interesting medium that.

I find Twitter is a good medium for me personally and professionally. While I remain on a few email discussion lists, I find I don't participate in them as much as I used to. I do however read and write/share thoughts via Twitter daily.

I added a gadget/widget thingy to the blogger layout here to show you my last few tweets. Twitter changed their API last year sometime so my tweets stopped appearing here. It's taken me too long to return to this blog site and fix the HTML code to redisplay my tweets on the side panel.

I spent about an hour tonight searching, scripting and playing with code to make those tweets appear again. I felt it was important to help the casual reader of this public blog understand that while I may share the occasional long thought on this site, I am micro-blogging on Twitter almost daily.

Several months ago, I also started to capture more thoughts and fieldstones for a book I plan to write on Testing. I haven't made it public yet, and when I do, I will announce it here and on Twitter.

So, I am currently writing a daily journal, capturing thoughts for a book in progress, tweeting daily, still on a few email discussion lists, participating in half a dozen conferences each year, and still trying to find time to blog every now and then. I have a huge backlog of ideas to share here. I promise to write no less than monthly here in 2013. I will share more ideas than I have in the past.

I have also been asked if I have an email mailing list. The short answer is no, not at this time. If I add that to the writing list, something else will likely have to come off. I am open to the suggestion though. Maybe later.

Read More

What is Exploratory Testing?

What is Exploratory Testing (ET)? I am asked this every once in a while and I hear a wide range of ideas as to what it is. This is one of those topics where Wikipedia doesn't really help much.

For some, ET is just "good" testing and the reason we say "exploratory" is to distinguish it from bad testing practices. Unfortunately, bad, lazy, haphazard, thoughtless, incomplete, and incompetent testing is quite popular. I won't go into the reasons or supporting evidence for this disgraceful blight on the Software Development industry at this time. Suffice it to say, I don't want to be mixed in with that lot either, so I am happy to describe what I do as something different - something that is far more successful and rewarding when done well.

Okay, so if ET = [good] testing, what is testing then? According to Cem Kaner, "software testing is a technical investigation conducted to provide stakeholders with information about the quality of the product or service under test." This definition took me a while to absorb but the more I thought about it the more I found it to be a pretty good definition.

If you ask Elisabeth Hendrickson, she would say that "a test is an experiment designed to reveal information or answer a specific question about the software or system." See, now I really like this definition! I studied Science in university and I love the way this definition reminds me of the Scientific Method. The more I learn about testing software, the more I find similarities with doing good Science. (By the way, if you want to learn more about how to do good testing, I highly recommend you read up on the Scientific Method. So much goodness in there!)

So, is that all there is to it? Testing = Science, blah blah blah, and we're done? Um, well, no, not really. ET has its own Wikipedia page after all!


I dislike the first line description of ET on the Wikipedia page. I dislike it because it is incomplete. It says that ET is "concisely described as simultaneous learning, test design and test execution." ... AND?!? And then what?! This definition is kind of missing what happens after you do the execution part. That's really important.

Elisabeth Hendrickson offers a better description (IMHO): "Exploratory Testing is simultaneously learning about the system while designing and executing tests, using feedback from the last test to inform the next."

I like this because it closes the loop between the purpose of the test and what you do with the results. In this case, when you learn something from the test you intentionally performed, you use that to decide what you will do next. It is kind of like playing the game of 20 questions. If you play the game poorly, you ask specific questions about what you think the answer is - e.g. "is it a turnip? No. Is it a bicycle? No. Is it a ...?" There's a slight, random chance you may guess it right, but that's really unlikely. If you play the game well, each question you ask helps you narrow down the possibilities until you can make a good guess with a high probability of getting it right.

I often use this diagram to explain the relationship between exploratory testing and test cases:

When we first look at a new feature or system, we don't know very much. We design experiments (or tests) to help us learn more about it. Initially, it is like the game of 20 questions, where we try many things and look at the system in different ways to try and discover what is important to someone who matters. That is, we explore the system for qualities and risks that we believe the customers, users, or other stakeholders may care about.

Test cases are different. When you have learned something about the feature (ET session complete), you may choose to document or automate important or representative paths (i.e. test cases) through the software for future reference (e.g. "regression testing"). You don't learn anything new from these test cases, so we sometimes refer to them as scripted "checks". We may use and reuse specific test cases for many different purposes - e.g. regression testing, performance testing user profiles, sanity checks, and so on.

At some point in history, the (1) intent or purpose and (2) test design behind the testing activities were lost and some idiot propagated the idea that test cases are the important part of the testing activity. This "bad practice" has doomed most of the technological world for a few generations now.

Let me be clear about this: test cases are not important. To anyone who knows how to test, we can create and choose new representative paths at any time, and, often times, the variations between the chosen paths through a system helps us uncover new risks and potential problems. Testing requires thinking. Checking, or blindly executing test cases, does not. If executing test cases doesn't require thinking, you may as well program a computer to run them because humans are famously bad at precisely following instructions.

An important difference between exploratory testing and scripted testing is that scripted testing blinds you to everything else going on in the system while exploratory testing aims to help you see more. To use a literary example, author Paulo Coelho posted a short story on "the secret of happiness" that illustrates this point. (NOTE: please read that story before continuing here - it'll only take a few minutes. I'll wait.)

I don't know if that is the secret to happiness, but I do know that in the first run through the palace, the young man was so focussed on the task that he missed everything else - this is exactly like scripted testing. The second time, the young man took in everything but forgot about the spoon - this is random or haphazard testing. Many people think this is exploratory testing but it is NOT! Exploratory testing would be how the wise man described the secret of happiness - complete your task and take in your surroundings.

It sounds hard, doesn't it? You know what, it IS hard. Good testing requires thoughtful effort and practice. If good testing was as easy as we are often led to believe then we wouldn't have all the software problems we have today, now would we?

Okay, so if doing good testing, exploratory testing, is hard, who can do it? Good question.

From one perspective, many people do this kind of testing naturally. BUT WAIT! Many people do this style of testing naturally, the same way many people can solve rate and calculus problems intuitively in their heads whenever you try to catch a ball. The mathematics behind the motion of a ball through the air (gravitational, kinetic and frictional forces) coupled with your movement relative to the ball in order to catch it is really quite complex. Not many people would say they understand or can do the math, but most people can catch the ball. So, part of your brain knows how to do the math even if your brain doesn't tell you how it does it.

It's the same with testing. There is a method to the madness. When someone goes looking for information, it is usually in response to some question in their head. Either someone asked them the question, or they thought it up based upon some related thought. That question drives you to poke, look, observe and evaluate what you learn to answer that question. That is testing. It has important elements: the question, intentional test design, observations, and analysis of results.

Some people are good at all of these elements, some are good at some of these elements, and some suck at all of them. To the latter group of individuals I say: please step away from the keyboard, and avoid management roles. Please.

There is an interesting side note related to Agile Software Development. Practitioners and coaches of agile methods may be familiar with the Agile Testing Quadrants. You will see that "Exploratory Testing" appears in quadrant 3, so what's that all about?

Funny you should ask. It is a bit misleading.

You may think that ET in Q3 means that it is something that is only done to critique the product with some business-facing tests. Not so. Exploratory testing will be performed in any and every quadrant as long as the person doing the testing is thinking, intentionally designing their tests, and learning from the results. Last time I checked, that happens in all the quadrants.

For example, when a programmer is creating unit tests to drive the development (Q1), they are thinking about the feature and design and making choices about what to automate. There is a lot of learning going on in this process and I would very much consider this discovery process as "exploratory". However, when the unit tests are coded and running automatically with every build, these are now "checks" and no more learning is taking place. So, executing these checks that were created in an exploratory way is no longer an exploratory testing activity. Get it?

Same thing with functional tests (Q2). You start off learning and exploring but once you decide upon and document a specific set of test cases, these test cases are no longer exploratory.

Quadrant 3 is an interesting place. It is the catch-all space for the million other tests that the system users and stakeholders may be interested in. The problem here is that complete testing is impossible and there is an infinite number of perspectives one may use to examine a particular system. The human brain is uniquely qualified to process a lot of different factors really quickly, integrating and adapting to new information, and eliminating and ignoring aspects that are not a priority to the stakeholders.

Computers cannot do this. Not even close. That's why the bubble in the corner of the matrix says "Manual" - because our brains are the most efficient tools to perform this kind of testing! Of course, we make use of tools and automation to help us gather information when appropriate; we just can't let ourselves fall into the trap of thinking that computers can do this for us.

So, while exploratory testing is a means to an end in the other agile testing quadrants, it is the primary approach in this particular quadrant (Q3). Got it?

So, if you fumble your way through the other three quadrants on your agile project and you are wondering why your quality still sucks, you may need to take a serious look at finding an awesome tester with some mad exploratory testing skills. Sorry to say that this is not widely taught in schools yet, so we are still something of a rare breed.

Does this help clear a few things about Exploratory Testing? Please let me know. Cheers!

Read More

Testing is a Medium

In a few days I will be giving a presentation to the local Agile/Lean Peer 2 Peer group here in town. The group has a web site - Waterloo Agile Lean, and the announcement is also on the Communitech events page.

I noticed the posted talk descriptions are shorter than what I wrote.  The Waterloo Agile Lean page has this description:

"This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it’s done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

The Communitech page has this description:

"Exploratory Testing is the explosive sound check that helps us see things from many directions all at once. It takes skill and practice to do well. The reward is a higher-quality, lower-risk solution that brings teams a richer understanding of the development project.
This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it's done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

This is what I submitted:

"Testing is the medium in which solutions are developed. The value of our delivered solutions depend upon how well we understand and utilize that medium. We can fly straight like an arrow or explode outwards like a spherical sound wave.
Traditional automated TDD "checks" help us fly straight in the direction we choose. Is it the right direction though? How do we know? Are you sure?
Exploratory Testing is the explosive sound check that helps us see things from many directions all at once. It takes skill and practice to do well. The reward is a higher-quality, lower-risk solution that brings teams a richer understanding of the development project.
This session will introduce the basic foundation of Exploratory Testing and run through a live, interactive demo to demonstrate some of how it's done. Bring your open minds and questions and maybe even an app to test. If ET is new to you, prepare to get blown away!"

This blog post is *not* about the differences in session descriptions. (In fairness, I really should learn to keep it to one paragraph. I hope to get better at writing session descriptions - it'll come with practice.) Reading the last one first, I can't help think that a bit of context is missing from the two posted descriptions for the final "blown away" statement. That is, that phrase comes from the sound wave analogy and not from some arrogant expectations I have for my presentation abilities. If I had known the descriptions would be shortened, I would have at least changed that sentence.

This blog post is about the idea that I try to convey in the first sentence that is unfortunately missing from both posted session descriptions: "Testing is the medium in which solutions are developed."

Software Development is a creative process. It is the intersection of people, skills, tools and experimentation to solve a people-problem with technology. As Jerry Weinberg once said: "all problems are people-problems." Therefore all developed products or services are "people-solutions."

Perhaps the main thought with "Testing is a medium" is that you cannot (successfully) solve any problem without trying to understand what the problem is in the first place. e.g.: Who is it a problem for? Where? When? How? -- these are all *Testing* questions. When you deliver a solution, you check it with something like: "how does this meet your needs?" -- again, another question.

Software Development begins and ends with questions. Somewhere in the middle of the creative development process are more testing questions. Lots of different kinds of questions, tests and checks depending on the people, skills and risks involved in developing each particular solution. One could say that Software cannot be developed without Testing.

Have you tried? Have you ever been on a project where you didn't first ask the customer what the problem was? You didn't check to see if what you are building is working towards that design? Or you didn't ask the customer afterwards if what you delivered meets their needs? How did that work for you?

If one cannot hope to hit the target without checking many different things, why is Testing often given so little attention or recognition on development projects? Too few individuals try to develop expertise in the Testing field to elevate their contributions to the development effort.

Anyone may ask a question. That doesn't make you an "expert" in asking questions. My 10-year-old son uses scissors to cut things, but that doesn't mean I want to let him cut my hair! Everyone I meet feels they know what Testing is. Okay, then why do so many projects fail?

So, what does it take to become really good at Testing? It appears to be somewhat important for the success of most software projects. Maybe it's time to take a good look at Software Development through the medium of Testing. How might you look at things differently then? What skills or knowledge would you want to learn more about? Who do you think should be involved?

I won't be talking about any of this stuff on Tuesday though. After all, it's not in the session description. =)

Read More

Quality Agile Metrics

I was asked recently what metrics I would collect to assess how well an agile team is improving. I paused for a moment to scan through 12 years of research, discussion, memories and experiences with Metrics on various teams, projects and companies - mostly failed experiments. My answer to the question was to state that I presently only acknowledge one Metric as being meaningful: Customer Satisfaction.

We discussed the topic further and I elaborated some more on my experiences. Regarding specific "quality" metrics, I explained that things like counting Test Cases and bug fix rates are meaningless. I also referred to the book "Implementing Lean Software Development" by Mary and Tom Poppendieck (which I highly recommend BTW) which warns against "local optimizations" because they will eventually sabotage optimization of the whole system. In other words, if I put a metric in place to try and optimize the Testing function, it doesn't mean the whole [agile] development team's efficiency will improve.

It needs to be a whole team approach to quality and value. Specific measurements and metrics often lead to gaming of the system and focus on improving the metrics rather than putting the focus on delivering quality and value.  If the [whole] team is measured on the customer satisfaction, then that is what they will focus on. I have long since stopped measuring individual performance on a team.

I haven't stopped thinking about this question though, so I put this question out on Twitter this morning:

Aside from Customer Satisfaction, are there any other Quality metrics you'd recommend in an #agile environment?

Here are the responses I received:

1. Churn or team turnover. (Real case: Product delivered on time, customer happy, whole team left.)
2. Escaped defects, inbound support calls/emails
3. Value created - Reference: "Lean Startup" by Eric Ries (I'm reading it right now)
4. Number of contributors to each story - not because exact count is meaningful but because it encourages collaboration & review.
5. Profitability of the project is one. Are the goals (whatever they may be) of the project met?
6. Lines of code changed during regression. That can expose some severe problems.
7. Production Defects in 15 days after 'Go Live'.
8. Cost of rework due to requirement changes.
9. Code churn as a measure of "quality" (e.g. System Defect Density) - Research, Code example, Discussion, and Sample Stats.

Hm, almost a Top 10 list. I am happy with the responses here and think many of them may be worth exploring further to see what insights they provide. Cautionary Note with all Metrics: Beware the impact they have on the team. If behaviour or performance starts to change in a negative way, STOP immediately!

One of the things I talked about in the conversation was the importance of Retrospectives to allow the team to own their improvement activities. If the team uses these opportunities, their improvements should be observable over several iterations. I think a Happiness Index reading during Retrospectives might be an interesting indicator of the overall effectiveness of the improvement strategies employed.

What do you think? Anything else I should consider?

Read More

Testers, Learn about Agile (and Lean)

Let me tell you about something called Dramatic Irony. You see it in movies, television shows, plays and in many other places. It happens when you (as the audience or observer) see or understand something that the main characters don't. Often times this is funny, sometimes it's not. Personally, I am one of those that likes to laugh when I see this happen.

On my learning/education quest over a decade ago, I took many different positions and roles within various IT organisations so that I could learn different aspects of Quality. I went through various phases, and the one I am least proud of was the "Quality champion." This wasn't a job title so much as a belief that (mis-)guided my actions. The role/part/perspective came mainly from believing what my employer(s) told me at the time - namely that "the QA/Test team was responsible for quality."

If you have worked in Software Development for a while, and perhaps for a larger organisation, you have likely seen someone who believes they are a Quality Champion. They don't want to see any (known) bugs go out; they check up on everyone in the team to see that they have done their reviews or had someone else inspect their work before passing it onto the next person/team; they join committees to create, document, maintain or present processes that will increase the quality of the delivered products/solutions; and so on.

Ah, the poor misguided fools.  Bless their hearts.


The first problem is in the company creating a scapegoat culture that puts the responsibility/blame of poor quality on the group of individuals who are least likely to help change the quality late in a development cycle - especially when the (test) team is under-informed, under-funded, under-staffed, under ridiculous time constraints, unappreciated and/or uneducated.

Quality is everyone's job. And I mean everyone. It starts with the president, moves through every person in the organisation and even includes the customers and users of your product/solution.

Returning to the naive tester who doesn't know or understand this, they do their part to motivate, inspire, nudge and, to some extent, manage the individuals affecting the quality of the released products. The effect of this is easy to predict. The nail that sticks out gets hammered down.

As it happened to me, I have seen it happen to other testers - they become disheartened, give up and withdraw back into their own work routine, ignoring everyone else and just focussing on their part.

I didn't give up entirely though - I can be persistent. I kept my eyes and ears open and looked in different places for ideas to help me understand how the organisation/system and "quality" fit together.

At the start of the 21st century, I stumbled upon something called the Agile Manifesto. I can't say that I completely understood it at the time but I was certainly excited about it. I mentioned it to my manager and he said it was a passing fad and that in 5 years no one would ever remember it. I felt that he was wrong and trusted my instincts on this one.

Over the next several years I learned about Agile and the different implementations. It all seemed very programmer/developer-centric to me as none of the models, articles, books or people ever seemed to talk about the testers. There was certainly a lot about devs taking responsibility for the quality of their work and incorporating testing practices into their regular routines. Hey, I'm all over that! Rah, rah, rah, sis-boom-bah, yaaaay Agile! :-D

Now, a decade later, I understand the Agile movement in a deeper, richer way. It is part of how I think and solve problems. It is part of how I encourage people to work together and to focus on the things that matter when delivering value to the customers. Just as I once described myself as someone who eats, sleeps and breathes 'Testing', I believe I would say the same thing about 'Agile'.

Here's the kick: the Agile movement is an ENTIRE COMMUNITY OF QUALITY CHAMPIONS! (oops, the caps lock got stuck for a moment there.)

That's right, listen up all of you testers out there who think you are alone and that no one is listening to your cries of "there must be a better way." There are people out there - Agile Coaches and Consultants - who are working to do just that. They find, create and teach better ways for development teams to work together to raise the quality/value of the delivered solutions.

If you don't know anything about Agile, start now. Read up on it, talk to others, attend a course, find online webinars, go to a conference - anything! Just get out there are start learning about Agile now! The same applies for Lean Software Development - learn about it.

Please keep in mind that there is a big difference between going through the motions of agile practices and actually being/thinking "agile". The agile mindset is more important to me than any particular set of practices.

This is especially important to keep in mind if someone tells you that testers have no place in Agile teams. Those people are what I like to call "wrong." (Get off my lawn.)

Testers can bring valuable insights to the agile software development process if the team works together and embraces the strengths of each team member. You, dear testers, must be open to change and adapt your role to work in new ways.

Warning, Warning, Danger, Danger: if someone tells you that you are "doing agile" and you (as a tester) should keep updating your manual regression test cases and test plans, please tell them to kindly "get off my lawn" for me. Thanks.

I still see this happen. I go into a client's office and look at how the software development team members are working. I see the testers off on their own, testing things in isolation and complaining about how no one seems to care about Quality because they only see and test the product at the end, just before it goes out the door.

Ooo, irony. I see something happening around you (within the community, industry, and sometimes within your own company) that you don't see.

Testers: if you feel like you are in this "Quality champion" role, be the hero and talk to an Agile Coach. Get one to come in and do an assessment. You aren't alone. You can help make a difference. Ask for the right help.

Be prepared to change yourself - to learn, to adapt to new ways of working with others, and to deliver a whole new level of quality and value that you didn't think was possible. Be a model team member and let the Agile coach help guide the rest of the team. Quality isn't your job - it's everyone's.

Read More

The Future(s) of Software Testing

This topic keeps coming up in various discussions so here are some thoughts on what I think a future may hold for us one day.  What does it mean to predict the future? Does it mean it is inevitable? If it is something appealing, maybe it is something we can work towards.

What does it mean to talk about the Future of Software Testing without talking about "Quality" (whatever that is)? I believe that Testing is a means to helping you attain the desired quality but that on its own it is not an indicator of what the delivered quality will be. I think it is fair to speculate on the practice of Testing in isolation of the idea of Quality. Just like it is fair to speculate on the kind of vehicles you use for travel without saying it is a clear indicator of the quality of the journey.

When it comes to predicting the future, how far ahead do we look? It used to bother me that H.G. Wells chose to go 800,000 years in the future in his famous work, The Time Machine. Some authors go 100 years or even just 5 or 10 to tell their tale. I will not be writing about what will happen in 5 years time and I really hope it doesn't take 100 years. I don't know how long some of these events will take to manifest. I do have some idea of 'markers' that we may see along the way though.

When it comes to Software Testing, two thoughts jump immediately to mind: 1) Software Testing is an inseparable part of the creative Software/Solution Development process; and 2) there will be many different possible futures depending on how you do testing today.  Put another way, there is no one right way to solve a problem, and creating software is a complex problem-solving activity performed in shifting organisations of human interaction so there are many ways to do it. In my opinion, the technical 'fiddly bits' like code, tests and user documentation pale in comparison to the difficulty of those two primary challenges: (1) solving the real problem, and (2) working with others to get the job done.

When we ask what is the future of software testing, we are really asking what is the future of software development. So what will Testing look like in the future? Well, what does Testing look like today? What does Development look like today?


Scenario 1: No (internal formal) Testing
For some, testing is only done by the end users. This may be a single individual, a small group of people or a larger population depending on the application. For example, if you have a highly-specialised application, the best testers may be the domain experts themselves (e.g. researchers, scientists, and so on). Aside from some quick checks made during development, no formal testing is performed prior to release of the software to the end user(s) for evaluation.

In some cases, non-critical software applications that people can live without if they fail are sometimes also released without any special internal testing phases. I have found that this often happens when a company is in a new niche market and they have no competitors yet. These applications may be prototypes or ways to try to figure out user/market value. If the apps stop working, people can continue on without them.

I don't see the future of this kind of software development changing much. No (internal) Testers are affected in the present or future of this kind of development shop. I believe with the advances in development technologies, we will see the quality of deliverables improve over time although the activity of determining suitability or fitness for purpose will always remain. That is, specialised software will always require the approval of the expert customers. They are the testers here.

Scenario 2: Programmer-driven-testing
I believe that "testers" are "developers" by definition, so rather than saying "developer-testing" (which would be confusing) I will say the code jockeys are the ones who own and perform the testing here. This is different from the scenario above in that formal test frameworks are in place and testing happens prior to release to the customer or users.  There are no separate testers or test teams within the company to take a second look at the system to make sure it "looks right" (big air quotes here). Many Agile Development shops operate this way, especially smaller ones.

Again, as there are no separate testers to speak of here, the future here won't seem very surprising. I expect things will pretty much look the same as they do today - only the tools will become more sophisticated. (more about that below, in the next scenario)

Scenario 3: Functional System Testing
Unfortunately, this makes up a large part of the software development companies out there with QA/Test teams - the "traditional" software testing. It depends on the context of course, however my experience has been that this activity is largely a waste of time and money and is performed more for show and to satisfy lawyers (i.e. copious test paperwork as "evidence" of due diligence) than it is to actually raise software quality.

Anyone today whose job is to create mountainous test documentation that slows the creative development process, creates division and mistrust between project team members, and only serves to check that what has already been built has been "built as specified" is completely wasting everyone's time and money.  I believe the term here is WOMBAT - waste of money, brains and time.

The future here is easy to predict. This tester role will completely disappear in the future as it is "wasteful" (in "Lean" development terms) and provides no additional value to the development process or solution quality. Sorry kiddos - adapt or die.

What will trigger this future? Simple. Responsible development practices and smarter development tools. When you take a look at the value that these test cases (i.e. usually very narrow-focussed functional "checks") provide, they are generally standard, straightforward things like:

1. Is the feature element (button, field, widget, text, whatever) that you said is supposed to be there actually there?

• Test-Driven Development (TDD) is a development practice (available today!) that completely eliminates the need to have separate testers do this kind of thing manually. There is simply no good reason for this testing activity to continue in a manual fashion into the future. It shouldn't even be done manually today!
• (TDD-style) Automated functional "checks" directly linked to the code are way more efficient to maintain. They also facilitate good quality deliverables that don't degrade unexpectedly over time.

2. Do input fields allow unexpected inputs?

• I believe that (future) advanced development tools (programming languages, compilers, etc.) will automatically include model-based testing (MBT) subroutines that will automatically scan for such trivial aspects of developed code.
• There is really nothing magical in this kind of testing activity. Yes, it finds a lot of really good bugs. The only perceived "magic" here comes from inexperienced programmers who don't know how to develop better solutions.

Generally speaking, I don't know when programmers stopped taking responsibility for such basic testing and checking activities as part of their coding tasks. If you go back to the 1960's and 70's, there were no separate testers - programmers did it all. This is easy stuff. I really believe that if programmers had continued to "own" this kind of testing, it would have been part of the development tools by now.

We took one step forward in the 60's and 70's and then two steps backwards in the 90's and 2000's. It's no wonder that the "agile movement" is trying to move programmers back in the right direction. When this quality/value ownership in development becomes more widespread, I will be happy to report that we will have achieved 1960's-level of development craftsmanship. Again. sigh.

Scenario 4: Business Analysts, Personas and Suitability Testing
Testing in some companies happens with BA's or other specialists who act on the customer's behalf to check that the application or system developed (SUT) meets the expected fitness for purpose. That is, does the SUT meet the business needs (rules, SOP's, statutes, industry standards, and so on) of the customers or users? These types of internal testers generally have some kind of industry or domain experience or knowledge.

In testing jargon, this is more "validation" kind of testing or "did we build the right thing?" Sometimes this is done with the help of "typical user profiles" called "personas." In the absence of formal requirements or tests, one can ask the question "what would user X do in this situation?" This kind of testing has more of a basis in business and psychology and doesn't presently lend itself to automation very well.

I believe that in the future, these kinds of tests will be automated as well, using intelligent systems and algorithms that can calculate the percentage probability of a developed solution falling within the desired user parameters.

I believe that the Eureqa system provides us with a glimpse of what intelligent computer systems can do today. With advancements in hardware technology, I believe it will be practical for humans one day to interact with computers via natural-language voice control and have the computers do these kinds of checks for us. The "comparing system" or "oracle" will group business rules together into a model, run through the SUT and heuristically compare the data output with the desired business model. At this point it is simple mathematics to let you know that the SUT meets approximately 72% of the desired model, include confidence limits, and tell you which rules the SUT fails to meet.

We're not there yet. If this describes your job, I'm pretty sure your job is safe for the next 5-10 years (if you are good at what you do). Development of this kind of "oracle" really depends on a lot of things, including how quickly we get through scenario 3 above.

Scenario 5: Life-Critical Systems
This is more than simply a variation of scenario 4 above. These kinds of systems are things like medical devices, nuclear/energy management systems, aerospace and deep-sea technologies, and so on. Basically, any system that if it fails someone will very likely die.

When people's lives are on the line, I believe the responsible action will be to always have people involved in the evaluation and assessment of the SUT.

Yes, I believe that advanced development tools and technologies such as what I mentioned in scenarios 3 & 4 will greatly improve the foundational quality of all systems developed - including life-critical systems. However, the role of the responsible development organisation here will be to have good, smart, skilled people own the field trials in a way that determines "fitness for use" at a level well beyond what I have already described above.

If my word processor is unavailable, I may get annoyed but I will find another way to write a message. If a pacemaker stops running after two weeks of use, you can be sure that the patient involved cares about this problem in a really big way. As will his or her family.

Scenario 6: Black-Box System Testing, Para-Functional Testing, Exploratory Testing
This is a superset of scenarios 3 & 4 and complementary to scenario 5.  In this role, the tester takes a look at more than just the functionality of the system and is trained to ask questions about the SUT and user expectations that sometimes challenges the current design of the developed solution.

A good Exploratory Tester identifies assumptions and asks open questions about them. For example, questions about the user experience, flow of data, security and privacy, internationalisation, reliability and many other facets that are often skipped or ignored in "traditional" software test teams.

I do this style of testing, and I know many good people who also do it around the world. Unfortunately, I also know that we represent a small percentage of all the test teams out there.

I believe that this testing role fills an important niche in the software development "creative problem-solving" activity that is currently lacking in many companies. That is, we go back to the important question: are we solving the right problem for everyone who matters?

I see two possible futures here. First, if people are still involved in the development process, I believe that this testing role will be split in two parts. The hands-on testing part will be automated using the advanced development tools I already described above. We can simply add new MBT subroutines to the tools to account for new personas, perspectives and potential problem types. The creative, investigative people-interaction part will still need a skilled person to go around, talk to people and ask the right questions. This will lead to creating the right tests for the tools to help us answer.

In the second possible future, if people are not involved in the software development process, this whole testing activity will be automated in a fashion similar to what I described in scenario 4 above.

Scenario 7: Specialised tests - e.g. Performance, Usability
This is a variation of scenario 6 above.  A specialised test is one that answers a specific class of questions. My experience in Performance Testing tells me that how I do this kind of testing is different from other kinds of testing. At the end of the day, there are a set of rules and models that apply to do this kind of testing properly. We will be able to program or teach these rules to an "oracle" system at some point in the future.

Human research and expertise will go into the problem-solving models that we will program into these computer systems.  Depending on the complexity of the development project, the oracle system may be able to ask the appropriate questions and execute the tests without further prompting. In some cases, I expect the initial questions will come from a person and the computer system will be able to perform the test and report the results as required.


To sum up, I see the future of software development as being much more plug-and-play than anything we have today. Software testing activities will be largely automated with the possibility that humans may still be involved in asking important questions that lead to suitability or fitness for purpose. In time, I think that learning computer systems will be able to anticipate those kinds of questions and they will free us up to do different, more interesting creative work.

Don't worry, testers. Your jobs are safe as long as the programmers' jobs are safe - provided you are contributing value to the development activities. I believe our "developer" roles will disappear in close tandem. Software development and engineering is still a relatively young field/industry. We still have a lot of growing up to do.

Read More