New Ruby SBTM scripts on my web site

Hi there, for anyone who is interested, I have updated the sbtm-ruby-tools zip file on my web site at: http://www.staqs.com/sbtm/

The current version says it's 1.2, but it's a bit of a mix. I made some updates to some of the scripts last Fall but didn't get around to pushing them onto my site. Just yesterday I ran into an annoying bug when I ran the scripts on a laptop with a newer version of Ruby. It was a one-line change to fix, but this change is worth posting because the bug may cause the 2 most important scripts to *not* run on some systems.

The reason I'm posting this on my blog is to solicit feedback on these scripts. It literally took me 4 hours to create this archive tonight. The reason it took me so long was because the gap between these v1.x files and the v2.x (with the 2.x folder structure that I use on a daily basis) is getting quite large.

You see, last year I completely changed the SBTM 'Sessions' folder structure to help us manage multiple projects simultaneously. To do that, I created new BATCH files and modified ruby scripts to help us work with the different project folders. It's pretty sweet actually. I'm currently managing 3-5 project simultaneously with the SBTM 2.0 framework and it's only a few clicks to switch between any project.

Is this new framework worth sharing? I don't think so. I'm bothered by all the text files and batch scripts (it's so 20th century)... although I have come to really like all the ruby scripts that I have for all my test management needs. From one perspective, it's like a file-based database. On the other hand, it's really a bunch of disjoint text files and command-line scripts (even when you do integrate them with the Windows Explorer).

So, new stuff aside, the help I'm looking for is from someone who is actually using the v1.x SBTM Ruby scripts. Since the gap is so large between my free ones and the ones I use on a daily basis, I'd really like to get some feedback from someone on a completely different system to let me know that the scripts work as advertised.

They should work. They're pretty simple. I'd just like to confirm that.

Any volunteers?

Read More

My illumination chamber

When I was a teenager, I used to bike to work to the downtown of the city. Sometimes a particular coworker/friend would bike home with me most of the way as we both lived in the same general part of the city.

I distinctly remember one night having a philosophical discussion as we pedalled quietly through the dark calm streets. I don't remember the particulars of the discussion anymore, but I recall that that was the first time when I was introduced to the term "Tao." He tried to describe it to me, and said that we were in a state of Tao while we cycled and talked, but that once we began discussing Tao we were no longer in a state of it.

Huh? So, we can be in a state of something until we realise that we're in a state of something and then we're no longer in that state? Is this a Schrödinger's cat kind of problem? Is it like The Game?

It took me a long time, a lot of reading, and personal experiences to begin to understand what my friend tried to explain to me that night all those years ago.

Oddly enough, I had a similar awareness moment just this morning.

First let's rewind about a year or two.. to a presentation I attended on Critical Thinking. Again, I don't remember the particulars of the talk, but I recall that the speaker described the Four Steps of Creativity at one point:

1. Preparation - Research: Collect information or data
   
2. Incubation - Percolation: Milling over collected information
   
3. Illumination - Light bulb idea: Aha moment
   
4. Implementation - Actual making/creating: Verification
   

I remember thinking at the time that somehow these steps related to the thinking processes involved when doing Exploratory Testing.

When we "prepare" to test a new feature, we research and discuss that feature. We explore our understanding and ideas and challenge every assumption we have. We *design* tests meant to explore our understanding and observe the results.

There are subtle and simple "aha" moments as we test to help concrete the information we began with. Things change from assumptions to facts, observations, trends and patterns that lead to recommendations.

And yet, things are not always that simple. Sometimes we get stumped when thinking through the problems we face. The answers do not come to us right away. Forcing more information into our heads is not usually the best way to solve a problem, I find. Taking time away from the problem is often what's required... i.e. enter the "Incubation" period.

If you are looking really hard for something and can't find it, sometimes you need to stop looking for it and move onto something else. Often you will find what you are looking for when you don't expect it.

Over the years, I've observed that even though I stop *actively* looking for solutions to a problem, as long as a problem is unsolved, my brain doesn't stop thinking about it. Sometimes, answers come to me in dreams, but I'm not a great sleeper so I don't often remember dreams. More often than not, answers come to me in those moments when I deprive myself of all sensory inputs and let my brain completely relax.

You see, I have a sensory deprivation chamber (of sorts) in my house that I amiably refer to as the Special Hydrogen hydrOxide Wave-particle Emission Room (or SHOWER, for short). I find that this SHOWER helps rejuvenate my energy, and provides a healthy glow to what hair remains on the top of my head.

While I'm in the SHOWER I generally try not to think about any particular problems. It's my only real time to myself all day, so I let the hydrogen hydroxide particles just bounce off my body and lose all sense of everything else.

And then it happens. Many times, under these conditions, ideas just pop into my head! Illumination! Aha moments! I see answers to questions that I had stopped thinking about.

I don't believe that your brain really stops working when you sleep, however, your consciousness needs a break and that's what we get when we sleep. Perhaps a good night's sleep is sufficient incubation period for our minds to mull over the collected information of the previous day(s).

I had noticed previously that I seem to get a lot of interesting ideas when I'm in the SHOWER. However, for some reason, colleagues are not always as happy and eager as I am when I tell them that I was thinking about them in the SHOWER. ;-) I don't know why. It's my illumination chamber.

This morning was slightly different. While in the shower I cheated. I tried to (actively/intentionally) *think* about the answer to a problem I've been working on ... and no new ideas came to me. I think I broke the rules of Tao on this one.

Illumination happens when you let it happen, not when you want it to happen. The other steps in Creativity (and Problem Solving) are a bit more straight-forward, they're done intentionally. This step has a tricky catch to it. Unlike the others, you can't force this one to happen on command.

Darn.

I look forward to my SHOWER session tomorrow. I'll give into the particles and just let them wash all my worries away.. if only for a short time. If illumination happens, that's cool. If not, then that's cool too. I love my showers. Sometimes I'm even moved to sing. =)

Singing is good. At least then I know that I'm using the creative part of my brain and not the analytical side. I've got all day to use my analytical skills. It's good to have time allotted daily to something creative too. There's something very Tao in that balance.

Read More

I Found This in My Underwear...

I purchased a new pair of underwear and small piece of paper fell out of the package. It was a note and it read:

STANFIELD'S

I have personally inspected this garment to be sure it meets the high quality standards that have made Standfields Limited famous. Exclusive fabrics and fine workmanship assure long-wearing comfort and style.

Maureen

Hm. That's interesting. Why is it we don't see notes like this in Software packages?

*Who* would have the courage to put their name as being responsible for assuring the product "meets high quality standards," has "fine workmanship" and that you will have long-term comfort in use?

That's pretty cool. I think the Software Industry still has a long way to go in achieving true customer satisfaction.

Read More

The Testing Paradox

I've seen much debate over the last few years regarding Scripted Testing vs. Exploratory Testing. I think I know the answer as to which approach is the one, correct, true way of doing Testing - it's "Yes."

Before I became a software tester I was a scientist and a High School Science Teacher. I recall a few lessons learned that helped shape how I do testing today.

When teaching (Inorganic) Chemistry, there was an experiment that I performed at the beginning of the year/term. This experiment served a few purposes. The first was to show the students an example of a chemical reaction. You know - the whizz, bang, cool aspect. The second purpose was to stress the importance of reading through the experiment carefully to understand what the important parts were - the warnings, dangers, and critical factors. You see the experiment is designed to fail. That's right, you follow the steps and nothing happens -- the big let-down, and students return to their desks. Denied. *Then* you add some water to dilute the chemicals before discarding the solution and .... POP! WHIZZ! SMOKE! SPARKS! Oooooo, ahhhhhh!

You see, Water is the catalyst required to make the reaction occur. The demonstration experiment is designed to challenge your beliefs that water is fine to dilute any failed experiment. Students need to understand that water is another chemical and that it is not always the best way to deal with a failed experiment. There is no always.

Firefighters understand this when dealing with different kinds of fires. You don't throw water on every type of fire. There are big differences between wood fires, electrical fires and chemical fires. They need to understand the situation before they can effectively deal with it.

When doing experiments in Science, there are times when you can improvise certain variables/steps and times when you clearly can't. So how can you tell the difference? You need to read everything carefully first. You need to understand what you're doing. Only then can you tell the critical steps and components from those that you have some freedom with.

So, what's the tie to testing?

When I first started testing software, many years ago, it was mostly scripted. In fact, I was responsible for an automation suite that tested different kinds of fax modems. The scripts ran through a series of functions in the software apps to test the compatibility of different hardware. Because I knew that, I was able to make variations to the software scripts as long as I knew that the hardware baseline information was still maintained. That is, there were critical functions that we needed to know about, and other, somewhat interesting things that were fine if we knew about them too (and fine if we didn't). I understood the purpose of the tests, so I was able to improvise as long as I didn't negatively affect the bottom line.

Over the last 6 years, I have been doing Exploratory Testing almost exclusively. Does that mean that we do it 100% of the time? No. Why not? Because I can think and tell the difference between when it's good to explore and when it's time to follow a script.

For example, when testing something new, we explore. We don't know anything about it and we don't know how well it meets the customer's needs. Scripting is not the way to go here. When we find problems we log bug reports.

Bug reports are interesting creatures. They are scripts. They indicate the conditions, data, exact steps and expected outcome(s) required to reproduce a very specific problem (or class of problems). Often, if you don't follow these steps as outlined, you will NOT see the unexpected behaviour. It's important that (1) a tester identifies this script as exactly as required and that (2) a programmer follow the steps as exactly as possible so that they don't miss the problem and say "it works on my machine."

When a bug is fixed and returned to our test team for testing, we do a few things. The first is to follow the script and see if the original/exact problem reported is indeed fixed. The second is to now use the bug report as a starting point and explore through the system looking for similar problems. Sometimes we have the time to do that when we first report a bug, sometimes we don't. It depends on what we were thinking/doing/exploring when we first encountered the problem. When a bug comes back to you, though, then that's the centre of your world and there's nothing to keep you from using it to give you additional ideas for finding similar or related problems.

When doing Performance Testing, it is important to understand that it is a controlled experiment.. a scripted test, if you will. You may have done some exploration of the system or risks to identify the particular aspect of the system that you want to observe, but now that you know what you're looking for, you need to come up with a specific plan to control the environment, inputs and steps as best as possible in order to observe and record the desired metrics. This is just good science. Understand your controls and variables. If you don't know what I'm talking about, DON'T DO PERFORMANCE TESTING. Leave it to the professionals.

I have a few stories about incompetent testers looking for glory who took my Performance Test Plans and improvised them in unintended ways or didn't even read the whole thing because they were lazy or thought they knew better .. just to have meaningless results that couldn't be used to infer anything about the system under test. My plans weren't the problem, the testers were.

So how do you do good testing? It starts with your brain. You have to think. You have to read. You have to understand the purpose of the activity you are being asked to perform and the kind of information your stakeholders need to make a good, timely decision.

Sometimes Exploratory Testing is the way to go, sometimes it's not. Note: I recognise that at this point there are still many, many testers out there who don't know how to do ET properly or well. Sigh. That's unfortunate. Those of us who do understand ET have a long way to go to help educate the rest so that we can see a real improvement in the state of the craft of testing.

Ditto for Scripted Testing. If you're going to follow the exact steps (because it is important to do so), then follow the steps and instructions exactly. Can't follow the steps exactly because they are incomplete or no longer relevant? Well, what do you think you should do then?

The point of this note is just to say that no one side is correct. There is no one true, correct, testing approach/method. They both are and they both aren't. It's a paradox. An important one. Practice good science and understand what you're doing before you do it. Improvise only when you know you can. Understand the strengths, weaknesses, and risks of any approach in the given situation and you should do fine.

Read More

Exploration in Literature

While reading a book recently, I came across a quote by T.S. Elliot. I looked up "Little Gidding" and found the whole poem on the internet. Here's a paragraph from the last quartet:

We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.

I tend to avoid deep poetry because I sometimes find it depressing more often than uplifting. This one made me reflective. In life, we explore so that we may gain wisdom and appreciate where we came from - where we started. As children and teenagers we may not appreciate things quite the same until we strike out on our own. "Home" always feels different when you've been away for a while.

Is there a tie here to software testing? Dunno. Haven't thought that far. If in life we explore to gain wisdom, how does that compare to exploring software? Do we appreciate the requirements more if we have done an effective job of test exploration? Do we gain wisdom? If so, about what? The people we're working with? The processes or technologies? The development practices or industry?

Just thought I'd share the find.

Read More

Great Example of Exploratory Testing Session notes

Exploratory Testing (ET) can be done well or it can be done poorly. How do you know how well you're doing ET? I believe you need 2 things: (1) you need to keep notes as you perform your testing, and (2) you need good feedback from a competent, experienced tester.

Taking good notes is not easy. It takes practice. A lot of it. In the process, you will learn something about how you organise your thoughts. You will also learn about biases, assumptions, techniques, critical thinking and communication of important facts.

Let's return to organisation of thoughts for a minute. One analogy that I often use is for a tester to think of your test notes like a Science Report. You know.. one of those reports that you likely had to do in elementary or high school for a science project, assignment or fair. There are basic and important sections/elements in a Science Report, and I believe those elements are also key for good test session notes.

I'm not going go into much detail about the comparisons here (for that, see my thoughts on my web site at http://www.staqs.com/sbtm/), rather I thought I'd share a link to a news story that I just came across on the Time.com web site. One journalist decided to perform a test of the new Google "Mail Goggles" feature.

Read the article here: http://www.time.com/time/business/article/0%2C8599%2C1849897%2C00.html?imw=Y

What do you think?

There are a number of things I like about that article. One of the things that struck me the most was how much I liked the "test notes" in that article. I think it's a great example of test notes that you should keep during an ET session. I've reviewed more session notes over the last 5 years than I can remember. The test notes in this Time article are very good.

The "science report" structure is both amusing and helps to efficiently communicate what the author did. I doubt the journalist knows anything about ET or session-based test management. The flow and clarity of the report is very good because the journalist has a skill that I often find lacking in the "poor" ET session reports I've reviewed over the years. What's that skill? A journalist knows how to focus on and communicate the facts. The science report structure helped the author organise her thoughts and communicate the facts efficiently. I liked it. I think it worked. And it was funny too! :)

Which brings me to another important point. Just because you add structure to a report doesn't mean you lose your personality. You can be both clear and funny. Emotion and impressions are important in good notes too because they help raise awareness of the "qualitative" aspects of testing. Too many people put too much value on the "quantitative" aspects of testing.

I like Albert Einstein's quote in regards to this point:

"Not everything that can be counted counts and not everything that counts can be counted."

A good ET session report/test notes should reflect both qualitative and quantitative elements. The science report structure might provide you with a good framework for organising your thoughts. Oh, and if you're looking to improve your technical writing in the Software Testing profession, perhaps you might consider a journalism course.

Read More

Testers Create Bugs, Not Bug Reports!

I've been thinking a lot about this phrase lately: "Your thoughts shape your reality." I think it's from a Buddhist teaching but I'm not sure because I read it as a second-hand quote somewhere that I can't recall right now.

I'm also familiar with the tester's lament: "hey, I didn't put the bug in the code, I just report it."

But what about the phrase: "if a tree falls in the forest and no one is around to hear it, does it make a sound?" That one has always annoyed me. The way I see it, the answer to the question depends mostly on how you define what a sound is. If a sound requires a listener to be real, then, no, no sound was made.

What about a bug? If there is a bug in the software but no one encounters it, is there really a bug in the software?

Now, I report bugs. A lot of bugs. I employ many creative and imaginative tricks to find them. They love me. They come to me even when I don't expect it. One of the tricks I use is to change my perspective on how I look at software. I also exploit vague areas, or areas of doubt within the development team, development processes or methods of communication. For example, if it looks like there might be some vague statement in a specification that could be confusing or interpreted in different ways, I'm pretty confident there will be lots of bugs there.

But hold on. What if I don't go looking for the bugs? What if I don't report them and no customer ever sees the problem or complains about it? Does the bug really exist?

Or does the mere act of identifying this discrepancy between intention and execution create the entity that I call a bug? Did I create it or was it already there?

Scenario: A spec-writer wrote a document that was interpreted in some way by one or more programmers. According to their interpretation, the implementation (the software) is working as intended. They check that their code measures up to their understanding (unit testing?) and then build the software into a deliverable product.

Then along comes a tester. A tester looks at it in a completely different way and suddenly the software is full of bugs. But wait a minute! The bugs weren't there until someone looked and said they were there.

This reminds me of Schrödinger's cat. Is the cat alive? Is it dead? Is it both alive and dead, or neither? You have to look to find out, but once you've looked you've changed everything. What a psych trip!

So, what if finding bugs falls into the same category? In that case, the mere act of observing has changed the reality of what's in the box. By looking inside and asking the question, we have shaped the reality by our thoughts.

Doesn't that mean that we have therefore created the bugs that we report? The programmers didn't put the bugs there. They interpreted the specs and design documents according to their thoughts and shaped the software according to their reality. They built no bugs.

The bugs were called into existence when we, the testers, said they were there with our mind tricks and clever tools.

The bug report, therefore, is just a record of how we have reshaped reality with our thoughts in a way that is different from how reality looked before we started testing. We created those bugs in the bug reports.

One might argue, then, that if you make the bug reports go away you make the bugs go away. If no one (customer) sees the bug, does it really exist?


That's a change in perspective from what I was taught about testing. I don't report bugs. I create them!

Read More