Never Test Before 4

Kind of a silly thought, I know, but it keeps coming back.

I work in a small agile development environment. Development works according to 2-week cycles to complete chunks of code. I keep noticing that anything prior to Cycle 4 or 5 is usually incomplete and unstable for testing. The first several cycles are when all the foundational architectural changes are usually happening.

So we can never really test before (cycle) 4. That's fine. I've got these Ruby scripts to keep me busy in the meanwhile. =)

Read More

Observation on the Proofreader Effect

I've been working on some performance test scripts using Ruby (Watir actually) over the last few weeks, and have been happily rewriting the scripts I first wrote a year ago. (Programmers call this activity refactoring.) I've learnt a lot about Ruby and scripting web apps over the last year. One of the biggest helps came when I read Brian Marick's new book "Everyday Scripting with Ruby". Thanks to that book, my performance test scripts are really slick now and look more like a programmer wrote them. But I digress..

The thing that I've been thinking about over the last few days is the problem of testing the scripts that I've written. Any good tester would never trust a programmer to write error-free code, so why should I trust myself to? But then who should test my scripts? Well, there really isn't anyone else around who can right now so I have to do it myself. Is that a problem? I don't think so.

I'm the biggest stakeholder who cares about these scripts working correctly, while my boss is mostly interested in the numbers and my analysis. So I ran the scripts and worked out the kinks one section at a time until I was able to run them straight through several times without error.

Is that good enough testing? Well, I got the coverage and measurements I wanted, so I guess so. The scripts don't have to be perfect, they just need to give me the data I need. So, it's all good.

Right. I completed the analysis for this run and then started to compare the numbers against the benchmark numbers from last year. It wasn't until several hours later that I noticed a typo in the output. Eek!

I'll just sneak back into the code and fix that. No one saw that. I'll just re-run the scripts and make sure the output looks "clean" this time. Great! Looks fine now.

So how did I miss that typo? I thought about this for a while. I think the proof-reader effect is like a FIFO buffer. That is, I don't think I could have seen this bug until I got the other bigger bugs out of the way.. you know, like the ones that prevented the script from completing or collecting the data I needed in the first place.

First in, First out. Get the big ones out of the way and then I can see the smaller ones that were hiding underneath. The typo was always there but I was just temporarily blinded to it because my attention was so focussed on the bigger fish.

So was I unqualified to test my own code? I don't think so. I caught all the bugs I cared about. It just took me a few days to find them. Would a separate tester have found the typo before me? Maybe, maybe not. The FIFO effect only affected *my* ability to see the little things until the bigger ones were out of the way because I was the one who wrote the scripts. A separate tester would have a different perspective and shouldn't be affected by this FIFO/Proofreader Effect in the same way.

We do Exploratory Testing almost exclusively on our products. When I test, I don't see the same effect happening to me. It's just a matter of time until I get to a feature or page and then I hit it like a whirlwind and move on. It's quite cool and effective. Defect finding rate starting to slow down? Switch to another Test Technique - voilà! More bugs. All the Risks addressed? Move on.

I've seen a number of conversations happening on some of the message boards questioning whether or not a programmer is able to test his or her own code. After this recent experience, I think if the desire is there and there is enough time, then yes, she should be able to find all the bugs that matter.

Once again, a separate pair of eyes not constrained by the FIFO effect would likely speed up the process. Nothing we didn't already know. A Tester helps you to find the bugs that matter sooner rather than later. Well, a good one will anyway.

Read More

Sometimes "Good Enough" isn't good enough

I've been a big fan of the idea of "Good Enough" software testing over the last decade. Rather than thinking that the problem of doing good Software Testing is akin to "Digital" technology with it's complete, precise values, I've thought of it more like "Analog" technology with the big dials and reasonable, approximate (and cheaper) signals.

This past week, I've watched my seven year old son play with a new LEGO set that he got for Christmas. It's a neat mechanical lego set that lets him build a motorised helicopter, cool car, or attack crab thingy. (ASIDE: I can't begin to imagine what the Marketing team's conversation was like when they thought up that last one!) I noticed when he completed the helicopter and turned on the motor, that it didn't sound right to me. So I went over and took a close look at his creation. It looked correct. There didn't seem to be any missing pieces, but when he turned it on again, I noticed that not all of the gears turned together consistently. I picked it up and took a really good look at it. Not knowing much about how it was built, I just randomly squeezed together lego pieces that weren't tightly packed together whenever I came across them.

There was one set of lego pieces that had a gap of about a millimetre. When I squeezed them together, it made a (good) snap sound. I asked my son to turn on the motor again and this time it not only sounded correct, but the gears all worked together in perfect synch also. Voila!

I thought about this for a few moments afterwards. Up until then, my son had worked on the premise that if the lego pieces were reasonably attached, that it was "good enough". He didn't need to have a tight fit between every single piece to see the finished product. I mean, it looked like the complete picture of the helicopter in the instruction manual, so what difference would a small gap between a few pieces make?

In this case it made a big difference. If it needs to work like clockwork, then "good enough" is probably not enough.

So what's the tie in to Software Testing? Well, just how scalable is the "Good Enough" approach? For me, it's always been about testing to the most important Risks and using whatever tools and techniques seem appropriate to the situation at hand. It's always seemed kind of foolproof to me.

Maybe my Digital/Analog analogy is a flawed one. I mean, Analog technology has its limits and is not very scalable. Digital technology is more precise and can handle more information. Is there a point when a Digital solution gets so large that it requires an Analog approach again? (I think the answer here is 'yes.')

Is there a time when "good enough" needs to be replaced with a more complete, structured or methodical approach to software testing? I can't think of any situations like that right now, but that doesn't mean there aren't any. That is, I can't think of a time when I wouldn't want to say that good software testing has to strike a balance between the economics, quality and time to market for a product or system. Shipping with bugs is okay if you know that they aren't critical or life-threatening.

So perhaps "good enough" doesn't always apply when we're dealing with real-world objects like lego creations, automobiles, watches, et cetera. I think that it still holds pretty well to the virtual world of software testing. Until someone can give me a good example or two of when "good enough" wouldn't be good enough for testing software, I think I'll chalk this up to another distinction between testing software and testing hardware.

Read More

Ruby at Work - Internet access through a Proxy Server

While reviewing Brian Marick's book "Scripting for Testers" (due out later this year or early next), I discovered that some of the internet access RUBY commands and scripts always failed when I tried them from work. Here's the weird thing - the same commands and scripts worked from my home computer. So what gives?

I found the answer when I googled comp.lang.ruby and tinkered around in IRB from my work computer. It turned out to be the Proxy Server at work. For lack of a better place to store this nugget of knowledge, this seems like a good place for now.

THE PROBLEM:

Open a command prompt on a Windows machine, and launch IRB. Issue the following commands:
----
irb(main):001:0> require 'open-uri'
=> true
irb(main):002:0> url = 'http://www.amazon.com/gp/product/0974514055'
=> "http://www.amazon.com/gp/product/0974514055"
irb(main):003:0> page = open(url)
Errno::EBADF: Bad file descriptor - connect(2)
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `initialize'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/timeout.rb:48:in `timeout'
from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/net/http.rb:555:in `do_start'
from c:/ruby/lib/ruby/1.8/net/http.rb:544:in `start'
from c:/ruby/lib/ruby/1.8/open-uri.rb:245:in `open_http'
from c:/ruby/lib/ruby/1.8/open-uri.rb:629:in `buffer_open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:167:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:165:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:135:in `open_uri'
from c:/ruby/lib/ruby/1.8/open-uri.rb:531:in `open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:86:in `open'
from (irb):3
----

THE SOLUTION:

You need to add the Proxy Server address (and user & password, if required) to the URL you are trying to access. Here's an example of something that worked for me from my work machine (replacing the relevant proxy address as required, of course):
----
require 'net/http'
host = 'www.amazon.com'
path = '/gp/product/0974514055'

proxy_addr = 'your.proxy.host'
proxy_port = 8080

response = Net::HTTP::Proxy(proxy_addr, proxy_port).get_response(host,
path)
----

=> response.body now returns the page source, just like it does from my home computer. (Response '200')

The second problem I had accessing a remote page was fixed with the following:
----
require 'open-uri'
url = 'http://www.amazon.com/gp/product/0974514055'

proxy_addr = 'http://your.proxy.host:'
proxy_port = 8080

page = open(url, :proxy => (proxy_addr + proxy_port.to_s))

text = page.read; nil
----

Note that the proxy_addr is slightly different here than with the 'net/http' commands above. (You could probably put it all in one variable here.)

If your Proxy server requires Username and Password, then the open command will be longer. (Luckily I don't need to enter that here at my work.)

I don't have any RUBY scripts that do this sort of thing yet, but I was happy to figure out the solution anyway. Nice change of pace. Ruby totally rocks. I have yet to find something that I can't do with it for testing automation.

Read More

"The Effects"

Patterns are funny things. Sometimes it is so easy to jump to silly conclusions because we notice the obvious even if we know there might be valid explanations for them.

Take for example patterns of behaviour. Here are a few things that I see on a regular basis:

• The Developer Effect - The software application always works as expected on a Developer's computer.
• The IT Effect - Any weird or unexpected computer behaviour (e.g. unable to print) will suddenly start to work correctly when you call an IT Helpdesk person over and they are looking over your shoulder as you retry the steps.
• The QA Effect - Any normally working system will suddenly stop working as expected when a Tester starts to look at something.
  

The "QA Effect" was my most recent addition to this list. A programmer sent me a link to a web site today and when I followed the link from the email message a new browser window appeared with the message: "We're sorry - an error has occurred. It has been logged and will be attended to shortly."

Ha ha. Is that Murphy's Law, or what?

I wonder how many other "Effects" there are in this industry? Write back a Comment to this Blog entry to let me know if you've come across any other immediate patterns. I'd be very interested to hear them.

Cheers!

Read More

I Know What I'm Doing!

This will sound weird but I had a strange realisation the other day. I thought to myself: "I know what I'm doing!"

Okay, this didn't just come up out of the blue and I wasn't practicing some sort of confidence-building exercise. It all started when I happened to notice a bug in the software I was testing and I decided not to report it.

What?! But you're a tester, man! Report the bugs you find!

Ah, well, yes, I plan to. But you see, I was busy at that particular moment thinking about something else, and I didn't want to distract myself by following up on a different thread that might make me lose my train of thought. To justify not reporting it I had to reflect for a moment as to how likely it would be that I would find the bug again. I had absolute confidence that I would find the bug again, so I simply forgot about it and went back to what I was working on.

I think it was later that night that the event started to settle in my mind and it started to make synapse connections with other memories in my brain. One particular memory came to me from my teenage years, some 20 years ago.

I recall one time when I was at a party and was talking to some of my older sister's friends. Coincidentally enough I think the topic of conversation was (creepy, crawly) bugs. There was this one guy who said that he was playing a video game in his home when he saw a millipede scurry across the floor. He said that his reaction was to pause for a moment, smile, and then keep playing his game. Someone asked him why he didn't just drop what he was doing and catch the icky bug right away?

His reply? He said that he knew those bugs and that he would easily find it later and take care of it then. He wasn't worried, so he didn't let it interrupt his game since there was no pressing need to take care of it right away.

I remembered that story for a long time and I always thought that if I ever saw an undesired bug I would simply take care of it then and there. Too many variables in life to let opportunities pass you by.

And so I have. For many years, I have always taken care of the bugs that I've found right away - both the creepy crawly kind, and the software kind.

So then, what happened this time? Why did I let a software bug go and not report it right away?

I realised that I was absolutely confident that I would find that bug again, and since there was no immediate need to take care of it, I let it go for the time being.

Sure enough, I got around to testing that particular area of the application and I had the opportunity to properly focus on the bug and report it with sufficient detail (it was a tricky one). I felt really great about it too.

When you do Exploratory Testing you are faced with the reality that none of your tests are pre-scripted. Exploratory Testing is kind of like what improvisation is to acting: there's no script. It certainly doesn't mean that the Improv people don't know how to act. In fact, it's quite the opposite. They are very talented actors. They just happen to know how to react to the circumstances that arise and can adapt to it with style.

I must confess that I've still been a bit nervous about Exploratory Testing. It doesn't help that I am bombarded with doubt by every other tester in the profession still doing it the traditional way: What if you miss something? How can you be sure of coverage? How do you know when to stop testing? and so on.

Well, I now *know* that I know what I'm doing. If I can totally forget about a bug and then stumble upon it again at a later date, then there must be something to this "improvisation" stuff after all.

Read More

Do you feel like the Stanford Bear?

I was asked just now if I felt like the Stanford Bear. I had to laugh, because sometimes I do.

What's the Stanford Bear, you ask? That's a good question. I'll tell you what I know..

I heard a story recently of an introductory computer class at Stanford University where the Teaching Assistants were so tired of answering the same questions over and over again that they programmed a machine to answer all the commonly asked questions. They added voice-recognition software to it and packaged it up nicely in a large teddy bear that sat on a shelf at one end of the class. Now, when a student had a problem or question, the TA's wouldn't help the student unless they knew that they had gone to the bear first. This relieved much of the workload from the TA's so that they could focus on the problems that really required their attention.

I'll admit that I don't know how much truth there is in this story. I spent some time scanning the internet and I couldn't find any pages to corroborate the tale. I'd like to think that it's true though.

True or not, it got me to wondering as to whether or not you could program something to give out canned answers on Software Testing to commonly asked questions.

I don't think that you could do it so that it would recognise every possible context and provide you with an appropriate response for every situation. (While a cool idea, let's be realistic here!) However, within a particular context - say revolving around the particular testing procedures for a particular class or industry, I'd think this might be do-able.

Better yet, I think it might be cool to program a Testing Bear that's hooked up to a central database somewhere. That way, people could program the Bear to answer all the common questions within their context, but then people all over the world would have access to the same questions and answers.

Now that might be something! Hmm.

Read More