TEDxWaterloo

I attended the first independent TED event in Waterloo (TEDxWaterloo) yesterday, 25 February 2010. The theme was "Tomorrow StarTED Yesterday." The web site and twitter account page have lots of great info if you're interested to know more about the event and speakers. There's even a nice photo blog of the event at http://www.longexposure.ca/2010/02/tedx-waterloo/

So what can I add that hasn't already been said? Well, I can tell you what the event meant to me.


I wasn't sure what to expect. "TED" stands for "Technology, Entertainment, Design," but that tells me nothing. I've seen and enjoyed a number of the videos that I have come across - i.e. that were recommended to me in one way or another (email, web page links, twitter, passing conversations, and so on). However, separate videos alone hadn't quite made the connection for me.

The opening remarks helped set the context for the event. In those few minutes, it all came together - it clicked. Aha! This event is about more than the sum of the individual words above. Even the motto "Ideas worth spreading" make sense to me now.

The people who spoke at the conference and the selected TED videos that were shown hit me somewhere I didn't expect - in my heart. Most of the conferences that I have attended speak to my mind; help me try to understand or learn something or other. That's nice, but that's not all there is to life.

There's only one conference that I have ever attended that I would compare to the TED event - the AYE Conference. The difference here is that AYE is built from a series of interactive workshops intended to help us understand and work better with each other - human beings, not machines. TED is a perfect complement in that the speakers share stories and ideas which inspire us.

Inspired. Yes, that's a word I would use to describe how I felt during and after the TEDx event.

I spoke with someone at the event whom I hadn't seen in over a year. She described to me how her current project is making her feel dumber every day, dealing with processes and bureaucracy that only serves to confuse and make the project more difficult. At the TEDx event she said she felt like her brain was expanding, opening up again. I think that's another good description of how I felt.

Following the TEDx event, I now have a better understanding of what the words mean when I see them. I have a context for this event and videos - which helps me know what to expect and where/how to process the information.

Here are some notes I took from some of the speakers at the event:

Terry O'Reilly

• The Age of Persuasion
• idea of friction - sometimes you need to slow down in order to sell and market new ideas. Brilliant examples
• He referenced the book "The Checklist Manifesto" by Atul Gawande. (Good example that has come up a few times in testing. I think we need more good, practical examples in our profession.)

Paul Saltzman

• (I was really surprised and blown away by this talk. Really left me with a lot to think about afterwards.)
  
• "magic" - that which is real but you do not yet understand. (I've seen magic used a number of times to help demonstrate certain testing concepts. Paul used it in a different sense here and I think there is a lot of magic in what we do every day.)
• "humility" - not making yourself feel small, rather, seeing yourself in perspective of the larger universe.
• "There is no end to love. Love is infinite. There is no end to creativity. Creativity is infinite."
• "Nothing changes until you do."
  

Caroline Disler

• "Western civilisation" - There is no Eastern vs. Western civilisation (unless you happen to be strictly speaking geographically). We are one world.
• "Those who are ignorant of the past are prisoners of the present."
• she gave a really interesting summary of where knowledge and language (and even numbers!) originated and how every part of the world has helped bring us to where we are today.
  

Madhur Anand

• Heh. I didn't know about Sudbury and the environmental restoration projects happening there. Cool! (My university degree was in Environmental Science, so I had an interest in this talk. As a mining town, Sudbury hasn't had the best rep over the years, so I'm pleased to see that it is leading the way in environmental restoration projects. The world needs more of this.)
  
• she used some interesting quotes. One that was new to me was: "To live in a place, you must first imagine it." by Jay Ruzesky
• The environmental crisis is also a crisis of imagination.
  

Darren Wershler

• First one to mention Marshall McLuhan (he said he pulled the short straw backstage before the event. ha ha.)
• The roles of different kinds of media (untimely, conceptual and impossible) and their influences on us. Something doesn't have to be real or even possible to have an effect/impact on us, to inspire us to new ideas.
• e.g. transporter technology from Star Trek, or the many inventions by Reed Richards of the Fantastic Four. (Nice comic book reference!)
  

Marty Avery

• Be soft to be strong. Namaste.
• Kayaking story really touching. What does it mean to be a hero?
• There is some of us in everyone we meet. "You am I and I am you."
  

Amy Krouse Rosenthal

• (Google search doesn't do her justice. This was [yet] another amazing talk! Here's a link to reach her: http://www.whoisamy.com/ - watch the videos!)
• The 7 [musical] notes on life:
• Always Trust Magic (ATM)
• Beckon the lovely. Whatever you beckon (attract, look for) will eventually find you.
  
• Connected. We are all connected.
  
• Do
• Empty space
• Figure it out as you go
• Go do it. Howard Thurman quote: "Don't ask what the world needs. Ask what makes you come alive, and go do it. Because what the world needs is people who have come alive."
  
• The [musical reference] 'key' is you
• "Make the most of your time here."
  

Wow. And that doesn't cover all the speakers and videos we watched!

This event was certainly something I needed! I was moved and feel like I want to do something even bigger now! Well done! I can't wait to attend the next one or create my own independent TED event! =)

If an independent TEDx event is coming near you, I highly recommend you take the time to attend. If you do, please tell me what you thought and felt.

Cheers!

Read More

Now with minty-fresh visitor counter

Someone suggested to me this past weekend that I add a visitor counter to this blog. It's one of the most common suggestions made to me over the years and I don't know why.

Back in the mid-90's I had a web site with a counter. It was novel then. I played around with different fonts and features and watched it go up over time. I don't have that site anymore. I set up a new web site about 7 years ago, but adding a counter wasn't one of the important things on my to-do list. Foolish? Dunno. Maybe. Maybe not.

Is Quality measured in numbers?

Anyhoo, I won't go there today. ;-) I'm not going to think philosophically about it right now. I'll reserve thinking and judgment about the visitor counter for a later date.

This is just a placeholder note to indicate that the counter started today.

Cheers! Paul.

Read More

SBTM is not ET

There's a subtle but important distinction that I'd like to talk about. Session-Based Testing is *not* Exploratory Testing. Please stop using those terms interchangeably because they're not.

Exploratory Testing (ET) is a testing approach that puts the emphasis on real-time learning, test design and test execution, as opposed to a more "scripted" approach that puts the emphasis on the separation of these activities - separated in time, space, and usually with copious amounts of documented artifacts.

When I first started in I.T. over 20 years ago, any testing I did as part of my programming contracts were exploratory in nature. I didn't call it 'ET' at the time and I certainly didn't approach it with the same discipline and formality that I do today. Back then, Programming was my main focus and testing was just something I did as required along the way. Ten years later (or about 12 years ago depending on your perspective), I took a workshop class on "Test Case Design" with Ross Collard. That was an amazing class that opened my eyes to a whole new world of analysis and problem solving that I didn't know before. Cool!

After that workshop, I had plenty of opportunities to practice what I learned, try new techniques and tools, and explore additional testing ideas thrown out onto the just-budding software testing mailing lists. One of the things we discussed in Ross' class was the role of "ad hoc" or informal testing. I don't have access to the data, but some study-or-other at the time (90's sometime?) showed that ad hoc testing failed to produce the same amount of testing coverage that formal test design analysis would.

Okay, I buy that. To paraphrase: guessing ideas off the top of your head consistently produced less coverage than having some structured analytical approaches/techniques/heuristics/models at your disposal. Okay. I don't need a formal study to tell me that.


So what's different with Exploratory Testing? Well, when I first learned about ET at the turn of this century, it instantly clicked with me. Rather than the "guessing" attitude normally associated with "ad hoc" testing, ET clearly defined the testing approach in a way that made you think. You learn something; you design something; you test and observe something; repeat. Note that nowhere in there does it say "take a wild-ass guess and call it good, complete or even 'good enough' testing coverage."

Before I was introduced to ET, I had spent several years practising and training other testers on test design techniques. That helped me fill in the "test design" step of ET. That step is the weakest link with most of the testers I have met and spoken with over the years who have tried and given up on (i.e. failed with) ET. You can't really fake your way through test design. That's why I make it an important part of my hiring/interviewing process (you can read the article online).

So, what *don't* I like about ET? There's just one thing really. The ET approach formalised the learning, test design and test execution aspects of testing, but not the interpersonal communication aspect of it.

The 'scripted' (waterfall) approach to testing relies on the documenting (and maintenance) of hundreds or thousands of test cases, each with their own set of pre-conditions, steps, expected results, and so on. While the value of these documented test cases may be questionable, one thing going for it is that you can share these test ideas with other people quite easily. (They're documented; pass it on.)

In ET, not so much. If the important parts of testing takes place in your head as you process all of the inputs and information, and compare them with explicit and implicit requirements and expectations, in order to assess the quality of the A/SUT, then when/how do you share those test ideas with other people (testers, developers, business analysts, etc.)? Well, you don't. Or rather, ET alone doesn't give you any advice for communicating test ideas or testing coverage with others.

Enter "Session-Based Test Management" (SBTM) or just '"Session-Based Testing".

Aha! After a year or two of using ET, I instantly found the merit in SBTM. SBTM provides the framework that you can wrap around an ET approach. It is a way that you can manage the testing effort. It has four main elements: develop specific charters, time-box an uninterrupted work session, create a reviewable result, and review/debrief the session afterwards.

Here's the catch: it is *not* a testing approach! It is a test management framework. Actually, when I teach/describe it to others, I sometimes refer to it as "Session-Based Task Management."

I have taught SBTM to programmers as a way to help them manage their time and reduce the number of interruptions during a work day. I have also successfully implemented SBTM in a waterfall organisation where very little ET was ever performed.

Yes, you read that correctly. I have even wrapped SBTM around a *scripted* testing approach.

Eek! Egad! Gadzooks! Isn't that blasphemy?

Well, actually, no.

You see, I have found that SBTM is an incredibly powerful tool for a test manager. It gives you insights into aspects of testing that you might never have without it.

The four main SBTM elements provide a solid foundation to managing your work, and can be transferred to activities other than just ET. For example: programming, writing, organising/cleaning your basement, any consulting work, and so on.

The original SBTM framework included some Perl scripts that I have long since stopped using. The original archive included a session sheet template, but like any template you can modify it and tailor it to your needs. (If in doubt, just ask James Bach for his thoughts on Test Plan templates! :)) That's the main reason I rewrote the SBTM scripts in Ruby - so that I could customise the session sheets to the needs of the projects I worked on. So, for one project I added a section to the session sheet, and for another project I completely removed the TBS metrics; my Ruby scripts are flexible and can handle the changes easily. (ASIDE: I haven't made this customisable script publicly available on my site yet. Send me a note if you are interested in trying it.)

In fact, if you follow the intent of SBTM, you don't need to use the session sheet template or scripts at all - as long as you have some agreed-upon reviewable result that you can later debrief. In this way, I have heard of some test teams that have implemented SBTM using Wiki's, and others that have integrated old Test Case Management systems into the process. Sounds cool and innovative to me!

So, what's my gripe? In the last several weeks, I have read several times that Exploratory Testing includes time-boxed, chartered sessions with reviewable results. Umm, no, I'm pretty sure you're confusing the framework with the approach, the wrapper with the content, the book format with the story.

If you have implemented SBTM on a project, I can make no assumptions about what testing approach you are following. Likewise, if you include ET in your overall testing strategy, I won't assume you are using SBTM to manage that effort.

If you want to talk about ET or SBTM, please try to describe them in the correct context. It will make it less confusing for beginners and other interested parties. Granted, together you have a very powerful combination. But Superman is still super in a different suit. =)

Read More

Time - Bane or Innovation Catalyst?

Time. What time is it? How much time do we have? When do you want/need it? What's the deadline? I need more time!

If we had all the time in the world for software development, would the delivered results really be of better quality?

A co-worker at a past employer wrote the following when someone sent an email submission for a fun, internal contest the day after the deadline:

The contest ended a long time ago. Trying to submit something now is like submitting your late university assignment.
One of my profs told me:
"I don't care if you have something that's better than all the works of Shakespeare. If you can't get it in before the deadline it's worth nothing to me."

Ha, ha. It was intended as a funny remark at the time but there's some truth in there too.

So, if someone submits an assignment "on time" but of lesser value/quality than they might produce if they had more time, would they still continue to work on their opus or would they give it up to move onto the next project? Do we (as a collective group of intelligent human beings) lose out by putting Time ahead of Quality?


The traditional "Project Management Triangle" puts the emphasis on: functionality/scope, cost and schedule. An experienced consultant can tell the employer: pick/fix any two and we can estimate the third.

I noticed years ago that "quality" isn't in this "triangle". As a novice, I took "scope" and "quality" to be part of the same point. Clearly I was mistaken. When people are focussed on delivering something, on time, at a fixed cost, everyone interprets "quality" in different ways.

I think the Agile manifesto/movement is an interesting response to the "traditional" (a.k.a. Waterfall) approach to software development. It takes the same 3 constraints (of scope, cost and time) and changes up the order of activities to integrate quality into the deliverable products. This is done by embedding customer involvement (via collaboration, user stories, automated acceptance tests) and rapid delivery releases to allow for quicker feedback into the design and implementation. For example, in a traditional/waterfall project, it may take anywhere from 6-18 months to find out your interface/implementation fails to meet the needs of the customers. Or, using agile methods, it might take anywhere from 2-14 days. Your choice.

So what about software testing?

In every waterfall project I have worked on, development always delivered software late into the "test" phase. This meant less time to provide feedback, because the release deadline was fixed. Time is my bane here. I've got less of it and need more of it! ... or do I?

If I stick to a waterfall approach to testing - i.e. develop & document test plans, test strategies, test cases, execute the tests, log the results and communicate the summaries - then, no, time is not my friend here.

But is it a requirement to do testing this way? Whose requirement? How much does their opinion really matter?

I watched my son play a game recently and describe the "glitches" (his lingo, not mine) to his younger brother so that he could try and work around them. I'm pretty sure my boys don't care whether the software team used waterfall or agile methods, or how well their test cases and processes were documented. They found bugs in their game, are annoyed by them, and figured out ways to work around them. Sometimes they just give up on a game altogether.

Personally, I'd say that the customer doesn't really care about how you do your testing - as long as the end result has good enough quality that doesn't interfere with their intended use of the software or system.

Here's a secret: Nobody cares.

Some lawyers may pretend to care when they are paid to do so, but the reality is that I don't know of a single tester who has ever been charged with manslaughter for failure to document critical test cases that may have caught the bugs that resulted in loss of life.

The FDA doesn't care. Their lawyers tell them that they should care about documented tests and results, so they impose regulations. But the FDA doesn't really care about your documented test cases or test processes. What they really care about is that a minimum standard of due diligence has been performed to demonstrate that a particular product will not harm anyone. That's it in a nutshell. You may not even need testers to achieve that level of quality either.

I could go on, but I think I made the point - nobody cares how you do your testing as long as the collective development effort produces a quality product. You remember "quality" - it's that thing that project managers leave off their project management triangle.

So, if we disregard the premise that testing needs to happen in a "waterfall" fashion, what's left? Well, what do we know? We know that (1) we don't have a lot of time, and (2) we have a lot of features to cover. Oh, and it's also very likely that (3) you have a limited number of resources and people - most likely less than what you'd probably like. (Hey, if we're screwed on the 'time' factor, why not get screwed on the 'cost' factor too, right? ;))

So where does that leave us? Time to innovate! Time to become agile! Talk to your customers; collaborate with your developers and business analysts/product managers; learn the software and functionality as you design and execute the tests because there really isn't time to do those things separately.

Risk-based testing (RBT) works on the premise that there might be something bad/undesirable that could happen, so why don't we start by looking in those places first. RBT is also an appropriate response to the statistical impossibility of complete testing coverage for any useful software program with more than 2 lines of code. That is, if it will take an infinite amount of time to test something, how about if we narrow it down to just some of the areas that we think might be risky in some way (i.e. popular, critical, complex, and so on).

What else can you do? You have a lot of features to cover in a short amount of time. Well, start by ditching all the test documentation requirements and focus on: what is necessary to establish a minimum level of understanding of what's going on.

Do you really need all those documented steps for every test case? No, you don't. Unintelligent automated systems and robots need step-by-step instructions, humans don't. And most humans don't follow the steps consistently either, so just let that one go. Instead, describe the scope of the testing you want to do using checklists and decision tables. The important things need to be discussed in person to ensure clarity of requirements and information, but everything else should be fine with using point form.

Worried about how you will capture the test results if you are denied the Pass/Fail test status column? Work it out! Figure out a solution that fits your project's (and organisation's) needs. There are a number of far more useful alternatives out there - e.g. application logging, screen captures, note taking, and so on.

If you don't have enough time to complete a project using the same approach you've used in the past, it's time to try something new. Time to think up of new solutions, new processes, and identify/create new tools to help you reach those goals.

The end goal is a high quality product.. or maybe just "good enough" quality depending on your situation. The end goal is not to produce sparkling, publishable test documentation. (If it is, consider changing your title from "tester" to "test biographer")

Don't lose sight of what's important. What will you do with the time you've been given? How will you choose to react to the situation?

Read More