Context-Driven Testing, An Education

Coming into my fifth year of Software Testing, I began to rethink it as a discipline. The current debate is between traditional methods of testing and more modern schools of thought:

• Entrenched methods are represented by the International Software Testing Qualifications Board (ISTQB) and its many local equivalents. Certification is their path to expertise, with accompanying wares of training sessions, books, tests and standards.
• Context-driven Testing focuses more on a set of tools and skills that typify testing.  Advocate offer classes at conferences, but certs and best-practice are four letter words.  They state that there are no best practices, and a tester knows best how to apply the tools to explore, experiment and learn about the system under test.

The conflict seemed from the sidelines like a pitched battle over the future of testing. The ISTQB and affiliated consultants had history and 300,000 certifications on its side.  The context-driven school was relatively young, but it had a few charismatic evangelists and professional results that could not be ignored.  It was plain that I needed to sort this out.

Something Just Didn't Feel Right

I strode onto this battleground in 2011 as a new manager and new tester.  Promoted from an application integration team, I was used to working with outside developers while using and abusing buggy product.  This did little to prepare me for the reality of testing: limited time and endless defects!  I dove into the online community in the hopes that it would help sort the good from the bad.  What I found to be the central influences were conferences, consultants and blogs.

At testing conferences, half the talks were advertisements for bug trackers, test case repositories and automation frameworks.  These were great for managing a project, but they didn't support the essence of testing: finding more defects.  Expensive tutorials before the conferences showed similar taint: how to use this tool, best practices for testing this specific type of thing, certification in a process and not a portable technique.  The cost was the most surprising thing: Thousands of dollars for something I couldn't justify to myself, the eager tester with a training budget to burn.

Delving into webinars brought more despair.  A demo of the latest automation tool invariably lead to a pitch to get me to purchase something.  The topic purported to discuss principles, but I ended up in the morass of industry jargon.  I learned how to write, automate, and justify my time, but I was no closer to actually finding bugs in a more efficient manner.  And what's more, I was spammed by vendors for months after.  I found zero techniques that were universal.

Finally, the blogs showed a glimmer of hope.  Some people wrote about techniques.  Others wrote about test cases and how they managed their overhead.  Still others advocated that QA should be involved earlier in the development process.  Nowhere did anyone extol the virtues of their test management system, bug tracker or automation too in helping them find bugs.  This was a breath of fresh air, but it still felt stunted and directionless.  My closest analogue, software development, spawned a generation of people applying agile to everything from families to manufacturing, but there wasn't a similarly powerful framework for testers.  I started to feel that no one was excited about my new career outside of the context of getting paid for it.

This muddled world left me questioning: Shouldn't there be some "ethos of testing" to unify our efforts just like in agile software development, lean manufacturing, and so forth?  Why do vendors have a place at the table at industry conferences?  Why isn't anyone embarrassed that the main competitor to major test management software is a spreadsheet?  Who cares about my expertise and not just my dollars?

"Answers" and Answers

For a long time, I thought the answer was in certification.  Surely, if ever there was an organization that could be a cheerleader for quality, it would be the ISTQB.  However, the reality is much different.  Manuals are filled with process, conferences host vendors and not practice sessions, and training classes are about extracting fees and not learning techniques.  The certification exam that guarantees your resume goes to the top of the pile and organization that proctors it is a laughing stock.

The alternative came through an unlikely route: Twitter.  Long a tool of celebrity publicists and companies looking to engage directly with individuals, Twitter also has a reputation as the way to communicate within a subcultures.  Have an idea?  Publish it in under 140 characters.  Want to learn the pulse of an industry?  Follow its leaders.  Computer security wonks, hackers, and now testers joined my follow list, and I was soon introduced to a new debate: is certification a waste of time?  I'd found my people.

The new school touted something called Context-driven Testing.  Instead of best practice, effective testing was supposed to be driven by context.  Instead of test cases, a set of tools were taught that could be used depending on the product (mainframes are tested differently than mobile devices).  Even among superficially similar products, the most effective testers would make judgement calls based on the needs of the customer and the time available.  Testing was not a set of rigid processes, but a scientific exploration of the software.  The knowledge gained by testing increases the confidence of the organization in the software.  In other words, testing challenges the assumptions made by the developers that the product works in the time we have available.  This sounded like the meat I was looking for, but the results were amazing too.

In an experiment, I had our work study group put down the ISTQB manual with its test cases, and I instead introduced them to exploratory techniques.  We first learned about the requirements and returned a bunch of questions to the developer.  Then we tested without scripts and tracked our coverage on a mind map.  It was the first time we had been prompted to field trial a technique in our study session.  The best part was that a person with very little previous experience in testing was able to pick up the technique almost organically.

This revelation about software testing was what we were all looking for, and it was delivered through experience instead of pronouncement.  James Marcus Bach, one of the proponents of Context-driven Testing compared the ISTQB and certification organizations to the medieval medicine of Galen.  People wrote down what testing was and proceeded to bleed their employers without knowing why it wasn't finding bugs.  Testers were outsourced instead of valued as their techniques were old or ineffective.  Yet in spite of all this, the consultants and conferences kept making money printing outdated works of questionable value.  Once context-driven techniques come to light, the old ways start dying.  We can only hope this continues so that meaningless certs are no longer valued by testers, managers and HR alike.

Where Next?

After stumbling through the world of testing for a few years, I have abandoned certification as a path to expertise.  As with computer security, network administration and technical support, certifications are a poor way of communicating true expertise.  This revelation places testers firmly in the camp of indispensable elements of the development organization.  They are not monkeys running scripts but knowledge workers with a valuable investigative skill that challenge the product from all angles.  They cannot be outsourced if you hope to be successful, and they cannot be replaced by automation.

I am beginning a new training regimen with my testing colleagues based around Context-driven techniques.  We hope to learn the techniques and apply them to our current projects and continually grow our skills in this new framework.

Further Reading

1. A Context-driven Testing manifesto of sorts
2. Black Box Software Testing, Coursework in Context-driven Testing
3. Rapid Software Testing, James Marcus Bach's courses on Context-driven approaches:
• General Information
• Course Slides and Commentary
• Appendices
4. Exploratory Testing techniques explained
5. Testing is not Automation.  Why Automation is another tool and not a cure all.

2014 Board Game Buying Guide

I've been playing games with people at the office, and it has shown me that you can get a game in with even a short lunch and new players.  I decided to put together a quick list of game recommendations based on venue and number of players.  Links to Amazon are provided, but most of these titles can be purchased from your Friendly Local Game Store.

Lunch

Bring co-workers together, let off some steam, and renew rivalries between departments with this selection of quick and easy games selected to allow you to teach and play in under 30 minutes.

• Love Letter is a card game of bluffing and card counting with just 16 cards.  It handles 2-4 players, but it is best with 4.  Also, it comes in a number of different re-themes, so you can get a non-threatening version to appeal more to co-workers.
• Tsuro of the Seas handles 2-8 players.  This "last man standing" tile game has compelling artwork, fast rounds, and dead simple rules.  A whole game seldom lasts more than 15 minutes so people can jump in with little fear of going over their lunch time.  There are simpler versions and expansions, so you can get as much depth as you want in the time allotted.
• Hey That's My Fish plays well for between 2 and 4 players.  You should be able to explain rules while setting it up.  All about area control, the gradually shrinking board is apt to cause panic even in the normally stone-faced players from Finance.

Date Night

Games that play best with or are designed for just two people can test the limits of your affection or bring you closer together (no warranty either way).  Though these games will work at work, teaching might take longer so an hour lunch is preferred for newer players.

• Hive is one of the few abstract games on this list (like chess or checkers).  It presents the elegance of chess without a board, and the pieces have heft and make a wonderful clacking sound like Majong tiles.
• Jaipur is a colorful trading game whose tokens and cards make a visually impressive setup.  The 2 of 3 mechanics means it is natural to play round after round.  This game used to be rare, but it has benefited from a recent reprint.
• Star Wars X-Wing Miniatures Game maxes out the nerd factor.  This is as close as you can get to a war game, so check with your significant other before taking the leap.  Games run about 45 minutes, and it plays in a space the size of your dining room table.  Best of all, there are endless expansions to add iconic and obscure characters to your fleet of ships.

Family/Friends Game Night

Have more than two?  Hate Monopoly as much as I do?  This mix of competitive and cooperative games are sure to show you what the modern board gaming renaissance is all about.

• Forbidden Desert is a cooperative game where you play against the game to avoid thirst, storms and the heat to get out of the desert alive.  The mechanics and components are top notch, and it plays in under an hour.
• In King of Tokyo, players take the role of giant monsters vying for control of Tokyo.  A fast paced dice game, the randomness and ridiculousness of it all is a hit.
• Card game The Builders: Middle Ages is a great worker placement game: hire and set workers to build a town.  The rules are simple, but the gameplay is complex.  This definitely benefits from replayability and the tin means it is sturdy enough to go anywhere.
• Love Scrabble but have that one friend that outstrips you every single game?  Try Qwirkle for scrabble like crosswords without the burden of words.  It is more puzzle than game, depending on how cut-throat you get.
• Ticket To Ride: Europe is rummy with a board, and this edition of the game is fantastic.  I learned more geography from this game than I did in grade school, and the artwork is fantastic.  Expansions are available and cover alternate maps and expanded ticket options.  It even has an App version you can pass-and-play if you don't want the fancy box.

Clark County, Nevada Elder Abuse Resources

I was concerned for the safety of a family member who is older not long after their spouse passed away.  Below are some things I learned about Elder Abuse, the resources available to help those in need (individuals or family), and things to do when investigating elder abuse.

Before going nuclear on someone new in your relative's life, first do the single most important thing: talk to your older relative.  Often, misunderstandings or matters of privacy can be sorted out without resorting to law enforcement, state assistance or subterfuge.  The matter of trust between you and your relative is the single most important factor in maintaining their long-term health and well-being.  If you lose their trust, you lose almost all ability to help them.

Local Police Resources

Police seem to only be able to make 'welfare checks' for elderly people that outsiders suspect are being abused. They can only visit the premises when the person is home. The Las Vegas Metropolitan Police Department Operator and Dispatch informed me that there are dedicated Elder Abuse detectives. Unfortunately, they only operate M-F, 7-4. As the crisis was after this time, we couldn't get a welfare check immediately. The numbers for these departments are below:

• Metro Operator: 702-455-8697
• Metro Dispatch 702-828-3307
• Elder Abuse Detectives: 702-828-3111 (Hours are 7-4, no voice mail)

State Resources

Though I I have not taken advantage of these resources, there may have been help available through the Aging and Disabilities Services connected through the county. Comparable services may exist locally in your neighborhood.  Perhaps there are some interventions that would be helpful going forward?

• http://www.shouselaw.com/nevada/elderabuse.html
• http://www.health.nv.gov/HCQC/SB129ElderAbuseTrainingforWebsites.pdf
• In Nevada, crimes committed against elderly command double the sentence.

Social Engineering

When trying to find out more about people that have entered a loved one's life unexpectedly, unexpected phone calls from unknown people are a great source of more information.  Generally, act as if the person is at home but not available.  The person on the other end of the line may divulge information that gives you clues about the intruder.  Effective phrases are below:

• "Yeah, he's here but he's busy.  He asks what you need."
• "Hold on, let me get her...She's here but can't pick up right now."
• "Who is this again?  I didn't get that down last time."
• "His phone is dead.  What number can he reach you at?"
• "What was it again that you're meeting for?"

While the person is out, check the circumstances of the house, but try not to disturb things too much.  Look for signs of drug abuse, behavior you know your relative would frown upon such as smoking and drinking in the house. If you know their location, ensure firearms are secure, and check the status of belongings, heirlooms and money caches.  Document and narrate your search by video.

If you must get the police involved, minimize the impact on your relative.  See if they will come around when you try to have a person escorted off the property.  Ensure your relative is not involved in any illegal activity before involving the police, and, most importantly, get the consent of your relative before escalating.  You must maintain their trust, and asymmetric reactions to otherwise benign or diffusable situations can ruin that bond and expose a vulnerable relative to harm from both the intruder and the police.

R00tz Asylum 2014

I took Ethan to the event run in parallel with DEF CON, R00tz Asylum.  I think he had a blast as they covered a lot of traditional hacker topics at multiple levels of complexity.  The highlights are below.

Structure

The event was held in the Crown Theater at the Rio.  It was about a 10 minute walk from DEF CON proper.  The separation was nice as it made for a more quiet and contained experience.  The stage was occupied by a speaker almost all the time.  Spread around the perimeter (mezzanine?) were tables with activities that changed every day.  Kids could choose to listen, play or work on challenges.  Most activities stayed the entire day, though some were more transient.

This setup was advantageous for my son.  He has little ability to focus on any one thing for an extended period of time, so the variety of activities was nice.  Much like its parent conference, R00tz Asylum did well when it focused on hands-on learning.  Toool, Google and Wickr held contests and learning opportunities that pushed attendees and their parents to participate together.  In particular, Ethan loved the puzzles, and I finally got him to solder something.  He did a bang-up job.

Speakers

The speaker experience was less than optimal with a few notable exceptions.  The stand-outs were Gene Bransfield's hilarious "Weaponizing your Pets" and Meredith Patterson's engaging activity "The Telephone Game" about Man-in-the-Middle attacks.  Special mention goes to @muffenboy and Esau Kang for being kid attendees and speakers.  For the rest, it would be good to learn that speaking to children is not the same as speaking to hackers, and most talks were too technical, lacked a hands-on component, and thus ended up being torture for the little ones.  From speaking with the organizers, I can tell this is something they are trying to focus on next year.

The Gift

R00tz Asylum is the opposite of DEF CON in one respect: it relies on sponsors to add pizzazz and to make ends meet.  One of those traditions that may or may not hold in coming years is the gift of a hackable piece of technology to attendees.  This year brought ASUS Chromebooks care of Google.  My son was enthralled, and I spent most of the conference convincing him to get off the Chromebook and out to the activities.  By the end of the conference, we had Linux in addition to Chrome, and we were running Wireshark thanks to perseverance by Joe and Chris, a father/son team.  This effort won Chris a trophy, even.  My son begged me to put Minecraft on there, but then quickly forgot how to get back to it and reformatted his Chromebook undoing all our hard work.  Hats off to Google, and congrats to Chris on the win.

Hardware Hacking

By far, my favorite part of the conference was the Hardware Hacking table.  Not only did the goodie bag include a HakTeam Throwing Star LAN Tap, but a table full of old equipment was available from which attendees could rip apart and salvage components.  The LAN Taps were used in an activity that taught wireshark and packet sniffing.  The hardware component salvage table was exploited for speakers, LEDs, gears and motors for all sorts of toys.  I am definitely bringing projects for Ethan next year.  I already recommended the salvage table to the official DEF CON Hardware Hacking Village.  Las Vegas thrift shops may see a run on their printers, VCRs and routers before next year's conference.

Lock picking

The one talk and table I was surprised that Ethan was interested in was from Toool.  Their interactive 101 talk caught his attention, and we worked on a lock at their companion activity table.  Though he ended up losing interest before successfully opening a lock, it gave me a clue of the type of activity he could do on his own between conferences.

Going Forward

I would definitely recommend any hacker parent to bring their child to R00tz Asylum.  Its expanding and evolving to be a great summer camp weekend that dovetails with the DEF CON experience.  As the organizers ger more experienced, I expect the content to grow and change to fit the kids and their interests.  We all started somewhere, and I hope R00tz is that start for the next generation.  I started a subreddit for R00tz, though it hasn't taken off.

As for Ethan and I, we are preparing a talk on how to hack Skylanders figures.  We hope it will be a fun combination of encryption, hardware hacking and games that will draw the attention of attendees and inspire them to really dig in and explore the technology that is used around them every day.

Resources to Check for Dead Links on a Website

http://www.brokenlinkcheck.com/
This external service hits your site and follows each link.  It is intelligent enough to check for loops.  Since it is an external service, it may artificially drive up hit count.


Check My Links Chrome Extension
This very handy chrome extension checks link status and places the results in-line.  Fairly turn-key, it even keeps a list of links to not follow that you can add to as you go along.  It seems to have trouble with blogspot controls and extensions, though adding them to the blacklist might be the solution.

Xenu's Link Sleuth
Heard about this one on UTest.  I am eager to try it out.  Free, long history, and automatable: all the right pieces for success.

Add more as you find them to the comments.

StarWest 2014

I viewed StarWest's Virtual Conference offering again.  This and the affiliated Better Software conference are run by Techwell.  A few observations.

1. I loved the talk on Healthcare.gov given by Ben Simo (@QualityFrog).  He communicates how easy it would have been to predict, find and fix the problems that would plague that site for more than a year.  It was a good choice putting him on keynote.
2. To attend one of these conferences will run you or your company into the thousands of dollars.  Attending the tutorials is even more.  This in spite of being sponsored by some of the biggest software providers in the industry.  We are bombarded by ads for the latest ALM or bug tracking tool and they are called talks.  What is such sponsorship getting the attendees?  Who is benefiting from this other than the organizers?  If the conference organizers were a not-for-profit, would they charge the same amount?
3. The online offering tries to simulate the networking opportunities for those who could not attend.  It tries to simulate the marketing side too by giving attendees contact info to vendors.  What about the testing opportunities?  With more than half the talks about web app testing, why aren't tutorial sites and learning apps available and promoted to virtual attendees?

Maybe DEF CON has spoiled me.  $200 for the most frenetic hands-on conference over twice the number of days?  A lot of that is a labor of love and volunteers, but then again most of it is also not sponsored by corporations too.  Maybe I need to bring DEF CON to testers, or testers to DEF CON.  See what shakes out.

Cards Against Mormonism

My brother and I grew up Mormon, but we no longer identify as such.  Nevertheless, the culture is unique, and there are many opportunities for squick, in-jokes and dark humor.  So, like everyone and their mothers, we decided to roll some of those up into an unofficial Cards Against Humanity expansion: Cards Against Mormonism.  A reminder: you may not get some of the jokes if you have never been Mormon, lived in Utah or or are otherwise considered a Gentile.  However, we took great pains to limit the amount of Utah-specific jokes.

The cards themselves were brainstormed under the influence of alcohol and put through the refiner's fire until about 100 unique cards (33 Black and 80 White) emerged.  Once the list was solid, we copied it into the brilliant Cards Against Humanity custom card batch processor.  It worked flawlessly the first time marking Pick 2's and adjusting fonts as needed.  This spat out two PDFs perfect for printing your own Mormon expansion to Cards Against Humanity.

This set is formatted to match the free print-and-play PDF still available on the CAH Website.  Like that set, it is shared under a Creative Commons BY-NC-SA 2.0 license.  You can see the original doc with those that did and didn't make it in Google Docs if you want to try your own hand at it and bring some apostasy to game night.

Sincerely,
DuncanYoudaho and FannyAlgersAbortion

Download Links:
White Cards
Black Cards