Test Early, Test Often

Of late, I have been enamored of testing techniques that come earlier and earlier in the development cycle. It can be called static analysis, design auditing, prospective testing, shift -left or the like, but the research is in: testing before you get something bears fruit in most organizations.  Here I present a few examples from my own experience.

At the start of a sprint, we leave Sprint Planning with the requirements.  The next interaction with developers is when we review their Developer Design Overview document that spells out the development approach and helps QA scope their testing effort.  This developer had chosen to put an error message into a file usually reserved for configuration.  QA saw the DDO and raised concerns immediately.  Why was a message being added to this file when they were usually reserved for the language DB?  With this one question, before QA saw the code, we changed the trajectory of development.  The fix was in before we got our first build, and the story closed with the Sprint instead of carrying over with the do-over.

An even earlier example came when we looked to implement secure communications between two servers.  While I couldn't code my own implementation, I was able to provide recommendations at design-time by staying educated and confirming my understanding with developers who had dealt with crypto.  By starting early, we were on surer footing when troubleshooting and confirming the implementation was sound.

As the examples above illustrate, QA often saves time for developers by defending standards and consistent implementation early in the cycle, but that is not the only savings that comes from shifting left.  Often, test environment issues can also be aided by an early understanding of requirements.  In one case, as story had carried over from a previous sprint which meant we were already behind.  The roadblock was a production issue pulling the developer away from the story.  Instead of sitting on our laurels, QA worked with the configuration manager to make sure our test environments were ship shape before the code was completed.  When the developer's changes passed build verification, we were off and running almost instantly.  Not only did our preparation help us get to the work of testing faster, but it also helped us close more stories as environments were made ready before they could become an obstacle. Not only was I able to test early, but it lead to me testing more and in greater depth.

Most modern test engineers have their own war stories from early testing.  For every story where requirements changed and early notes became meaningless, there are ten stories where early questions lead to greater clarity, fewer bugs, and more time for digging in.  I consider projects that foster this early access for QA to be among the most fruitful and least volatile.

RadioShack LED Strip Driver

I modified the Pololu RGB LED Strip drivers from version 1.2.0 to support Radio Shack's behind the times model that is 30 LEDs controlled in 3-diode sections.  I had to swap the colors around to match this pinout, and I changed the struct to a class (because why not).

The fix was to physically reorder the declaration of red/gree/blue variables in the struct declaration.  This way, when the information is written to the strip, it is sent in a different (and now correct) order.  You can make the fix yourself by changing the file PololuLedStrip.h:

typedef struct rgb_color  {    

   unsigned char red, green, blue;  

} rgb_color;

becomes:

typedef struct rgb_color  {    

   unsigned char green, blue, red;  

} rgb_color;

And here it is on GitHub: https://github.com/RangerDan/RadioShackTricolorLEDStrip

I should probably talk to Pololu on licensing concerns here.  I found the license from the original driver and copied it into my repo.  I couldn't figure out how to fork this properly, so I just re-uploaded it until I understand git a bit better.

C3BO: Proof of Concept using Timbermanbot Schematic

This post is part of a series about building electro-mechnical PIN-cracking robots, R2B2 and C3BO.



This is a proof of concept for @JustinEngler's C3BO (https://github.com/justinengler/C3BO) using transistor controlled relays. It was prototyped by modifying Blink from the Arduino sample project.

The schematic was obtained from Timbermanbot (https://github.com/vheun/ArduinoPlays...) as seen on Hackaday (http://hackaday.com/2014/07/26/pwning...).

In the video, You'll notice I've replaced the touchpad for your finger with a wire to the headphone jack's ground as the circuit ground. The two pieces of copper tape were no longer sticky enough to stay by themselves, so I am holding them down. They press two and 5 with about 8 key presses per second.

OFBC: Putting it All Together

Note: This is part of the Project Write-up for OFBC: One Fluorescent Beer Coaster

After months of effort, we had a circuit, PCB and shell design to accomplish our goal.  Putting it all together meant solving some unique challenges in the home stretch.  By far the most communal part of the project was finishing the circuit.  Parts were bought by three different people.  It took hours of trial runs and four different nights in my shop to finally get the circuit assembled and ready.  In all, the project taught us to keep moving in spite of obstacles.

Internals

The main obstacle was PCB manufacture.  As detailed in that post, uncooperative copper and etchant lead to abominations not fit for solder.  Drill bits broke in PCBs, holes were misaligned, and traces were torn up as we worked and reworked the boards.  The major blunder was the reversed PCBs, but it was tempered by the lack of polar components.  Only the transistor and MOSFET needed to be adjusted when we realized our mistakes.  The quality checks and encouragement as we worked as a definite plus.  There were several times I wanted to just give up and abandon the project.  Truly, I get by with a little help fro my friends.

After the PCBs were in our hands, the task of soldering all the components was a team effort.  One person ran continuity tests on newly etched boards.  Another bridged scratches and pasted down traces.  Buttons (functional and fake) were inserted and crimped at one station while a fourth person began to solder on components.

That moment of truth when the LED lit up was breathtaking all nine times it happened.  When it, more often than not, didn't work on the first try, the scramble to troubleshoot was a team effort as well.  A loose connection, bad trace or through hole in need of a reflow was rooted out in minutes. I can't describe the feelings from closing the box with nine functional copies of the idea sketched out on a picnic table the year before.

Externals

Shell manufacture forced choices between what we wanted versus what we needed.  The mechanical ideas at the outset gave way to manufacturing considerations.  Features were pared back to match timelines, work schedules and summer vacation.  Anyone reading this who has worked in an Agile Development environment will recognize similar decisions they make every Sprint.  To borrow a cliché, "Perfect is the enemy of good enough."  With this in mind, we have an eleventh hour compromise ready: should the 3D printer prove a roadblock, we have arranged for a Wednesday night Hail Mary meeting to turn Ziploc Containers into eternal glory.

The Ziploc idea produced 4 "just in case" models.  We stabilized them with glass beads and hot glue.  The containers became the shell and mount for the PCB.  The beverage lid was provided by another ziploc container hot-glued onto the buttons.  Hot glue for grip and stabilization of the platform finished the job.  See the result in the pic below next to the finished shells.

Luckily, the 3D Printer roadblock was cleared just two days before the BBQ.  Poor quality filament lead to clogged extruders.  After a good cleaning, we were back in business.  5 shells total were produced with various upgrades.  We got a top that nested well with the shell, and the mouse-hole in the shell was added to allow the USB to be passed out of the body.  We did not get impressions in the top to get the lid closer to the lens of the LED.  We also did not get any part of the body held together by magnets.

Final assembly took place at Toxic BBQ itself.  The lights stayed on this year, but we started conversations and passed out some business cards with links here.  We placed a few on the tables farther out that didn't have light, and we presented two to the organizer in a Utilikilt.  Furthermore, it went on display in r00tz and the HHV for most of the convention.

Final Word

I left DEF CON for two years running with a profound sense of my own shortcomings.  I saw people around me doing amazing things, but I couldn't point to similar achievements for myself.  Though not terribly complex (most ideas came from Instructables, after all), the process and coordination required to pull off this simple idea has been eye opening.  It all started by pivoting from planning to doing.  It finished with an 80's-montage-worthy string of late nights and high fives.

Already, these efforts are fertile ground from which numerous other ideas have sprung.  Facing another DEF CON, I'm looking for the next big project instead of lamenting my noob credentials.  Only time will tell how many of these work their way to reality.

OFBC: Shell Fabrication

Note: This is part of the Project Write-up for OFBC: One Fluorescent Beer Coaster

Design

In parallel with the circuit, we designed an enclosure that would be more sturdy than hot glue and disposable containers.  The general idea was a shell with a lid that had some travel.  Our first designs focused on a mechanical clip to lock the lid in place.  Further ideas were a rail to keep drinks stead, a drain channel for condensation, and an interlocking base/top for easy stacking.  Our lack of expertise with the 3D design software and the complexity of the print made us go back to basics.

More Design Shots

Similar products used a coaster shape, so we started there. As the whole point of this project is to show off, we wanted to make it easy to disassemble top and bottom. To make this happen, we settled on magnets instead of screws for both top and bottom.

First Full-size Print (with Frenchman Mountain in the Background)

Problems

First, the 3D Printer needed to be calibrated, then the extruder needed to be cleaned, then the Kapton tape needed to be replaced.  After a much better test print, we got our first dimensional fit.  Once the PCB was together, we realized two things: 1) the buttons we bought had a much heavier mechanical action than the test buttons and 2) the LED package was too tall for the way we printed the top.  Combine the two, and a lot less light was reaching our girly drinks.  Something had to be done.  The above problems were noted, and various other edits were written directly on the 3D printed shell (a sharpie on white ABS works wonders for clarity).  They were handed off to our man with the printer while the rest of the team worked on PCB fabrication.

Between orders of Kapton Tape - Let's try Painter's Tape!

It was about 10% too small.  Looks like a job for Superm*n!

Not nearly as bright as it should be

Solutions

The final prototype came together the weekend before DEF CON.  Edits to allow the charging cable to escape from the bottom of the case, a drip cover to prevent condensation from entering the shell, more accurately nested tops and bottoms, and a host of other small changes came together for the final prints. In all, the first run is bulkier than we imagined, but we have discussed ways to miniaturize and reduce costs across the board.  It will be something we are proud of showing off.

The 3D printer was the final obstacle.  From miscalibration to a clogged extruder head to a stepper motor burning out, we had our fair share of problems getting the final package in a physical format.  If the repairs don't come through, we'll be manufacturing stand-ins for the Toxic BBQ.  Nothing can stop us at this point. 

The current Sketchup files will be available on the OFBC project on Github.

Great Size, Less Filling

OFBC: Random Design Shots

Note: This is part of the Project Write-up for OFBC: One Fluorescent Beer Coaster

Puzzling Out the Protoboard

Can you Smell the Confusion?

Shell Designs

Initial clip-together design with rails for stable drinks

Circuit, post and spring and modified clip-together designs

Hockey Puck design emerges, Boolean shape building discussions

First practical attempt at dimensional design

OFBC: PCB Fabrication

Note: This is part of the Project Write-up for OFBC: One Fluorescent Beer Coaster

From Protoboard to PCB

The next step along the path was to turn a gawky mess of a protoboard into an elegant example of good design.  This step took a lot less time than I thought it would thanks to Fritzing.  Billed as "Electronics Made Easy", I installed and got up to speed in under an hour.  Conversations with my compatriots helped me tweak and massage the design to our satisfaction.  The end result is a 2" (58mm) PCB for through-hole components.  This will secure the buttons, driver and LED while connecting to the battery.  The experienced among you are probably thinking how absurdly large that it.  It could be a lot smaller, but I consider it acceptable for a first run.  As with other projects, the latest version of the fritzing file will be available on github.

First Run

If I had to pick one part of this project that made me more uncomfortable than any other, it would be the PCB fabrication steps.  I took chemistry in High School and College.  I know the basics.  However, I don't know enough to do it confidently.  I took my queues from MAKE's excellent video tutorial, acquired chemicals at Frys, harvested glass from a recently disabled printer/scanner, and printed transparencies at FedEx Kinkos.  My exposure light was a 26W CFL in a desk lamp.  My red light was a red LED straddling a button cell.

Sneak peek at the Shell prototype

I removed the board from the developer too early or exposed it to too much light. This caused a large region of copper to not develop. There is no way to align and cut a ton of these after the fact. The PDFs exported from Fritzing come out one per page. This means they need to be done one at a time unless you have the skill to post-process the PDF into multiples per page. I pre-cut the PCB's during subsequent runs. My table saw made short work of the big board, and the pre-sensitized copper has a sticker over it that allowed me to cut the board to pieces without compromising its ability to accept an image.

Oops 

Once the etching was complete, I drilled out the traces.  The prototype board was drilled using a 1/16" bit.  This was way too big.  Out of all the bits I tried, normall through-hole components worked great with a 1/32".  A 1/16" bit was required for the MOSFET, however.  The best set I found was one for the Dremel.

Modern Silk Screening ain't got nuthin' on Sharpies

The only part of the process that ended up being perilous was the disposal of the ferric chloride.  The leftovers are back in the bottle.  I'll take them to the waste disposal place soon.  My driveway has a nice big rust spot on it from where I washed off the etchant.  How am I going to explain that to the HOA?  Ultimately, the problems with Ferric Chloride lead me to a different etchant entirely: Cupric Chloride.  See below.

Production Run 1

With the test run experience in hand, we were set to make an attempt at our first full run.  I chopped the boards on a table saw first.  This was a stunning success.  I also redesigned the PCB to include a slide switch to kill the circuit.  This allows long-term storage as a shifting bag or box won't depress the pressure switches and drain the battery.  You might see on the random design shots how we were planning on mounting the LED to the lid.  This changed before the final production PCB run, and we moved all traces outward to make room in the center of the PCB for the LED module on heatsink.  I also added a copper pad in the center to maximize heat transfer.  Some thermal paste will seal the deal.

The exposure took place in the half bathroom.  With access to water and no exterior windows, it was perfectly suited to etching.  Our supplies were:

1. Tape, transparencies and pane of glass.  The circuit image is taped to the glass in a double-thick layer.
2. Positive Developer mixed 10 to 1 in a glass pyrex.  When in doubt, use pyrex to ensure things won't melt through the container.
3. Desk lamp with a sifficiently bright CFL bulb.

And our steps were:

1. In darkness, peel off protective layer from light-sensitive copper clad board.
2. Position the board over the top of your image and tape down.
3. Flip the glass pane over and double-check the board is positioned correctly.
4. Expose for 8 minutes using the lamp.
5. In darkness, remove the board from the glass, and place it in the positive developer.
6. Swirl the PCB in the solution until the image appears.  If your developer is sufficiently diluted, longer development times will be experienced.  It is better to over-develop and start to lose the image than it is to under-develop and end up with no traces at all.
7. Wash off the board when it is sufficiently developed,

Final Exposure Workstation (The Guest Bathroom)

After exposure, good boards were placed into the etchant.  The Ferric Chloride was a great big mess.  It was hard to see how the process was coming without fully removing the board.  In addition, it needs to be heated to be truly effective.  Las Vegas has an ambient temperature of 100F/ 38C, and it still took 30 minutes per board.  You can see the etched boards below.  Before soldering, the etch-resist is removed via acetone.

Finished Product on the Plate

Great Success!

Production Run 2

With DEF CON a week away, we didn't have enough boards to complete our goal yet of nine complete lights.  We met for what we thought was our last etching party.  Much like the previous time, we decided to expose and etch using Ferric Chloride.  These boards looked great.  It was obvious we were starting to figure out how to do this effectively.  Unfortunately, we also forgot to check the boards as they were produced.  All 3 good boards were mirror images of what they should have been.  DEF CON loomed large, and we went with the more radical solution: switch etchants and try again.

The new etchant relied on Cuperic Chloride.  Once again, I turned to Instructables for a helpful tutorial.  The key ingredients were muriatic acid and hydrogen peroxide.  As the link shows, the acid and hydrogen peroxide oxidize the copper to form CuCl.  This in turn oxidizes to form 2CuCl by stealing copper from the PCB.  What's better, it needs an acid refresh much less often, and it is completely reusable.  A bubbler or aggressive mixing causes oxygen to oxidize with the 2CuCl and start the cycle again.  We obtained new PCBs (see my rant about Fry's below) and went to town.  We ended up with the 9 boards we needed; we began assembly in earnest.

Lessons Learned

The first board we did had the light placed too close (1-2 inches).  Also, the positive developer was extremely strong.  This caused all of the etch-resistant coating to wash away.  A little more water and moving the light 6-8 inches solved that problem.  When we were doing it right, we waited for the image to appear and then become crisp.  It is extremely difficult to tell in the dark if the image is still cloudy or not.  When in doubt, dilute your developer and leave the boards in there longer.

As with my woodworking posts, the matra is, "Measure Twice, Cut Once."  After every critical step, we had one person check another's work.  This saved us time and again from poorly aligned boards, undrilled holes and bad decisions at the bench.  It didn't save us from all screw-ups (a full crop of mirrored boards), but it saved us other embarrassments.  It also exposed every team member to each point in the process.  In total, five people participated in the manufacture of these boards.  Most have projects lined up that take advantage of things learned along the way.

The boards themselves were a problem, and they highlighted a weakness in the supply chain.  All copper obtained from Fry's failed at least a third of the time.  Online, the consensus was that the boards were old or improperly handled.  The positive developer was much stronger due to being partially evaporated.  The buttons we obtained were stiff and of differing quality that those used in prototypes.  All in all, I would recommend avoiding Fry's if you can help it.  They might have enough materials to get you going, but Amazon or similar suppliers can get you what you need fast enough that it makes no difference.