We Don't Know What We Don't Know

I observe much of the world through the lens of risk and quality. As a student of what makes some things better than others, I often discover common characteristics and attributes that indicate something is of good quality. Similarly, there are those things that give a hint that we are looking at an illusion of quality.

I was recently helping a friend assemble a piece of flat-packaged furniture. I removed the various components, screws, dowels, and an instruction manual and began the assembly journey.

The veneer on the panels was a beautiful beach wood, something most would find appealing, but just a millimeter under that veneer lay a compression of wood particles, resembling what a carpenter might discard after a busy day.

I imagine you have assembled one of these pieces of particle board furniture, maybe in your college days or possibly the furniture surrounding you at this very moment. The veneer lament sets an expectation of quality, but the flimsy layer below reveals the actual quality of the product.

To assess the quality of something, it’s necessary to look beyond the marketing photos and observe the product from different perspectives, such as:

Was this designed to last a lifetime or a few years?
Is its appearance consistent with reality?
Will it perform and function as advertised?
Does the manufacturer have a reputation for quality products?

The Quality Illusion

There’s an art to the illusion of quality. Most of us have enough experience to know you’re likely not getting lavish furniture on sale for $59.99 at your local superstore. Remember, there was a day when you were not as insightful and experienced. You purchased a product believing it was more than it was.

There is also an art of illusion when it comes to the testing of software. Most customers need to realize that what has been delivered to them in the name of good quality testing might be a veneer of quality adhered to a particle board foundation.

I once worked with a Test Engineer who refused to perform test planning. His approach was one of the laziest forms of testing I know, exclusive ad-hoc. While he had a sense of where defects existed, he robbed his customers by never planning his testing approach and instead dazzled them with a good bug now and then. The unfortunate part of this story is that the customer bought it. They trusted this engineer to do a good, quality job but were actually sold a bill of goods, a cheap, quality veneer on a particle board foundation.

This is an example of veneer test engineering, and it set precedence with a customer and other test engineers that this lazy approach is acceptable testing. The accolades this tester received in the absence of trustworthy testing went to their manager’s head as it also made them look good. I call this a Shift-Up defect as the management of the Quality Team had now introduced risk into the enterprise and product lifecycle by allowing the veneer of quality to become the acceptable standard. The story finishes with the promotion of the lazy tester to manager, incompetent in quality engineering, and now encouraging quality veneering practices to his direct reports.

What went wrong?

Leading a Quality Engineering team is one of the most challenging roles in the technology organization. The manager is responsible for ensuring their team has customers to work for with budgets to cover their pay. They also have the reality that their team’s work may result in defects discovered that the customer would rather not know about. The QA Manager needs to be firm in character and integrity to deliver accurate test results to customers.

Every quality manager stands at a crossroads of important decisions in their career.

1) Do they support and train their test engineers to perform in-depth quality testing and report what they find even if the results are not what the customer wants to hear?

2) Do they skew the presentation of the results to keep customers returning even if the quality of work performed is nothing more than a thin veneer of quality?

Training and Test Planning

Ongoing training is a necessity for Test and Automation Engineers. There is little formal training in our field and plenty of poor examples of how to perform it. I believe strongly in test planning, which does not mean a fifty-page Word document every time. I believe the Test Engineer needs to have a strategy for their work and not simply perform ad-hoc testing (as fun as that might be). Test planning does not need to be tedious and something a Test Engineer dreads. Have an established approach to applying reason and strategy to what needs testing in the given timeframe. If you’re unfamiliar with the METS (Minimal Essential Testing Strategy) method, this could be an excellent place to start your team with basic test planning. You can learn more about METS at METSTesting.com.

Setting Expectations

A critical responsibility of the Quality Manager role is setting realistic expectations of what their team will deliver. So many individuals find themselves at the helm of leading a QA Team, rarely with any background in testing yet given the assignment to hire people and test stuff. The team frantically tests all sorts of things in all kinds of ways to find “all” the bugs in an endeavor I call “The Testing Miracle.” Add automation to the mix, and you have testing madness.

All this turmoil could have been alleviated by having a real-world expectation of what a QA Team should bring to the development lifecycle. The following is how I would summarize what a QA team should be offering. A QA Team should perform testing experiments executed on an application and the changes made to that application, analyzing and identifying where risks exist within a given timeframe.

All software contains defects, even well-tested software. The work of a Test Engineer is to apply their training, curiosity, and experience to uncover and report on the defects discovered. If a different expectation is cultivated, such as “find all the bugs,” which initially seems like assessing risk, the team will depart from sound testing practices towards a testing fantasy. Cue the music.

Product Owner Confidence

The product owner needs to have confidence that the Test Engineer will present their findings, whether good or bad. They also need confidence that those managing the QA Engineers are providing continuous training to their team, ensuring they test and deliver the most accurate results possible in the timeframe given. If the QA management is compromised, the team is sure to follow.

The Fast Farse, Encouraging Quality Veneering

A madness is driving more organizations to encourage quality veneering practices, and it’s rooted in a single word, “Fast.” When did you hear anyone want to get something done slowly? We all generally desire to get things done as quickly as is reasonable. When a company culture is driven to perform things fast for fast sake, you have a farse on your hands and likely a compromised manager or executive in the ranks, delivering something regardless of its quality. This fast-for-fast-sake mindset has probably adopted a nonsensical explanation that we will fix substandard products offered to our customers faster. Those who lead an organization in this manner encourage compromise and quality veneering.

Consider an assessment of your goals and those that lead your quality organization. Are they encouraging their Test Engineers to grow in their craft or speed things up regardless of what they deliver to their internal customers? Is their leadership valuing the honest reporting of risk discovered or error on the side of rewarding fast-for-fast sake irrespective of the product’s quality?

The craft of delivering quality and risk insights seems to be in a state of compromise. Delivering a quality veneer hurts our entire test engineering community and leads many to ask, was that worth the cost? If you’re looking for your Test Engineer to deliver a valuable product for the monies allocated, look for more than simple pass-or-fail results. Look for details that show risk analysis and good test planning.