It’s a common age-old adage, the ongoing war between Software Engineer and QA Engineer, the eternal struggle between the bright-eyed, and those tasked with enforcing practical reality.

I mean not to demonise either party here, this is exactly the picture that can be seen if both are performing their roles adequately.

It is very important to understand, at least from a Software Engineering perspective, that QA Engineers, not out to make your life difficult, and often will be happy to help you work out the kinks… mostly.

Neither is a Software Engineer too lazy to verify all the potential routes a user may or may not take, it’s only natural when building, to imagine only how something should be used.

As with many things, balance is the key, it is important to balance QA requirements with development capacity, and practical architecture standards. The reason for this comment is that the overuse of testing and “quality assurance” can result in product stagnation and a less-than-ideal development velocity.

I remember something I had been told quite early on in my career, by my then CEO, on the importance of shipping a first version. Whilst I initially agreed that this was true, it took me some time to ponder on the real benefits and the thought process behind this advice.

The topic had arisen whilst discussing side-projects, and delaying releasing an initial version, due to apprehension in the resulting version satisfying my arbitrary standards. This wasn’t a recommendation to relax my principles or become lazy, but to optimise where I spend my energy, especially since these were side-projects for which I had a limited capacity whilst working full-time.

I had fallen victim to two of my most disliked practices in modern software development.

  • Executive Driven Development
  • Nirvana Fallacy

Executive Driven Development

You may or may not already be familiar with the phrase, but you will most certainly be familiar with the practice. This is where the decision on what features to build come from the whim of an executive, or other stakeholder, because perhaps they saw this feature in another app and thought it was cool; or more likely, it came to them in a “dream”.

Executive Driven Development is one of the most dangerous issues facing most companies, and has a direct impact on the company’s viability in the marketplace. All too often, have companies striven to create a fanciful application, managed to eventually hit the market, only to find there was no demand in the first place. Then, having wasted all their resources on getting there, are forced to either rebuild, pivot, or liquidate.

How not to do this? Build the smallest possible version of your product, and ship it. Deliver your MVP, or your MFP, or whatever luxurious sounding acronym you’d like to use for the smallest possible unit of work.

Once it’s out there, work on getting feedback, use analytics, user interviews, user testing, whatever tool is most appropriate for you, but most importantly, gather DATA.

If we have data, let’s look at the data. If all we have are opinions, let’s go with mine.

— Jim Barksdale, the former CEO of Netscape

The importance of usage data cannot be understated, with data supporting your argument, you can overcome even the most difficult of opinions.

Nirvana Fallacy

The Nirvana Fallacy (also known as the Perfect Solution Fallacy) is most commonly known with the quote “perfect is the enemy of good” — Voltaire.

In software development, this often manifests with engineers spending hours and hours, chasing down hard to produce bugs, or ultimately waiting to release a product because it’s not yet perfect.

Perfect is an arbitrary standard, one that is specific to the person, team, stakeholder, or executive involved. This standard, will almost certainly, not be held true for your users, and by withholding your product, in an often futile attempt to reach this nirvana, you will be losing out on vital usage data.

But not only this, if you have truly found a niche market slice, and could end up with a market viable product, by delaying your initial release, you’re prolonging your time-to-market. Then if and when you manage to build something that is perfect, you may have already been beaten to the market by your competitors, who had been able to deliver a product quicker and already gather a user base.

Usage Data

I’ve mentioned usage data here a couple of times, it should be remembered that this is not the same as customer data, and should only be collected in a safe and secure manner, whilst abiding by all GDPR requirements. You may see this option when using an app, that may ask you to opt-in for usage data, it can be difficult to get users to opt-in, but don’t be tempted by Dark UX Patterns to trick your users.

Gathering usage data for your product, allows you to react much quicker, and with more accuracy. If the only people testing your product are the people that built it and work with it internally, you will almost certainly miss real user requirements.

A QA engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. Orders a ueicbksjdhd.

First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone.

Furthermore, should you delay your time-to-market, and then only receive this information after a “perfect” product has launched, you may find it too late to do anything about it. Gathering this data early, gives you the opportunity to pivot the product roadmap; or more importantly, in the case or any potential software bugs, will allow your Software Engineers to work on the most pressing issues first. Those issues that affect the largest portion of your user base, much sooner than they need to fix that annoying, hard-to-reproduce bug affecting only one very persistent user.

Defensive Design

Finally, I’d like to cover how this can affect software design, by discussing Defensive Design; there are many things designed in our society, with fail-safes, put in place, for good reason.

  • A lift (elevator) will have brakes that are held away from the brake pads by the tension in the cables, thus should the cable break, the brakes latch immediately and the lift does not fall.

  • Many avionics equipment will have redundant systems to perform the same computation in unison, thereby being able to identify if there is a fault in sensors, systems, or software.

  • Traffic light controllers will monitor faults or conflicting signals in order to display an error signal instead of dangerous conflicting signals.

The commonality between these, is that they are life critical, the failure of these systems could (and has), resulting in a catastrophic loss of life. Whilst I’m willing to bet you may feel your product is fairly important, but in most cases, it’s not life critical.

This of course is not to say that we want our software to crash, nobody wants that, and nobody builds code intentionally to crash, at least not whilst sober. But I do want to know if my code is failing, how significant, and how widespread this failure is, so I can act on it, with specificity, and with haste.

Over-testing, can often result in engineering-fatigue, meaning that instead of ensuring every possible failure case is handled, with a pleasant user experience journey, giving informative feedback and allowing them to recover from the error, errors are simply swallowed, or ignored.

This behaviour is deceptively dangerous, because your app might appear to be working correctly, but unbeknownst to you and your team, is actually failing in the background, or worse, behaving destructively. In doing so, you effectively throw away this very useful data, about your users habits, and the most effective opportunities your engineering team can focus on.

An effective QA process can be very helpful in finding some issues, but it cannot replace an effective feedback cycle. Falling victim to seeking a perfect solution by compromising this, can cost more than just engineering time. Inversely, an over-scrutinising QA process can slow engineering, and lead to poorly architected defensive codebase, hiding misbehaviour.

If your product does misbehave, using effective (or offensive) design can help your engineers find the source of the failure, quickly, effectively, and precisely. Resulting in fewer side effects, and a faster feedback cycle.

Fail fast. Fail often. Fix quicker.