Should QA Exist

Should engineering organizations have quality assurance (QA)? A lot of engineering leaders say they should not. I review the topic, and provide some suggestions for QA leaders, and some nuance on the problems with the way QA and engineering work together. 🌶️ ahead!

QA should not exist

In some engineering leadership circles, the answer is “of course not”. QA is largely seen as an old fashioned practice: “Engineering should own quality”.

I tend to be skeptical of this type of engineering hubris. But there are some good arguments for this:

QA slows things down. There is too much back and forth when you’re passing work in a gated process. QA finds an issue, dev fixes it. Then QA rechecks it and finds other issues. This continues forever, greatly slowing down engineering velocity. Throw it over the wall. There is a moral hazard in having other people responsible for whether your work is correct. You’re creating bad behavior for your engineers by not having them responsible for their quality. Incentives are poor for QA. If your job is to find issues, you’re going to find issues. This automatically slows down engineering, regardless of whether the issues are important or not.

QA is often compared to Operations. Having engineers write code and operators put it in production and care for it has similar problems: handoffs and incentive problems.

In a lot of modern and startup environments, engineering leaders view that as a harmful practice. And they’re often dismissive of the value of QA as a result.

QA should definitely exist

Other leaders are firmly in support of QA. They argue: Testing is a skill. Looking for issues is a real skill, and having an adversarial approach does result in finding deeper issues. Engineers sometimes exhibit an arrogance that they can do everyone else’s job, but the truth is this is a real skill, and not one that is easy for developers to pick up. Automed tests are valuable. Automated tests, when embedded in your workflows, are incredibly valuable. And they can be done by people who are less expensive than engineers, making their work high leverage. Some situations absolutely require QA. There are some high stakes situations where a skilled expert can be worth their weight in gold. They can save the company from considerable risk and ensure a better product experience.

I find this attitude more often in traditional companies, but I see it in a lot of places. Perhaps the strongest argument nowadays for QA is that with AI, automated verification is a leverage maximizer.

The testing pyramid

Before I weigh in further, I’d like to make sure you’re familiar with the testing pyramid. The basic ideas is that there is a natural pyramid of value with testing. The base is unit tests, which are fast and deterministic. Then above that you have integration tests, and then above that, slower UI tests. You want to have lots of fast unit tests. And fewer integration tests, and fewer still UI tests.

Everyone agrees that the engineers should be responsible for unit tests and integration tests. And the UI tests are up for debate. In a lot of organizations, that’s where the split is: QA owns end to end UI testing, and is testing it from the end user perspective. Other organizations share ownership of the end to end tests.

My (slightly) more nuanced take?

I polled my engineering leadership peers, and 100% of a sample of 10 people said, “You mostly should not have QA on software teams, with some edge cases and exceptions.”

I mostly agree with that, but I do have some nuance I’d like to add, and some ideas on ways that QA might be made more high leverage. And I have some advice for QA leaders to make sure they’re as valuable as possible in their companies.

Don’t start with QA

If I am building out an organization from scratch, I wouldn’t start out with QA on a team. Everyone on the team would know that engineering owns quality. Engineering would do unit, integration, and UI tests as a regular part of their work. CI/CD and test automation would be built in from the beginning. Most teams should not start with QA unless they have some very specific reason for doing so.

Don’t have a handoff

I agree that structurally, you don’t want to have a handoff between QA and engineering, if you have QA. It’s much better to have QA “shift left” and be a part of the team itself, doing the work more continually with the engineering team. So my first suggestion is that if you have QA, you embed them on a team and aim to get their work into the same cadence as the engineering team.

Have engineering own quality

If you do have QA, I believe it’s best to still have engineering “own” quality. Metaphorically, I like to treat QA as the quality experts, similar to SREs who are reliability experts. SREs should not be doing all the reliability work, just like QA should not be doing all the quality work. They are experts who can help the team be better at quality, and make sure the team is doing quality work.

Require automation

You almost always want to focus on automated testing. Any manual testing should be for highly unusual situations. I’ve seen it be useful for brittle architectures, or for basically probing for quality issues by a highly experienced QA person.

So if I was building a team from the ground up, and evolving it over time, I’d lean hard on automation and the team being customer focused, and try to solve those problems before reaching for QA.

When would I reach for QA?

There are a few times where QA is high leverage. I’ve seen brittle architectures that are hard to test in an automated way. (Again, I would try to automate first, but this could be a backup plan while we fix the architecture).

I’ve also seen very skilled QA people be deployed as a strategic roaming asset within the company, to focus on the most sensitive projects. I remember working with a couple of people in the past who were just amazing at making sure projects were successful and were shipping with quality. Having roaming QA can be a good pattern, because it can only operate within a larger engineering culture of engineering being responsible for the quality of their own work.

I also think there is a lot of untapped potential with QA. The space is changing rapidly with new AI based tools, and I’ll talk below about some potential ways you might tap QA for high leverage benefits.

What about when there is already a QA team?

I often join teams that have an existing QA team. What do I do when I join a situation like that?

Sometimes it’s not the top priority

First of all, it’s often not my top priority. If there are other higher priorities, I’ll focus on them first. But I can usually improve things a bit with just a few nudges, that don’t require a lot of extensive work. I’ll often meet with the Head of QA and talk these over as goals:

Shift QA left

Get rid of handoff between dev and QA. Have QA members embedded on the team and attend meetings. Try to get the workflows of the QA people onto pull requests (PRs) rather than later in the process. Emphasize that engineering owns quality, but QA are quality experts. Engineers can do QA tasks as well. Often QA can help with test plans that engineers can be involved in doing. Engineers can also write test plans and have QA review. Less process gates and more working together. Share concept of testing pyramid, and make sure engineers are writing unit tests, and that PR review requires unit tests. Ironically, I find that most companies that have had QA for a while also don’t have a good unit test suite. I’m not sure why that is, but I think it’s because engineering thinks QA is there as a safety net. One challenge is with QA teams in different time zones, or where they are offsite and the team is on-site.

Focus on automated testing

Emphasize that testing needs to be automated. Ask the leader to recommend training or replacing team members to move toward automation.