Deliver 30% more PRs to production by having developers own their own testing

Published on December 01, 2019

Tagged: #management #engineering

Follow me on twitter for more posts like this

Is testing “slow” in your team? Do tickets pile up in the “ready to test” column every sprint?

If you think that the testers on your team need to fix this you’re wrong!

In a cross-functional product development team balancing the cadence and handover between development and testing is often a pain point. Read on as to why this is team problem and it’s best resolved by getting developers to thoroughly test their own work.

Broken Kanban

A modern cross-functional team is expected to maintain a steady velocity of getting work out to production. i.e. the team as a whole is responsible for getting work out to production, not the last “function” before production.

Unfortunately many teams still operate with a fixed role mindset and a strict process of Design -> Dev -> Test -> Deploy. The last function before production is often QA and they get work piled on them to test and get out ASAP.

In particular if you’re running Kanban then your team should only be working on a volume of work that matches the part of delivery that takes the team the most effort.

If one of your team’s “columns” is filling up then you need to stop adding more stuff in to the pipeline! Developers are usually the folks starting new tickets so they should stop and instead help clear the test backlog.

It needs to be instinctual in the team to stop working and help get current work out to production. If this isn’t happening in your team you might need to semi-formalise and evangelize Developer Testing.

Developer testing?

Developer testing doesn’t mean filling in forms and clicking buttons (although that might be the thing that’s needed to unblock your team’s test queue sometimes!).

You need to go a bit deeper to figure out why it really takes so long for another team member to prove that your code solution works as expected and doesn’t breaking anything else.

Developer testing is a mindset change. Think of it like you’re trying to get your code through a quality assurance function faster. There are a few general ways to do this.

You can make help the function itself faster, you can reduce how many times the same code needs to go through the function or you can make the work item take less effort to pass through.

Consider these common causes of testing pain

  • Regression testing the existing functionality is manual
  • There are more devs producing more change than testers to verify it
  • Devs don’t test their own stuff properly so there are issues in PRs
  • Handover is a jira ticket number in Slack
  • The scope of change is massive
  • The tester gets all changes at once with 3 days to go in the sprint 👎

These are all things that can and should be the responsibility of the developer. You can work on improving these for your current piece of work rather than starting a new item.

The levels of dev testing in a team

These levels describe some states of your developer testing. You might recognise your team in one of these! I believe all teams should operate at level 3 and this is what I try to get any team I work on to do.

Level 0

In teams where developers don’t do much of their own testing they will handover broken software to testers. A developer will “complete” a task and set it to “In Test” or similar and the developer quickly grabs the next task.

There’s no automated testing here.

The test team members have to perform enormous amounts of regression testing for ALL code changes. Developers keep pushing work to some test queue and an undesirable testing backlog quickly forms. The test team is stressed because they are forced to be gatekeepers.

When the test team need to ask the devs questions the dev needs to pull context from a few days ago because they’ve moved on to something else already.

The test team can only be reactive at this level because they’re so busy testing and retesting the same stuff. They have no time to do proactive quality assurance work or future planning for product complexity (things like investigating and selecting an integration test framework).

Level 1

At this level the developer is writing unit tests for their code and the quality is better. The engineer and tester might have a good discussion about a change so there is some handover of knowledge between the two roles in person.

The test team might have had just enough slack so they were able to add an integration test framework to the project.

The test team is still busy manually testing all the work. They’re finding all the issues that the developers miss and the team is having to redo work all the time.

The test team don’t really have time to add many integration tests so the initiative isn’t seen as a successful and the team get disheartened.

There is still lots of manual work, knowledge handover and context switching back and forth between tester and developer causing inefficiency.

Level 2

This team has unit tests, integration tests to cover the main scenarios or any scenarios that take a long time to set up and they discuss changes before during and after working on them.

The developers are spending time adding integration tests that prove their work solves the problem and doesn’t cause unintended side-effects. This effort to write integration tests seems to slow down developer throughput initially. In fact it halves how many tickets the developers do per week.

Quickly the test team notices there are far fewer issues in the code changes they receive. They can regression test the main app functionality in 2 minutes where it used to take 30 minutes to an hour. Developers are discussing the tests to implement with the QA expert early so there is adequate coverage.

The test team have time to review what’s coming in the product pipeline and suggest concerns or ideas to the team before the coding even begins. They can work on introducing more quality automation and 10x’ing their contribution to the product rather than just doing the same regression testing everyday.

Level 3

Now the team has a full suite of integration tests for the product. There are so many that the tests are split into suites. The smoke test suite is run on every deploy in CI so you don’t have people manually checking the QA or production environment anymore. There is no context switching across days for developers and testers now. Everyone gets feedback on their work in minutes.

There is no test backlog requiring a combined release train. The developers know they can release whenever their code is ready, so they make smaller changes! A developer can arrive in the morning, make a change, get the change and their tests reviewed in the afternoon and deploy it.

Guess what? There is no QA column on your Jira board anymore. Testing is just a part of development like writing code and doing code reviews. The team can confidently put changes in to production safely and calmly multiple times per day.

This is at least where every team should be! This should be the baseline for your quality and delivery practices in software product engineering.

Evangelizing testing

Now that you know where you’re at you can start evangelizing testing in your team. Every team is different but there are some things I’ve found useful in the past. Developers can be given some lessons on what QA is.

All of the team must agree on what is risky for your product. Developers and test analysts often have different ideas of what is high risk. You should have a discussion about all the risk factors that affect code in your system and make sure that there is general understanding and empathy across the team for what each team member considers risky.

Have a test analyst in the team run some sessions on how to test for common issues in your application. Have everyone on the team work through some of the items in the following two links so they are all aware of some of the tools and techniques available for testing software.

Have the developers review the test heuristics cheat sheet here: http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

You can run a session with the team about Exploratory Testing based on this: https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2013/jj620911(v=vs.120)?redirectedfrom=MSDN#bkmk_tours

Specific changes you can make to the ways you work

First thing is to measure where you’re at. The number of defects that make it to test stage currently or defects in production are good things to measure before making changes. Your measure doesn’t have to be perfect because the improvements from shifting to testing earlier in the process will be significant.

Choose a unit test or integration test framework (depending on what level you’re at). Don’t over think this, anything is better than nothing. If developers are going to be writing most of the tests then a code first framework is probably better than a GUI test framework.

Communicate what’s happening to all the stakeholders. When teams are changing the way they work and are learning to work cross functionally it can be hard to convince everyone else in an organisation to follow that pattern.

This means that stakeholders on the fringes of the team might still expect waterfall type progress and might be frustrated that developers are spending so much time writing tests. This will fade over time with the improved team throughput but it needs to be managed carefully during the transition.

This change needs to go both ways - if devs are getting comfortable testing then testers need to get comfortable with code. Even with developers writing tests they may only test the happy path or will sometimes make bad assumptions. A tester needs to code-review the code and tests that a developer writes.

You will have a much better experience if you can have one change per deploy. This is easier for web based applications and harder for apps on mobile devices that need to go through approval. You CI system should make deploying to production automatic.

You should decouple deploying code and releasing code. The business side of the organisation should be able to control releasing new features and that shouldn’t dictate how you engineer the feature. You can use a feature toggle framework to accomplish this.

Focus on integration tests instead of unit tests if you’re just starting out with testing and there is no existing coverage. Integration tests will provide more value per test. Make sure you add tests every time a defect gets to production. The point here is to reduce regressions from occurring.

Conclusion

If testing is a bottleneck in your team your first reaction might be something like - “We need more testers to handle all the code we’re writing”.

Instead consider that your test function is swamped just testing regressions in the code. They have to do this because they regularly find regressions in your code! It’s a viscous cycle. Next problem is that testing regressions in your system is probably manual, and therefore slow.

Developers have the ability to fix a significant amount of these issues at the root by proving early that their own code works.

Modern testing frameworks make proving that your code works relatively easy to implement. This proof can also act as excellent living documentation for yourself and others later.

If you’re a developer take ownership and test your own code fully with automated tests before handing it to anyone else!

Darragh ORiordan

Hi! I'm Darragh ORiordan.

I live and work in Sydney, Australia building supporting happy teams that create high quality software for the web.

I also make tools for busy developers. Have you ever spent a week getting your dev environment just right?

My DevShell tooling will save you 30+ hours configuring your dev environment with all the best modern tools. Get it here

https://darraghoriordan.gumroad.com/l/devshell


Read more articles like this one...

List of article summaries

#management

Hiring engineers in a candidate-driven marketplace

I’m writing this at the start of 2022 and it’s never been tougher to hire engineers. There is a very strong candidate market in software engineering at the moment.

There are roughly 1 million open software engineering roles in the USA and somewhere around 200,000 candidates. The rest of the world is having similar issues hiring engineers. Most people seem to think it will be this way for quite some time. There just aren’t enough engineers as software becomes more important to every industry.

When I started my career getting hired was skewed in favour of the hiring organisation. A candidate had to have a degree and there was no remote work so your choices for where to work were limited.

Now, in 2022 candidates with a couple of years of experience are in high demand and practises for hiring have changed significantly. You can work anywhere across a few time zones and university degrees thankfully aren’t necessary any more.

I’m mostly on the other side of interviews these days, so my are tips for other folks trying to hire engineers during these tricky times!

#developer-experience

How engineers can help deliver software effectively

Delivery managers and team leads have the responsibility to deliver a software system via an engineering team.

Your customer wants every feature to work perfectly and they want it delivered yesterday. Your team wants to learn and grow.

It’s a tough role managing all the stakeholders and creators in a project.

Engineers can help drive great delivery by empathising with and supporting the delivery manager or leads in a project team.

#engineering

Engineering systems for consistency and impact

Your most impactful engineering is done before you write any code.

It’s important to have some systems around how you approach problems to make sure you’re consistent every time.

These are some of the techniques I use to make sure I’m covering as many angles as possible when doing my pre-coding engineering.

#developer-experience

How to identify a great tech organisation before you join

No one wants to waste time working for an organisation that’s a bad fit. But how do you know a good place from a bad place until you actually work there?

It’s important to spend the same amount of time analysing a potential organisation as they spend analysing you!

During an interview process an organisation has personality testing, resumes, tech testing, behavioural testing, reference checks, bank checks, police checks and more.

If it sounds like a very one-sided process that’s because it is! But there are ways you can bring more balance and here are some important things to research before accepting any engineering job offer.