Note: This post was written by Filip Hric. Filip is a QA lead at Slido, a lifelong IT enthusiast with a background in psychology, and a Cypress ambassador.
Cypress recently released version 5.0.0 of their open source test runner. The release introduced native support for test retries (off by default) which generated some really interesting and healthy discussion about the right and wrong ways to leverage test retries in your test suite. Today I want to share my personal opinion on using test retries without hiding or ignoring problems with your test cases.
Giving a test an option to retry is not a brand new concept. Retries have been around for a long time, and even the Google Chrome team uses them. That means they are good, right? Well, no. Simply seeing someone using a cool tool does not make it good for everyone. It’s the same for retries. It’s a tool that does not fit everyone and does not fit every situation. That may be the reason for mixed reactions. It’s neither good nor bad. It’s what you make of it. And you can make it really good—I mean really good—if your approach is right.
Flaky tests are still an anti-pattern
They should not be fixed with retries. Not in Cypress nor in any other tool. The danger of "solving" test flakiness with retries is real though. I can understand how pressure to release a new feature pushes developers or testers to flag that annoying test as "fix later" and just make it retry until it passes. That is an example of a bad approach. Let’s be honest though - if this is a pattern you are willing to choose, you’ll find a way to stay on that pattern with or without retries. But no one reading this is like that. We’re all good people here 🙂
Looking at what the Cypress team does, I’m very doubtful that adding retries to the product was an encouragement to write bad tests. The Cypress team has always argued for deterministic tests. A whole chapter on avoiding conditional tests can be found in the Core concepts section in their documentation. Also, to find out more about how to write stable Cypress tests, I recommend "Where can the test X" series by Gleb Bahmutov:
What happens when a test fails?
When testing a web application, there are actually a lot of things that can go wrong:
- app didn’t render properly
- http request was not fired, timed out or did not return proper data
- staging/testing server is experiencing downtime
- exception was thrown
- 3rd party integration did not work properly
- test was not written properly
- CI pipeline threw an error, tests did not finish
- something else
Each of these problems may need a different solution. Some of these require work of DevOps, some of these need cooperation of different teams, and some just require a frontend developer to fix a bug.
For example, if your staging server is experiencing an issue, what would you do after your tests fail? There’s a good chance you would just run your tests again. Or, maybe you would ask your DevOps team to fix that and go for a coffee. In any case, it is a good practice to take interest into what is really happening when a test fails. Once you know that, you can make a decision. The decision is your own.
What Cypress has provided with retries functionality is the option to make a retry. If that is something that helps your process, I encourage you to use it.
But a test should never retry, should it?
Well, as with everything - it depends. In my case, 98% of my failed tests were because my tests were running too fast. If I read Gleb’s blogs more often, I would be able to fix them sooner - I closed the modal too soon, didn’t wait for a request to finish, didn’t wait for an element to render, page to load, input to be enabled... you name it. Whenever I tried to run those tests manually, they passed. So when our dev team asked me: “What was wrong with the user story?” I had to answer - “Nothing. It’s all fine.”
I’ve been using retries for quite a long time, with a plugin that is now deprecated in favor of Cypress v5. The cool update now is that retries are actually reflected in the Cypress Dashboard too. I can now look into all tests that have retried and passed. I spent the last few weeks fixing these to make them more stable. I’m reaching 0 retries on hundreds of user stories end-to-end tests. So I *do* in fact agree that ideally, tests should not retry. But I’m trying to be pragmatic, too and test with my eyes wide open.
Do it right
My advice is - if you are using retries, do it right. You can look ahead with Cypress test analytics to see which tests are more likely to retry. You can toughen those up, so that they don’t fail in the future. Make sure you look into those retries regularly and don’t become blind to them. Don’t give up on understanding why a test retried. This will help you learn about your app as well as your tests. Don’t make retries a cure for lazy code, but a powerful tool to make you release faster, with greater confidence, better understanding of your code and continuous improvement.
If you are interested in learning more about Cypress, I have a quick course that will teach you the basics. Enjoy!