Test Isolation as a Best Practice

This is a guest post from Ambassador David Ingraham.

Hello, I’m David. I’m a Senior SDET with a passion for quality and training engineers on best practices in the industry. I’ve implemented robust testing frameworks for three years with Cypress.io and I’m looking forward to many more. Find and connect with me at https://www.linkedin.com/in/dingraham01/!

What is Test Isolation?

Test isolation is a best practice in which all tests should always be run independently from one another while still passing reliably. As Cypress continues to be a champion for testing processes, they now enforce test isolation by default.

Improved in version 12, now the test and browser state before each test is reset to guarantee that consecutive tests are not affected by previous ones. Cypress clears each test state, such as aliases and intercepts, and cleans the entire browser context including the DOM state, cookies, and local and session storage. This in-depth reset ensures every test controls its own state fully, without risk that another test’s state leaking and impacting it. The test isolation core-concept documentation contains a full list of everything that Cypress resets between tests.

Test isolation is a key tenant to testing because it helps ensure tests are independent and reproducible, which is essential for reliable test results. When tests are not isolated, they may depend on the results or state of other tests, leading to false positives and making it harder to debug issues when there are failures. Additionally, having tests confidently run in any order enables parallelization within CI/CD, which vastly improves test execution time.

For instance, consider the example below where two tests validate two separate pieces of functionality within the Cypress guide. The first test navigates the user to the URL and validates some text. The second test assumes the user is already at the URL and attempts to type in the search box. If the first test fails, the second test will also fail, even though it is testing something completely unrelated.

describe('Dependent Test Example', () => {
   it('Test 1', () => {
       cy.visit('https://docs.cypress.io/guides/core-concepts/test-isolation')
       cy.contains('Test Isolation')
   })

   it('Test 2', () => {
       cy.get('.DocSearch-Input').type('Can Cypress Do Everything?')
   })
})

Refactoring Tips

To determine if a test is independent, run it with .only() and validate if it passes without the state of any tests before it.

For example:

 
it.only("should reliably pass when run on it's own", () => {})
 

If the test fails, it means that it has dependencies on other tests or some external state that needs to be set up first. One approach of transitioning a test to becoming independent is by defining all shared setup and test state in before and beforeEach hooks.

Above, the second test was previously dependent on the first test. Demonstrated in the example below, the tests were made completely independent by utilizing the beforeEach hook to define the same setup of URL navigation. Now both tests will successfully pass if run on their own or in a random order.

Before:

describe('Dependent Test Example', () => {
	it('Test 1', () => {
    	   cy.visit('https://docs.cypress.io/guides/core-concepts/test-isolation')
    	   cy.contains('Test Isolation')
	})
 
	it('Test 2, dependent on Test 1', () => {
    	   cy.contains('Core Concepts')
	})
})

After:

describe('Independent Test Example', () => {
	beforeEach(() => {
    	   cy.visit('https://docs.cypress.io/guides/core-concepts/test-isolation')
	})
 
	it('Test 1', () => {
    	   cy.contains('Test Isolation')
	})
 
	it('Test 2, now independent of Test 1', () => {
    	   cy.contains('Core Concepts')
	})
})

Additionally, utilizing and understanding the cy.session() command is particularly useful for login commands. Even with Test Isolation enabled, the cy.session() command preserves all cookies, local, and session storage for all subsequent calls of the command with the same provided session ID. In plain terms, this means that a test suite can log in once while also having Cypress continue to clear the state and page between tests. This not only speeds up test runs but ensures all tests are run in a consistent state while maintaining independence.

In the example below, the custom login command demonstrates how a simple UI login command can be wrapped by cy.session().

Cypress.Commands.add('loginWithUI', (username, password) => {
   cy.session(username, () => {
       cy.visit('/login')
       cy.get('#username-input').type(username)
       cy.get('#password-input').type(password)
       cy.get('#submit-button').click()
   })
})

Then in the test suite, the custom login command is called before each test with the beforeEach hook. This ensures that after the first session is created, subsequent tests will reapply the initial session.

describe('Test Isolation with cy.session', () => {
   beforeEach(() => {
       cy.loginWithUI('myUsername', 'myPassword')
       cy.visit('https://docs.cypress.io/guides/core-concepts/test-isolation')
   })

   it('Test 1, initial session is created first', () => {
       cy.contains('Test Isolation')
   })

   it('Test 2, restores the session', () => {
       cy.contains('Core Concepts')
   })
})

Disabling Test Isolation

Now that every test is forced to redefine its own state, overall performance of a test suite may be impacted. For slow groups of tests or costly test setup, test isolation can easily be disabled globally or per test suite by setting the testIsolation option to false.

Globally, in the cypress.config.js configuration file:

e2e: {
    testIsolation: false,
},

For a single test suite:

describe('Disabled Test Isolation', { testIsolation: false }, () => {
	it('Test 1', () => {
    	   cy.visit('https://docs.cypress.io/guides/core-concepts/test-isolation')
	})
 
	it('Test 2, which now inherits state from Test 1', () => {
    	   cy.contains('Test Isolation')
	})
})

Disabling test isolation can be useful for the overall performance of end-to-end tests, but there are trade-offs and risks with test reliability. Dependent tests run out-of-order can cause misleading failures and introduce complexity when debugging issues.

While it might be tempting to disable test isolation for performance optimization, there are some tips and tricks that can be utilized to improve test performance before doing so:

• Avoid explicitly waiting for arbitrary periods of time. Instead, use Cypress’s built-in retry and wait mechanisms to avoid flakey tests and unnecessary execution time. More information can be found here.
• Use cy.intercept() to stub unnecessary network requests with mock responses.
• Control the application’s state programmatically using API calls instead of relying solely on the UI. Utilize the cy.request() command to create or cleanup data in before or beforeEach hooks.
• Incorporate Cypress into the CI/CD pipeline and run tests in parallel across multiple machines. More information can be found here.

Disabling Test Isolation ≠ Pre-v12

Prior to v12, Cypress still cleared the local storage and cookies for the current domain, but didn’t clear sessions storage nor the DOM state. By contrast, now disabling test isolation fully clears everything.

Below is a table explaining the differences and what Cypress clears for each configured state.

While not recommended, if pre-v12 behavior can be replicated by

• Disabling Test Isolation
• Running cy.clearLocalStorage() and cy.clearCookies() in a beforeEach hook

An example of this is demonstrated in the migration guide here.

While test performance is important, an unreliable or unsupportable test suite negates any time saved by faster test runs. Having full confidence that each test passes (or fails) on its own is critical as a test suite scales. Remember, the less time you spend debugging and supporting intertwined test suites means that you gain more time writing new tests and increasing test coverage. Adding a little bit more time up front to write isolated tests can save numerous headaches later. The new test isolation behavior provides a reliable framework for this best practice and allows more time spent on your core activities.

Happy testing!