Webinar Recap: Speed up your E2E Testing with ephemeral environments from Shipyard

Recently the team at Shipyard joined us for a webinar on speeding up end-to-end testing with ephemeral environments. We’ve captured the highlights, and you can watch the recording here on YouTube. 🎉

This is a guest post from Benjie De Groot, Co-founder at Shipyard.

Hello, I'm Benjie. I'm a co-founder with a passion for DevOps and releasing code I know will work! I have been an avid Cypress.io fan and user for years and love spreading the gospel around ephemeral environments + e2e testing. Find and connect with me at https://www.linkedin.com/in/bueller/!

What's slowing down your e2e tests?

Many things can slow testing, and it's essential to be aware of them to optimize your process. Some pain points might sound familiar: broken and failing tests, flakiness, or waiting for testing environments. Let's dig into these a little. 👀

Broken or failing tests

The source of a failing test can be a challenging thing to find. Is it a race condition? Is it a stale test? Does your test need to be updated, or did your feature actually break something? Figuring out what’s actually going on in a shared testing environment makes all this even harder, especially when you might have multiple pull requests (PRs) being merged and tested simultaneously. Determining which PR or code change is responsible for the failing/broken test turns into forensic analysis.

Test suite bloat is a real thing and can really start to slow down your ability to deploy quickly. While having enough tests to validate new deployments is critical, too many tests can lead to even more bloating and flakiness. As your test suite grows, you commonly have multiple tests validating the same feature. Maintaining good test hygiene is critical but hard to stick to.

Waiting for testing environments

The staging bottleneck: waiting for testing environments to free up can also be a pain in the neck. 😉 When you're stuck waiting for a staging or pre-production environment to become available, it throws a wrench in your ability to deploy and move quickly. When developers are waiting hours or even days for feedback, the context switching costs become immense.

Getting data right in your test environments

Setting up databases for testing can be tedious and time-consuming. Having good seed data to run tests against is a must if you want to be able to count on your tests. Maintaining that data and keeping it clean and reusable is a huge challenge for all QA teams. When tests fail, your data can become corrupted, forcing testers to manually try and reset data to a particular point. This is not only tedious but error prone. Teams get stuck in a seemingly endless loop of constantly resetting databases, which can take up entire chunks of your day.

Use ephemeral environments to speed up your testing process 🎉

At Shipyard, we live and breathe ephemeral environments! They are often referred to as preview, testing, review or PR environments. The simple definition is full-stack, independent copies of your production infrastructure that you can run tests against in parallel on every code change. Think of them as full versions of your production environment where all your services are running (front end, back end, databases, etc.), but with the code changes that you want to test before you release.

While there are many flavors of ephemeral environments, the key features are full isolation, short-lived managed life cycle, securely accessible by all stakeholders, and automatically built on every code change. While many teams can implement these platforms using their existing internal DevOps/SRE teams, using a platform like Shipyard automates all of this for you.

To use Shipyard, you need to be containerized and you get all of the aforementioned features and more (data snapshotting anyone?).

Since ephemeral environments are isolated, engineers can work on code in parallel without stepping on each other’s toes. These preview environments are easy to access, with the latest changes available through a URL link or programmatically through a CLI. Furthermore, who wants to pay for infrastructure that you’re not using? No one! So, they also spin up and down on demand– you only pay for what you need.

Ephemeral environments can "shift left" testing in the development process. They can be spun up automatically at the PR level, so every time a new code change comes in from a developer, the environment detects it and runs the test suite against the PR environment, thereby Speeding production time in general. This changes the testing workflow, allowing for testing early and often before merging into the main branch. 💃

If you would like to run your end-to-end tests with Shipyard’s preview environments and Cypress, follow these 6 steps:

1. Go grab the Shipyard Github Action (here) Put it in to your github repo for detailed instructions check out our docs.
2. Add your Shipyard API Key to GitHub Secrets: Log into your Shipyard account and go to your Organization Settings page to find your API KEY. Add this key to your GitHub repository secrets as SHIPYARD_API_TOKEN.
3. Create a GitHub Actions workflow file: In your GitHub repository, create a new file at .github/workflows/run-cypress-e2e.yml.
4. Add the GitHub Actions workflow code: In the run-cypress-e2e.yml file, set up the workflow to run on specific events (e.g., push or pull_request). See the following example for details:

on: [push]

jobs:
  cypress:
    runs-on: ubuntu-latest
    name: Run Cypress end-to-end tests
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Install Node.js
        uses: actions/setup-node@v1
        with:
          node-version: '16.x'
      - name: Install dependencies
        run: npm install
        # Fetch Shipyard Environment fetches the SHIPYARD_ENVIRONMENT_URL
        # and SHIPYARD_BYPASS_TOKEN for the specific Pull Request
      - name: Fetch Shipyard Environment
        uses: shipyard/github-action/[email protected]
        with:
          # Your Shipyard Org API token
          # Add to your GitHub Secrets
          api-token: ${{ secrets.SHIPYARD_API_TOKEN }}
          # How long to wait for the environment
          timeout-minutes: 10
      - name: Cypress run
        uses: cypress-io/github-action@v5
        with:
          browser: chrome
          # Pass in the env vars to Cypress
        env:
          CYPRESS_BASE_URL: ${{env.SHIPYARD_ENVIRONMENT_URL}}
          CYPRESS_BYPASS_TOKEN: ${{env.SHIPYARD_BYPASS_TOKEN}}

5. Add the env vars in Cypress: In your Cypress end-to-end tests, pass in the BYPASS_TOKEN (the BASE_URL is added by default to cy.visit() calls). Example:

describe("shipyard flask react example app", () => {
  beforeEach(() => {
    const pathWithShipyardToken = path => {
      return path + "?shipyard_token=" + Cypress.env("BYPASS_TOKEN");
    };
    cy.visit(pathWithShipyardToken("/"));
  });

    it("Displays the logo with correct alt text", () => {
    cy.get("#page-logo").should("have.attr", "alt", "Shipyard logo");
  });
});

6. Test it out: Create a new pull request to your repo for the GitHub Actions code, and you should see your changes tested in your GitHub Action.

Wrapping Up

Switching to ephemeral environments in your developer pipeline is just a smart thing to do. It saves your team time and money on infrastructure while facilitating easier collaboration between engineers, QA, testers, product teams, and stakeholders. Our team uses Shipyard to build Shipyard because we don't just preach ephemeral environments; we live it.

And we have the data from our customers to support our claim to fame. During the webinar, we shared some success stories from teams and companies that transitioned to using Shipyard for their environment management. Overall, companies that made the switch experienced a host of benefits, including:

• Accelerated developer velocity by 50%
• Saving 67% in cloud infrastructure costs
• Reduction of bloated and flakey tests in your test suite by 42%

Also, don’t forget that for the first 100 people to sign up for a Shipyard trial, get an extra 15 days free if you use the promo code CYPRESS2023. To learn more about ephemeral environments, visit the open-source community maintained https://ephemeralenvironments.io/. Stay updated on the latest in the Kubernetes and container world by listening to the Kubelist podcast or signing up to the Kubelist community newsletter. Both can be found at https://kubelist.com/. Of course follow us on our Github, Twitter, Slack or Linkedin and feel free to reach out for more information.