How to Run Changed Specs First in a Pull Request

Imagine a large project with hundreds of Cypress spec files. If you work on just one feature and open a pull request with small code changes and corresponding updates to 1 Cypress spec file, it makes sense to try running that changed spec file first. Chances are high that any new issues will be introduced in the changed code, so by running the changed files first, we will discover the problem more quickly.

After running Cypress with changed specs first, we still should run all spec files. The second test run ensures that any changes have not introduced bugs somewhere else. In this blog post I will show how to quickly set up such "double run" continuous integration workflow. I will use GitHub Actions, but the main shell code to determine and run the changed specs uses plain Git commands, and thus this blog post can be implemented on most CI vendors.

Note: you can find the finished project in repository bahmutov/changed-cy-tests

First, I will create a GitHub workflow to run on every pull request.

name: ci
on: [pull_request]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

The above ci workflow uses GitHub's own actions/checkout to bring the code from GitHub to the CI container. On every pull request, the checked out code is a merge request between the head (aka the current) and the base (aka the target) branches. For example, this pull request #2 has the head branch named update-test-b and the base branch master.

By default, only the merge commit is checked out - to save time. Thus, if we want to see what has changed between the update-test-b and master branches, we need to pull them from the remote origin to the local source.

Tip: you can see the remote branches by running git ls-remote command.

$ git ls-remote
From [email protected]:bahmutov/changed-cy-tests.git
e5007aa819980ffd469dd1f893678970592be5d1	HEAD
acde0808e5c303136e95e6f6c039eb52d1c73d62	refs/heads/changed-tests
e5007aa819980ffd469dd1f893678970592be5d1	refs/heads/master
27ac3826934eb54cf4f119fb08afe64f5dd92011	refs/heads/update-test-b
acde0808e5c303136e95e6f6c039eb52d1c73d62	refs/pull/1/head
27ac3826934eb54cf4f119fb08afe64f5dd92011	refs/pull/2/head
503c73efc52f8c88db8316f174cb0a3afbd1bdf0	refs/pull/2/merge

Let's get the two branches from the remote origin (GitHub) into local CI container. We can hardcode the branch names or use GitHub Action Expressions to get the names from the current Pull Request event.

git fetch --no-tags --depth=1 origin ${{ github.base_ref }}
git fetch --no-tags --depth=1 origin ${{ github.head_ref }}
# get the actual source for two branches
git checkout origin/${{ github.base_ref }}
git checkout origin/${{ github.head_ref }}
# get back to the merge commit
git checkout ${{ github.sha }}

Now that the local Git has the information for both branches, let's find the changed files between two branches using git diff command.

git diff --name-only origin/${{ github.base_ref }} origin/${{ github.head_ref }}

We are only interested in the filenames, but we also want to limit ourselves to names of changed Cypress specs. In my example, all specs are in cypress/integration folder. Let's pass this folder name to git diff to limit the returned list to files in that subfolder.

git diff --name-only \
  origin/${{ github.base_ref }} origin/${{ github.head_ref }} \
  -- cypress/integration

Note the "--" separator between the Git branch labels from the folder name.

Once we have found the changed Cypress specs, if this list is non-empty, we should run them first.

CHANGED_SPECS=$(git diff --name-only origin/${{ github.base_ref }} origin/${{ github.head_ref }} -- cypress/integration)
if [ -n "$CHANGED_SPECS" ]; then
  echo "Running the following changed specs"
  echo $CHANGED_SPECS
  npx cypress run --spec $CHANGED_SPECS
fi

We can see the quick first Cypress run limited to spec-b.js

After running just the changed specs, we still have to run all specs to thoroughly test the entire system. In this case, we can use Cypress GH Action. This action allows us to split the install and test run steps, thus we don't have to worry about installing and caching Cypress in each run - it is done for us automatically.

- name: Install Cypress and dependencies 📦
  uses: cypress-io/github-action@v1
  with:
    runTests: false

# quickly run just the changed specs to fail fast
- name: Maybe run just the changed Cypress tests ⏱
  run: |
    CHANGED_SPECS=$(git diff --name-only origin/${{ github.base_ref }} origin/${{ github.head_ref }} -- cypress/integration)
    if [ -n "$CHANGED_SPECS" ]; then
      echo "Running the following changed specs"
      echo $CHANGED_SPECS
      npx cypress run --spec $CHANGED_SPECS
    fi

# run all tests to be sure
- name: Run all Cypress tests 🧪
  uses: cypress-io/github-action@v1
  with:
    # we have already installed all dependencies above
    install: false

Let's say while we are working on branch update-test-b, someone changes test file spec-c.js on master. Our check finds those two files and runs them both!

Once we have the test jobs working, let's record results on the Cypress Dashboard - so we can see the passing or failing tests and quickly debug their failures. I have added projectId to cypress.json file, and set the CYPRESS_RECORD_KEY as a secret in GitHub repository settings. I will need to set the record key as an environment variable for two run steps that need it. Then we can use CLI arguments and GitHub Action parameters to turn on the recording mode. Let's give them group names and the same CI build id so both steps record into a single Dashboard "Run".

- name: Maybe run just the changed Cypress tests ⏱
  env:
    # pass the Dashboard record key as an environment variable
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
  run: |
    CHANGED_SPECS=$(git diff --name-only origin/${{ github.base_ref }} origin/${{ github.head_ref }} -- cypress/integration)
    if [ -n "$CHANGED_SPECS" ]; then
      echo "Running the following changed specs"
      echo $CHANGED_SPECS
      npx cypress run --record --group "Changed specs" --ci-build-id ${{ github.sha }} --spec $CHANGED_SPECS
    fi

# run all tests to be sure
- name: Run all Cypress tests 🧪
  uses: cypress-io/github-action@v1
  env:
    # pass the Dashboard record key as an environment variable
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
  with:
    # we have already installed all dependencies above
    install: false
    record: true
    group: All specs
    ci-build-id: ${{ github.sha }}

You can see the recorded runs at https://dashboard.cypress.io/projects/aobpjx/runs/

Question: the above shell script looks a lot like Jest --changedSince flag. Why isn't this a CLI flag in the Cypress test runner?

Answer: Yes, this experiment and blog post were inspired by Jest and the --changedSince flag in the GitHub Actions CI blog post. But adding any new feature to the Cypress test runner core is a decision we do not take lightly. Every flag adds new code, new tests, new documentation. Every feature requires maintenance and slows down further development. Thus if one can quickly implement the feature outside the test runner, then it seems to be the appropriate solution.