It is simple to run Cypress both locally and on CI. And it is simple to point your tests to a different server. If you are using immutable deploys, Cypress can be the ultimate “health check”. If the tests pass locally, then on CI, and then at a newly deployed environment - the chances are very high that you can point production DNS record at the new deployment and it will work.
Here is an example. Let us take a “stock” React TodoMVC app and see all the moments we can run Cypress end to end tests against it.
git clone [email protected]:bahmutov/todomvc.git
cd todomvc
npm install
npm start
Open http://localhost:8888 in the browser; the application should be running.
Run end to end tests locally
I grabbed the existing end to end tests from the Cypress example repo. The tests run very quickly inside the Cypress desktop application.
We can start practicing test-driven development by keeping the Cypress application open and updating our test code - the tests will be rerun automatically. If we update the application code, we need to reload the Cypress window - just press Command-R. I like concentrating on a feature by allowing only a block of tests to run with describe.only or even just a single test with it.only.
Seeing the UI update during the test is a powerful drug 😉, but we can run Cypress without the UI too. Assuming the TodoMVC application server is running we can just call cypress run to execute end to end tests.
Notice that the headless run records a video, so you will always know what has happened during the test execution. You can control video recording and compression using Cypress options.
Let us write a single command that starts the server, runs end to end tests and then closes the server. I am going to use the npm-run-all module for this.
$ npm i -D npm-run-all
Then let us define a script command in package.json
{
"scripts": {
"start": "http-server -p 8888 -c-1",
"e2e": "cypress run",
"test": "run-p --race start e2e"
}
}
From now on, if you run npm test it will start the server, execute all end to end tests and then will close the server.
$ npm t
> todomvc@ test /Users/irinakous/git/todomvc
> run-p --race start e2e
Starting up http-server, serving ./
Available on:
http:127.0.0.1:8888
http:10.47.29.203:8888
Hit CTRL-C to stop the server
Started video recording: /todomvc/cypress/videos/im8kg.mp4
... test output ...
http-server stopped.
Perfect.
Run tests before pushing code
We can execute E2E tests on each commit and before pushing the code to the master repo, just to make sure the master is never broken. In this case, I will use pre-git to run the tests before pushing my changes to the origin.
$ npm install -D pre-git
{
"devDependencies": {
"npm-run-all": "^4.0.2",
"pre-git": "^3.14.0"
},
"config": {
"pre-git": {
"commit-msg": "simple",
"pre-commit": [],
"pre-push": ["npm test"],
"post-commit": [],
"post-checkout": [],
"post-merge": []
}
}
}
I set pre-push hook commands to be just npm test. Let me commit and push this code to the origin.
$ git commit -m "feat(ci): run tests on pre-push"
running bin/pre-commit.js script
pre-commit Nothing the hook needs to do. Bailing out.
post-commit Nothing the hook needs to do. Bailing out.
[master acc83af] feat(ci): run tests on pre-push
1 file changed, 14 insertions(+), 2 deletions(-)
Nothing extraordinary happens during commit, I am just following simple commit message format. Let us push the code to the remote server.
$ git push
pre-push Detected files in diff: 1
executing task "npm test"
> todomvc@ test /Users/irinakous/git/todomvc
> run-p --race start e2e
> todomvc@ e2e /Users/irinakous/git/todomvc
> cypress run
> todomvc@ start /Users/irinakous/git/todomvc
> http-server -p 8888 -c-1
Starting up http-server, serving ./
Available on:
http:127.0.0.1:8888
http:10.2.60.121:8888
Hit CTRL-C to stop the server
Started video recording: /Users/irinakous/git/todomvc/cypress/videos/478b6.mp4
(Tests Starting)
... test output ...
http-server stopped.
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 439 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
To gitlab.com:bahmutov/todomvc.git
c378c1d..acc83af master -> master
Great, the E2E tests were executed before our code was pushed to the remote repository. This sanity test pass is one less thing we have to worry about now. If I am really in a rush, and willing to live dangerously I can skip Git pre-push hook using --no-verify CLI option.
Run tests on CI
Executing Cypress E2E tests on CI is as simple as running unit tests. As long as your CI has the right dependencies, you are good to go. Our customers are successfully using Cypress on all major CI systems: Jenkins, Travis, Circle and others. Recently, we also built Cypress Docker image that you can use in a containerized CI environment.
But I am a Cypress insider, so there are some perks 😋 like building a Docker image with dependencies and Cypress pre-installed. While we are still trying to figure out the best long-term solution, I am using cypress/internal:cy-0.19.2 image. This image makes it so simple to run E2E tests on GitLabCI for example, that it is almost unnoticeable. Here is my .gitlab-ci.yml file, which mirrors my local development flow 100%.
image: cypress/internal:cy-0.19.2
cypress-e2e:
stage: test
script:
- npm install
- npm test
You can see the CI running the tests on GitLab todomvc/builds. For each test run, it
- Starts the local server
- Runs the E2E tests against the local server
The system is self-contained; the CI runs both the server and the tests.
Immutable deploy
Running TodoMVC server locally is nice, but we want to share it with the world. The simplest way to do this that I know of is to serve it via Zeit.co now tool. For example, from the local terminal, I can just invoke now and get my server up and running!
How cool is that? Notice the unique URL created during deploy - each now invocation is creating a brand new site; the old deployments are unaffected. This is called immutable deployment. But is the application working as expected after deployment? Let us test it using Cypress of course.
We are telling Cypress test runner what to load via the baseUrl option in the cypress.json file. By default it is matching our local server address: http://localhost:8888. We can override this setting when starting the desktop application or running the tests headlessly. Either cypress run --config baseUrl=... or CYPRESS_baseUrl=... cypress run would work in this case. Let us try it out.
$ cypress run --config baseUrl=https://todomvc-jodtvxbtuh.now.sh
Started video recording: /Users/irinakous/git/todomvc/cypress/videos/xjl7n.mp4
... test output ...
The tests finish and if we open the generated video file cypress/videos/xjl7n.mp4 we can confirm that Cypress tested the deployed site. Perfect, our immutable deploy is working according to spec.
If we can deploy and test our application from the local terminal, we should be able to deploy and test the same way on CI. And it almost works, except now tool fails inside the very limited Docker environment.
$ npm install -g now
/usr/local/bin/now -> /usr/local/lib/node_modules/now/download/dist/now
> [email protected] postinstall /usr/local/lib/node_modules/now
> node download/install.js
> Retrieving the latest CLI version...
> For the sources, check out: https://github.com/zeit/now-cli
/usr/local/lib
`-- [email protected]
$ now
> Error! Unexpected error. Please try later. (Authentication error – stdin lacks setRawMode support)
Luckily, I have a work around that uses Zeit API to drive the deployment. The tool is called now-pipeline and just needs NOW_TOKEN environment variable to work. You can make a new token at https://zeit.co/account/tokens; it is a good idea to give it a very descriptive name, like <repo url> - gitlab ci deploy string.
now-pipeline will run the deploy for your project, then will invoke a test command against the newly created url. To avoid variable interpolation problems, we can tell now-pipeline to pass the newly created url using environment variable named CYPRESS_baseUrl. Our CI job definition (I am using deploy stage for this) looks like this:
deploy-and-test:
stage: deploy
script:
# we now longer need to install package.json dependencies
- npm install -g now-[email protected]
# we do not need to start local server
# since it is already running in the cloud
- now-pipeline --as CYPRESS_baseUrl --test 'cypress run'
artifacts:
expire_in: 1 week
paths:
- cypress/screenshots
- cypress/videos
The job runs and successfully tests a new unique deploy every time it runs.
$ now-pipeline --as CYPRESS_baseUrl --test 'cypress run'
READY todomvc-swtllwnzbh.now.sh limit 600 seconds
deployed to url https://todomvc-swtllwnzbh.now.sh
testing url https://todomvc-swtllwnzbh.now.sh
passing it as env variable CYPRESS_baseUrl
test command "cypress run"
running "cypress" with extra env keys [ 'CYPRESS_baseUrl' ]
Added this project: /builds/bahmutov/todomvc
... test output ...
deployed url https://todomvc-swtllwnzbh.now.sh is working
found 0 deploy(s) with aliases
there is no existing alias
will skip updating alias to https://todomvc-swtllwnzbh.now.sh
Uploading artifacts...
WARNING: cypress/screenshots: no matching files
cypress/videos: found 2 matching files
Uploading artifacts to coordinator... ok
Job succeeded
We just ran our E2E tests against freshly deployed instance, verified that it works, yet have not affected the existing production deployments in any way.
Switch production DNS
Pointing a production URL / DNS record at a fresh and tested deployment is the last step we are going to do. Just add --alias todomvc parameter to the now-pipeline command to switch the todomvc.now.sh (or any custom domain controlled by Zeit World) to point at the new deployment if and only if tests passed. We can also prune old deploys after switching the alias.
deploy-and-test:
stage: deploy
script:
- npm install -g now-[email protected]
- now-pipeline --as CYPRESS_baseUrl --alias todomvc --test 'cypress run'
- now-pipeline-prune
Finally, we can verify that the application is still working after changing the DNS record. Who knows, maybe we had some hard-coded URLs or domain white lists? Just run Cypress tests against the production URL! By now you know how to do this; we have been using the one and only cypress run command every time, and just gave it different base urls. I will run the test-production job during verify stage which executes after deploy stage, because that is the order I listed them in the “stages” list, you can see the complete .gitlab-ci.yml file for reference.
image: cypress/internal:cy-0.19.2
stages:
- test
- deploy
- verify
# cypress-e2e job as before
# deploy-and-test as before
test-production:
stage: verify
script:
- cypress run --config baseUrl=https://todomvc.now.sh
artifacts:
expire_in: 1 week
paths:
- cypress/screenshots
- cypress/videos
Perfect, let us try it one more time to confirm. The entire pipeline runs … and passes.
Final thoughts
Immutable deploys bring peace of mind to continuous deployment, at least for simple cases without shared dependencies. Trying to deploy new code does not affect the existing production system, and we can fully test and verify the fresh version before switching users to it. Zeit.co makes immutable deploys incredibly simple and fast. Our tool, Cypress, makes it similarly easy to verify that the entire web application is working correctly. Taken together, Zeit and Cypress verify your code at multiple check points. To review, we run E2E tests:
- Locally as we are working on new features, or refactoring existing code.
- Locally before pushing new code to the remote repository to avoid accidentally sending broken or incomplete code.
- On CI using local isolated server.
- After deploying new code to its own unique cloud instance.
- After switching production DNS to point at the latest deployed instance.
Of course, I just showed different times when we could run E2E; it might seem like an overkill. I would suggest at least running tests when the code crosses a boundary: from local machine to CI, and from CI to production. With this approach, our continuous deployment pipeline can deploy hundreds of changes per day, yet giving us peace of mind. Easy win.
Links
- TodoMVC example project
- Zeit.co Now tool
- Cypress CI docs
- Cypress Changelog
- now-pipeline deployment tool
Happy testing!