Bitbucket Pipelines

With its integrated integrated CI/CD, Pipelines, Bitbucket offers developers "CI/CD where it belongs, right next to your code. No servers to manage, repositories to synchronize, or user management to configure."

Detailed documentation is available in the Bitbucket Pipelines Documentation.

Basic Setup

The example below shows a basic setup and job to use Bitbucket Pipelines to run end-to-end tests with Cypress and Electron.

image: node:latest

pipelines:
  default:
    - step:
        script:
          # install dependencies
          - npm ci
          # start the server in the background
          - npm run start:ci &
          # run Cypress tests
          - npm run e2e

How this bitbucket-pipelines.yml works:

• On push to this repository, this job will provision and start Bitbucket Pipelines-hosted Linux instance for running the pipelines defined in the pipelines section of the configuration.
• The code is checked out from our GitHub/Bitbucket repository.
• Finally, our scripts will:
  • Install npm dependencies
  • Start the project web server (npm start:ci)
  • Run the Cypress tests within our GitHub/Bitbucket repository within Electron

Testing in Chrome and Firefox with Cypress Docker Images

The Cypress team maintains the official Docker Images for running Cypress locally and in CI, which are built with Google Chrome and Firefox. For example, this allows us to run the tests in Firefox by passing the --browser firefox attribute to cypress run.

image: cypress/browsers:node12.14.1-chrome85-ff81

pipelines:
  default:
    - step:
        script:
          # install dependencies
          - npm ci
          # start the server in the background
          - npm run start:ci &
          # run Cypress tests in Firefox
          - npx cypress run --browser firefox

Caching Dependencies and Build Artifacts

Per the Caches documentation, Bitbucket offers options for caching dependencies and build artifacts across many different workflows.

To cache node_modules, the npm cache across builds, the cache attribute and configuration has been added below.

Artifacts from a job can be defined by providing paths to the artifacts attribute.

image: cypress/browsers:node12.14.1-chrome85-ff81

pipelines:
  default:
    - step:
        caches:
          - node
        script:
          # install dependencies
          - npm ci
          # start the server in the background
          - npm run start:ci &
          # run Cypress tests in Firefox
          - npx cypress run --browser firefox
        artifacts:
          # store any generates images and videos as artifacts
          - cypress/screenshots/**
          - cypress/videos/**

Using the definitions block we can define additional caches for npm and Cypress.

definitions:
  caches:
    npm: $HOME/.npm
    cypress: $HOME/.cache/Cypress

Parallelization

The Cypress Dashboard offers the ability to parallelize and group test runs along with additional insights and analytics for Cypress tests.

Before diving into an example of a parallelization setup, it is important to understand the two different types of jobs that we will declare:

• Install Job: A job that installs and caches dependencies that will be used by subsequent jobs later in the Bitbucket Pipelines workflow.
• Worker Job: A job that handles execution of Cypress tests and depends on the install job.

Install Job

The separation of installation from test running is necessary when running parallel jobs. It allows for reuse of various build steps aided by caching.

First, we break the pipeline up into reusable chunks of configuration using a YAML anchor, &e2e. This will be used by the worker jobs.

image: cypress/base:14.16.0

## job definition for running E2E tests in parallel
e2e: &e2e
  name: E2E tests
  caches:
    - node
    - cypress
  script:
    - npm run start:ci &
    - npm run e2e:record -- --parallel --ci-build-id $BITBUCKET_BUILD_NUMBER
  artifacts:
    # store any generates images and videos as artifacts
    - cypress/screenshots/**
    - cypress/videos/**

Worker Jobs

Next, the worker jobs under pipelines that will run Cypress tests with Chrome in parallel.

We can use the e2e YAML anchor in our definition of the pipeline to execute parallel jobs using the parallel attribute. This will allow us to run multiples instances of Cypress at same time.

## job definition for running E2E tests in parallel
## ...

pipelines:
  default:
    - step:
        name: Install dependencies
        caches:
          - npm
          - cypress
          - node
        script:
          - npm ci
    - parallel:
      # run N steps in parallel
      - step:
          <<: *e2e
      - step:
          <<: *e2e
      - step:
          <<: *e2e
definitions:
  caches:
    npm: $HOME/.npm
    cypress: $HOME/.cache/Cypress

The complete bitbucket-pipelines.yml is below:

image: cypress/base:14.16.0

## job definition for running E2E tests in parallel
e2e: &e2e
  name: E2E tests
  caches:
    - node
    - cypress
  script:
    - npm run start:ci &
    - npm run e2e:record -- --parallel --ci-build-id $BITBUCKET_BUILD_NUMBER
  artifacts:
    # store any generates images and videos as artifacts
    - cypress/screenshots/**
    - cypress/videos/**

pipelines:
  default:
    - step:
        name: Install dependencies
        caches:
          - npm
          - cypress
          - node
        script:
          - npm ci
    - parallel:
        # run N steps in parallel
        - step:
            <<: *e2e
        - step:
            <<: *e2e
        - step:
            <<: *e2e
definitions:
  caches:
    npm: $HOME/.npm
    cypress: $HOME/.cache/Cypress

Using the Cypress Dashboard with Bitbucket Pipelines

In the Bitbucket Pipelines configuration we have defined in the previous section, we are leveraging three useful features of the Cypress Dashboard:

1. Recording test results with the --record flag to the Cypress Dashboard:

   • In-depth and shareable test reports.
   • Visibility into test failures via quick access to error messages, stack traces, screenshots, videos, and contextual details.
   • Integrating testing with the pull-request process via commit status check guards and convenient pull request comments.
   • Detecting flaky tests and surfacing them via Slack alerts or Bitbucket PR status checks.

2. Parallelizing test runs and optimizing their execution via intelligent load-balancing of test specs across CI machines with the --parallel flag.

3. Organizing and consolidating multiple cypress run calls by labeled groups into a single report within the. Cypress Dashboard. In the example above we use the --group "UI - Chrome" flag to organize all UI tests for the Chrome browser into a group labeled "UI - Chrome" in the Cypress Dashboard report.

Cypress Real World Example with Bitbucket Pipelines

A complete CI workflow against multiple browsers, viewports and operating systems is available in the Real World App (RWA).

Clone the Real World App (RWA) and refer to the bitbucket-pipelines.yml file.