Contract testing

Contract tests consist of two parts — consumer tests and provider tests. A simple example of a consumer and provider relationship is between the frontend and backend. The frontend would be the consumer and the backend is the provider. The frontend consumes the API that is provided by the backend. The test helps ensure that these two sides follow an agreed upon contract and any divergence from the contract triggers a meaningful conversation to prevent breaking changes from slipping through.

Consumer tests are similar to unit tests with each spec defining a requests and an expected mock responses and creating a contract based on those definitions. On the other hand, provider tests are similar to integration tests as each spec takes the request defined in the contract and runs that request against the actual service which is then matched against the contract to validate the contract.

You can check out the existing contract tests at:

spec/contracts/consumer/specs for the consumer tests.
spec/contracts/provider/specs for the provider tests.

The contracts themselves are stored in /spec/contracts/contracts at the moment. The plan is to use PactBroker hosted in AWS or another similar service.

Write the tests

Run the consumer tests

Before running the consumer tests, go to spec/contracts/consumer and run npm install. To run all the consumer tests, you just need to run npm run jest:contract -- /specs. Otherwise, to run a specific spec file, replace /specs with the specific spec filename. Running the consumer test will create the contract that the provider test uses to verify the actual API behavior.

You can also run tests from the root directory of the project, using the command yarn jest:contract.

Run the provider tests

Before running the provider tests, make sure your GDK (GitLab Development Kit) is fully set up and running. You can follow the setup instructions detailed in the GDK repository. To run the provider tests, you use Rake tasks that can be found in ./lib/tasks/contracts. To get a list of all the Rake tasks related to the provider tests, run bundle exec rake -T contracts. For example:

$ bundle exec rake -T contracts
rake contracts:merge_requests:pact:verify:diffs_batch                                   # Verify provider against the consumer pacts for diffs_batch
rake contracts:merge_requests:pact:verify:diffs_metadata                                # Verify provider against the consumer pacts for diffs_metadata
rake contracts:merge_requests:pact:verify:discussions                                   # Verify provider against the consumer pacts for discussions
rake contracts:merge_requests:test:merge_requests[contract_merge_requests]              # Run all merge request contract tests

Verify the contracts in Pact Broker

By default, the Rake tasks will verify the locally stored contracts. In order to verify the contracts published in the Pact Broker, we need to set the PACT_BROKER environment variable to true and the QA_PACT_BROKER_HOST to the URL of the Pact Broker. It is important to point out here that the file path and filename of the provider test is what is used to find the contract in the Pact Broker which is why it is important to make sure the provider test naming conventions are followed.

Publish contracts to Pact Broker

The contracts generated by the consumer test can be published to a hosted Pact Broker by setting the QA_PACT_BROKER_HOST environment variable and running the publish-contracts.sh script.

Test suite folder structure and naming conventions

To keep the consumer and provider test suite organized and maintainable, it's important that tests are organized, also that consumers and providers are named consistently. Therefore, it's important to adhere to the following conventions.

Test suite folder structure

Having an organized and sensible folder structure for the test suite makes it easier to find relevant files when reviewing, debugging, or introducing tests.

Consumer tests

The consumer tests are grouped according to the different pages in the application. Each file contains various types of requests found in a page. As such, the consumer test files are named using the Rails standards of how pages are referenced. For example, the project pipelines page would be the Project::Pipelines#index page so the equivalent consumer test would be located in consumer/specs/project/pipelines/index.spec.js.

When defining the location to output the contract generated by the test, we want to follow the same file structure which would be contracts/project/pipelines/ for this example. This is the structure in consumer/resources and consumer/fixtures as well.

The naming of the folders must also be pluralized to match how they are called in the Rails naming standard.

Provider tests

The provider tests are grouped similarly to our controllers. Each of these tests contains various tests for an API endpoint. For example, the API endpoint to get a list of pipelines for a project would be located in provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb. The provider states are grouped according to the different pages in the application similar to the consumer tests.

Naming conventions

When writing the consumer and provider tests, there are parts where a name is required for the consumer and provider. Since there are no restrictions imposed by Pact on how these should be named, a naming convention is important to keep it easy for us to figure out which consumer and provider tests are involved during debugging. Pact also uses the consumer and provider names to create the locally stored contract filenames in the #{consumer_name}-#{provider_name} format.

Consumer naming

As mentioned in the folder structure section, consumer tests are grouped according to the different pages in the application. As such, consumer names should follow the same naming format using the Rails standard. For example, the consumer test for Project::Pipelines#index would be under the project folder and will be called Pipelines#index as the consumer name.

Provider naming

These are the API endpoints that provide the data to the consumer so they are named according to the API endpoint they pertain to. Be mindful that this begins with the HTTP request method and the rest of the name is as descriptive as possible. For example, if we're writing a test for the GET /groups/:id/projects endpoint, we don't want to name it "GET projects endpoint" as there is a GET /projects endpoint as well that also fetches a list of projects the user has access to across all of GitLab.

To choose an appropriate name, we can start by checking out our API documentation and naming it the same way it is named in there while making sure to keep the name in sentence case. So GET /groups/:id/projects would be called GET list a group's projects and the test filename is get_list_a_groups_projects_helper.rb. GET /projects would be called GET list all projects, and the test filename get_list_all_projects_helper.rb.

There are some cases where the provider being tested may not be documented so, in those cases, fall back to starting with the HTTP request method followed by a name that is as descriptive as possible to ensure it's easy to tell what the provider is for.

Conventions summary

Tests	Folder structure	Naming convention
Consumer Test	Follows the Rails reference standards. For example, `Project::Pipelines#index` would be `consumer/specs/project/pipelines/index.spec.js`	Follows the Rails naming standard. For example, `Project::Pipelines#index` would be `Pipelines#index` within the `project` folder.
Provider Test	Grouped like the Rails controllers. For example, `GET list project pipelines` API endpoint would be `provider/pact_helpers/project/pipelines/provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb`	Follows the API documentation naming scheme in sentence case. For example, `GET /projects/:id/pipelines` would be called `GET list project pipelines`.