The Cycle Appliance provides a platform that facilitates continuous testing and continuous integration through cloud-based infrastructure running Jenkins in Azure. A key aspect of continuous testing and continuous integration is giving your developers and testers a platform to test their changes early and often so that any changes merged into the main branch of the code repository have been tested and those changes can be deployed with confidence.

Jenkins pipelines provide many options for managing your automated tests. This article describes an example of using logic within the Jenkinsfile to determine which tests your Jenkins build should run during pipeline execution.

Example Scenario

For the example in this article, we will assume there are tests associated with existing functionality that have already been deployed. Those tests are in the main branch of our code repository. When we run a Jenkins build on the main branch, we will want to run all of the tests for approved changes that have already been deployed.

Meanwhile, a developer has been tasked with adding and testing new functionality. The developer creates a branch in the repository and includes a new test feature in that branch. When the developer runs a build against that feature branch of the repository, only tests associated with that change are executed.

Once, the changes have been approved and merged into the main branch, then that new test will become part of the approved tests and all tests will run when running a pipeline build on the main branch.

Jenkins Pipeline Syntax

We will be using several options available to us in Jenkinsfile syntax to drive our pipeline test execution. Detailed information on Jenksinfile pipeline syntax and usage can be found here.

BRANCH_NAME Environment Variable

We will use the Jenkins environment variable BRANCH_NAME to determine which tests we need to run during any given build. This BRANCH_NAME variable is set by Jenkins and matches the branch name checked out from the code repository for that build execution.

when/anyOf and when/not/anyOf

The Jenkins pipeline uses the “when” directive to determine whether or not a stage should be executed depending on a specified condition. In our example, we will be using the “when” directive to control which tests are executed based on the repository branch name. Using “anyOf” and the combination of “not” and “anyOf”, we are able to include or exclude stages during the build based without having to modify our Jenkinsfile every time we run a build for a specific branch.

Example Syntax:

In the example above the stage labeled “Main and Development Branch Tests” will execute if the branch the build is running on is either the “master” branch or the “development” branch.

The stage labeled “Feature Branch Tests” will only execute if the build is running on a branch that is not “master” or “development”.

Running Multi-Branch Tests using Cycle

The following approach can be used to control which Cycle tests run based on the repository branch the build is executed on. This is a simple example and it may be necessary to use more complex test and code management in a production environment, but this example can serve as a foundation to build more complex pipelines.

Repository Contents

In this example, we have a repository with two branches:

Master Branch

The master branch currently contains two test features in a folder called web_tests.

These two tests represent web-based tests to login and to sign-up for a new account.

The master branch also contains a playlist. This playlist is set to execute all of the features in the “web_test” folder.

Feature Development Branch

The example repository also contains a development feature branch called “TEST-1-multibranch-pipeline”. This branch was created from the master branch, so it contains the same files listed in the master branch above. It also contains development related to a new test that is not yet part of the master branch.

There is an additional feature test file in the “web_tests” folder of this branch. This new feature tests the password reset function.

There is a branch specific playlist in the “playlists/branches” subfolder. We will use this playlist to ensure our password reset feature is executed when the Jenkins build is run against the feature specific branch.

Notice that the playlist for the branch specific feature will execute the test password_reset.feature rather than executing all of the features in the web_test folder as specified in the web_test_playlist on the master branch.

Jenkinsfile Configuration for Multi-branch Testing

We can use the when/anyOf and when/not/anyOf Jenkins pipeline directives along with the repository branches to control which tests execute for each branch build in Jenkins.

A full copy of the Jenkins file used for this example is available at the bottom of this article. We will look at the portion of the pipeline file that allows us to run different tests for each branch.

In the pipeline snippet below, we have two different stages that execute playlists using Cycle-CLI. The details of what the code is doing are listed in the following sections.

Master branch test

The “All Approved Tests” pipeline will run on a build of any branch named “master”. This stage runs Cycle-CLI and executes the playlist web_test_playlist.cycplay found in the “playlists” directory.

Running a build in Jenkins against the master branch results in a pipeline execution like the one below:



Notice that the “All Approved Tests” stage was executed, but the “Development Feature Branch Tests” stage was skipped. This was due to running the build against the master branch of the repo. The logic in the Jenkinsfile directed our pipeline to run the tests for our master branch while skipping execution of our tests currently in development on the feature specific branch.

You can also see that the pipeline ran the two features for “Login Test” and “Sign Up Test”.

Feature branch test

The “Development Feature Branch Tests” stage will execute on a build of any branch not named “master”. Notice that this stage runs Cycle-CLI against a playlist found in the “playlist\branches” subfolder. Specifically, we are instructing Cycle-CLI to look for a playlist named after the repository branch we are building.

We use the BRANCH_NAME Jenkins environment variable to specify the name of the Cycle playlist so we do not need to update the Jenkinsfile each time we want to test a different feature branch. As long as we have a cycplay file in the branches folder that is named after the branch being built, Cycle-CLI will run  the playlist specified for that branch.

Running a build in Jenkins against the feature specific branch results in a pipeline execution like the one below:

Notice that for this build, the “All Approved Tests” stage was skipped, but the “Development Feature Branch Tests” was executed. This is because this build was against the “TEST-1-multibranch-pipeline” branch of the repository rather than the master branch.

Also, notice the output shows that Cycle-CLI executed the TEST-1-multibranch-pipeline.cycplay playlist. The flexibility this approach provides allows you to test your development changes and ensure those tests are working before merging code into your master branch and running a full suite of tests across all functional areas.

Merging Branches

After development and testing are complete on the feature specific branch, that branch is merged with the master branch. The approach detailed in this example does not require any changes to the Jenkinsfile in order to include the new password reset test as part of the master branch playlist. Merging the feature branch with the master branch will place the password_reset.feature in the web_tests folder of the master branch, and the web_test_playlist.cycplay playlist will automatically include the feature as part of the “All Approved Tests” stage.

After merging the branches, all three Cycle test features are now in the web_tests folder.

Executing a build on the master branch will now run all three tests as part of the “All Approved Tests” stage.

Now all three tests execute as part of the “All Approved Tests” stage.