Automating your test suite: Integration tips for common CT/CI/CD tools

Automating your test suite: Integration tips for common CT/CI/CD tools

What is test automation?

Test automation is used to automate repetitive tasks and other testing tasks which are difficult or too time consuming to perform manually. Creating a Feature file that performs the testing task a human would normally perform is an example of test automation.

What is CT/CI/CD?

CT/CI/CD are variants of  test automation orchestration. This is the method of scheduling and executing your automated test suite  based on a variety of factors such as time, repository commits and manual executions.

While the specific  tools used to perform the automation vary in features and complexity, the overall purpose of each construct remains the same. 

Continuous Testing (CT) is the act of running tests on a schedule only. The source of the code base or the tests isn’t specified. It could be a static, standalone instance that has the ability to execute Cycle Feature or a Playlist on a schedule.

Continuous Integration (CI) is a development practice that teams use to regularly merge code into a shared repository. Each merge triggers an automatic execution of Cycle Features or Playlists.

Continuous Deployment (CD) takes CI one step further and automates the deployment of the solution into the designated environment. That can equate to moving validated code from test to QA and QA to Prod.

What does that mean in Cycle?

Automation orchestration in Cycle is running Features, either standalone or in a playlist, through cycle-cli on a scheduling platform such as Windows Task Scheduler or continuous testing application such as Jenkins, Bamboo or Team City. Automation orchestration applies more to regression projects than Volume or Load projects due to the live monitoring Volume and Load typically require.

For detailed instructions on cycle-cli utilization refer to this Knowledge Base article.

Automation orchestration  is a technical construct that may include, dedicated servers, dedicated personnel, a dedicated production sized fully functional WMS environment, and additional software on top of Cycle. It will require IT expertise and resources different from Cycle Feature authoring.

When do we start thinking about Continuous Testing (CT)?

The simple answer, the sooner the better.  This section details some of the reasons why thinking about CT at the beginning of a project is smoother than trying to squeeze it in at the end.

System Needs

There could be addition infrastructure and software procurement to support CT and getting that request in as early as possible will prevent any delays later in the project.

Potential additional infrastructure

Application server matching production

This is needed to prevent conflicts, ensure testing is done on a consistent environment, not affect development, ensure results and reports generated are accurate and can be reasonably extrapolated to production.

Database server matching production

In the case of JDA this is needed to support the instance. Same reasons as the instance.

Potential additional software

Project tracking (Jira, Trello, YouTrack)

To link test cases to project artifacts including rework issues.

Version Control (Git, SVN, CVS)

To maintain and track historical changes. Version control can be used to trigger automated tests.

CI/CT Tool (Jenkins, Bamboo, Team City)

Engine driving automated tests through a Master and Agent relationship. This tool may require its own infrastructure.

System Set Up

 CT takes time to set up and link with Cycle. Often times in projects there is a little more time on the front end for tasks then trying to stuff something new in the middle or worse yet at the end of a project. Once set up very little will need to change with the CT environment even with all the changes going on in the system under test.

Project Status

 Using BDD style Features in CI/CT will provide an easy to see status of completed executable Features. Initially all the Features will “fail” since they are just BDD steps without the executable support Scenarios. As those Scenarios are completed the number of passed Features will increase up to 100%. One thing to keep in mind is not all Functional areas are created equal. While this approach will highlight coverage percentage it does not show complexity percentage.

Project Priorities

 When working in CT environment it is beneficial to create Features for core functionality first. The most important aspects of the business will be completed and will marinate the longest in the changing environment.

Feature Design

Knowing you will be or may be working with a CT environment will affect Feature design. Features will need to follow the desired process efficiently with as little execution overlap as possible. Setting up Utilities that execute the functional processes will allow for Features to run simultaneously with only data elements being different.

Data Design

CT will require advanced preparation to the data elements necessary to support a test case. One simply cannot use all of the same parts, or locations, or user for every test. Just like one cannot manipulate data elements that could affect other process that could also be potentially running. An example of this would be a test case suspending all work in a system in order to cherry pick a certain activity. That artificial suspension of work could cause other Features to fail unnecessarily. This concept is no different to manually managing a shared environment with a team of developers.

Playlist Design

 Creating groups of Features into Playlists by Functional Area will reduce the amount of potential conflict when running Playlists across a CT environment. For example, a Playlist of Work Order Features and a Playlist of Inventory Counting Features have a far less chance of impacting each other, provided proper data set up, than a Playlist of Web Receiving Features and a Playlist of Terminal Receiving Features. 

Continuous Testing (CT) Approaches

 The goal of your Continuous Testing effort will help determine the approach. Independent of approach, cycle-cli can be called directly or through a batch file or another executable program to provide additional flexibility when calling it. Also, using a batch file makes editing cycle-cli configurations easier than when in a Scheduler directly. The batch or executable file is what the Continuous Testing trigger would call.

 Scheduler Approach

 The most basic way to enable Continuous Testing is through cycle-cli and a scheduler, like Windows Task Scheduler. This can be set up to run Features or Playlists that will run in sequence. The machine would need to have all appropriate access and applications installed and need to be available at the time the scheduler runs. See the Knowledge Base and the User Manual for cycle-cli configurations.

The pros to this approach are:

  •  No additional software required
  •  Easy set up
  • All Cycle reports are available

The cons to this approach are:

  • Features or Playlist run in succession
  •  Can take a very long time depending on the complexity and amount of Features
  • The process takes over the machine running the schedule
  •  Windows Task Scheduler is limited in its output
  • Windows Task Scheduler can be difficult if attempting to run Features headless with native applications such as JDA WMS Local Client (non-Web).

This approach is most effective if the goal is to run a handful of Features nightly to make sure nothing catastrophic is happening. This approach can be used for an entire suite if the execution time is within the operational acceptable range.

For instructions on using the Scheduler approach check out the Cycle User Manual

CI/CT Tool (Jenkins, Team City, Bamboo) Approach

Enabling cycle-cli through a CI/CT server begins the same way as with a scheduler by configuring cycle-cli. The advantage to using a tool is the power they provide. They all have the concept of “Agents” which are controlled by a Master. Agents are instance copies that are used to divide and conquer test suite execution. They also all have the concept of jobs or stages. Jobs or stages are worker bees that execute a particular function such as moving files or calling cycle-cli. They can integrate with code and Feature repositories to check out and execute against. They can be configured to deploy to additional environments. They can be configured to clean up files and directories.

The pros to this approach are:

  • Greater flexibility with:
    • Schedule
    • Triggers
    • Projects
    • Deployment
  • Ability to more easily execute Features Headless
  • Ability to use agents to run the test suite more efficiently
  • Ability to run Features and/or Playlists in Parallel
  • Ability to see trend reporting in the tool
  • Ability to link with Repositories
  • Ability to link with issue tracking software
  • Ability to link with messaging platforms like Slack
  • All Cycle reporting is available

The cons to this approach are:

  • Requires additional software
    • Jenkins is open source
    • Bamboo has a cost
    • Team City includes 3 agents free but additional has cost
  • Tryon doesn’t own nor manage 3rd party CI/CT software
  • Tryon has limited expertise in the backend of such applications
  • Requires standalone environment
  • Requires set up and technical capability to configure
  • Requires upfront Feature and Playlist considerations

How many agents running Cycle are needed?

These major factors will determine how many agents your CT setup will require.

  • How long the test suite takes to run:
    • If we need to run our tests every night, and the full test suite takes 20 hours to run on a single agent - that's too long to run daily. Add agents and divide up the test suite until the time it takes to run the full test suite is suitable for daily execution.
  • How many different platforms are being targeted:
    • If we need to test on Windows versions x,y,z - Linux flavor a,b,c - etc - we would likely need a separate agent for each of those platforms.
  • Necessary frequency/timing of test suites:
    • Once we use Jenkins/TeamCity as part of the code merge process, every time a developer needs to merge code, they need to run through Jenkins/TeamCity. If there is a bottleneck there, more agents will reduce it so that multiple people can get their tests done and get code merged more quickly.

 

Below is an example using Jenkins pipeline and cycle-cli. In this example the Jenkins stage (Verify Environment) is running 3 Features in parallel using cycle-cli. The Features are validating the 3 connection types the suite will be using. This setup is designed to run on the Jenkins agent where Cycle is installed and the Features reside either manually or through a repository checkout.

 

stage('Verify Environment') {

  parallel {

stage('Verify MOCA') {

  steps {

bat '"C:\\Program Files (x86)\\TryonSolutions\\Cycle\\cycle-cli" -a <token> -p C:\\PROJECT_DIR\\PROJECT_FILE.cycproj features/Verify_MOCA_Environment.feature'

  }

}

stage('Verify Terminal') {

  steps {

powershell 'Start-Sleep -s 5'

bat '"C:\\Program Files (x86)\\TryonSolutions\\Cycle\\cycle-cli" -a <token> -p C:\\PROJECT_DIR\\PROJECT_FILE.cycproj features/Verify_RF_Environment.feature'

  }

}

stage('Verify Web UI') {

  steps {

powershell 'Start-Sleep -s 8'

bat '"C:\\Program Files (x86)\\TryonSolutions\\Cycle\\cycle-cli" -a <token> -p C:\\PROJECT_DIR\\PROJECT_FILE.cycproj features/Verify_WEB_Environment.feature'

  }

}

  }

}




    • Related Articles

    • Securely Injecting Secrets and Passwords into Cycle

      Handling secrets, like passwords used to login to the the system under test, safely and securely is an important part of maintaining and sharing your tests. Hard-coding passwords in your features in plain text makes them readable to anyone who you ...
    • How to manage Multi-Branch Pipeline Testing with the Cycle Appliance

      The Cycle Appliance provides a platform that facilitates continuous testing and continuous integration through cloud-based infrastructure running Jenkins in Azure. A key aspect of continuous testing and continuous integration is giving your ...
    • Are There Prerequisites to Deploy the Cycle Appliance?

      The Cycle Appliance provides a platform that facilitates continuous testing and continuous integration through cloud-based infrastructure running Jenkins in Azure. The main objective of the Cycle Appliance is to make getting started with these ...
    • How To Update Jira Statuses With Jenkins Pipeline In Cycle Appliance

      The Cycle Appliance provides a platform that facilitates continuous testing and continuous integration through cloud-based infrastructure running Jenkins in Azure. A key aspect of continuous testing and continuous integration is giving your ...
    • How To Post Jenkins Pipeline Results from the Cycle Appliance to Slack

      The Cycle Appliance provides a platform that facilitates continuous testing and continuous integration through cloud-based infrastructure running Jenkins in Azure. A key aspect of continuous testing and continuous integration is giving your ...