Test automation is used to automate repetitive tasks and other testing tasks which are difficult or too time consuming to perform manually. Creating a Feature file that performs the testing task a human would normally perform is an example of test automation.
CT/CI/CD are variants of test automation orchestration. This is the method of scheduling and executing your automated test suite based on a variety of factors such as time, repository commits and manual executions.
While the specific tools used to perform the automation vary in features and complexity, the overall purpose of each construct remains the same.
Continuous Testing (CT) is the act of running tests on a schedule only. The source of the code base or the tests isn’t specified. It could be a static, standalone instance that has the ability to execute Cycle Feature or a Playlist on a schedule.
Continuous Integration (CI) is a development practice that teams use to regularly merge code into a shared repository. Each merge triggers an automatic execution of Cycle Features or Playlists.
Continuous Deployment (CD) takes CI one step further and automates the deployment of the solution into the designated environment. That can equate to moving validated code from test to QA and QA to Prod.
Automation orchestration in Cycle is running Features, either standalone or in a playlist, through cycle-cli on a scheduling platform such as Windows Task Scheduler or continuous testing application such as Jenkins, Bamboo or Team City. Automation orchestration applies more to regression projects than Volume or Load projects due to the live monitoring Volume and Load typically require.
For detailed instructions on cycle-cli utilization refer to this Knowledge Base article.
Automation orchestration is a technical construct that may include, dedicated servers, dedicated personnel, a dedicated production sized fully functional WMS environment, and additional software on top of Cycle. It will require IT expertise and resources different from Cycle Feature authoring.
The simple answer, the sooner the better. This section details some of the reasons why thinking about CT at the beginning of a project is smoother than trying to squeeze it in at the end.
There could be addition infrastructure and software procurement to support CT and getting that request in as early as possible will prevent any delays later in the project.
Application server matching production
This is needed to prevent conflicts, ensure testing is done on a consistent environment, not affect development, ensure results and reports generated are accurate and can be reasonably extrapolated to production.
Database server matching production
In the case of JDA this is needed to support the instance. Same reasons as the instance.
Project tracking (Jira, Trello, YouTrack)
To link test cases to project artifacts including rework issues.
Version Control (Git, SVN, CVS)
To maintain and track historical changes. Version control can be used to trigger automated tests.
CI/CT Tool (Jenkins, Bamboo, Team City)
Engine driving automated tests through a Master and Agent relationship. This tool may require its own infrastructure.
CT takes time to set up and link with Cycle. Often times in projects there is a little more time on the front end for tasks then trying to stuff something new in the middle or worse yet at the end of a project. Once set up very little will need to change with the CT environment even with all the changes going on in the system under test.
When working in CT environment it is beneficial to create Features for core functionality first. The most important aspects of the business will be completed and will marinate the longest in the changing environment.
Knowing you will be or may be working with a CT environment will affect Feature design. Features will need to follow the desired process efficiently with as little execution overlap as possible. Setting up Utilities that execute the functional processes will allow for Features to run simultaneously with only data elements being different.
CT will require advanced preparation to the data elements necessary to support a test case. One simply cannot use all of the same parts, or locations, or user for every test. Just like one cannot manipulate data elements that could affect other process that could also be potentially running. An example of this would be a test case suspending all work in a system in order to cherry pick a certain activity. That artificial suspension of work could cause other Features to fail unnecessarily. This concept is no different to manually managing a shared environment with a team of developers.
Creating groups of Features into Playlists by Functional Area will reduce the amount of potential conflict when running Playlists across a CT environment. For example, a Playlist of Work Order Features and a Playlist of Inventory Counting Features have a far less chance of impacting each other, provided proper data set up, than a Playlist of Web Receiving Features and a Playlist of Terminal Receiving Features.
The goal of your Continuous Testing effort will help determine the approach. Independent of approach, cycle-cli can be called directly or through a batch file or another executable program to provide additional flexibility when calling it. Also, using a batch file makes editing cycle-cli configurations easier than when in a Scheduler directly. The batch or executable file is what the Continuous Testing trigger would call.
The most basic way to enable Continuous Testing is through cycle-cli and a scheduler, like Windows Task Scheduler. This can be set up to run Features or Playlists that will run in sequence. The machine would need to have all appropriate access and applications installed and need to be available at the time the scheduler runs. See the Knowledge Base and the User Manual for cycle-cli configurations.
The pros to this approach are:
The cons to this approach are:
This approach is most effective if the goal is to run a handful of Features nightly to make sure nothing catastrophic is happening. This approach can be used for an entire suite if the execution time is within the operational acceptable range.
For instructions on using the Scheduler approach check out the Cycle User Manual
Enabling cycle-cli through a CI/CT server begins the same way as with a scheduler by configuring cycle-cli. The advantage to using a tool is the power they provide. They all have the concept of “Agents” which are controlled by a Master. Agents are instance copies that are used to divide and conquer test suite execution. They also all have the concept of jobs or stages. Jobs or stages are worker bees that execute a particular function such as moving files or calling cycle-cli. They can integrate with code and Feature repositories to check out and execute against. They can be configured to deploy to additional environments. They can be configured to clean up files and directories.
The pros to this approach are:
The cons to this approach are:
These major factors will determine how many agents your CT setup will require.
Below is an example using Jenkins pipeline and cycle-cli. In this example the Jenkins stage (Verify Environment) is running 3 Features in parallel using cycle-cli. The Features are validating the 3 connection types the suite will be using. This setup is designed to run on the Jenkins agent where Cycle is installed and the Features reside either manually or through a repository checkout.
stage('Verify Environment') {
parallel {
stage('Verify MOCA') {
steps {
bat '"C:\\Program Files (x86)\\CycleLabs\\Cycle\\cycle-cli" -a <token> -p C:\\PROJECT_DIR\\PROJECT_FILE.cycproj features/Verify_MOCA_Environment.feature'
}
}
stage('Verify Terminal') {
steps {
powershell 'Start-Sleep -s 5'
bat '"C:\\Program Files (x86)\\CycleLabs\\Cycle\\cycle-cli" -a <token> -p C:\\PROJECT_DIR\\PROJECT_FILE.cycproj features/Verify_RF_Environment.feature'
}
}
stage('Verify Web UI') {
steps {
powershell 'Start-Sleep -s 8'
bat '"C:\\Program Files (x86)\\CycleLabs\\Cycle\\cycle-cli" -a <token> -p C:\\PROJECT_DIR\\PROJECT_FILE.cycproj features/Verify_WEB_Environment.feature'
}
}
}
}