Using StormForger with GitLab

This guide shows an example GitLab CI/CD workflow utilising StormForger to perform automated load tests after each deployment. We also setup a scheduled load tests (e.g. once a week) against your application.

A validation run is also performed against the staging environment for every push to ensure the load test definition is up to date.

GitLab CI/CD Job Execution

For this guide we assume you have general knowledge of GitLab CI/CD and how it works. You also need the permissions to configure variables in your repository or organisation.

The code is available on GitHub at stormforger/example-gitlab-ci. Our example service is written in Go, but you don't need to know Go, as we will only discuss the StormForger related steps.

Note that this is just an example and your actual development workflow may differ. Please take this as an inspiration how to use StormForger with GitLab CI/CD.

Preparation

Please follow the Getting Started with the Forge CLI guide to create an API Token and configure it as the STORMFORGER_JWT variable in your repository.

Follow the steps described in GitLab's Creating variables in the UI guide. Since this is a secret, we recommend ensuring it is masked and protected. It should look like this when you are done:

GitLab CI/CD Variable Settings Screenshot

Note: You can also configure secrets on the group level. If you have multiple repositories using StormForger, this might be easier to manage.

Inside the workflow this secret can now be referenced as ${STORMFORGER_JWT}, but you don't need to reference it directly. Our CLI will pick it up automatically.

Steps

Our workflow file .gitlab-ci.yml consists of five stages: build, staging:deploy, staging:test, production:deploy and production:test. Everything except the production:* stages are executed on every commit.

Our goal is to run the load test in validation mode against the staging target in the staging:test job and against the production environment in the production:test job. For both environments we follow the same steps:

  1. Manage Data Sources
  2. Build the Test Run and Launch It

There is also a ping step in the build stage to check if the forge cli can communicate correct with the API.

Let's checkout the ping job first, then go through each step one by one.

Ping with the StormForger CLI

As a first step, we check that the forge CLI works and successfully talks with the StormForger API:

.forge:
  image:
    name: stormforger/cli:latest
    entrypoint: [""]

sf:ping:
  stage: build
  extends: .forge
  script:
  - forge ping # check that we can reach the API and our token is correct

Here we define a template job .forge that we use in the sf:ping job. Since we will reuse the docker image definition, this avoids having to copy the same image definition multiple times. The stormforger/cli:latest is the docker image that is released as part of our StormForger CLI. You can also copy this image to your private registry, but make sure to update it regularly. The actual work is in the forge ping command: This performs an authorized ping against the StormForger API and verifies that the token is valid and usable.

Manage Data Sources

Data Sources allow a Test Run to pick random data out of a predefined pool, e.g. a product out of all the available inventory. If you don't use data sources, you can skip this step.

In our workflow we use the script ./scripts/data-source.sh to generate a CSV file for our test, but this can be easily changed or extended to download the latest inventory data from a database.

./scripts/data-source.sh "${TARGET_ENV}"
forge datasource push ${SF_ORG} ./*.csv --name-prefix-path="${SF_DS_PREFIX}/${TARGET_ENV}/" --auto-field-names

We prefix all uploaded CSV files with our repository name and the target environment to make them easily distinguishable in the data source management of app.stormforger.com.

This step is the same for both the staging and production environment, except for the value of the TARGET_ENV variable.

Build the Test Run and Launch It

Heads Up! Testing in production can be dangerous. Make sure the configured arrival rate/load in your Test Case repo is low enough so you don't trigger any incidents. The goal here is not to run stress tests regularly but to ensure we do not experience any major performance regression with nominal load.

As the last step we launch the Test Run. We use one script here: scripts/compile-loadtest.sh. It combines the load test (in loadtest/loadtest.js) with an environment specific prefix (loadtest/staging.js or loadtest/production.js). Building the Test Case dynamically for each environment allows us to modify the target system urls, specify global variables and even define different arrival phases as needed. Afterwards we use forge test-case launch to launch a Test Run.

This step can be more or less sophisticated, like generating more complex Test Cases, compiling them from TypeScript or using a bundler like gulp/webpack.

The Combined GitLab Job

Combined into a job, the scripts and steps look like this in our .gitlab-ci.yml.

sf:loadtest_staging:
  stage: staging:test
  extends: .forge
  needs: ["sf:ping"]
  script:
    - ./scripts/data-source.sh "${TARGET_ENV}"
    - forge datasource push ${SF_ORG} ./*.csv --name-prefix-path="${SF_DS_PREFIX}/${TARGET_ENV}/" --auto-field-names

    - ./scripts/compile-loadtest.sh "${TARGET_ENV}" "/tmp/testcase.js"
    - forge test-case launch "${TESTCASE}" --test-case-file="/tmp/testcase.js" --title="${TITLE}" --notes="${NOTES}" ${LAUNCH_ARGS}
  variables:
    TARGET_ENV: "${TARGET_ENV_STAGING}"
    LAUNCH_ARGS: "--validate"
    NOTES: |
      Name | Value
      ---- | -----
      Ref  | ${CI_COMMIT_REF_NAME}
      Sha  | ${CI_COMMIT_SHA}
      Job#    | [${CI_JOB_ID}](${CI_JOB_URL})
      Concurrent# | ${CI_CONCURRENT_ID}
      User    | ${GITLAB_USER_LOGIN}

      Message:
      ${CI_COMMIT_MESSAGE}
    TITLE: "#${CI_CONCURRENT_ID} (${CI_COMMIT_SHORT_SHA})"
    TESTCASE: "${SF_ORG}/example-gitlab-ci-${TARGET_ENV_STAGING}"

In this step we use a lot of environment variables. This is mainly for formatting reasons. NOTES and TITLE are stored in the Test Run and provide metadata based on the current commit. This allows easier retracing and comparing later on, as we also link back to the current GitLab job. You can check the predefined variables GitLab documentation if you want to customize this.

With LAUNCH_ARGS: "--validate" we are launching the test run only in validation mode for our staging environment. For the production environment we are instead passing in LAUNCH_ARGS: "--nfr-check-file=./loadtest/loadtest.nfr.yaml" which performs the Non-Functional Requirement checks after the test run has finished.

Formatted the notes and title look like this:

Screenshot showing the GitLab Metadata in a StormForger report

Scheduled Test-Run Execution

Finally, we want to run the load test once a week automatically via GitLabs pipeline schedules.

Since we don't want to run through all phases (build, test, deploy) nor redeploy to production for this, we use the except statement to disable all jobs. The sf:loadtest_production job gets no except: statement, as this is the job we want to actually execute by the scheduled execution.

For the deploy_staging job, this looks like this:

deploy_staging:
  stage: staging:deploy
  except: ["schedules"]

Screenshot showing the create pipeline schedule form

You can hit the "Play" button after saving the scheduled pipeline to perform a test execution:

Screenshot showing a scheduled pipeline running the load test

This pipeline will now execute every Sunday night!

Summary

To summarise, we used GitLab CI/CD to automatically upload data sources and launch a Test Run for every environment in our development cycle. A weekly job verifies that the load test continuously works and no other factors introduce regressions. By using NFR checks, we automatically verify that our non-functional requirements are fulfilled.

Icon Support Are you stuck? Or do you have any feedback? Get in touch with us – we are happy to help you.
Icon Schedule a demo Schedule a personal, customized demo. We'll show you around and introduce you to StormForger.
Icon Talk to a human To build and run reliable applications is complex – we know. Schedule a call and we’ll figure things out.

We are using cookies to give you the best online experience. If you continue to use this site, you agree to our use of cookies. By declining we will disable all but strictly required cookies. Please see our privacy policy for more details.


Accept Decline