Non-functional Requirements

You can define formal, non-functional requirements using YAML. Here is a short example

version: "0.1"
requirements:

# require overall error ratio to be below 0.5%
- http.error_ratio:
    test: ["<", 0.005]

# require overall 99th latency percentile to be below 1.5 sec
- http.latency:
    select:
      type: percentile
      value: 99.0
    test: ["<=", 1500]

You can use this definition to check a completed test run against these requirements.

NOTE This feature is currently in beta and only available via our command line tool. Usage is forge test-run nfr acme-inc/shop2018/42 stormforger/nfr/blackfriday.yml. The output will look like this:

CLI NFR Output

Requirement Definition

We currently support the following requirement types:

  • test.completed, to check if your test was completed (and not aborted)
  • test.duration, to check the test duration
  • test.cluster.utilization.cpu, to check the test cluster utilization
  • checks, to assert your checks
  • http.error_ratio, to check for HTTP error ratios (status >= 400)
  • http.status_code, to check for HTTP status codes
  • http.latency, to check for HTTP request latencies
  • http.apdex, to check for Apdex Scores

Every requirement has up to three parts:

  1. Select metric (select)
  2. Filter subject (where)
  3. Test to be performed (test)
- http.latency:
    select:
      type: percentile
      value: 99.0
    where:
      status: ["<", 400]
      tag:
      - checkout
      - register
    test: ["<", 1500]

Select Metric

With select you can specify what metric or aggregation you are interested in.

In this example we are interested in the 99th percentile.

Filter Subject

With where you can specify filter for example to select specific requests.

In the example above, we want to have requests, which status code is below 400 and request tag is checkout or register; Think SELECT percentile(duration, 99) FROM http.latency WHERE status < 400 AND tag IN('checkout', 'register').

Test

With test you can specify the test criteria that should be applied to the subject.

In the example we want the subject to be below 1500 ms (subject being 99th percentile of requests).

Currently you have to specify the test in form of ["$operator", "$value"]. Operator can be =, !=, <, >= or <=.

Longer Example

version: "0.1"

requirements:

# Check if the test was completed and not aborted
- test.completed: true

# Require that the test cluster CPU was not under too high load
- test.cluster.utilization.cpu:
    select: average
    test: ["<=", 60]


# Checks
# =============================================================================

# make sure that we have have executed exactly 1000 checks
- checks:
    select: total
    test: 1000

# allow up to 100 KOs (failed checks) overall
- checks:
    # possible: 'ok', 'ko', 'success_rate' or 'total'
    select: ko
    test: ["<=", 100]

# require success rate of over 95% for
# 'checkout_step_1' checks
- checks:
    select: success_rate
    where:
      name: checkout_step_1
    test: [">", 0.95]


# HTTP Status & Errors
# =============================================================================

# require overall error ratio to be below 0.5%
- http.error_ratio:
    test: ["<", 0.005]

# require http error rate of requests tagged with 'checkout_step_1' or
# 'checkout_step_2' to be at most at 1%.
- http.error_ratio:
    where:
      tag: 
      - checkout_step_1
      - checkout_step_2
    test: ["<=", 0.01] # <= 1%

# require all requests tagged 'checkout_step_1' to be HTTP 302 redirects.
- http.status_code:
    select:
      type: ratio
      value: 302
    where:
      tag: checkout_step_1
    test: ["=", 1]

# require almost all requests tagged 'register' or 'checkout' to be HTTP 301
# redirects.
- http.status_code:
    select:
      type: ratio
      value: 301
    where:
      tag:
      - register
      - checkout
    test: [">=", 0.99]

# require HTTP status code 504 (Gateway Timeout) overall to be under 0.1%.
- http.status_code:
    select:
      type: ratio
      value: 504
    test: ["<", 0.001]


# HTTP Latency
# =============================================================================

# require the overall average to be below 1 sec
- http.latency:
    select: average
    test: ["<", 1000]

# require overall 99th latency percentile to be below 1.5 sec
- http.latency:
    select:
      type: percentile
      value: 99.0
    test: ["<=", 1500]

# require 95th latency percentile of 'checkout' to be below 5 sec
- http.latency:
    select:
      type: percentile
      value: 95.0
    where:
      tag: checkout
    test: ["<", 5000]

# require the 99th percentile of requests with status code below 400 and tag
# 'checkout' or 'register' to be below 1500 ms.
- http.latency:
    select:
      type: percentile
      value: 99.0
    where:
      status: ["<", 400]
      tag:
      - checkout
      - register
    test: ["<", 1500]


# HTTP Apdex Score
# =============================================================================
# require overall Apdex (750ms) to be at least 0.5
- http.apdex:
    select:
      value: 750
    test: [">=", 0.5]

# require Apdex (750ms) score of requests tagged with 'search' to at least 0.8.
- http.apdex:
    select:
      value: 750
    where:
      tag: search
    test: [">=", 0.8]
Get started icon

Get Started

New to the StormForger?
With these guides you’ll be up and running in no time!

FAQ icon

FAQ

Already took a look at our FAQs?

Support icon

Support

Are you stuck? Talk to us! We're humans.

We are using cookies to give you the best online experience. If you continue to use this site, you agree to our use of cookies. By declining we will disable all but strictly required cookies. Please see our privacy policy for more details.


Accept Decline