Frequently Asked Questions

Reports

What is Apdex?

Apdex (Application Performance Index) is an index that summarizes the performance of a system under test. For more information, see https://en.wikipedia.org/wiki/Apdex#Apdex_method.

What is median?

Median is a statistical value, describing the measurement where 50% of all measured values lie below and 50% above. A median response time of 100 ms means that 50% of all requests have been completed within 100 ms and 50% took longer then 100 ms.

What is percentile?

A percentile (or a centile) is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall. – https://en.wikipedia.org/wiki/Percentile

Example: Your test showed that your 95th percentile is 42 ms and your 99th percentile is 150 ms. This means that 95% of your requests were completed within 42 ms (and 5% took longer), and 99% of your requests took 150 ms or less (and only 1% of all requests took longer).

A word on Mean and Standard Deviation

Only looking at mean values for performance measurements is dangerous because you have no indication of how your actual response times are spread around the mean value.

In statistics, the standard deviation […] is a measure that is used to quantify the amount of variation or dispersion of a set of data values. A standard deviation close to 0 indicates that the data points tend to be very close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values. – https://en.wikipedia.org/wiki/Standard_deviation

Standard Deviation gives an indication how close the measured response times are to the mean. This gives you some idea how far spread your response times are.

We highly recommend to not use mean as your primary performance indicator. You should take a look at median, 95th and 99th percentile instead.

Making Requests

Which HTTP Headers does StormForger always send?

To identify the user who sends the requests to a given target we always send the users signature as header.

headers: {
  "X-StormForger-User": user.signature,
}

Which HTTP Headers can not be overwritten?

You cannot set headers beginning with X-StormForger-. Also the user agent can not be overwritten.

Why is the number of requests not exactly as defined?

There are several reasons for this.

Most likely the test duration is over before a client has made all its requests. It also appears if a client waits to long for a response and thus can not make the following requests.

Also take a look at Why did my test end early?

Are there any details to the requests defined in a test case?

Yes there are. On the TestCase detail page all defined sessions are listed in the sessions box below the test case summary. If you click on a session a more detailed view will expand. This view contains all defined requests as curl requests which you can and should validate by copying and pasting them in the shell of your choice.

Why are the defined requests not arriving at once in my system?

Although you define a rate in terms of new clients per second in the test case definition StormForger uses a poisson distribution to launch the defined clients.

This is due to the goal of creating a more realistic behavior.

For more details check out the arrival phases reference.

Can I specify which session should use which target?

No you can't. For each session StormForger will use all specified targets in the setTargets definition in a round robin approach. If you need to define different targets in specific requests you have to hardcode an absolute URL in the request like so:

session.get("https://api2.mytarget.com/login")

Miscellaneous

Where will the traffic/requests come from?

We currently only utilize resources from the Amazon AWS cloud (regions eu-central-1 and eu-west-1). If you need to know what IP ranges will be used, consult AWS's documents on AWS IP Address Ranges.

Why did my test end early?

Your test ends when the following criteria are fulfilled:

  1. no more clients should be launched and
  2. all previously launched clients have completed their session

A common reason for irritation comes from max_clients value in your arrival phase. If you launch 10 clients per second, but only allow 100 max_clients your test might end within 10 seconds (if your session is short).

Why do I see more requests in my logs compared to your report?

We only analyze completed requests currently. So if your test ends before all requests have been completed you might see slightly more requests hitting your target compared to our reporting.

What is Preflight Mode?

Preflight mode means that you will get a fully functional test, but only very limited resources will be allocated to your test. Measurements and reports have to be taken with a grain of salt.

In preflight mode we limit the maximum amount of clients to 300 in your test case.

If you need more load you can send a request via the Request more power button.