Changelog

2020-Q2

Highlight: We have completed the migration of all customers to our new engine! 🥳 The legacy engine is no longer available.

In case you are curious about our engine transition, you can check out our article on Load Testing Engine Evolution. Note that the jsonpath extraction behaves slightly different in some cases. This and other differences can be found in the engine reference.

  • Added/TCDSL: Using the new regexpheader you can extract data from response headers
  • Added/TCDSL: Cookie Domain Mapping
  • Added: We added graphs on "active connections" and "new connection rate" to the test run report
  • Added: A table with "other errors and events" is now shown in the report
  • Fixed: regexp results are no longer trimmed for whitespace
  • Fixed: The test case outline no longer incorrectly adds a --http1.1 option to the curl example
  • Improved/TCDSL: Validation of request headers is now more strict
  • Improved/TCDSL: cors: false is now supported as an request option to disable CORS (this is the default)
  • Improved: Validation Runs are now allowed in parallel to one normal test-run per test-case
  • Improved: Test Runs can now be aborted while they are still launching
  • Improved: Abnormally terminated test runs are now collected in a new list
  • Improved: Traffic Dumps filter out binary responses by replacing them with an omission marker
  • Improved: Graphs in test run reports are now faster to use (navigate and zoom)
  • Improved: Durations in the test report are now "human readable" and split into minutes, seconds and milliseconds (e.g. "1m 23s") in case they exceed a certain threshold
  • Improved: "Other errors" are now available for filtering in the latency distribution section of the report
  • Improved/CLI: NFR validation and traffic dumping are no longer mutually exclusive
  • Improved/CLI: Using test-case create --update <file> will either create a new test case or update an existing one
  • Improved/CLI: There is now some more documentation on Getting Started with the forge CLI.
  • Improved/CLI: test-case launch now also allows to update the test case with a new JavaScript file in one operation by passing --test-case-file=<file>.

2020-Q1

Highlight: We are previewing our new load testing engine. See our engine guide for details.

Highlight: All legacy data sources have been auto-archived on March the 31th of 2020 (EOD).

  • Added: New session.redirectTarget() helper function for easier following of redirects (only available in the new engine)
  • Removed/TCDSL: Relative request paths are no longer allowed if multiple targets are defined. Before, a random target was chosen which lead to unexpected behaviour
  • Removed: Legacy data sources are no longer supported
  • Improved/HAR Converter:
    • improved scaffolded test case
    • added filter options for hosts and assets
    • allow to ignore global filter lists
    • more debug options
  • Improved: Various checks and timeouts have been added to ensure that test runs don't get stuck (which rarely happened anyway). We also improved the time-to-test start significantly
  • Improved/TCDSL: Added more validation for CORS shorthand
  • Improved/TCDSL: Improved validation of incorrect JSONPath expressions
  • Improved/TCDSL: Improved validation for setTestOptions when configuring cluster types
  • Fixed: In certain cases the test run call log could not be downloaded
  • Fixed: Some validation errors were hidden uploading fixtures / data sources using the web UI
  • Fixed: Validation runs and preflight tests did not work in eu-north-1 (AWS Stockholm)

2019-Q4

Highlight: We sunsetted our old data source API. See our [migration guide] for details. All legacy data sources will be automatically archived on March the 31th of 2020 (EOD).

  • Removed/TCDSL: Test Cases with more than one Target could use request paths to load balance traffic between these targets. Instead define multiple sessions (e.g. per target) with absolute URLs to imitate such load balancing.
  • Added: Allow variables for session.wait() and session.waitExp(), e.g. session.wait(session.getVar("retry_in"))
  • Added: Function timestamp to get the current time as a unix timestamp [docs]
  • Improved/Extractions: Better error messages for invalid JSONPath expressions for $[0]_id
  • Improved/Editor: ECMAScript 6 is now supported
  • Fixed: Allow single character tag, variable and check names
  • Fixed/TCDSL: Fixed validation for session.addTarget()
  • Fixed/SSL: Clients can now change the client certificate multiple times, e.g. to check resigning flows
  • Improved/Editor: Use CodeMirror for our web editor
  • Improved/TrafficDump: Traffic Dumping a session-validation test-run now records all requests correctly
  • Improved/TCDSL: Improved validation around max_clients in the arrival-phases to alert on non-integers
  • Improved/TCDSL: Improve support for multi-line error messages
  • Fixed/Test-Run Dashboard: Do not show null requests during test-run dashboard

2019-Q3

Highlight: session.calcInt() and session.calcFloat() to calculate dynamic values based on extraction results [docs]

  • Added/TCSDL: Added "pinned TLS version" support (definition.setTestOptions({ special: { pinned_tls_version: "tlsv1.1" } });)
  • Improved/HAR Converter: Generated test cases contain more details about skipped requests, if blocked due to the HTTP method
  • Improved/HAR Converter: UI now links to the corresponding guide in the docs
  • Improved/Test-Cases: test case outline now shows the Request-ID header [docs]
  • Fixed/Test-Cases: auto terminate test run for rare cases when cluster failed to start
  • Fixed/Test-Cases: test cases can now be differed with the first version
  • Fixed/TCDSL: Fixed setAuthentication typo
  • Fixed/Test-Cases: Region selection contained wrong US state names for Oregon / Ohio

2019-Q2

  • Added/TCDSL: Added session.addTarget() [Docs]
  • Added/Data Sources: Show uploaded file history
  • Improved/Requests: additional_cookies structure validation has been improved
  • Improved/HAR Converter: OPTIONS requests are supported and no longer filtered
  • Improved/Test-CaseLaunch: Region selector now shows the AWS region ID first, then the name
  • Improved/Test-CaseUpload: Deny uploads with a zero file size
  • Improved/Test-Runs: Made handling of empty variables more robust
  • Fixed/Test-Case Launch: Skip disabled sessions in validation mode
  • Fixed/Latency Distribution: Fixed shown data being truncated when certain filters where applied
  • Fixed/Latency Distribution: Fixed error when not enough data is available
  • Fixed/Test-Cases: Fixed internal server error when showing aborted test runs with enabled traffic dumps
  • Fixed/HAR Converter: Generate valid variable names for csrf_token instead of illegal csrf-token

2019-Q1

  • Added: Configurable latency distribution analysis. You can filter requests based on arrival phase, tag and status and get insights on the response time distributions.
  • Added: A X-Request-ID header is now send with all requests. This can be configured (or disabled). The value is guaranteed to be unique per request.
  • Added: cookie extraction to extract the value of a Set-Cookie response header
  • Added: JavaScript definition can be downloaded for completed test runs
  • HAR Converter/Added: The conversion can now be performed in context of an organisation to allow for black and white-listed targets etc.
  • Added: session.withProbability() to execute steps with a given probability
  • Added: Test reports now contain free memory on load generators
  • Improved: Integration Links now support more timestamp variants
  • Improved: Test Case error stack traces are now truncated by default to hide internal frames
  • Improved: session.setVar(name, value) now allows for dynamic values
  • Improved: The test case outliner now includes data source pick, check and assert operations
  • Improved: Time to start a test run is now below 60 seconds for 98% of all tests (time from trigger to first client launched); 50% (median) start time is 39 seconds.
  • HAR Converter/Improved: Detection of XMLHTTPRequests (XHR) has been improved
  • HAR Converter/Improved: Better support for non-browser HAR recordings like Charles Proxy
  • Fixed: Path substitutions were not applied for CORS request
  • Fixed: In certain situations checks based on session.lastHttpStatus() would not fail even if the request failed
  • Fixed: Allow request payload for DELETE requests, see RFC7231 section 4.3.5
  • Fixed: session.chooseBy{Probability,Weight}() now have improved validations
  • Fixed: In certain situations request options passed to session.{get,post,…}() could be mutated leading to unindented side-effects.
  • HAR Converter/Fixed: CONNECT requests are now filtered correctly

2018-Q4

  • CLI/Added: Add //#include comment to allow simple inclusion of test case snippets (https://github.com/stormforger/cli/pull/58)
  • CLI/Fixed: Improved error messages for test case evaluation errors (https://github.com/stormforger/cli/pull/57)
  • Fixed: Sorting of test case list was broken
  • Fixed: In certain situations, the test case overview showed the same target multiple times
  • Fixed: For test runs started in Session Validation Mode, the "Archive all Preflight" action did not work
  • Fixed: For CORS request, URL params were not properly substituted
  • Fixed: Validations were added for session.if expressions to make sure the variable name is correct

2018-Q3

  • Added new launch option for test runs: You can now overwrite configuration options like cluster sizing and region as well as other, mostly debugging related options. This is available when launching tests via the web UI as well as the command line tool.
  • Added session validation mode: This will ignore arrival phases and cluster configuration and instead launch a single user per defined scenario one after another. This can be used to verify that a test case works properly.
  • CLI: Added various new options around launching test runs and listing resources. See README or use --help for more details.
  • Added Traffic Dump option for test runs: You can now request a traffic dump for a preflight test for debugging purposes
  • Added: Organisations may now be renamed
  • Fixed HEAD requests. Under certain conditions session.head(…) requests caused clients to hang for up to 60s. This has now been resolved.
  • Fixed: JSONPath expressions may now contain dynamic parts
  • Fixed: session.abort() did not always set an abort reason properly
  • Fixed: The value in session.if(…) may now be dynamic. It had to be a static value before.

2018-Q2

  • NOTE We changed the URL schema for many resources. Previous links should continue to work. The new URLs are more consistent and are easier to share.
  • Added session.check(), session.abort() and session.assert(): Session Checks (OK/KO criteria) documentation for more details
  • Added non-functional requirements (NFR) check feature, see our Non-functional Requirements documentation
  • Added visualiation of test cluster utilization to reports
  • Enabled ability to (soft) delete file fixtures
  • CLI: Refactored all commands to work with organisation/test-case/test-run-id as references, see https://github.com/stormforger/cli#usage for usage information

2018-Q1

  • Added converter feature for legacy file fixtures to help with migration to our new data source API, see File Fixture Migration Guide
  • Added average session duration to report (only clients that finished)
  • Added sharable reports: You can now create a read-only share link for a test run report
  • Added scheduled execution of test runs
  • Added session.lastHttpStatus() to get the last HTTP status code
  • Deprecated session.matchedValue() (use session.getVar() now)
  • Various releases of our StormForger Command Line Client (including support to manage test cases, start/stop/monitor test runs), check out https://github.com/stormforger/cli/releases for details

2017-Q4

  • Reworked test case data source API and management of file fixtures, see https://docs.stormforger.com/reference/data-sources/ for details and https://docs.stormforger.com/reference/data-sources/#migration for information on how to migrate your existing test cases
  • Various releases of our StormForger Command Line Client (including support to manage test cases, start/stop/monitor test runs), check out https://github.com/stormforger/cli/releases for details
  • HAR Converter: Improved handling of HTTP2 pseudo headers (they get filtered out now)
  • Test Run Reports: Improved loading speed of data for charting
  • Fixed an issue that could cause a test run analysis to take longer then necessary
  • Fixed an issue with definition.setSessionWeights: In case you define weights, sessions not explicitly listed now get a weight of 0 instead of 1
  • Fixed an issue with the test case outliner: Random strings were truncated after 33 characters. This had no impact on test executions.

2017-Q3

2017-Q2

  • Added more AWS regions from which you can run tests: https://docs.stormforger.com/reference/test-cluster/#cluster-region
  • Fixed issue with wait steps in loops
  • Fixed validation for data source names (currently no dashes allowed)
  • Fixed a rare case where the organisation management page would cause an error
  • Improved detection of problems for newly launched test runs
  • Improved automatic tag generation for HAR converter

2017-Q1

2016-Q4

2016-Q3

2016-Q2

  • Significantly speed up Test Run analysis
  • Improved StormForger API and made Closed-Beta™ version available
  • new and highly improved search for documentation
  • added basic support for TLS client certificates

2016-Q1

  • Added a bulk of new content to our documentation at https://docs.stormforger.com
  • Added limit of concurrent tests per target host (can be raised on request)
  • Test cases and runs can now be archived
  • Improved UI for launching and running test runs
  • Various table views throughout the app can now be sorted and filtered
  • Significantly improved charting performance for longer test runs
  • Fixed issue with URL encoding of payloads containing data sources

2015-Q4

  • Fixed issue with test runs getting sometimes stuck in "analyzing" state
  • Improved feedback on various sessions and file fixture related errors
  • Fixed error reporting of incorrect probabilities of sessions
  • Introducing our first, consolidated documentation at https://docs.stormforger.com
  • Improved test cluster resource allocation
  • Improved test case editor shortcuts
  • Reworked life validation and error messages for test case editing
  • Fixed issue with payloads getting ignored for PUT and PATCH requests
  • Reworked UX and UI around active test runs and test cases
  • Fixed an issue concerning: under certain conditions traffic metrics were not calculated correctly
  • Open sourced a rough version of an HAR to StormForger JavaScript DSL Test Case Definition converter.

2015-Q3

  • Improved file fixture management
  • Added a simple counter for matching response contents (useful for debugging)
  • Added reporting of non-HTTP errors and events
  • Added our current Terms of Service
  • Added Intercom as our Chat and Support solution to assist customers even better
  • Improved test run report stacked graphs

2015-Q2

  • Major UI rework and improvements
  • Fixed an issue with data source usage when using control structures
  • Introducing "Preflight Mode" to test and debug test cases
  • Starting with Open Beta! WOOHOOO \o/
Icon Support Are you stuck? Talk to us! We're humans.
Icon Schedule a demo Schedule a personal, customized demo. We'll show you around and introduce you to StormForger.
Icon Talk to a human To build and run reliable applications is complex – we know. Schedule a call and we’ll figure things out.

We are using cookies to give you the best online experience. If you continue to use this site, you agree to our use of cookies. By declining we will disable all but strictly required cookies. Please see our privacy policy for more details.


Accept Decline