Integrating in CI

While you can write one-off test cases using tlsfuzzer to test a specific issue, tlsfuzzer caters to CI environments.

Preparation

Configuration of the server has significant impact on its behaviour. For example, you can’t test ECDSA cipher suites without an ECDSA certificate available for the server.

To verify all features of a server implementation you need to enable all of them in the server. In case the features conflict with each-other, you need to test them separately by running the relevant tests with every configuration.

As a full-featured implementation has a lot of independent parameters, you can find yourself in a situation where you don’t have enough computer resources to test all combinations of parameters. In such case you may want to apply combinatorial testing, or pairwise testing, to keep the required amount of server configurations manageable.

Configuration variables

You should take into consideration the following aspects of server configuration:

  1. How many and what types of certificates the server has set up
    • especially relevant for differences between rsaEncryption and rsassa-pss in Subject Public Key Info
  2. Test with SNI enabled and disabled
  3. Testing the default host vs SNI host of a SNI-enabled server
  4. Test with different security levels enabled in the library
    • for OpenSSL: cipherstring with @SECLEVEL=3 vs @SECLEVEL=0
    • for GnuTLS: priority string with NORMAL vs SECURE256:%PROFILE_HIGH
  5. No client certificates, requesting client certificates, or requiring client certificates
  6. Test with ALPN (or NPN) enabled and disabled
  7. Session tickets enabled or not
    • For example in OpenSSL it influences the kind of tickets issued by OpenSSL in TLS 1.3
  8. Test with support for 0-RTT enabled and disabled
    • for example, even with 0-RTT disabled, the server still must process 0-RTT ClientHello but ignore the early data
  9. Enabled protocol versions (e.g. with TLS 1.2 implemented but only TLS 1.1 enabled, the server must not abort connection starting with a TLS 1.1 ClientHello with TLS_FALLBACK_SCSV
  10. Changes to configuration before session establishment and resumption (e.g. resuming a session with a now disabled cipher; or processing 0-RTT data with its cipher disabled)
  11. Integrated with different applications—callbacks can change important parts of implementation behaviour
  12. Large client or server certificates
  13. The private key residing in an HSM or a smart card
    • different features supported in the smart card—things like RSA-PSS or SHA-384
  14. Testing with specific extension or feature of the protocol disabled, enabled, or required
  15. Server running under valgrind, compiled with ubsan, asan, etc.
  16. Force-enabled features deprecated in later protocol versions (e.g. PKCS#1 v1.5 SHA-1 signatures enabled through configuration should not enable them in TLS 1.3)
  17. Server running on different hardware (different assembly implementations in use, AES-NI support, SSE3, etc.)
  18. Interactions between implementation versions and session resumptions— test what happens when a client resumes a session from old library with new server (and vice versa, to simulate server downgrade)

Running tests

Since the included tests expect strict adherence to RFCs, you can expect that executing them for the first time will find a lot of issues. As such, you should start with running them one by one, manually, inspecting test results and checking if they pass.

Many tests verify behaviour unrelated to main feature under test, indicated by the name and the summary printed at the end of execution. Such tests provide extra command line options to make them more aligned with behaviour of the tested implementation.

Warning

Some tests allow changing expected alert description for the negative tests. Before introducing any such modifcations you should have a good understanding of TLS and oracle attacks—you need to verify that similarly malformed messages result in the exact same alerts. Please note that some features (e.g. padding in CBC mode ciphers) have more than one script that tests them, so you need to adjust invocations of all relevant scripts.

If you find differences between script-expected behaviour and actually observed behaviour of the system under test, inspect the source code to determine the root cause of the issue. Commonly the implementation detects the wrong behaviour of the peer but returns a wrong alert. While technically those are compliance issues and you should fix them (see also the section describing reasons for strict alert description checking), they don’t cause interoperability or security issues, so you can postpone fixing them.

While working on a test script, you should adjust its parameters so that it matches the server configuration. If a script expects different behaviour, you can either disable running the failing test case by specifying its name as a parameter to -e option or mark it as an “expected failure” by specifying its name as a parameter to -x option. In the latter case, you can also specify a substring to match the printed error against with the -X option. Using -x ensures that resolving the bug causes the test suite to “notice” the new behaviour. Pairing it with -X option ensures that the way the test fails doesn’t change.

Scripts by default require less than 10 seconds to execute against a local server (using a mid-range CPU from 2020). You can use the -n option to control how many tests to execute in a script. To execute all tests in a script, specify -n 0.

Automation scripts

Tlsfuzzer ships with a few scripts to make using it in CI easier: verify-scripts-json.py, scripts_retention.py, and verify-multiple-jsons.py.

The scripts_retention.py one starts servers based on passed configuration file. To ensure reliable execution, it verifies that the server can accept connections before running the test cases. The script uses server_hostname and server_port to verify readiness of the server to accept connections.

Tip

The server_command can specify the server to run or a command necessary to reconfigure the server. In latter case, it needs to run for as long as scripts_retention.py executes test scripts for— scripts_retention.py aborts testing when this command exits.

A test case marked with "exp_pass": false needs to fail, otherwise the script counts it as a failure.

You can find example configuration files in tests directory: tlslite-ng.json and tlslite-ng-random-subset.json. The latter one is part of the CI for tlsfuzzer.

The verify-multiple-jsons.py and verify-scripts-json.py check if specified json files reference all tests in the scripts directory. You should use them when migrating to new version of tlsfuzzer to verify that you don’t skip any newly added scripts.