Integrating in CI
While you can write one-off test cases using tlsfuzzer to test a specific issue, tlsfuzzer caters to CI environments.
Preparation
Configuration of the server has significant impact on its behaviour. For example, you can’t test ECDSA cipher suites without an ECDSA certificate available for the server.
To verify all features of a server implementation you need to enable all of them in the server. In case the features conflict with each-other, you need to test them separately by running the relevant tests with every configuration.
As a full-featured implementation has a lot of independent parameters, you can find yourself in a situation where you don’t have enough computer resources to test all combinations of parameters. In such case you may want to apply combinatorial testing, or pairwise testing, to keep the required amount of server configurations manageable.
Configuration variables
You should take into consideration the following aspects of server configuration:
How many and what types of certificates the server has set up
especially relevant for differences between
rsaEncryption
andrsassa-pss
in Subject Public Key Info
Test with SNI enabled and disabled
Testing the default host vs SNI host of a SNI-enabled server
Test with different security levels enabled in the library
for OpenSSL: cipherstring with
@SECLEVEL=3
vs@SECLEVEL=0
for GnuTLS: priority string with
NORMAL
vsSECURE256:%PROFILE_HIGH
No client certificates, requesting client certificates, or requiring client certificates
Session tickets enabled or not
For example in OpenSSL it influences the kind of tickets issued by OpenSSL in TLS 1.3
Test with support for 0-RTT enabled and disabled
for example, even with 0-RTT disabled, the server still must process 0-RTT ClientHello but ignore the early data
Enabled protocol versions (e.g. with TLS 1.2 implemented but only TLS 1.1 enabled, the server must not abort connection starting with a TLS 1.1 ClientHello with
TLS_FALLBACK_SCSV
Changes to configuration before session establishment and resumption (e.g. resuming a session with a now disabled cipher; or processing 0-RTT data with its cipher disabled)
Integrated with different applications—callbacks can change important parts of implementation behaviour
Large client or server certificates
The private key residing in an HSM or a smart card
different features supported in the smart card—things like RSA-PSS or SHA-384
Testing with specific extension or feature of the protocol disabled, enabled, or required
Server running under valgrind, compiled with ubsan, asan, etc.
Force-enabled features deprecated in later protocol versions (e.g. PKCS#1 v1.5 SHA-1 signatures enabled through configuration should not enable them in TLS 1.3)
Server running on different hardware (different assembly implementations in use, AES-NI support, SSE3, etc.)
Interactions between implementation versions and session resumptions— test what happens when a client resumes a session from old library with new server (and vice versa, to simulate server downgrade)
Running tests
Since the included tests expect strict adherence to RFCs, you can expect that executing them for the first time will find a lot of issues. As such, you should start with running them one by one, manually, inspecting test results and checking if they pass.
Many tests verify behaviour unrelated to main feature under test, indicated by the name and the summary printed at the end of execution. Such tests provide extra command line options to make them more aligned with behaviour of the tested implementation.
Warning
Some tests allow changing expected alert description for the negative tests. Before introducing any such modifcations you should have a good understanding of TLS and oracle attacks—you need to verify that similarly malformed messages result in the exact same alerts. Please note that some features (e.g. padding in CBC mode ciphers) have more than one script that tests them, so you need to adjust invocations of all relevant scripts.
If you find differences between script-expected behaviour and actually observed behaviour of the system under test, inspect the source code to determine the root cause of the issue. Commonly the implementation detects the wrong behaviour of the peer but returns a wrong alert. While technically those are compliance issues and you should fix them (see also the section describing reasons for strict alert description checking), they don’t cause interoperability or security issues, so you can postpone fixing them.
While working on a test script, you should adjust its parameters so that it
matches the server configuration.
If a script expects different behaviour, you can either disable running
the failing test case by specifying its name as a parameter to -e
option or
mark it as an “expected failure” by specifying its name as a parameter to
-x
option.
In the latter case, you can also specify a substring to match the printed
error against with the -X
option.
Using -x
ensures that resolving the bug causes the test suite to “notice”
the new behaviour.
Pairing it with -X
option ensures that the way the test fails doesn’t
change.
Scripts by default require less than 10 seconds to execute against a local
server (using a mid-range CPU from 2020).
You can use the -n
option to control how many tests to execute in a
script.
To execute all tests in a script, specify -n 0
.
Automation scripts
Tlsfuzzer ships with a few scripts to make using it in CI easier:
verify-scripts-json.py
, scripts_retention.py
, and
verify-multiple-jsons.py
.
The scripts_retention.py
one starts servers based on passed configuration
file.
To ensure reliable execution, it verifies that the server can accept
connections before running the test cases.
The script uses server_hostname
and server_port
to verify readiness
of the server to accept connections.
Tip
The server_command
can specify the server to run or a command necessary
to reconfigure the server. In latter case, it needs to run for as long
as scripts_retention.py
executes test scripts for—
scripts_retention.py
aborts testing when this command exits.
A test case marked with "exp_pass": false
needs to fail, otherwise
the script counts it as a failure.
You can find example configuration files in tests
directory:
tlslite-ng.json
and
tlslite-ng-random-subset.json.
The latter one is part of the CI for tlsfuzzer.
The verify-multiple-jsons.py
and verify-scripts-json.py
check if
specified json files reference all tests in the scripts
directory.
You should use them when migrating to new version of tlsfuzzer to verify that
you don’t skip any newly added scripts.