diff --git a/ssl/test/PORTING.md b/ssl/test/PORTING.md new file mode 100644 index 00000000..8cd147b9 --- /dev/null +++ b/ssl/test/PORTING.md @@ -0,0 +1,106 @@ +# Porting to Other Implementations + +## Introduction + +This document provides an overview of the test runner and how to +integrate it with other stacks. So far we have it working with +BoringSSL and some incomplete integrations with NSS and OpenSSL. + +Note that supporting non-BoringSSL implementations is a work in +progress and interfaces may change in the future. Consumers should pin +to a particular revision rather than using BoringSSL’s master branch +directly. As we gain experience with other implementations, we hope to +make further improvements to portability, so please contact +davidben@google.com and ekr@rtfm.com if implementing a new shim. + + +## Integration Architecture + +The test runner integrates with the TLS stack under test through a +“shim”: a command line program which encapsulates the stack. By +default, the shim points to the BoringSSL shim in the same source +tree, but any program can be supplied via the `-shim-path` flag. The +runner opens up a server socket and provides the shim with a `-port` +argument that points to that socket. The shim always connects to the +runner as a TCP client even when acting as a TLS server. For DTLS, +there is a small framing layer that gives packet boundaries over +TCP. The shim can also pass a variety of command line arguments which +are used to configure the stack under test. These can be found at +`test_config.cc`. + + +The shim reports success by exiting with a `0` error code and failure by +reporting a non-zero error code and generally sending a textual error +value to stderr. Many of the tests expect specific error string (such +as `NO_SHARED_CIPHER`) that indicates what went wrong. + + +## Compatibility Issues + +There are a number of situations in which the runner might succeed +with some tests and not others: + +* Defects in the stack under test +* Features which haven’t yet been implemented +* Failure to implement one or more of the command line flags the runner uses with the shim +* Disagreement about the right behavior/interpretation of the spec + + +We have implemented several features which allow implementations to ease these compatibility issues. + +### Configuration File + +The runner can be supplied with a JSON configuration file which is +intended to allow for a per-stack mapping. This file currently takes +two directives: + + +* `DisabledTests`: A JSON map consisting of the pattern matching the + tests to be disabled as the key and some sort of reason why it was + disabled as the value. The key is used as a match against the test + name. The value is ignored and is just used for documentation + purposes so you can remember why you disabled a + test. `-include-disabled` overrides this filter. + +* `ErrorMap`: A JSON map from the internal errors the runner expects to + the error strings that your implementation spits out. Generally + you’ll need to map every error, but if you also provide the + ` -loose-errors` flag, then every un-mapped error just gets mapped to + the empty string and treated as if it matched every error the runner + expects. + + +The `-shim-config` flag is used to provide the config file. + + +### Unimplemented Features +If the shim encounters some request from the runner that it knows it +can’t fulfill (e.g., a command line flag that it doesn’t recognize), +then it can exit with the special code `89`. Shims are recommended to +use this exit code on unknown command-line arguments. + +The test runner interprets this as “unimplemented” and skips the +test. If run normally, this will cause the test runner to report that +the entire test suite failed. The `-allow-unimplemented` flag suppresses +this behavior and causes the test runner to ignore these tests for the +purpose of evaluating the success or failure of the test suite. + + +### Malloc Tests + +The test runner can only be used to stress malloc failure +codepaths. If passed `-malloc-test=0`, the runner will run each test +repeatedly with an incrementing `MALLOC_NUMBER_TO_FAIL` environment +variable. The shim should then replace the malloc implementation with +one which fails at the specified number of calls. If there are not +enough calls to reach the number, the shim should fail with exit code +`88`. This signals to the runner that the test has completed. + +See `crypto/test/malloc.cc` for an example malloc implementation. + + +## Example: Running Against NSS + +``` +DYLD_LIBRARY_PATH=~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/lib go test -shim-path ~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/bin/nss_bogo_shim -loose-errors -allow-unimplemented -shim-config ~/dev/nss-dev/nss-sandbox/nss/external_tests/nss_bogo_shim/config.json +``` diff --git a/ssl/test/README.md b/ssl/test/README.md index 7a46c323..7da29eb6 100644 --- a/ssl/test/README.md +++ b/ssl/test/README.md @@ -33,3 +33,6 @@ If adding a new test, these files may be a good starting point: * `test_config.h`, `test_config.cc`: the command-line flags which control the shim's behavior. * `bssl_shim.cc`: the shim binary itself. + +For porting the test suite to a different implementation see +[PORTING.md](./PORTING.md).