Add a script to refresh fuzzer corpora.

This is getting to be a nuisance to do by hand.

Change-Id: If3aa7c666c4cc0bf97615564f258ff740a561766
Reviewed-on: https://boringssl-review.googlesource.com/12281
Reviewed-by: David Benjamin <davidben@google.com>

FUZZING.md

@@ -71,25 +71,13 @@ This is to prevent the fuzzer from getting stuck at a cryptographic invariant in
 
 ## TLS transcripts
 
-The `client` and `server` corpora are seeded from the test suite. The test suite has a `-fuzzer` flag which mirrors the fuzzer mode changes above and a `-deterministic` flag which removes all non-determinism on the Go side. Not all tests pass, so `ssl/test/runner/fuzzer_mode.json` contains the necessary suppressions. To run the tests against a fuzzer-mode `bssl_shim`, run:
+The `client` and `server` corpora are seeded from the test suite. The test suite has a `-fuzzer` flag which mirrors the fuzzer mode changes above and a `-deterministic` flag which removes all non-determinism on the Go side. Not all tests pass, so `ssl/test/runner/fuzzer_mode.json` contains the necessary suppressions. The `run_tests` target will pass appropriate command-line flags.
 
-```
-cd ssl/test/runner
-go test -fuzzer -deterministic -shim-config fuzzer_mode.json
-```
+There are separate corpora, `client_corpus_no_fuzzer_mode` and `server_corpus_no_fuzzer_mode`. These are transcripts for fuzzers with only `BORINGSSL_UNSAFE_DETERMINISTIC_MODE` defined. To build in this mode, pass `-DNO_FUZZER_MODE=1` into CMake. This configuration is run in the same way but without `-fuzzer` and `-shim-path` flags.
 
-For a different build directory from `build/`, pass the appropriate `-shim-path` flag. If those tests pass, record a set of transcripts with:
+If both sets of tests pass, refresh the fuzzer corpora with `refresh_ssl_corpora.sh`:
 
 ```
-go test -fuzzer -deterministic -transcript-dir /tmp/transcripts/
+cd fuzz
+./refresh_fuzzer_corpora.sh /path/to/fuzzer/mode/build /path/to/non/fuzzer/mode/build
 ```
-
-Note the suppressions file is ignored so disabled tests record transcripts too. Then merge into the existing corpora:
-
-```
-cd build/
-./fuzz/client -max_len=50000 -merge=1 ../fuzz/client_corpus /tmp/transcripts/tls/client
-./fuzz/server -max_len=50000 -merge=1 ../fuzz/server_corpus /tmp/transcripts/tls/server
-```
-
-There are separate corpora, `fuzz/client_corpus_no_fuzzer_mode` and `fuzz/server_corpus_no_fuzzer_mode`. These are transcripts for fuzzers with only `BORINGSSL_UNSAFE_DETERMINISTIC_MODE` defined. To build in this mode, pass `-DNO_FUZZER_MODE=1` into CMake. These corpora are updated the same way, but without the `-fuzzer` and `-shim-config` flags.

fuzz/refresh_ssl_corpora.sh

@@ -0,0 +1,112 @@
+#!/bin/bash
+# Copyright (c) 2016, Google Inc.
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
+# SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
+# OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+set -ex
+
+if [[ $# -ne 2 ]]; then
+  echo "Usage: $0 fuzzer_mode_build_dir no_fuzzer_mode_build_dir"
+  exit 1
+fi
+
+fuzzer_mode_build_dir=$1
+no_fuzzer_mode_build_dir=$2
+
+
+# Sanity-check the build directories.
+
+if ! grep -q '^FUZZ:' "$fuzzer_mode_build_dir/CMakeCache.txt"; then
+  echo "$fuzzer_mode_build_dir was not built with -DFUZZ=1"
+  exit 1
+fi
+
+if grep -q '^NO_FUZZER_MODE:' "$fuzzer_mode_build_dir/CMakeCache.txt"; then
+  echo "$fuzzer_mode_build_dir was built with -DNO_FUZZER_MODE=1"
+  exit 1
+fi
+
+if ! grep -q '^FUZZ:' "$no_fuzzer_mode_build_dir/CMakeCache.txt"; then
+  echo "$no_fuzzer_mode_build_dir was not built with -DFUZZ=1"
+  exit 1
+fi
+
+if ! grep -q '^NO_FUZZER_MODE:' "$no_fuzzer_mode_build_dir/CMakeCache.txt"; then
+  echo "$no_fuzzer_mode_build_dir was not built with -DNO_FUZZER_MODE=1"
+  exit 1
+fi
+
+
+# Sanity-check the current working directory.
+
+assert_directory() {
+  if [[ ! -d $1 ]]; then
+    echo "$1 not found."
+    exit 1
+  fi
+}
+
+assert_directory client_corpus
+assert_directory client_corpus_no_fuzzer_mode
+assert_directory server_corpus
+assert_directory server_corpus_no_fuzzer_mode
+
+
+# Gather new transcripts. Ignore errors in running the tests.
+
+fuzzer_mode_shim=$(readlink -f "$fuzzer_mode_build_dir/ssl/test/bssl_shim")
+no_fuzzer_mode_shim=$(readlink -f \
+    "$no_fuzzer_mode_build_dir/ssl/test/bssl_shim")
+
+fuzzer_mode_transcripts=$(mktemp -d '/tmp/boringssl-transcript.XXXXXX')
+no_fuzzer_mode_transcripts=$(mktemp -d '/tmp/boringssl-transcript.XXXXXX')
+
+echo Recording fuzzer-mode transcripts
+(cd ../ssl/test/runner/ && go test \
+    -shim-path "$fuzzer_mode_shim" \
+    -transcript-dir "$fuzzer_mode_transcripts" \
+    -fuzzer \
+    -deterministic) || true
+
+echo Recording non-fuzzer-mode transcripts
+(cd ../ssl/test/runner/ && go test \
+    -shim-path "$no_fuzzer_mode_shim" \
+    -transcript-dir "$no_fuzzer_mode_transcripts" \
+    -deterministic) || true
+
+
+# Minimize the existing corpora.
+
+minimize_corpus() {
+  local fuzzer="$1"
+  local corpus="$2"
+
+  echo "Minimizing ${corpus}"
+  mv "$corpus" "${corpus}_old"
+  mkdir "$corpus"
+  "$fuzzer" -max_len=50000 -merge=1 "$corpus" "${corpus}_old"
+  rm -Rf "${corpus}_old"
+}
+
+minimize_corpus "$fuzzer_mode_build_dir/fuzz/client" client_corpus
+minimize_corpus "$fuzzer_mode_build_dir/fuzz/server" server_corpus
+minimize_corpus "$no_fuzzer_mode_build_dir/fuzz/client" client_corpus_no_fuzzer_mode
+minimize_corpus "$no_fuzzer_mode_build_dir/fuzz/server" server_corpus_no_fuzzer_mode
+
+
+# Incorporate the new transcripts.
+
+"$fuzzer_mode_build_dir/fuzz/client" -max_len=50000 -merge=1 client_corpus "${fuzzer_mode_transcripts}/tls/client"
+"$fuzzer_mode_build_dir/fuzz/server" -max_len=50000 -merge=1 server_corpus "${fuzzer_mode_transcripts}/tls/server"
+"$no_fuzzer_mode_build_dir/fuzz/client" -max_len=50000 -merge=1 client_corpus_no_fuzzer_mode "${no_fuzzer_mode_transcripts}/tls/client"
+"$no_fuzzer_mode_build_dir/fuzz/server" -max_len=50000 -merge=1 server_corpus_no_fuzzer_mode "${no_fuzzer_mode_transcripts}/tls/server"