tls-canary/README.md

# TLS Canary version 3
Automated testing of Firefox for TLS/SSL web compatibility

Regression scanning results live here:
http://tlscanary.mozilla.org

## This project
* Downloads a branch build and a release build of Firefox.
* Automatically runs thousands of secure sites on those builds.
* Diffs the results and presents potential regressions in an HTML page for further diagnosis.
* Does performance regression testing
* Extracts SSL state information
* Can maintain an updated list of TLS-enabled top sites
* Requires a highly reliable network link. **WiFi will not do.**

## Requirements
* Python 2.7
* virtualenv (highly recommended)
* 7zip
* git
* Go compiler
* OpenSSL-dev
* libffi-dev

The script [linux_bootstrap.sh](linux_bootstrap.sh) provides bootstrapping for an Ubuntu-based EC2 instance.

## Linux and Mac usage
```
git clone https://github.com/mozilla/tls-canary
cd tls-canary
virtualenv .
source bin/activate
pip install -e .
tls_canary --help
tls_canary --reportdir=/tmp/test --debug debug
```

## Windows support
Windows support targets **PowerShell 5.1** on **Windows 10**. Windows 7 and 8
are generally able to run TLS Canary, but expect minor unicode
encoding issues in terminal logging output.

### Run in an admin PowerShell
First, [install Chocolatey](https://chocolatey.org/install), then
```
choco install 7zip.commandline git golang openssh python2
choco install python3  # Optional, provides the virtualenv cmdlet
pip install virtualenv  # Not required if python3 installed
```

### Run in a user PowerShell
```
git clone https://github.com/mozilla/tls-canary
cd tls-canary
virtualenv -p c:\python27\python.exe venv
venv\Scripts\activate
pip install -e .
```

### Command line arguments
Argument | Choices / **default** | Description
----------|----------|----------
-b --base | **release**, nightly, beta, aurora, esr | Baseline test candidate to test against. Only used by comparative test modes.
-d --debug | | Enable verbose debug logging to the terminal
-f --filter | 0, **1** | The default filter level 1 removes network timeouts from reports which may appear spuriously. Filter level 0 applies no filtering.
-h --help | | Longer usage information
-i --ipython | | Drop into an IPython shell after a run
-j --parallel | 4 | Number of parallel firefox worker instances the host set will be distributed among
-l --limit | | The number of hosts in the test set is limited to the given number. The default is to scan all the hosts in the set.
-m --timeout | 10 | Request timeout in seconds. Running more requests in parallel increases network latency and results in more timeouts.
-n --requestsperworker | 50 | Chunk size of hosts that a worker will query in parallel.
-o --onecrl | **production**, stage, custom | OneCRL revocation list to install to the test profiles. `custom` uses a pre-configured, static list.
-r --reportdir | **$PWD** | Directory for report output. Default is the current directory. Each report is written to a subdirectory there by date and time. Writing to TLS Canary's Python module directory is prohibited.
-s --source | **top**, list, ... | Set of hosts to run the test against. Pass `list` to get info on available test sets.
-t --test | release, **nightly**, beta, aurora, esr | Specify the main test candidate. Used by every run mode.
-w --workdir | **~/.tlscanary** | Directory where cached files and other state is stored
-x --scans | 3 | Number of scans to run against each host during performance mode. Currently limited to 20.
MODE | **performance**, regression, scan, srcupdate | Test mode to run, given as positional parameter

### Test modes

Test modes are specified via the positional `mode` parameter.

Mode | Description
-----|-----
performance | Runs a performance analysis against the hosts in the test set. Use `--scans` to specify how often each host is tested.
regression | Runs a TLS regression test, comparing the 'test' candidate against the 'baseline' candidate. Only reports errors that are new to the test candiate. No error generated by baseline can make it to the report.
scan | This mode only collects connection state information for every host in the test set.
srcupdate | Compile a fresh set of TLS-enabled 'top' sites from the *Umbrella Top 1M* list. Use `-l` to override the default target size of 500k hosts. Use `-x` to adjust the number of passes for errors. Use `-x1` for a factor two speed improvement with slightly less stable results. Use `-b` to change the Firefox version used for filtering. You can use `-s` to create a new database, but you can't make it the default.

## Testing
* nosetests -sv