Improving usability and documentation

* Making the 'mode' argument mandatory
* Removing ipython option and dependency
* Rewording documentation
* Version bump to 3.1.0a15

README.md

@@ -4,19 +4,19 @@
 
 TLS Canary is a [TLS/SSL](https://en.wikipedia.org/wiki/Transport_Layer_Security) testing framework for the
 [Mozilla Firefox](https://www.mozilla.org/firefox) web browser. It is used by developers to run regression and
-performance tests against a large number of live HTTPS-enabled hosts on the Internet.
+performance tests against a large number of HTTPS-enabled hosts on the Internet.
 
 Results of the regression scans are published in HTML format here:
 * http://tlscanary.mozilla.org
 
 ## This project
 * Downloads a test build and a base build of Firefox for comparison.
-* Automatically runs thousands of secure sites on those builds.
+* Automatically queries thousands of secure sites with those builds.
-* Diffs the results and presents potential regressions in an HTML page for further diagnosis.
+* Diffs the results and presents potential regressions in an report for further diagnosis.
 * Does performance regression testing.
 * Extracts SSL state information.
-* Can maintain an updated list of TLS-enabled top sites.
+* Can maintain an updated list of HTTPS-enabled top sites.
-* Requires a highly reliable network link. **WiFi will not do.**
+* Requires a highly reliable network link. **WiFi will give unstable results.**
 
 ## Requirements
 * Python 2.7
@@ -25,16 +25,15 @@ Results of the regression scans are published in HTML format here:
 * OpenSSL-dev
 * libffi-dev
 
-### Dependencies for Debian/Ubuntu users
+### Dependencies for Debian and Ubuntu users
-Assuming that you want to run TLS Canary on a regular graphical desktop machine, these are the packages that
+Assuming that you run TLS Canary on a regular graphical desktop machine, these are the packages it requires:
-you require:
 ```
 sudo apt-get install python python-dev gcc golang-go p7zip-full libssl-dev libffi-dev
 ```
 
 The script [linux_bootstrap.sh](bootstrap/linux_bootstrap.sh) provides bootstrapping for a headless Ubuntu-based EC2
-instance. This requires the installation of a few GUI libraries required to run Firefox that are already available
+instance which requires installation of a few standard GUI libraries for running Firefox.
-on a regular desktop machine. The script may or may not work for your other favourite Debian-based distribution.
+The script may or may not work for your other favourite Debian-based distribution.
 
 ### Dependencies for Mac users
 Assuming that your're using [Homebrew](https://brew.sh/) for package management, this should set you up:
@@ -53,18 +52,21 @@ choco install 7zip.commandline git golang openssh python2
 ```
 
 ## For end users
-TLS Canary can be installed directly as a stable package from PyPI and as experimental package directly from GitHub.
+TLS Canary can be installed as a stable package from PyPI and as experimental package directly from GitHub.
 The following command will install the latest stable release of TLS Canary to your current Python environment:
 ```
-pip install --upgrade tlscanary
+pip install [--user] --upgrade tlscanary
 ```
 
-If you require the bleeding-edge developer version with the latest features and added instability, you can run
+Whether or not you require the `--user` flag depends on how your Python environment is set up. Most Linux distributions
+require it when not installing Python packages as root.
+
+If you prefer the bleeding-edge developer version with the latest features and added instability, you can run
 ```
-pip install --upgrade git+git://github.com/mozilla/tls-canary.git
+pip install [--user] --upgrade git+git://github.com/mozilla/tls-canary.git
 ```
 
-After that the `tlscanary` binary will be available in your Python environment:
+Once it finishes the `tlscanary` binary is available in your Python environment:
 ```
 tlscanary --help
 ```
@@ -72,7 +74,7 @@ tlscanary --help
 ## Usage examples
 ```bash
 # Run a quick regression test against the first 50000 hosts in the default `top` database
-tlscanary -r /tmp/reports -l 50000
+tlscanary -r /tmp/reports -l 50000 regression
 
 # Compile a fresh 'top 1000' host database called `mini`
 tlscanary -s mini -l 1000 -x1 srcupdate
@@ -81,7 +83,7 @@ tlscanary -s mini -l 1000 -x1 srcupdate
 tlscanary -s list
 
 # Use your fresh `mini` database for a quick regession test and see lots of things happening
-tlscanary -s mini -r /tmp/report --debug
+tlscanary -s mini -r /tmp/report regression --debug
 ```
 
 Please refer to the complete argument and mode references below.
@@ -92,10 +94,11 @@ For development you will additionally need to install:
 * git
 * virtualenv (highly recommended)
 
-*git* can be installed with your favourite package manager. *virtualenv* comes with a simple `pip install virtualenv`.
+*git* can be installed with your favourite package manager. *virtualenv* comes with a simple
+`pip install [--user] virtualenv`.
 
 ### Developing on Linux or Mac
-These are the commands that should set you up for development work:
+These are the commands that set you up for TLS Canary development work:
 ```
 git clone https://github.com/mozilla/tls-canary
 cd tls-canary
@@ -104,12 +107,12 @@ source venv/bin/activate
 pip install -e .[dev]
 ```
 
-The latter command should be used regularly to install new Python dependencies that a pulled update may require.
+The latter command should be used regularly to install new Python dependencies that a TLS Canary update might require.
 
 ### Developing on Windows
 Developing TLS Canary on Windows is not something we practice regularly. If you encounter quirks along the way,
 please do not hesitate to open an issue here on GitHub. The following commands, executed in a PowerShell session
-with user privileges, should set you up for development:
+with user privileges, should set you up for TLS Canary development:
 ```
 git clone https://github.com/mozilla/tls-canary
 cd tls-canary
@@ -129,14 +132,14 @@ They are largely equivalent, but the former takes care of missing test dependenc
 directly offers more control.
 
 ### Installing the pre-commit hook for git
-There's a pre-commit hook for git that you can use for automated checking for
+There's a pre-commit hook for git that you can use for automated [PEP 8](https://www.python.org/dev/peps/pep-0008/)
-[PEP 8](https://www.python.org/dev/peps/pep-0008/) violations. You can install it by running
+violations checking. You can install it by running
 ```
 ln -sf ../../hooks/pre-commit .git/hooks/
 ```
 in the top-level project directory. By using a symbolic link, you will automatically get updates once the hook
 in the repo changes. This is highly recommended. You can also copy the script manually, but then you have to
-tkae care of updates yourself.
+take care of updates yourself.
 
 ### Command line arguments
 Argument | Choices / **default** | Description
@@ -145,7 +148,6 @@ Argument | Choices / **default** | Description
 -d --debug | | Enable verbose debug logging to the terminal
 -f --filter | 0, **1** | The default filter level 1 removes network timeouts from reports which may appear spuriously. Filter level 0 applies no filtering.
 -h --help | | Longer usage information
--i --ipython | | Drop into an IPython shell after a run
 -j --parallel | 4 | Number of parallel firefox worker instances the host set will be distributed among
 -l --limit | | The number of hosts in the test set is limited to the given number. The default is to scan all the hosts in the set.
 -m --timeout | 10 | Request timeout in seconds. Running more requests in parallel increases network latency and results in more timeouts.
@@ -156,15 +158,15 @@ Argument | Choices / **default** | Description
 -t --test | release, **nightly**, beta, aurora, esr | Specify the main test candidate. Used by every run mode.
 -w --workdir | **~/.tlscanary** | Directory where cached files and other state is stored
 -x --scans | 3 | Number of scans to run against each host during performance mode. Currently limited to 20.
-MODE | **performance**, regression, scan, srcupdate | Test mode to run, given as positional parameter
+MODE | performance, regression, scan, srcupdate | Test mode to run, given as positional parameter. This is a mandatory argument.
 
 ### Test modes
 
-Test modes are specified via the positional `mode` parameter.
+Test modes are specified via the mandatory positional `mode` parameter.
 
 Mode | Description
 -----|-----
 performance | Runs a performance analysis against the hosts in the test set. Use `--scans` to specify how often each host is tested.
 regression | Runs a TLS regression test, comparing the 'test' candidate against the 'baseline' candidate. Only reports errors that are new to the test candiate. No error generated by baseline can make it to the report.
 scan | This mode only collects connection state information for every host in the test set.
-srcupdate | Compile a fresh set of TLS-enabled 'top' sites from the *Umbrella Top 1M* list. Use `-l` to override the default target size of 500k hosts. Use `-x` to adjust the number of passes for errors. Use `-x1` for a factor two speed improvement with slightly less stable results. Use `-b` to change the Firefox version used for filtering. You can use `-s` to create a new database, but you can't make it the default.
+srcupdate | Compile a fresh set of TLS-enabled 'top' sites from the *Umbrella Top 1M* list. Use `-l` to override the default target size of 500k hosts. Use `-x` to adjust the number of passes for errors. Use `-x1` for a factor two speed improvement with slightly less stable results. Use `-b` to change the Firefox version used for filtering. You can use `-s` to create a new database, but you can't make it the default. Databases are written to `~/.tlscanary/sources/`.

setup.py

@@ -5,12 +5,11 @@
 from setuptools import setup, find_packages
 
 PACKAGE_NAME = 'tlscanary'
-PACKAGE_VERSION = '3.1.0a14'
+PACKAGE_VERSION = '3.1.0a15'
 
 INSTALL_REQUIRES = [
     'coloredlogs',
     'cryptography',
-    'ipython',
     'worq'
 ]
 

tlscanary/main.py

@@ -55,10 +55,6 @@ def get_argparser():
                         choices=[0, 1],
                         action='store',
                         default=1)
-    parser.add_argument('-i', '--ipython',
-                        help='Drop into ipython shell after test run',
-                        action='store_true',
-                        default=False)
     parser.add_argument('-j', '--parallel',
                         help='Number of parallel worker instances (default: 4)',
                         type=int,
@@ -125,11 +121,9 @@ def get_argparser():
                         action='store',
                         default=3)
     parser.add_argument('mode',
-                        help='Test mode to run. (default: `%s`)' % modes.default_mode,
+                        help='Test mode to run (mandatory)',
                         choices=modes.all_mode_names,
-                        action='store',
+                        action='store')
-                        nargs='?',
-                        default=modes.default_mode)
     return parser
 
 
@@ -276,8 +270,4 @@ def main():
         logger.critical("\nUser interrupt. Quitting...")
         return False
 
-    if args.ipython:
-        from IPython import embed
-        embed()
-
     return True

tlscanary/modes/__init__.py

@@ -23,5 +23,3 @@ def __subclasses_of(cls):
 all_modes = dict([(mode.name, mode) for mode in __subclasses_of(basemode.BaseMode)])
 all_mode_names = all_modes.keys()
 all_mode_names.sort()
-default_mode = "regression"  # name of the default mode as listed in <mode class>.name
-assert default_mode in all_mode_names