The platform currently uses SQL throughout all of the instrumentation.
This makes it impossible to move to provide the option of an alternate
data format. This commit moves all SQL logic into a SQLiteAggregator and
uses simple dictionaries for passing data through the rest of the
platform. The BaseAggregator and BaseListener classes can be used to
support alternate output formats.
Add [isort] to setup.cfg, per docs
Fix flake8-isort errors in /test/*
Add mapped flake8-isort to pkg_name_mapping()
Fix one isort error
See https://travis-ci.org/citp/OpenWPM/jobs/407022965
Try fixing second flake8-isort error
See https://travis-ci.org/citp/OpenWPM/jobs/407388964
Moar fixes for flake8-isort validation
See https://travis-ci.org/citp/OpenWPM/jobs/407390660
This |isort --diff| run made flake8-isort happy, locally
Add future, six, and six.moves to isort's known_future_library section
Try outputting suggested flake8-isort fixes
Address a slew of flake8-isort issues, in Travis
Another round of flake8-isort 'fixes?
Add configs to isort; fix a few flake8-isort issues
Should fix isort errors
Hopefully final isort fixes
I think I cracked it: fixed isort issues
Remove debug flake8-isort command in Travis builds
Just update known_first_party with the couple I know
Most of the HTTP Instrument tests broke due to a new column being added
in the middle of the http request and response tables. Our tests
shouldn't have such a strong dependence on column order. Let's move
query responses over to the `sqlite3.Row` datatype, which allows access
by name. This is work in progress, I expect a bunch of other tests to
fail on calls to `db_utils::query_db`.
Add a new utility module for DB related
functions. This is to be used by test and data analysis code.
Move domain_utils.py to automation/utilities.
Remove repeating PSL related code.
Read the python modules to be tested from requirements.txt
Add tests to make sure firefox-bin directory and the Firefox
binary within exist.
Fix some PEP8 warnings in the test code.
Remove unused imports.
proxy instrumentation.
The naming of sql tables and browser params have been updated to reflect
that the extension HTTP instrumentation is preferred to the proxy. A few other
notable changes:
(1) Extension HTTP instrumentation is preferred, but still off-by-default
(2) The proxy is now off-by-default and shouldn't be used.
(3) browser_params['save_javascript'] uses the extension, proxy-based
javascript saving is controlled with browser_params['save_javascript_proxy']
(4) The "post processing pipeline" (which was only used to parse HTTP
cookies) has been removed and the TaskManager::close API updated.
englehardt
Stephen Donner
Zack Weinberg
Gunes Acar