OpenWPM/.gitignore

# Docker volume
docker-volume/

# pytest
.pytest_cache/

# mypy
.mypy_cache

# firefox directories
firefox-bin/
Firefox*.app/
Nightly*.app/
firefox-profile

# VIM tmp files
*~
.*.sw*

# A bug in selenium creates this on unix systems
C:\\nppdf32Log\\debuglog.txt

# PyCharm
.idea/*
*.idea
*/idea
.Trash-0

#VSCode
.vscode

# Local Python virtual environments
venv/

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]

# C extensions
*.so

# Distribution / packaging
.Python
env/
bin/
build/
develop-eggs/
dist/
eggs/
#lib/
#lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.cache
nosetests.xml
coverage.xml

# Translations
*.mo

# Mr Developer
.mr.developer.cfg
.project
.pydevproject

# Rope
.ropeproject

# Django stuff:
*.log
*.pot

# Sphinx documentation
docs/_build/
docs/apidoc/

# npm packages
node_modules

# built extension artifacts
openwpm/Extension/firefox/dist
openwpm/Extension/firefox/openwpm.xpi
openwpm/Extension/firefox/src/content.js
openwpm/Extension/firefox/src/feature.js

datadir
Move Dockerfile to root directory and remove unnecessary files 2018-07-20 01:42:13 +03:00			`# Docker volume`
			`docker-volume/`

Ignore pytest cache directory 2018-06-11 21:19:14 +03:00			`# pytest`
			`.pytest_cache/`

Added callback to constructor of CommandSequence 2020-02-25 13:35:19 +03:00			`# mypy`
			`.mypy_cache`

initial commit of frozen selenium/firefox versions 2015-04-15 01:59:56 +03:00			`# firefox directories`
Updating gitignore to match firefox directory rename 2015-12-21 22:02:24 +03:00			`firefox-bin/`
Ignore all downloaded Firefox mac app directories 2018-10-02 12:03:57 +03:00			`Firefox*.app/`
Update macos install script 2019-04-08 17:38:56 +03:00			`Nightly*.app/`
Initial Draft for saving callstacks 2020-01-10 16:59:37 +03:00			`firefox-profile`
initial commit of frozen selenium/firefox versions 2015-04-15 01:59:56 +03:00
0.1.0 initial public release 2014-06-10 21:41:03 +04:00			`# VIM tmp files`
Initial commit of socket interface. Tested and working with proxy 2014-03-31 10:19:27 +04:00			`*~`
			`..sw`

			`# A bug in selenium creates this on unix systems`
			`C:\\nppdf32Log\\debuglog.txt`

0.1.0 initial public release 2014-06-10 21:41:03 +04:00			`# PyCharm`
updated to write crawl history to DB 2014-04-12 00:00:42 +04:00			`.idea/*`
			`*.idea`
			`*/idea`
Fixed s3 test after removing * 2020-02-28 17:54:38 +03:00			`.Trash-0`
Add .dockerignore, somewhat synchronized with gitignore but dedicated to ignoring the largest files and directories that may by lying around in a developer's openwpm folder 2019-06-25 14:22:03 +03:00
added .vscode to .gitignore 2019-10-29 19:24:48 +03:00			`#VSCode`
			`.vscode`

Add .dockerignore, somewhat synchronized with gitignore but dedicated to ignoring the largest files and directories that may by lying around in a developer's openwpm folder 2019-06-25 14:22:03 +03:00			`# Local Python virtual environments`
Attempts at getting tests to run on Mac, both locally and via Docker for Mac and XQuartz 2018-09-17 17:39:48 +03:00			`venv/`
0.1.0 initial public release 2014-06-10 21:41:03 +04:00
Initial commit 2014-06-10 20:59:28 +04:00			`# Byte-compiled / optimized / DLL files`
			`__pycache__/`
			`*.py[cod]`

			`# C extensions`
			`*.so`

			`# Distribution / packaging`
			`.Python`
			`env/`
			`bin/`
			`build/`
			`develop-eggs/`
			`dist/`
			`eggs/`
initial port of FourthParty 2015-03-17 19:44:13 +03:00			`#lib/`
			`#lib64/`
Initial commit 2014-06-10 20:59:28 +04:00			`parts/`
			`sdist/`
			`var/`
			`*.egg-info/`
			`.installed.cfg`
			`*.egg`

			`# Installer logs`
			`pip-log.txt`
			`pip-delete-this-directory.txt`

			`# Unit test / coverage reports`
			`htmlcov/`
			`.tox/`
			`.coverage`
			`.cache`
			`nosetests.xml`
			`coverage.xml`

			`# Translations`
			`*.mo`

			`# Mr Developer`
			`.mr.developer.cfg`
			`.project`
			`.pydevproject`

			`# Rope`
			`.ropeproject`

			`# Django stuff:`
			`*.log`
			`*.pot`

			`# Sphinx documentation`
			`docs/_build/`
Generate module index (#900) * Fix documentation module index Populate the module index by setting up Sphinx to automatically run sphinx-apidoc for every build. Also, move readthedocs dependencies under docs/ and make prune-environment.py automatically generate the environment-rtd.yaml file whenever we run repin.sh. * Fix black and mypy errors 2021-04-26 18:05:30 +03:00			`docs/apidoc/`
Attempts at getting tests to run on Mac, both locally and via Docker for Mac and XQuartz 2018-09-17 17:39:48 +03:00
			`# npm packages`
			`node_modules`
Don't track built extension in git 2019-04-15 18:29:35 +03:00
Add .dockerignore, somewhat synchronized with gitignore but dedicated to ignoring the largest files and directories that may by lying around in a developer's openwpm folder 2019-06-25 14:22:03 +03:00			`# built extension artifacts`
Renaming automation module to openwpm (#793) 2020-11-14 18:06:51 +03:00			`openwpm/Extension/firefox/dist`
			`openwpm/Extension/firefox/openwpm.xpi`
			`openwpm/Extension/firefox/src/content.js`
			`openwpm/Extension/firefox/src/feature.js`
Data Aggregator Rewrite (#753) * First steps in the rewrite * Fixed import paths * One giant refactor * Fixing tests * Adding mypy * Removed mypy from pre-commit workflow * First draft on DataAggregator * Wrote a DataAggregator that starts and shuts down * Created tests and added more empty types * Got demo.py working * Created sql_provider * Cleaned up imports in TaskManager * Added async * Fixed minor bugs * First steps at porting arrow * Introduced TableName and different Task handling * Added more failing tests * First first completes others don't * It works * Started working on arrow_provider * Implemented ArrowProvider * Added logger fixture * Fixed test_storage_controller * Fixing OpenWPMTest.visit() * Moved test/storage_providers to test/storage * Fixing up tests * Moved automation to openwpm * Readded datadir to .gitignore * Ran repin.sh * Fixed formatting * Let's see if this works * Fixed imports * Got arrow_memory_provider working * Starting to rewrite tests * Setting up fixtures * Attempting to fix all the tests * Still fixing tests * Broken content saving * Added node * Fixed screenshot tests * Fixing more tests * Fixed tests * Implemented local_storage.py * Cleaned up flush_cache * Fixing more tests * Wrote test for LocalArrowProvider * Introduced tests for local_storage_provider.py * Asserting test dir is empty * Creating subfolder for different aggregators * New depencies and init() * Everything is terribly broken * Figured out finalize_visit_id * Running two event loops kinda works??? * Rearming the event * Introduced mypy * Downgraded black in pre-commit * Modifying the database directly * Fixed formatting * Made mypy a lil stricter * Fixing docs and config printing * Realising I've been using the wrong with * Trying to figure arrow_storage * Moving lock initialization in in_memory_storage * Fixing tests * Fixing up tests and adding more typechecking * Fixed num_browsers in test_cache_hits_recorded * Parametrized unstructured * String fix * Added failing test * New test * Review changes with Steven * Fixed repin.sh and test_arrow_cache * Minor change * Fixed prune-environment.py * Removing references to DataAggregator * Fixed test_seed_persistance * More paths * Fixed test display shutdown * Made cache test more robust * Update crawler.py Co-authored-by: Steven Englehardt <senglehardt@mozilla.com> * Slimming down ManagerParams * Fixing more tests * Update test/storage/test_storage_controller.py Co-authored-by: Steven Englehardt <senglehardt@mozilla.com> * Purging references to DataAggregator * Reverted changes to .travis.yml * Demo.py saves locally again * Readjusting test paths * Expanded comment on initialize to reference #846 * Made token optional in finalize_visit_id * Simplified test paramtetrization * Fixed callback semantics change * Removed test_parse_http_stack_trace_str * Added DataSocket * WIP need to fix path encoding * Fixed path encoding * Added task and crawl to schema * Fixed paths in GitHub actions * Refactored completion handling * Fix tests * Trying to fix tests on CI * Removed redundant setting of tag * Removing references to S3 * Purging more DataAggregator references * Craking up logging to figure out test failure * Moved test_values into a fixture * Fixing GcpUnstructuredProvider * Fixed paths for future crawls * Renamed sqllite to official sqlite * Restored demo.py * Update openwpm/commands/profile_commands.py Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com> * Restored previous behaviour of DumpProfileCommand Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com> * Removed leftovers * Cleaned up comments * Expanded lock check * Fixed more stuff * More comment updates * Update openwpm/socket_interface.py Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com> * Removed outdated comment * Using config_encoder * Renamed tar_location to tar_path * Removed references to database_name in docs * Cleanup * Moved screenshot_path and source_dump_path to ManagerParamsInternal * Fixed imports * Fixing up comments * Fixing up comments * More docs * updated dependencies * Fixed test_task_manager * Reupgraded to python 3.9.1 * Restoring crawl_reference in mp_logger * Removed unused imports * Apply suggestions from code review Co-authored-by: Steven Englehardt <senglehardt@mozilla.com> * Cleaned up socket handling * Fixed TaskManager.__exit__ * Moved validation code into config.py * Removed comment * Removed comment * Removed comment Co-authored-by: Steven Englehardt <senglehardt@mozilla.com> Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com> 2021-02-22 19:51:32 +03:00
			`datadir`