mozilla
/
OpenWPM
зеркало из https://github.com/openwpm/OpenWPM.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
492 коммитов 18 Ветки 47 Теги 138 MiB
Граф коммитов

492 Коммитов

Автор SHA1 Сообщение Дата
englehardt 103fab33cd Preventing an update ping for OpenH264 codec 2015-10-15 17:46:49 -04:00
englehardt f047fb47e3 Adding in a number of new prefs to stop Firefox from pinging Mozilla and 
to disable unnecessary features. Close #22
 2015-10-15 17:35:37 -04:00
englehardt 4c0de642c5 Version bump 2015-10-15 15:51:17 -04:00
englehardt 0f131db375 Bugfix: JSON serialize headers, fixing cookie parsing. 
Mitmproxy changed the way it stores headers to use
netlib.odict.ODictCaseless. This type has a string representation of a
list of lists that must be eval()'ed to parse the headers. Instead,
let's use json serialization in combination with the save/load functions
provided by the ODictCaseless class. This fixes the bug that would cause
http cookie tables to be empty with newer versions of mitmproxy.
 2015-10-15 15:43:27 -04:00
englehardt 03f9b4efaa Merge pull request #21 from citp/javascript 
Merging Javascript + Extension Support to master branch
 2015-10-14 07:57:35 -07:00
englehardt 7311649dda Version 0.5.0 Bump 2015-10-14 10:54:24 -04:00
englehardt feabf48931 Adding adblock-plus to default params to fix crash 2015-10-13 15:06:22 -04:00
englehardt 3a5d02f339 Add a warning message that tracking protection doesn't work 2015-10-13 15:05:58 -04:00
englehardt 913bb11f53 Adding Ad Block Plus support 2015-10-13 14:44:13 -04:00
englehardt 3d85e812ae More flexible javascript Content-Type detection. 2015-09-29 15:39:06 -04:00
englehardt edce6d8803 Better javascript content detection 2015-09-29 15:16:16 -04:00
englehardt c8be9bc864 Install setuptools via pip before other dependencies. Fixes failures 
caused by an outdated version of setuptools.
 2015-09-22 15:59:34 +00:00
englehardt f4575044d6 Improvements to profile dumping and loading 
1. Added in success logging for profile untaring
2. Made compression optional when dumping/loading profile
3. Removed "slim profile" option, since the full profile isn't much bigger
 2015-09-21 16:01:56 +00:00
englehardt c0cea836f9 Replacing subprocess calls with python functions in profile_commands 2015-09-21 06:01:07 -07:00
englehardt 9bfc144006 Including the browser ID in every call to the logger 2015-09-21 05:51:31 -07:00
englehardt b8e4668a06 Fixing the profile commands to check if webapps and storage exist as 
directories (instead of files).
 2015-09-18 12:56:06 -07:00
englehardt 86e4c17f46 Adding logging to profile commands. 
Logging via MPLogger has been added to the profile commands, including
flash loading and saving. dump_profile has also been altered to require
cookies, history, and localStorage. If these don't exist the function
will now throw an error.
 2015-09-18 08:30:04 -07:00
englehardt 29f4137fd5 Have the browser command use the logger rather than print 2015-09-17 14:05:35 -07:00
englehardt 482e1ed2ae Reworking browser restart pipline to fix crashes and deadlocks. In 
support of #19

Rather than having multiple threads managing the same browser, each
thread will simply set a flag if the browser needs to be restarted.
Then, the main thread managing the browser will restart if necessary.
This fixes a bunch of race conditions caused by managing the process in
two separate threads. Also bumped the failure threshold as it was too
low for instances with a low number of browsers.
 2015-09-17 13:56:24 -07:00
englehardt ac82b209f1 Adding back in check for browser freshness before restarting 2015-09-16 20:12:20 -07:00
englehardt d603979e37 Stopgap simiplification of browser restarts in support of #19 2015-09-16 20:08:22 -07:00
englehardt 6e384db5a0 Bugfix: Don't clear profile when browser exceeds memory limits 2015-09-16 18:45:02 -07:00
englehardt 40d976b275 Bugfix. Browse command will not properly call get command 2015-09-16 14:23:50 -07:00
englehardt d8995ce2cc updating configure firefox description 2015-09-16 14:20:44 -07:00
englehardt 38c895eee0 Removing random extensions in support of #17 2015-09-16 14:20:06 -07:00
englehardt d56d3c095b Support for Ghostery + HTTPS-Everywhere. 
This also includes the start of a refactor of the code to launch
Firefox. The goal of this is to simply deploy_firefox.py and separate
the configuration options into logical pieces.
 2015-09-16 14:19:11 -07:00
englehardt 9c2473e133 Add error handling for zlib decompression errors when decompressing javascript from proxy 2015-09-16 01:22:27 +00:00
englehardt 8240733da6 Changing javascript files to use 64 bit murmur3 hash. 
The 32 bit hash is likely to have at least a few collisions
over 1 million sites. To avoid this we use the fast murmur3_x64_128
hash from pyhash and mask it to 64 bits. This requires a few
additional dependencies, which are included in the install script.
 2015-09-15 22:36:52 +00:00
englehardt b207c28450 Bumping Firefox and Selenium Version 2015-09-15 18:42:08 +00:00
englehardt c39e8000b8 bugfix to properly label hashes in database 2015-09-15 14:06:54 -04:00
englehardt 5dce2f35d6 Check if js file exists before compressing and writing 2015-09-15 10:41:40 -04:00
englehardt ad0bf250f5 Adding gzip compression to javascript files. 2015-09-15 10:31:12 -04:00
englehardt 048748d592 New psutil API get_process_list -> process_iter 2015-09-14 16:44:26 -04:00
englehardt c80db4a203 adding changes that should have been in previous commit 2015-09-14 16:19:07 -04:00
englehardt 54c39915aa Create data_directory if it doesn't exist + drop an init file in main 
directory to make it possible to use OpenWPM as a submodule
 2015-09-14 16:14:21 -04:00
englehardt db97598d96 Control saving of javascript with a browser pref 2015-09-14 12:12:31 -04:00
englehardt 407a03c72a Adding in a manager_params dictionary 
The goal of this change is to provide a better way of passing
per-crawl parameters. Since the browser_params dictionary is meant for
per-browser settings, adding in any additional parameters (like
logger_address) would require adding the address to all copies of the
dictionary. With manager_params, a single additional dictionary is
passed around to hold the crawl-wide configuration settings. This
includes things like the location of the crawl database and the log
file.
 2015-09-14 11:05:50 -04:00
englehardt 9eef476395 Initial support for saving raw javascript files. 2015-09-11 14:21:03 -04:00
Steven Englehardt 2caabc256f update for API changes 2015-06-26 14:04:39 -07:00
englehardt 4cd5f78952 adding back in mitmproxy to install script 2015-06-12 16:01:40 -04:00
englehardt 903bc395eb v0.4.0 Release 2015-06-12 16:00:13 -04:00
englehardt 9d69dba0f7 Process watchdog + Command failure threshold 2015-06-11 14:27:58 -04:00
englehardt dbce9f1c99 Fix encoding issues in cookie parsing 2015-06-10 16:09:19 -04:00
englehardt 3ef8276245 Adding a process watchdog to kill orphaned processes 2015-06-10 15:29:35 -04:00
englehardt 08f8d1b810 Merge branch 'stability' of https://github.com/citp/OpenWPM into stability 2015-06-09 20:48:20 -04:00
englehardt d22bc6498e Merge branch 'master' into stability 2015-06-09 20:48:00 -04:00
englehardt 09eb3cd410 prevent incorrect query sizes from crashing dataaggregator 2015-06-09 20:46:49 -04:00
englehardt bfdf11c23c Providing bibtex for citation 2015-06-09 19:21:24 -04:00
englehardt ed13495454 updating install file 2015-06-09 21:15:59 +00:00
englehardt fe3aa49308 bugfix in install script 2015-06-09 20:08:27 +00:00