Граф коммитов

66 Коммитов

Автор SHA1 Сообщение Дата
Gunes Acar df50fb2a15 Fix issues raised in the review.
Make timestamp the last column again.
Add debug logs for stream types.

Add more extensive comments for POST data parsing and streams.

Return original formData without modification, if the POST
data is not parseable into key=value form.

Remove missing onload handler in the post data test.
Add test for parsing file uploads.
Add test for sending base64 encoded data.
2016-12-28 21:26:39 +01:00
Gunes Acar f4cc4ef180 Store POST request data as JSON in the database.
Incorporated some code from: https://github.com/redline13/selenium-jmeter

Handle various content encodings for the POST data (encType).
This includes `text/plain`, `application/x-www-form-urlencoded` and
`multipart/form-data`.

Store "request headers from upload stream" in the database.
We add additional headers from POST body to existin headers column.

Discard OCSP (POST) requests to prevent noise.

Add tests for form submissions with different encodings by JavaScript.
Add tests for making POST requests with AJAX (jQuery) and FormData.
Add test for byte array POST data.

Handle exception when looking up the `Content-Type` response header.

Add shared/utils.js file. We can add common test functions to this file.

Fix some style inconsistencies.
2016-12-28 19:22:05 +01:00
Gunes Acar 60f3ac37ec Add instrumentation for document.cookie.
Log cookie read and writes, capture call stack.
2016-12-21 23:24:40 +01:00
Gunes Acar c75a7f7906 Use py.test tmpdir fixture instead of mkdtemp.
py.test tmpdir directory names are more convenient.
We add a property to access the tmpdir fixture from get_test_config.
2016-12-19 23:13:23 +01:00
Gunes Acar 16d67cd803 Move boilerplate get_config code to the parent test class.
Read the python modules to be tested from requirements.txt

Add tests to make sure firefox-bin directory and the Firefox
binary within exist.

Fix some PEP8 warnings in the test code.

Remove unused imports.
2016-12-18 20:52:23 +01:00
englehardt 9bb9b2e570 Merge branch 'custom_ff_bin_terminal_colors' of https://github.com/gunesacar/OpenWPM into gunesacar-custom_ff_bin_terminal_colors 2016-12-15 13:51:45 -05:00
Gunes Acar c4f33ac9f2 Don't use the system Firefox binary in the manual tests.
Log table name for DB inserts to terminal.
Update terminal colors used in the manual test to be more readable.

Switch to test directory before starting the web server.

Regenerate openwpm.xpi.
2016-12-15 17:29:02 +01:00
Gunes Acar 50a0330d51 Store timestamp for javascript instrumentation.
We use `canvas_fingerprinting.html` test page to test both
property access and function call instrumentation (logCall and logValue).
2016-12-15 14:53:59 +01:00
dreisman e0593d1462 Added test of TaskManager.browse() 2016-12-13 12:09:01 -05:00
englehardt 5d86590149 Making extension-based HTTP instrumentation default and deprecating
proxy instrumentation.

The naming of sql tables and browser params have been updated to reflect
that the extension HTTP instrumentation is preferred to the proxy. A few other
notable changes:
(1) Extension HTTP instrumentation is preferred, but still off-by-default
(2) The proxy is now off-by-default and shouldn't be used.
(3) browser_params['save_javascript'] uses the extension, proxy-based
    javascript saving is controlled with browser_params['save_javascript_proxy']
(4) The "post processing pipeline" (which was only used to parse HTTP
    cookies) has been removed and the TaskManager::close API updated.
2016-12-05 12:32:55 -05:00
englehardt 476d292127 Bugfix for incorrect naming after merge 2016-12-05 10:22:46 -05:00
englehardt 6a23b6ca06 Merge branch 'http-extension' of github.com:citp/OpenWPM into http-extension 2016-12-05 09:42:18 -05:00
englehardt e3f434f305 Fixing a few style issues in support of #111. 2016-12-05 09:41:17 -05:00
Gunes Acar af03b187d9 Fix the problems with the test page URL and asserts in the HTTP stack trace test.
DRY in the expected.py.
Fix tab-space mix up in the test page.
2016-12-05 15:28:47 +01:00
englehardt 8aa9cd6218 Merge remote-tracking branch 'origin/master' into http-extension
Conflicts:
	automation/Extension/firefox/openwpm.xpi
2016-12-02 16:29:26 -05:00
Gunes Acar c7535dee0d Store function name and eval/Function details for the JS calls.
Remove regex based call script URL parsing.
Remove redundant array access and parenthesis.
Add test for the call stack parsing.

Fixes #49.
2016-12-02 15:14:45 +01:00
englehardt 58e11527e5 Adding test for cached requests and responses 2016-11-29 21:04:39 -05:00
englehardt ddf91fd564 Bugfixes and tests of javascript saving from the extension 2016-11-29 19:20:14 -05:00
englehardt ffc64ed5a3 Fixing broken http tests from API change 2016-11-29 15:43:47 -05:00
englehardt 452ac2815b Merge branch 'master' of github.com:citp/OpenWPM into http-extension
Conflicts:
	automation/DeployBrowsers/deploy_firefox.py
	automation/Extension/firefox/index.js
	automation/default_browser_params.json
2016-11-28 12:06:48 -05:00
englehardt 6c8107f719 Refactor of current webdriver id removal tests + new test.
Added a bunch of new tests to ensure Object.defineProperty still works
as expected after our instrumentation runs and removes Selenium's
webdriver property. Other tests refactored to better handle a few
conditions.
2016-11-23 17:18:37 -05:00
englehardt 670364e42a Test should be headless 2016-11-23 14:16:09 -05:00
englehardt 3d7443cfef Moving webdriver attribute test to its own file and adding a test with option enabled and disabled 2016-11-23 13:42:13 -05:00
englehardt d9ded62d3c Remove document and navigator webdriver self-identification attributes.
The document attribute is removed with a DOMAttrModified eventListener
that removes itself after the first call. The navigator attribute is
prevented from being set by altering Object.defineProperty until
Selenium attempts to set the attribute (at which point the alteration is
reversed).
2016-11-23 13:09:19 -05:00
englehardt 54765c1c2c [WIP] Adding handling to remove selenium's self identification
Also includes automated tests. It seems the order of content script
execution is non-deterministic. Sometimes selenium will run first and
sometimes openwpm's will run first. We should be able to handle this by
monitoring the DOM for changes, but need to confirm the performance
degredation isn't too high.
2016-11-22 20:04:44 -05:00
englehardt 2ac606f6f9 Remove unused import 2016-11-21 19:25:50 -05:00
englehardt da2b5e0de2 Refactor extension configuration and make extension on by default. Close #107 2016-11-21 19:22:40 -05:00
englehardt 057ab5d325 Updating broken test from new encoding changes 2016-11-15 19:55:47 -05:00
Gunes Acar d3d59ae839 Exclude chrome callers from http stackstrace, add new tests.
Add a test for parse_http_stack_trace_str.
Add a test for http stackstrace instrumentation where we use different
 methods to inject an element to trigger a request (appendChild and
 innerHTML).
Test http stackstrace with an external script.
Combine http stackstrace tests in a single page.
2016-11-15 19:51:20 -05:00
Gunes Acar 3272842482 Add test for a page with no script initiated calls. 2016-11-15 19:50:39 -05:00
Gunes Acar 64a7660048 Fix expected import error. 2016-11-15 19:50:16 -05:00
Gunes Acar d659792766 Instrument the stackstrace for the HTTP requests.
Use a stack trace format similar to the javascript instrumentation.
Add a function to parse HTTP stack trace strings.
Add a test for the HTTP callstack instrumentation.

Conflicts:
	automation/platform_utils.py
2016-11-15 19:49:42 -05:00
englehardt fef6c32c50 [WIP] Adding extension-based HTTP instrumentation.
Includes flattened commits by @dreisman:
53ff16c37fdf6d2a47917d32bbb9d645402c5ab2
ecd719beaae979e1292d33e1b4002801d0951eb7
2a57dd985fe977dd64509e4941cde9214c7d183b

Conflicts:
	automation/DeployBrowsers/deploy_firefox.py
	automation/Extension/firefox/index.js
	automation/default_browser_params.json
2016-11-15 19:39:16 -05:00
englehardt 1f720c98f1 Moving run_custom_function command to its own test file 2016-11-14 17:20:33 -05:00
englehardt 5acb7d911e Adding tests for custom command and moving `multiprocessing` -> `multiprocess`
I chose to use `dill` instead of `pickle` for additional flexibility
in inline functions. Python's built-in `multiprocessing` library uses
`pickle`, so it was still a bottleneck. In particular functions defined
within class methods (e.g. within a test method) couldn't be pickled.
`multiprocess` is a fork of `multiprocessing` which mainly differs in
replacing `pickle` with `dill`. It's maintained by the same developers
as `dill`, so it seems like a safe replacement.

Conflicts:
	test/expected.py
2016-11-14 17:19:35 -05:00
dreisman a3734b32d5 disabling audiocontext test for travis CI 2016-10-04 15:23:11 -04:00
dreisman 6f8bdcc1e3 Modified test of save screenshot 2016-10-04 14:43:01 -04:00
dreisman 9c59781885 Forgot to add expected source for test 2016-10-04 14:28:31 -04:00
dreisman 683216f792 Added dump_page_source and tests 2016-10-04 14:25:13 -04:00
dreisman 7733912fd4 save_screenshot tests done 2016-10-04 12:20:24 -04:00
dreisman d105bfc24f Modified test expected values for audio fingerprinting. Prior to commit, was working locally but not on travis for some reason 2016-09-29 15:25:35 -04:00
englehardt 4a9f18a86b Adding Audio API instrumentation and tests 2016-09-12 11:18:22 -04:00
englehardt ae21cd26c2 Fix browse command to use PS+1, minor bugfix, and additional tests. Closes #85 Closes #78 Closes #82 2016-09-08 23:24:28 -04:00
dreisman 8aff632e59 Added test for visit_id in firefox extension, minor style/comment changes 2016-05-05 11:38:56 -04:00
dreisman ce6f266ca3 Adding test pages to fix broken tests 2016-05-04 12:24:16 -04:00
dreisman ee7503e40e Adding test_simple_commands.py 2016-05-04 12:03:12 -04:00
dreisman 4cc559fd8e Added test of simple commands and table integrity, bug fixes 2016-05-04 11:46:15 -04:00
dreisman 4b107d17c5 Fixed tests for command_sequence compatibility 2016-05-03 16:54:42 -04:00
Gunes Acar c20d9b548f Add travis.yml file to run continuous integration tests.
Install pytest for Travis tests.
Move python dependencies to requirements.txt.

Use mmh3 instead of pyhash

pyhash cannot be installed on Travis due to an outdated check
for libboost_python.so. mmh3 seems to be better maintained and
more widely used.
2016-04-23 14:55:39 +02:00
Gunes Acar 992673d82e Add timeouts to some tests. 2016-04-12 02:58:02 +02:00