62 Коммитов

Автор	SHA1	Сообщение	Дата
englehardt	9b41396ed9	Adding back in tblib which I accidentally removed	2015-11-08 06:38:39 +00:00
englehardt	83992a1355	Install script cleanup. mitmproxy==0.13, and adding a prompt to install flash in support of #32	2015-11-08 06:08:09 +00:00
englehardt	d186a38e0d	Save javascript in LevelDB database instead of flat files ... Flat files are fine for smaller crawls, however the structure begins to break down for long crawls. Directories with millions of small files are slow to interact with, especially on traditional hard disks. LevelDB requires a few extra dependencies. A third (unrelated) dependency is added to fix the install of Pillow.	2015-10-19 15:58:27 -07:00
englehardt	c8be9bc864	Install setuptools via pip before other dependencies. Fixes failures ... caused by an outdated version of setuptools.	2015-09-22 15:59:34 +00:00
englehardt	8240733da6	Changing javascript files to use 64 bit murmur3 hash. ... The 32 bit hash is likely to have at least a few collisions over 1 million sites. To avoid this we use the fast murmur3_x64_128 hash from pyhash and mask it to 64 bits. This requires a few additional dependencies, which are included in the install script.	2015-09-15 22:36:52 +00:00
englehardt	b207c28450	Bumping Firefox and Selenium Version	2015-09-15 18:42:08 +00:00
englehardt	4cd5f78952	adding back in mitmproxy to install script	2015-06-12 16:01:40 -04:00
englehardt	ed13495454	updating install file	2015-06-09 21:15:59 +00:00
englehardt	fe3aa49308	bugfix in install script	2015-06-09 20:08:27 +00:00
englehardt	3791d11605	upgrade selenium to 2.46.0 and firefox to 38	2015-06-08 21:32:29 +00:00
englehardt	c51db6b9f1	Adding in firefox since we need the dependencies	2015-05-04 15:07:14 -04:00
englehardt	1d6a1f6e6c	adding install script for dependencies	2015-04-14 23:57:03 +00:00