microsoft
/
ghcrawler
зеркало из https://github.com/microsoft/ghcrawler.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
89 коммитов 15 Ветки 0 Теги 1,7 MiB
Граф коммитов

89 Коммитов

Автор SHA1 Сообщение Дата
Jeff McAffer 59c3febe90 more changes for reprocessing 2016-11-27 14:37:26 -08:00
Jeff McAffer bf962c93e0 implement more and more dynamic options 2016-11-26 12:12:50 -08:00
Jeff McAffer 467a32455a tighten up page/collection handling 2016-11-25 21:36:11 -08:00
Jeff McAffer b6770b2de1 extend queueSet, add relations to processor 2016-11-25 18:47:47 -08:00
Jeff McAffer 5e0afca516 more tests 2016-11-24 11:58:38 -08:00
Jeff McAffer 93c6ba2a75 Add support for reprocessing and update transitivity 2016-11-24 05:46:39 -08:00
Jeff McAffer 736895a20b 0.1.9 2016-11-22 22:58:46 -08:00
Jeff McAffer 5f89c62551 try out non-exclusive priority queue popping 2016-11-22 22:57:56 -08:00
Jeff McAffer f00d5766cd 0.1.8 2016-11-20 23:52:50 -08:00
Jeff McAffer 5500f6d82c enable org configuration 2016-11-20 23:48:27 -08:00
Jeff McAffer 9c83442c27 0.1.7 2016-11-20 23:12:09 -08:00
Jeff McAffer a3df9e0f49 fix up empty context 2016-11-20 23:11:39 -08:00
Jeff McAffer 2e7e5c08e4 0.1.6 2016-11-20 22:45:58 -08:00
Jeff McAffer 5de53a5501 Improve error handling 2016-11-20 22:45:18 -08:00
Jeff McAffer 40713f46b7 0.1.5 2016-11-20 09:07:27 -08:00
Jeff McAffer 42695a0ae6 Merge pull request #11 from Microsoft/jm/productionTweaks 
Add crawler service
 2016-11-20 09:06:37 -08:00
Jeff McAffer 9177e47555 Add crawler service 2016-11-20 01:13:13 -08:00
Jeff McAffer f1d61b4570 Merge branch 'jm/delayUntil' into develop 2016-11-18 10:00:21 -08:00
Jeff McAffer e682dcc431 0.1.4 2016-11-18 09:57:48 -08:00
William Bartholomew cf16eeb35f Merge pull request #10 from Microsoft/jm/delayUntil 
update queue structure
 2016-11-18 09:46:08 -08:00
Jeff McAffer 085f581572 more tests and a header setting fix 2016-11-17 19:55:12 -08:00
Jeff McAffer 275052413c make the crawl loop more resilient 2016-11-17 17:03:11 -08:00
Jeff McAffer 5c39058887 collapse queues into a queueset 2016-11-17 15:33:44 -08:00
Jeff McAffer 9a1546439e fixup delayUntil and still more tests 2016-11-15 22:21:33 -08:00
William Bartholomew 2e0d059bfe Fix tests 2016-11-15 17:19:34 -08:00
William Bartholomew 4d71502292 Fix bug in promise handling 2016-11-15 17:05:20 -08:00
William Bartholomew 1c044cedb5 Fix logging 2016-11-15 17:02:52 -08:00
Jeff McAffer 844424f8f9 logging attempts 2016-11-15 16:36:58 -08:00
Jeff McAffer a73b632712 Merge branch 'develop' of https://github.com/Microsoft/ghcrawler into develop 2016-11-15 14:23:41 -08:00
Jeff McAffer 9721b94b8c add 403 and throttling handling 2016-11-15 14:23:28 -08:00
William Bartholomew dbe9ac4e13 Logging 2016-11-15 14:11:58 -08:00
William Bartholomew 7e26cfe971 Fix bugs 2016-11-14 16:17:58 -08:00
Jeff McAffer 4deb1a0e92 Merge pull request #9 from Microsoft/jm/executionFixes 
object walking and queuing fixes
 2016-11-14 14:23:11 -08:00
Jeff McAffer 52d247246b add schema version to docs 2016-11-14 09:15:08 -08:00
Jeff McAffer a9817362ae move process() to Processor and more tests! 2016-11-13 23:38:43 -08:00
Jeff McAffer 05443509c6 Add end to end tests, sinon and istanbul 2016-11-13 21:19:27 -08:00
Jeff McAffer f720584d90 Add istanbul code coverage 2016-11-13 15:34:50 -08:00
Jeff McAffer afa27e3ce4 more tests and fix up try() and failure to return test promises 2016-11-13 15:10:45 -08:00
Jeff McAffer 630ffb6997 convert to Q.try() and add more requeue tests 2016-11-13 12:25:25 -08:00
Jeff McAffer a5be766b91 more tests and a bug fix in requeue 2016-11-12 22:34:48 -08:00
Jeff McAffer 18599eb7fa improve error and promise handling 2016-11-12 18:36:14 -08:00
Jeff McAffer 1924857ebf Add more tests 2016-11-12 13:06:06 -08:00
Jeff McAffer 98e3a64378 add crawler tests 2016-11-12 12:14:24 -08:00
Jeff McAffer 7b903d5ca6 fix event tests 2016-11-12 09:46:00 -08:00
Jeff McAffer 6c01a9f49d Add promise completion 2016-11-12 08:46:31 -08:00
Jeff McAffer c6c5752648 remove need for request.page property 2016-11-12 08:32:33 -08:00
Jeff McAffer 7f7c76b3d4 object walking and queuing fixes 2016-11-12 00:13:04 -08:00
William Bartholomew 5ed3e0d868 Pair programming 2016-11-11 16:46:31 -08:00
Jeff McAffer 3e50460fd0 add better error management 2016-11-11 14:06:10 -08:00
William Bartholomew 34da424d70 Add missing dependency 2016-11-11 11:21:57 -08:00