microsoft
/
ghcrawler
зеркало из https://github.com/microsoft/ghcrawler.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
201 коммит 15 Ветки 0 Теги 1,7 MiB
Граф коммитов

136 Коммитов

Автор SHA1 Сообщение Дата
Jeff McAffer 5c3fdcfcb5 add stargazers, reviews and tweak collaborators 2017-01-24 19:34:51 -08:00
Jeff McAffer 359793e686 deal with 403 and retry after better 2017-01-24 19:34:10 -08:00
Jeff McAffer 562eafb095 adjust retry-after checking 2017-01-23 06:21:34 -08:00
Jeff McAffer d9f213d2ff add regex filtering for doc repos 2017-01-23 05:40:33 -08:00
Jeff McAffer e634995758 fix handling of benched token scenario 2017-01-22 20:12:04 -08:00
Jeff McAffer f99e3a7328 increase doc metadata version 2017-01-22 08:38:56 -08:00
Jeff McAffer 2651d6bc4d more traversal robustness 2017-01-21 22:55:56 -08:00
Jeff McAffer 8a3e07fd68 revamp traversal so to use maps 2017-01-20 17:49:25 -08:00
Jeff McAffer c28fe8d153 get all issues and fix throttling wait 2017-01-19 11:00:53 -08:00
Jeff McAffer 154c1254bc remove async 2017-01-16 15:23:08 -08:00
Jeff McAffer 57b5632150 remove commented code 2017-01-16 15:22:32 -08:00
Jeff McAffer 20638810f1 add a ratelimiter to the fetcher 2017-01-16 15:21:27 -08:00
Jeff McAffer 2b2872d32a fix slice problem 2017-01-14 22:24:33 -08:00
Jeff McAffer 574ef4812f improve deadletter queuing support 2017-01-14 19:30:25 -08:00
Jeff McAffer 1a264a6edc improved error handling 2017-01-14 18:28:06 -08:00
Jeff McAffer eb1efe486e stop attempts after 404, 403 and 401 2017-01-14 18:27:50 -08:00
Jeff McAffer a9d9df10a0 cut out children requests know to be empty 2017-01-14 18:15:10 -08:00
Jeff McAffer c919576cc1 fix page() queue call args 2017-01-13 14:08:24 -08:00
Jeff McAffer c37381246c update tests and commit processing 2017-01-13 10:23:25 -08:00
Jeff McAffer ee9984a32f add mutables notion and optimize commit/comment processing 2017-01-13 09:29:53 -08:00
Jeff McAffer bb80ed725b tweaks to new policy and tests 2017-01-10 23:20:53 -08:00
Jeff McAffer e7f8f6e9e1 add filtering of seen requests 2017-01-10 23:20:30 -08:00
Jeff McAffer 07517e47d9 fix completeRequest return values 2017-01-10 23:19:29 -08:00
Jeff McAffer 883f8e03b5 move token from header to query param 2017-01-10 23:18:32 -08:00
Jeff McAffer 5cde7a3708 first cut refactor for new transitivity model 2017-01-09 21:24:02 -08:00
Jeff McAffer 1d15b5580f handle benched tokens 2017-01-06 23:34:03 -08:00
Jeff McAffer f1962b6046 fix up should* tests and deep policies. add org teams 2017-01-05 23:00:32 -08:00
Jeff McAffer 0d0160a9f9 ondemand promises property on request 2017-01-02 11:42:52 -08:00
Jeff McAffer fec1b4b215 clean up events page handling 2017-01-02 11:42:09 -08:00
Jeff McAffer 744b86c488 add fetch metrics 2017-01-02 11:41:17 -08:00
Jeff McAffer 6966e8e41c update for _emitter change 2016-12-29 16:53:09 -08:00
Jeff McAffer 1717718490 open source cleanup: copyrights, stale files, ... 2016-12-29 14:58:25 -08:00
Jeff McAffer d1495020ee fix startup and no-request bugs 2016-12-28 22:41:36 -08:00
Jeff McAffer af85298f9a remove collectPatches 2016-12-28 16:52:25 -08:00
Jeff McAffer c5b64bf2ff integrate refreshing config 2016-12-27 09:27:03 -08:00
Jeff McAffer 184328c52b improve error handling 2016-12-26 08:47:57 -08:00
Jeff McAffer bd04694a37 add events to repos 2016-12-26 08:47:46 -08:00
Jeff McAffer 004e90e682 tweak location of traffic processing 2016-12-21 15:11:30 -08:00
Jeff McAffer 1c46bd1afa Merge pull request #23 from Microsoft/magpint/addtraffic 
add traffic
 2016-12-21 15:08:21 -08:00
Jeff McAffer 0c0d776b0f update payload mechanism 2016-12-21 14:55:51 -08:00
Jeff McAffer 67b5ff59a2 fix tests 2016-12-20 23:44:35 -08:00
Maggie Pint 23056c8dc1 pr changes 2016-12-20 13:32:13 -08:00
Maggie Pint 6ed95beb71 add traffic 2016-12-19 18:02:41 -08:00
Jeff McAffer e2fa57c164 revert origin 2016-12-19 16:26:27 -08:00
Jeff McAffer 27455a7c2c change originQueue to be a named "retryQueue" 2016-12-19 13:04:46 -08:00
Jeff McAffer 0f0acd68a3 change org and team members urn 2016-12-19 10:02:41 -08:00
Jeff McAffer 044a34994d tweak event processing 2016-12-18 11:58:08 -08:00
Jeff McAffer 3024ce00b2 add scads more events, entities and tests 2016-12-18 11:12:26 -08:00
Jeff McAffer 45e30ca0dc improve options change lifecycle 2016-12-16 16:06:13 -08:00
Jeff McAffer b74f65bbb7 integrate event processing into crawler itself 2016-12-16 10:26:19 -08:00