microsoft
/
ghcrawler
зеркало из https://github.com/microsoft/ghcrawler.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
433 коммитов 15 Ветки 0 Теги 1,7 MiB
Граф коммитов

433 Коммитов

Автор SHA1 Сообщение Дата
Darío Hereñú 1c152900a2
Fix formatting on paragraph #68 2018-08-12 12:35:51 -03:00
Gene Hazan 56c42362cd Remove unnecessary import 2018-07-31 10:30:08 -07:00
Gene Hazan f1ed8ae8f5 Upgrade dependencies 2018-07-31 10:10:50 -07:00
Gene Hazan d1aa8444c1
Merge pull request #137 from grooverdan/docker_ports 
Docker ports - use expose and limit 'ports' to 127.0.0.1
 2018-07-19 09:41:05 -07:00
Gene Hazan 06d851502d
Merge pull request #136 from grooverdan/docker_volumes 
docker-compose: include persistent volumes for redis, rabbitmq, mongo…
 2018-07-19 09:39:00 -07:00
Gene Hazan 6e29c2b2f9
Merge pull request #140 from Microsoft/gh/dockerhub 
Upgrade Node version and add labels in Dockerfile
 2018-07-19 09:30:46 -07:00
Gene Hazan bdea480a89 Upgrade Node version and add labels in Dockerfile 2018-07-19 09:02:15 -07:00
Jeff McAffer 71ebba525b
Merge pull request #135 from grooverdan/index_mongodb 
Taking your work for it that this approach makes sense. we do not have any significant Mongo expertise.
 2018-07-15 21:40:52 -07:00
Jeff McAffer 228149fdc4
Merge pull request #131 from gauntface/google-cloud-storage 
Looking great. Sorry for the delay. I think this came in while I was on vacation. Merging now.
 2018-06-25 20:01:50 -07:00
Matt Gaunt bc4260f9d4 Adding support for google cloud storage as a storage provider 2018-06-13 14:02:09 -07:00
Jeff McAffer eb967a3d5c
Merge pull request #138 from kant/patch-1 
Thanks @kant. Related, I'm interested to know about your ghcrawler scenarios.
 2018-06-10 12:55:08 -07:00
Darío Hereñú ca58ff63a2
Typo on #08 - plus some minor formatting proposals 2018-06-09 12:54:55 -03:00
Daniel Black 37dafb45d2 docker-compose: minimize network expose 
Use 'expose' for internal only services.

Restrict external services to 127.0.0.1 by default
as someone running this on a public service will
have them all exposed. Rabbitmq management has some
default credentials. Crawler could be reconfigured to trigger
githubs throttling service.
 2018-06-10 00:17:07 +10:00
Daniel Black 6a74d904c9 docker-compose: include persistent volumes for redis, rabbitmq, mongodb, metabase 2018-06-09 22:14:20 +10:00
Daniel Black 24aa211a83 MongoDB - index commit _metadata.url and _metadata.links.self.href 
Frequent access via these fields can lead to slow and cpu intensive
access particularly on large repositories.

Creating the index with the access methods may seem odd however its
a noop if the index occurs according to Stack Overflow and mongo
documentation. It also help migration.
 2018-06-09 22:05:04 +10:00
Gene Hazan cd0c3abe4c Upgrade vulnerable moment dependency 2018-03-26 10:56:25 -07:00
Gene Hazan 4900d1c9f7 Correct winston-azure-application-insights breaking version 2018-03-26 10:33:30 -07:00
Gene Hazan 39631b184c
Merge pull request #130 from Microsoft/gh/msdocs 
Do not filter repos with names that start with OfficeDocs
 2018-03-26 09:39:40 -07:00
Gene Hazan d99cc9ac92 Do not filter repos which names start with OfficeDocs 2018-03-23 21:15:57 -07:00
Gene Hazan df03f7fa97
Update README.md 
Add Node version
 2018-03-13 21:45:06 -07:00
Gene Hazan fc1053d521
Merge pull request #128 from Microsoft/gh/repoFiltering 
Remove /azure-docs/ filtering
 2018-03-13 11:18:29 -07:00
Gene Hazan c85cf71e18 Remove /azure-docs/ filtering 2018-03-12 16:07:45 -07:00
Gene Hazan c6ac137b4f
Merge pull request #119 from Microsoft/gh/amqpPersistence 
Persist messages when sending to AMQP queue
 2017-11-27 15:32:10 +00:00
Gene Hazan 1fc8f0fd0c Persist messages when sending to AMQP queue 2017-11-22 15:03:45 -08:00
Gene Hazan cf61ceed03 Queue issue requests for pull request events 2017-11-22 09:58:57 -08:00
Gene Hazan 00927515ac Revert PullRequestEvent changes for now 2017-11-20 09:29:09 -08:00
Gene Hazan 24d5a1609f
Merge pull request #118 from Microsoft/gh/prEvent 
Queue issue in addition to PR when PR event is received
 2017-11-18 01:09:52 +00:00
Gene Hazan 6a6ddf4569 Queue issue in addition to PR when PR event is received 2017-11-17 16:21:32 -08:00
Gene Hazan f69c199792
Merge pull request #116 from Microsoft/gh/issuesEvent 
Process all issues events, not just for actions opened, closed & reop…
 2017-11-16 23:16:59 +00:00
Gene Hazan 0456cfaf16 Process all pull_request events 2017-11-16 14:45:12 -08:00
Gene Hazan ec34a75e8b Process all issues events, not just for actions opened, closed & reopened. 2017-11-16 14:15:49 -08:00
Jeff McAffer 40474922eb Merge pull request #113 from stuartlangridge/develop 
Fix the config/tokens endpoint
 2017-09-08 12:27:01 -07:00
Stuart Langridge d90f508f20 Shorten token details available from the REST API, as per PR #113 discussion 2017-09-08 18:59:22 +01:00
Stuart Langridge fb077863fd Fix the config/tokens endpoint, which does not work from bin/cc as it currently stands. 
bin/cc sends a list of tokens as a simple JSON string, colon-separated. This gets rejected by strict body-parser JSON parsing (which expects JSON to be an array or an object).
Add a GET verb to the /config/tokens endpoint to list the tokens that have already been added.
 2017-09-08 15:19:12 +01:00
Gene Hazan 9c193ce6a6 Improve regex for _testRegex 2017-08-22 08:38:22 -07:00
Gene Hazan cbacbb0ec0 Correct regex 2017-08-16 08:27:12 -07:00
Gene Hazan 8ee2964496 Filter out traffic requests 2017-08-14 14:49:41 -07:00
Gene Hazan db5caa4edc Merge pull request #111 from Microsoft/gh/extraDeltaStore 
Add pluggable delta store
 2017-08-04 14:01:14 -07:00
Gene Hazan 41fe68dd6f Add pluggable delta store 2017-08-04 13:51:59 -07:00
Gene Hazan 9671af623b Correct traffic migration script for paths & referrers 2017-07-17 08:59:01 -07:00
Gene Hazan 20aed8061f Merge pull request #110 from Microsoft/gh/trafficMigration 
Add onetimeMigrateTrafficData job
 2017-07-11 16:42:56 -07:00
Gene Hazan 95d698a288 Add onetimeMigrateTrafficData job 2017-07-11 16:24:32 -07:00
Gene Hazan 5249db7c74 Merge pull request #108 from Microsoft/gh/excludeRepos 
Correct regex. Add repos to regex.
 2017-07-05 14:54:07 -07:00
Gene Hazan b05dd003b0 Add comment 2017-07-05 14:51:21 -07:00
Gene Hazan a9bd66cedc Correct regex. Add repos to regex. 2017-07-05 14:14:16 -07:00
Gene Hazan e158979ac5 Merge pull request #105 from Microsoft/gh/supportLookupByUrn 
Gh/support lookup by urn
 2017-06-28 10:09:58 -07:00
Gene Hazan 685e6f1d16 Correct inmemory store 2017-06-28 09:40:30 -07:00
Gene Hazan f195272f98 Code review 2017-06-27 17:09:28 -07:00
Gene Hazan 10fe6b5362 Correct mongodocstore 2017-06-27 14:46:40 -07:00
Gene Hazan 9172fc3325 Update mongodocstore 2017-06-27 14:37:37 -07:00