microsoft
/
ghcrawler
зеркало из https://github.com/microsoft/ghcrawler.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
458 коммитов 15 Ветки 0 Теги 1,7 MiB
Граф коммитов

56 Коммитов

Автор SHA1 Сообщение Дата
Gene Hazan dd76ad8b54
Events (#168) 
* Add handling of star and repository transferred events

* Upgrade dev dependencies

* Handle new organization event's renamed and deleted actions

* Upgrade dependencies
 2019-05-02 15:47:28 -07:00
Gene Hazan 0c8cf3ff5a Add comment 2019-01-09 15:40:00 -08:00
Gene Hazan 2108f9becb Resolve TypeError: Cannot read property 'id' of null for PullRequestReviewCommentEvent 2019-01-08 14:14:29 -08:00
Gene Hazan 68e986196c Handle deleted & transferred issues 2018-12-12 12:33:19 -08:00
Gene Hazan 56c42362cd Remove unnecessary import 2018-07-31 10:30:08 -07:00
Gene Hazan f1ed8ae8f5 Upgrade dependencies 2018-07-31 10:10:50 -07:00
Jeff McAffer 71ebba525b
Merge pull request #135 from grooverdan/index_mongodb 
Taking your work for it that this approach makes sense. we do not have any significant Mongo expertise.
 2018-07-15 21:40:52 -07:00
Matt Gaunt bc4260f9d4 Adding support for google cloud storage as a storage provider 2018-06-13 14:02:09 -07:00
Daniel Black 24aa211a83 MongoDB - index commit _metadata.url and _metadata.links.self.href 
Frequent access via these fields can lead to slow and cpu intensive
access particularly on large repositories.

Creating the index with the access methods may seem odd however its
a noop if the index occurs according to Stack Overflow and mongo
documentation. It also help migration.
 2018-06-09 22:05:04 +10:00
Gene Hazan 1fc8f0fd0c Persist messages when sending to AMQP queue 2017-11-22 15:03:45 -08:00
Gene Hazan cf61ceed03 Queue issue requests for pull request events 2017-11-22 09:58:57 -08:00
Gene Hazan 00927515ac Revert PullRequestEvent changes for now 2017-11-20 09:29:09 -08:00
Gene Hazan 6a6ddf4569 Queue issue in addition to PR when PR event is received 2017-11-17 16:21:32 -08:00
Gene Hazan 0456cfaf16 Process all pull_request events 2017-11-16 14:45:12 -08:00
Gene Hazan ec34a75e8b Process all issues events, not just for actions opened, closed & reopened. 2017-11-16 14:15:49 -08:00
Stuart Langridge fb077863fd Fix the config/tokens endpoint, which does not work from bin/cc as it currently stands. 
bin/cc sends a list of tokens as a simple JSON string, colon-separated. This gets rejected by strict body-parser JSON parsing (which expects JSON to be an array or an object).
Add a GET verb to the /config/tokens endpoint to list the tokens that have already been added.
 2017-09-08 15:19:12 +01:00
Gene Hazan 685e6f1d16 Correct inmemory store 2017-06-28 09:40:30 -07:00
Gene Hazan f195272f98 Code review 2017-06-27 17:09:28 -07:00
Gene Hazan 10fe6b5362 Correct mongodocstore 2017-06-27 14:46:40 -07:00
Gene Hazan 9172fc3325 Update mongodocstore 2017-06-27 14:37:37 -07:00
Gene Hazan 253f6ef73a Correct mongodocstore 2017-06-27 13:22:12 -07:00
Gene Hazan 5a0f0fb705 Support etag and get by URN 2017-06-27 11:15:37 -07:00
Gene Hazan c830f0751d Code review changes 2017-06-08 09:23:17 -07:00
Gene Hazan 2c87136362 Code review changes 2017-06-07 14:14:26 -07:00
Gene Hazan 2d15805828 Add PR commit & commit comments types. Process synchronize PullRequestEvent. 2017-06-02 14:29:31 -07:00
Jeff McAffer c45c0facf9 simplify simple token scenarios 2017-05-26 11:10:08 -07:00
Gene Hazan ef9d5327c1 Do not queue PushEvents if there is no policy. 2017-05-08 10:31:19 -07:00
Gene Hazan b194c5d2e6 Update PushEvent's commits href 2017-05-03 21:47:27 -07:00
Gene Hazan 342a08a07a Code review 2017-05-03 15:41:09 -07:00
Gene Hazan bfee4144ed Add handling of PushEvent commits. 2017-05-03 13:56:52 -07:00
Jeff McAffer ca135a9252 update guid to be deliveryId 2017-04-11 12:45:48 -07:00
Jeff McAffer 8858197a0b uniquify the request event_trigger request url 2017-04-11 10:27:08 -07:00
Gene Hazan 214ca4bd2e Added tests. Corrected url for events invisible in the timeline. Corrected MembershipEvent. 2017-04-07 16:56:02 -07:00
Gene Hazan de09c0314e Merge remote-tracking branch 'origin/develop' into jm/fixTest 2017-04-07 13:21:04 -07:00
Gene Hazan 2851389fa6 Gh/deleted entities (#74) 
* Initial changes for deleted entities support. Correct repo test.

* Add tests

* Do not queue certain events in case of deletion

* Add placeholders for handling relation changes

* Relation handling for team and member events

* Initial code review

* update deletion handling and repo test

* Correct indentation

* Process repository only if TeamEvent contains repository payload

* Correct TeamEvent & deletion tests

* Try queueing repo for MemberEvent's added action which does not contain repository in the payload

* Decrease qlimit for one time job.

* Add history. Capture removed MemberEvent.

* Added storage count script. Added token arg.

* Final corrections

* Correct tests

* Add test

* More tweaks

* Mark review_comment deleted - could be via PullRequestReviewCommentEvent

* Add OrgBlockEvent

* Code cleanup

* Remove duplicate functions

* Crawl a repo when it is first created.

* Update MemberEvent handling

* Crawl team when it is first created

* Update fetch policy for relations.

* Correct linting warnings

* Correct merge

* Correct merge

* Move deleteAt marking out to crawler

* Correct context copying

* tweak processing outcome for delete

* Carry history through
 2017-04-07 11:34:16 -07:00
Jeff McAffer bc73b4bc16 Fix remaining tests 2017-04-06 16:51:00 -07:00
Jeff McAffer defd4f0da2 fix a mess of test errors 2017-04-06 15:53:11 -07:00
Jeff McAffer 10b45402ff fix event type in etag issue 2017-04-04 21:25:39 -07:00
Jeff McAffer b145c960e2 add root request to log meta object 2017-04-04 20:18:25 -07:00
Gene Hazan 138d4158d1 Replace Redis URL & URN mapping store with Azure table 2017-03-31 08:17:08 -07:00
William Bartholomew d31e9ffa6c Fixed incorrect storage API 2017-03-15 16:52:29 -07:00
William Bartholomew 21789799a8 Add comments on deadletter-only APIs 2017-03-15 16:07:30 -07:00
William Bartholomew b1af03a946 Make stores return a consistent format for list 2017-03-15 15:40:46 -07:00
William Bartholomew 220a3eee85 Store metadata lowercase 
Azure Storage will store it in lowercase so minimize surprises by having the code read that way.
 2017-03-15 13:58:15 -07:00
William Bartholomew 59ff3a07fb Make deadletter APIs consistent in use of URN 2017-03-15 13:37:47 -07:00
William Bartholomew fff2a743c2 Make argument names consistent 2017-03-15 12:28:41 -07:00
William Bartholomew 11b3e3a013 Add MongoDB store support for deadletter refactor 
There's some oddities attempting to mimic the behavior of the Azure Blob store. I want to come back and resolve these but it's more to have CIAB running successfully.
 2017-03-14 16:16:10 -07:00
William Bartholomew 42a7429546 First pass at MongoDB APIs to support deadletter 2017-03-14 16:16:09 -07:00
Gene Hazan 380be04da3 Remove unused variable 2017-03-14 13:29:46 -07:00
Gene Hazan e5469fcfda Insert into both Redis & Azure table stores. Read only from Redis for now. 2017-03-14 11:20:33 -07:00