Граф коммитов

2470 Коммитов

Автор SHA1 Сообщение Дата
SergeyGaluzo 2720e1aa8c
Tools changes to support bundles test (#3708)
* Removing dead code from $import

* Remove progress and rebuld indexes

* var

* No use of progress

* Removed progress again

* Removing call to heavy heartbeats

* Remove test relying on heavy heartbeats

* removed usage of heavy heartbeats

* Removed progress from IJob

* removed progress more

* tests

* removed obsolete comment

* Removed obsolete fields.

* write back to be on par

* json convert

* removed obsolete properties

* Removed disable indexes from the docs

* more changes in import doc

* Added bundle update method

* Bundles

* Add System.IdentityModel.Tokens.Jwt package and update BearerTokenHandler to print claims from JWT token

* Writing output bundles

* - sleep 40

* Added sync Send to BearerTokenHandler for Import Tool

* blob rewriter

* threads*threads

* .json

* Added deduplication for "single" types

* multiple

* short exe names

* direct bundles

* Removed order by

* full names back

* short on one

* write mixed ndjson

* Fix codeql in bearer token handler

* code an changes

* Fix codeql

* SearchByIds

* correction in searchbyids

* count in correct place

---------

Co-authored-by: Mikael Weaver <mikaelw@microsoft.com>
2024-02-16 17:28:45 -08:00
Fernando Henrique Inocêncio Borba Ferreira 7b5ca42228
Healthcare Shared Components v7.1.13 (#3717) 2024-02-16 09:08:52 -08:00
Mikael Weaver a677456664
Added exception handling to WaitAll in DeletionService (#3719)
* Added exception handling to WaitAll in DeletionService

* updated comment

* fix issue/filter out noise from job record

* Removing unneeded exception block

* Revert "Removing unneeded exception block"

This reverts commit 5fce8ecc30.
2024-02-15 14:30:32 -08:00
Brendan Kowitz 774e600611
Updates dotnet Security patch (sdk:8.0.200) (#3716) 2024-02-15 08:56:04 -08:00
rajithaalurims 4d5dfb93ae
Additional logs added (#3710)
* Additional logs added

* Fixed printing Object
2024-02-14 16:27:57 -06:00
SergeyGaluzo cc5576f4a9
Controller and tests (#3713) 2024-02-14 08:24:16 -08:00
Mikael Weaver c317606340
Revert [Pipelines] Add Integration/E2E Retry (#3642) (#3714)
* Revert "[Pipelines] Add Integration/E2E Retry (#3642)"

This reverts commit d23bf8fd07.

* Add back repaired tests
2024-02-14 07:43:25 -08:00
Fernando Henrique Inocêncio Borba Ferreira 869dbabbe3
FHIR Model and temporarily cached data (#3705)
* Change SqlServerFhirModel to use FhirMemoryCache
* Improvements in FhirMemoryCache.
* Added support to multi case cache.
* Test improvements.
* New tests. Removing lock to be aligned with the MemoryCache.
2024-02-09 17:06:53 -08:00
Mikael Weaver d23bf8fd07
[Pipelines] Add Integration/E2E Retry (#3642)
* Added trx logging and test publishing

* Fixed test reporting path

* changed to vstest

* rollback build change

* Fixed newline issue

* Fixed export tests to use VSTest

* Changed over last dotnet test

* Fixed export tests

* pipeline test fix

* formatting fix

* Fixed test filter typo

* pinning vstest to fix issue

* trying different version of vstest

* Testing batch size to fix data test issues

* Testing custom batch size fix

* removed vstest batching

* Testing changing the TestClassWithFixtureArgumentsTypeInfo name

* testing blame timeout

* Testing new blame params

* Changing SQL exception number

* Removed restriction for testing message only in Win

* Added publishing of test files on failure on error to help debugging

* Updated upload test failure task

* Fixed integration test artifact name

* Added fix for sql test

* Fox normalize in SqlServerSchemaUpgradeTests

* Fixed SQL int test newline integration

* Updated dump type / timeout for tests

* Reduced blame back down to 7

* Added pool name to windows jobs

* Changing to correct windows pool
2024-02-09 15:36:36 -08:00
Brendan Kowitz 56abac01d5
Job Factory provides Job tasks with an ambient scope (#3463)
* Job Factory provides Job tasks with an ambient scope
* Addresses Singletons that use Func<> as ServiceProvider
2024-02-08 10:12:51 -08:00
v-iyamauchi 2abb97aaa3
Logging an exception message only for blob LeaseAlreadyPresent. (#3704)
* Logging an exception message only for blob LeaseAlreadyPresent.

* Adding a blob resource uri in the message.
2024-02-08 17:04:36 +00:00
v-iyamauchi f2d7fedee4
Adding more logging for import job. (#3702) 2024-02-08 08:05:07 -08:00
Brendan Kowitz 353657c320
Updates .net sdk minor versions (#3703) 2024-02-07 13:57:32 -08:00
SergeyGaluzo f7c6ce66b1
Removing dead code from $import (#3696)
* Removing dead code from $import

* Remove progress and rebuld indexes

* var

* No use of progress

* Removed progress again

* Removing call to heavy heartbeats

* Remove test relying on heavy heartbeats

* removed usage of heavy heartbeats

* Removed progress from IJob

* removed progress more

* tests

* removed obsolete comment

* Removed obsolete fields.

* write back to be on par

* json convert

* removed obsolete properties

* Removed disable indexes from the docs

* more changes in import doc
2024-02-07 08:20:10 -08:00
rajithaalurims c3a7ce2b36
Removed resource type expansion for global search (#3701)
* Removed resource type expansion for global search

* Removing resource type expansions only for non Smart global search
2024-02-06 14:51:11 -06:00
Mikael Weaver adc81cb85f
Default to non-parallel $export for API for FHIR (#3700)
* Added handling for dynamic isParallel result

* Small formatting fixes
2024-02-05 11:09:40 -08:00
Brendan Kowitz 4610f05780
Adds permissions to codeql.yml (#3698)
+semver: skip
2024-02-05 10:10:06 -08:00
mahajan-xor 8ce4e6f596
Adds exception block to handle invalid storage URL (#3695)
* Adds exception block to handle invalid storage url.It will log exception as Information.

* Added string interpolation
2024-02-05 22:47:48 +05:30
SergeyGaluzo e59f8b1fba
Enable replicas for search (#3699) 2024-02-02 16:12:57 -08:00
Fernando Henrique Inocêncio Borba Ferreira 3f0e0a4bd9
[Jobs] Extending existing logs (#3693)
* Adding new logs

* Log improvements.

* Extending new logs to ImportOrchestratorJob

* Improving tests

* More log details

* Adding more details

* Undo Global.json

* More changes expanding the use of the new logging logic.

* Added missing log.

* Improving the logs in Job Hosting.

* Adjusts to fix errors after merge with main.
2024-01-31 16:30:10 -08:00
Brendan Kowitz 8b9207c271
Update codeql.yml with query pack (#3694) 2024-01-31 11:02:39 -08:00
SergeyGaluzo 3046bd0c0f
Remove +1 assumption for input versions in $import (#3685)
* Remove +1 assumption for input versions in $import

* spelling

* order

* Abilty to add meta

* time generation

* better time

* var name

* _ => x
2024-01-30 13:48:25 -08:00
Teng Li 53f5153126
Update convert libs to add the performance telemetry logger. (#3686)
Update convert libs to have the performance telemetry logger available.
2024-01-30 10:00:01 -08:00
v-iyamauchi 2754739a4c
Fixing a DI issue. (#3692) 2024-01-29 11:53:33 -08:00
Fernando Henrique Inocêncio Borba Ferreira 24d67e839c
[Bundle] Fix racing condition during parallel processing (#3593)
* Clean config file.

Bundle orchestrator improvements.

* Logging to conditional updates

* Adding more logs to conditional operations

* Added new tests to test the issue with MergeAsync and multiple versions being updated in parallel

* Improve the new test adding more asserts to it.

* Add espace condition in BundleOrchestratorOperation's looping

* New tests and comments

* Increase SQLClient version to v4.8.6

* Enable tests to more scenarios. Fix typo.

* Removed not used local variable in test

* Updating the external cancellation timeout of a test.
2024-01-29 11:40:06 -08:00
SergeyGaluzo d653a28c5a
Add retries to deal with small GP databases. (#3691) 2024-01-29 08:26:10 -08:00
Mikael Weaver 7b599763b6
Add Parallel Cosmos Export (#3584)
* checkpoint for export with lastModified count

* Reduced scope to parallelization by resource.

* Added tests for cosmos export orchestrator

* checkpoint to move my code to another machine

* Fixed cosmostdb export integration test expected values

* Added sharding to CosmosDB export

* Revised grouping for export

* small bug fix/enhancements

* bug fixes found during testing

* Rolled back ExecutePagedQueryAsync

* Updated export query to use limit and count

* Fixed logic for continuing export in thread

* Fixed export bugs

* Updated time incrementing logic to milliseconds

* Simplifed range lookup

* Moved large async export search logic to search service

* Use HasValue vs is null for CosmosExportOrch

* Export largest resource counts first

* Modified export to use feedrange

* Fixed tests

* Fixed tests

* small simplification on cosmos orch

* Added initial logic for Cosmos history/delete in $export

* Fixed build issue with SQL export orchestrator

* Fixed cosmos export bugs found via manual testing

* Added soft delete extension to export

* Add first pass on Export E2E Tests

* Updated export E2E tests to use test data from fixture.

* Updated long export tests for individual history/delete scenarios

* Updated gitignore to ignore azurite temp files

* Small azure storage explorer / azurite comment update for export tests

* Fixed ExportJobTaskTests to not rely on position of queryParameterList

* Added ExportJobTask unit test for history/delete

Also cleaned up some left over comments from previous commit

* Removing changed settings.json file

* Added historical and soft delete export to rest file

* Added import of soft deleted resources

* Updates per PR review

* Removed import changes - going to another PR

* Removed launch.json changes accidentally committed

* Added initial SQL export with soft delete / history

* oopsies undo change to importresourceprocessor breaking tests

* Fixed logical error in new SQL historical search

* Added searchoptionsfactory test for include history/deleted

* Fixed failing export test failure.

* Removed long running export flag to see if it runs in pipeline

* restructured export test location for readability

* Fixed export issues found in testing

* Changed SQL script version for merge

* Updated SQL schema version

* Code style cleanup

* Updated export history/deleted query params

* Fixed export included data test

* Changed SQL exporter to use export configuration vs magic numbers in code

* Rolling back unneeded SQL changes

* Fixed parallel export with history/soft delete

* Updated export E2E tests for parallel export multi-job

* fixed merge regression

* Removed unnecesary usings

* Fixed tx issue in export data tests

* testing central export perf

* Rolled back central execution of export tests

* Removing exportlongrunning for pipeline perf test

* Fixed build

* Optimized test structure

* merged from main

* Updated schema version to iterate after merge

* Fixed issue with merge

* Fixed export create call

* decoupled search options from export

* Fixed test issue

* Removing NA SQL comment

* PR comments on tests fix

* "un-fancified" ResourceTypeVersion enum

* Fix STU3 build error

* Fixed unit tests that no longer use export/history query params

* fixed test mistake for resourcetypeversion

* Fixed cosmos export search functoin

* Fixed export includeAssociatedData error message

* Default ResourceVersionType to latest

* Fixed history search - added deleted

* fix export count issue?

* small code cleanups

* fixed merge issue

* Fixed logic stopping from parallel jobs being enqueued

* Fixed parallelization issue

* testing w/o job dup logic

* Added cosmosdb parallel back

* Handle canceled cosmos export issues

* Fixed cosmos export cancel issues

* Fixed for job not completing 100%

* fixed logic to queue new cosmos jobs

* omg I can't believe I missed this

* Added proper handling of cosmos task canceled exceptions

* Fixed cancel retry

* Test cancel export ignore etag

* Added job task tests

* Added export processing job tests

* Updated tests

* Fixed cosmos orchestrator unit tests

* Review improvements

* Fix small isues with cosmosdb export integration tests

* test simplification

* changes from feedback

* undo test changes that aren't needed

* undo spacing changes to resx file

* PR review changes

* undo whitespace changes

* cleanup

* Fixed new export bug

* Used scope search service for Cosmos parallel export

* Split cancelled tests via theory

* Fixed cosmos export int tests

* fixed bad test

* small changes from PR
2024-01-26 14:51:12 -08:00
Robert Johnson e0d665b52f
Bulk delete fixes (#3689) 2024-01-26 14:49:33 -08:00
Richa Bansal fdac1e657f
Revert "Revert endpoint routing changes" (#3682) 2024-01-24 15:00:33 -08:00
SergeyGaluzo a9902ddc5b
Adding missing throw to GetAsync method. (#3688) 2024-01-23 19:11:19 -08:00
Mikael Weaver ad3ff3ff3c
Added TrimStart to FhirClient SearchAsync (#3681) 2024-01-23 11:49:18 -08:00
SergeyGaluzo ac1852a13b
New secondary indexes plus plus (#3674)
* Secondary indexes

* Update to allow *:* _revinclude search

* Remove IsHistory filter from reference table WHERE clause

* 73

* removed ResourceType expansion and isHistory for global searches

* fix lines after merge

* ref index

* changed GivenACrossSystemQuery_WhenRewritten_GetsAllResourceTypes(

* reverted generator changes

* 27 -> 3

* temp disable _rev

* _rev back

* Skip wring history clause for search params

* sql client

* Stop resource type expansion

* resource type expansion is back

* View access

* transactions and unit test code

* Commented test script

* Extra IsHistory

* comment

---------

Co-authored-by: Jared Erwin <jaerwin@microsoft.com>
Co-authored-by: rajithaaluri <rajithaaluri@microsoft.com>
2024-01-22 07:48:08 -08:00
Mikael Weaver e0adbbdbde
Removed prefix slash on history tests (#3680) 2024-01-21 07:27:52 -08:00
Mikael Weaver 2e475b919f
Fix flaky history count test (#3675)
* Removed offset

* Updated System.Data.SqlClient

* test cleanup

* Updated test to explicitly read datetime from server

* Added output string to flaky test

* simplified code

* Testing using before with since

* Updated text output so it's not truncated

* adding thread.sleep
2024-01-19 10:36:46 -08:00
Fernando Henrique Inocêncio Borba Ferreira 9cc895e0ed
[Cosmos DB] Query max concurrency flag (#3669)
* Extending queries to use max concurrency parallelism
* Add new Action Filter for Query Latency Over Efficiency
* Adding new tests to ensure that FhirController has the expected attributes. Adding more tests on top of the new filter. Limiting the execution of the filter to Azure API for FHIR only.
2024-01-18 13:18:55 -08:00
Richa Bansal 9bbe6cb7c7
Revert changes (#3678) 2024-01-18 13:10:52 -08:00
Mikael Weaver 20c138b0b7
Update SqlClient Package Version 2024-01-17 13:05:24 -08:00
rajithaalurims dcdbfc35c9
tests to ImportResourceLoader (#3673) 2024-01-17 11:42:32 -06:00
SergeyGaluzo c27944fcda
$import parser with \n line ends on windows VMs (#3670)
* Make $import parser work with \n line ends

* comment

* Removed batching

* removed new buffer

* static methods and tests

* Do not return empty lines

* corrected spelling

* Removed skip empty line logic and removed "incorrect" test

* extra test plus skip empty
2024-01-16 12:57:03 -08:00
SergeyGaluzo 3b31406553
unsetting read only on retries (#3671) 2024-01-16 09:21:29 -08:00
Mikael Weaver 4ae7bec855
Add Summary Count for History Endpoints (#3661)
* Added count for all history

* Added necessary extra tests

* Added E2E Test

* Addressed PR comments

* Fixed E2E Tests

* fix test build issue

* Fixed test name

* Updated tests

* Fixed test issue

* Fix test date/time format
2024-01-12 16:18:20 -08:00
dependabot[bot] c2ea2a56d4
Bump actions/github-script from 6 to 7 (#3591)
Bumps [actions/github-script](https://github.com/actions/github-script) from 6 to 7.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Richa Bansal <ribans@microsoft.com>
2024-01-12 16:12:26 -08:00
dependabot[bot] 8fd6483b6e
Bump github/codeql-action from 2 to 3 (#3631)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-12 14:01:21 -08:00
dependabot[bot] 69c236ae1c
Bump actions/labeler from 4 to 5 (#3612)
* Bump actions/labeler from 4 to 5

Bumps [actions/labeler](https://github.com/actions/labeler) from 4 to 5.
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](https://github.com/actions/labeler/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/labeler
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update labeler file as per new documentation

* Testing change

* For testing

* Testing change verified, undo the change

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Richa Bansal <ribans@microsoft.com>
2024-01-12 12:11:35 -08:00
SergeyGaluzo 883d6a8698
Enable use of read only replicas (#3663)
* read only

* read only

* bug fix

* minus using

* passing builder

* HttpGet

* return conn in open state

* comments
2024-01-12 10:01:34 -08:00
Mikael Weaver 6d11c641f5
Fixed SQL Search Service Deleted/History Handling (#3666)
* Fixed SQL Search Service deleted history

* Added testing for SQL query generation of history/soft delete

* Removed SqlSearchType as it was redundant

* Added test case for history and soft deleted without latest
2024-01-12 07:00:30 -08:00
Richa Bansal bf6d6a094e
Adds check for basePath (#3668) 2024-01-11 13:30:41 -08:00
Robert Johnson 5d880f1e8b
Fix null definitions (#3662) 2024-01-10 16:21:22 -08:00
Robert Johnson 54c1ff98c2
Fix Cosmos Bulk Delete (#3657)
Fix scoping for bulk delete
2024-01-10 11:25:13 -08:00
mahajan-xor 7e0f423774
Fixes Invalid URL exception message response. (#3644)
* changed invalidurl exception status code to 400 insteadof 500

* Added Validation Filter for ResourceType and Id.
Removed Argumentnull exception block from OperationoutcomeExceptionFilterAttribute class

* Validation added to few other scenarios

* Added unit test case for Filete validation and rename filter attribute to ValidateIdSegmentAttribute

* E2E for VRead BadRequest

* reverting test

* decorated newly added test case class with Trait attribute

---------

Co-authored-by: rajithaaluri <rajithaaluri@microsoft.com>
2024-01-10 15:05:05 +05:30