Граф коммитов

395 Коммитов

Автор SHA1 Сообщение Дата
David Justo 928f933a40
disambiguate Netherite constructor (#401) 2024-06-05 13:10:51 -07:00
David Justo 6ee62e3f27
Complete FASTER sessions before releasing them during query processing (#397) 2024-06-04 23:30:57 -07:00
David Justo f3e5f2621f
Add code mirror pipeline (#399)
* add code mirror pipeline

* move within eng/ci as per convention
2024-05-30 16:41:36 -07:00
Jaliya Udagedara 122885a5f8
Add initial integration tests for Durable Entities. (#391) 2024-05-30 15:52:22 -07:00
David Justo 51c6b957c7
Auto-generate AssemblyInfo + refactor version into common props. (#393) 2024-05-09 15:19:36 -07:00
David Justo aa124cdf50
rev FASTER.Core to 2.6.5 (#395) 2024-05-07 16:30:59 -07:00
Jaliya Udagedara beb84a0a7c
Cleanup unnecessary references. (#388) 2024-05-03 16:31:10 -07:00
Naiyuan Tian f8b5634746
Update Durable Dependencies to Latest Version (#389)
* Update DurableTask.Netherite.AzureFunctions.csproj

* increase durabletask extension ver and patch

* incre durabletask.core ver

* remove typo
2024-04-30 13:49:52 -07:00
Sebastian Burckhardt 24efd5478d
Add support for isolated entities (#358) 2024-04-26 09:31:15 -07:00
Sebastian Burckhardt 7ca60baec2
update to v1.5.0 (#386) 2024-04-22 14:43:44 -07:00
Varshitha Bachu 687071326d
Upgrade FASTER dependency to v2.6.4 (#344)
* initial commit

* added comment

* added readCopyOptions

* addressed PR feedback

* updated CheckpointVersionShift()

* updated Netherite version

* Update src/DurableTask.Netherite/StorageLayer/Faster/FasterStorageProvider.cs

* Update src/DurableTask.Netherite/DurableTask.Netherite.csproj

* update to FASTER v2.6.3

* update to FASTER 2.6.4

---------

Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com>
2024-04-22 14:43:30 -07:00
Sebastian Burckhardt 40a1f8c68f
Deduplicate CreationRequestReceived events (#381)
* always deduplicate CreationRequestReceived events

* Update src/DurableTask.Netherite/PartitionState/InstanceState.cs

Co-authored-by: David Justo <david.justo.1996@gmail.com>

---------

Co-authored-by: David Justo <david.justo.1996@gmail.com>
2024-04-22 13:07:27 -07:00
Sebastian Burckhardt 7e52cbe912
go back to version 1.4.3 for a patch release (#382) 2024-04-17 09:35:56 -07:00
Sebastian Burckhardt a582ca576e
Make adjustments and add tracing (#374)
* adjust based on observations in stress testing, and add more tracing to FASTER callbacks

* another trace line
2024-04-16 11:26:56 -07:00
Sebastian Burckhardt ab706eb79e
Fix blob batch persistence (#369)
* draft

* fix handling of batchposition, and tolerate race condition for deleted blob

* address PR feedback

* fix incorrect handling of skipped events

* fix check that removes confirmed events and add detail-level tracing

* fix broken trace statement

* fix handling of blob deleted

* add more tracing to blob download

* do not process any more events if already shutting down

* address PR feedback (add comments)

* fix handling of missing blobs

* Update src/DurableTask.Netherite/TransportLayer/EventHubs/BlobBatchReceiver.cs

Co-authored-by: David Justo <david.justo.1996@gmail.com>

---------

Co-authored-by: David Justo <david.justo.1996@gmail.com>
2024-04-16 11:24:38 -07:00
Sebastian Burckhardt 96b3ca5e97
Ignore duplicate client requests (#379)
* filter duplicate client requests

* Update src/DurableTask.Netherite/PartitionState/PrefetchState.cs

Co-authored-by: David Justo <david.justo.1996@gmail.com>

* fix whitespace

* address PR feedback: add warning when event is dropped

---------

Co-authored-by: David Justo <david.justo.1996@gmail.com>
2024-04-16 09:38:11 -07:00
Sebastian Burckhardt 2f94e765d5
fix race condition on OrchestrationRuntimeState.NewEvents (#380)
* fix race condition on mutable list of new events inside OrchestrationRuntimeState.

* Apply suggestions from code review

Co-authored-by: David Justo <david.justo.1996@gmail.com>

---------

Co-authored-by: David Justo <david.justo.1996@gmail.com>
2024-04-15 11:29:07 -07:00
dependabot[bot] b885bd4a8c
Bump undici in /samples/Hello_Netherite_with_TypeScript (#373)
Bumps [undici](https://github.com/nodejs/undici) from 5.28.3 to 5.28.4.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v5.28.3...v5.28.4)

---
updated-dependencies:
- dependency-name: undici
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-12 11:34:24 -07:00
dependabot[bot] 86d36a16f4
Bump Azure.Identity in /samples/TokenCredentialDTFx (#378)
Bumps [Azure.Identity](https://github.com/Azure/azure-sdk-for-net) from 1.10.4 to 1.11.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-net/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-net/compare/Azure.Identity_1.10.4...Azure.Identity_1.11.0)

---
updated-dependencies:
- dependency-name: Azure.Identity
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-12 11:34:11 -07:00
Chris Gillum 5a2fdc3e38
Use OrchestrationAlreadyExistsException instead of InvalidOperationException (#375) 2024-04-05 14:46:11 -07:00
Sebastian Burckhardt 3399888400
update version to 1.5.0 (#368) 2024-03-18 16:26:11 -07:00
Sebastian Burckhardt 0c1ccbec4a
Add variable-sized generic entity benchmark (#366)
* add a store entity to the performance benchmarks, for testing

* update http triggers
2024-03-18 15:41:46 -07:00
dependabot[bot] a72a1abdba
Bump follow-redirects in /samples/Hello_Netherite_with_TypeScript (#367)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.5 to 1.15.6.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.15.5...v1.15.6)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-18 09:54:38 -07:00
Sebastian Burckhardt 84b45a89b2
add README.md and some historic results to the performance tests folder (#362) 2024-03-18 09:49:40 -07:00
Sebastian Burckhardt 7e16a84d12
Improve error handling for FASTER IO completion callbacks (#349)
* wrap FASTER callbacks so exceptions are logged, and release semaphores prior to callback.

* address PR feedback, add two more lines of tracing, and use simpler trace message

* refactor control path for FASTER callbacks. Eliminate callbacks with nonzero error code.

* immediately cancel all requests waiting for the semaphore when partition is terminated

* cancel faster callbacks prior to disposing FasterKV and FasterLog (to prevent hangs)

* revert reorganization of cancellation since it appears to lead to issues with hanging dispose calls
2024-03-18 09:49:13 -07:00
Sebastian Burckhardt a74178bc6b
loosen azure storage semaphores (#365) 2024-03-18 09:48:09 -07:00
Sebastian Burckhardt d71a5a1ba8
Fix hanging blob creation (#351) (#364)
* fix hanging blob creation

* remove unnecessary try-catch
2024-03-15 13:21:42 -07:00
Sebastian Burckhardt a6924c01dd
handle long or infinite timeout argument in WaitForOrchestrationAsync (#357) 2024-03-15 12:59:34 -07:00
Sebastian Burckhardt 37f49adde5
Add RecoveryTester feature (#341)
* refactor partition management options, remove Scripted partition manager, and add RecoveryTester feature

* prevent creation of fresh taskhub in partition recovery mode
2024-03-15 12:57:01 -07:00
dependabot[bot] 230ada9c11
Bump the npm_and_yarn group group (#360)
Bumps the npm_and_yarn group group in /samples/Hello_Netherite_with_TypeScript with 2 updates: [durable-functions](https://github.com/Azure/azure-functions-durable-js) and [axios](https://github.com/axios/axios).


Updates `durable-functions` from 1.5.4 to 3.1.0
- [Release notes](https://github.com/Azure/azure-functions-durable-js/releases)
- [Changelog](https://github.com/Azure/azure-functions-durable-js/blob/v3.x/azure-pipelines-release.yml)
- [Commits](https://github.com/Azure/azure-functions-durable-js/compare/1.5.4...v3.1.0)

Updates `axios` from 0.21.4 to 1.6.7
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.21.4...v1.6.7)

---
updated-dependencies:
- dependency-name: durable-functions
  dependency-type: direct:production
  dependency-group: npm_and_yarn-security-group
- dependency-name: axios
  dependency-type: indirect
  dependency-group: npm_and_yarn-security-group
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-05 10:00:54 -08:00
dependabot[bot] 2215708925
Bump follow-redirects in /samples/Hello_Netherite_with_TypeScript (#340)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.9 to 1.15.4.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.9...v1.15.4)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-05 09:35:24 -08:00
Sebastian Burckhardt e6487b9cec
update README and samples to latest veersion (1.4.2) of Netherite, update other dependencies in the samples, and use consistent taskhub names for all samples (#355) 2024-03-05 09:26:40 -08:00
David Justo 7921c1e4e5
Merge pull request #353 from microsoft/dev
Promote dev to main for 1.4.2 release
2024-02-15 18:54:15 -08:00
Sebastian Burckhardt 60c7d39e03
revise how dispose tasks are organized so that they can all use the same timeout check and error reporting mechanism (#330) 2024-02-12 16:20:10 -08:00
David Justo 0abc16d3e8
Add new logs to recovery path (#337)
* add simple log

* add detail column to FasterAzureStorageAccessCompleted

* refactor read range log

* pass just the position parameter instead of printing full details for FasterStorageAccessCompleted

* put position and id information into details

---------

Co-authored-by: sebastianburckhardt <sburckha@microsoft.com>
2024-02-12 16:19:40 -08:00
David Justo 66762a381d
rev version to 1.4.2 (#346) 2024-02-12 14:20:17 -08:00
Sebastian Burckhardt 2d3e5ea8e3
fix EH batch load cancellation issue (#347)
* do not cancel EH batch loading when partition is terminated, but only when EH processor shuts down

* Update src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs

Co-authored-by: David Justo <david.justo.1996@gmail.com>

---------

Co-authored-by: David Justo <david.justo.1996@gmail.com>
2024-02-12 13:53:57 -08:00
Sebastian Burckhardt b2431e62f7
Enhance validation of deserialization and reporting of errors (#345) 2024-02-12 13:09:18 -08:00
Sebastian Burckhardt b9ed29ada7
update to FASTER v2.0.23 (#264)
* draft commit

* update to 2.0.23

---------

Co-authored-by: David Justo <david.justo.1996@gmail.com>
2024-01-16 10:23:30 -08:00
Sebastian Burckhardt b47eae7f40
Always release partition lease when a partition shuts down (#335)
* always release lease when a partition shuts down

* address PR feedback (add comment)

* recognize another empiric case of transient storage exception

* change default of partition checkpointing

* do not release lease if lease was already lost.
2024-01-03 09:48:52 -08:00
Sebastian Burckhardt fc777fed0a
Revise partition startup (#332)
* add "waiting" transition and timeout to partition startup

* address PR feedback
2023-11-03 11:03:18 -07:00
sebastianburckhardt f24f6de01a Merge remote-tracking branch 'origin/main' into dev
# Conflicts:
#	src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs
2023-10-24 13:33:05 -07:00
Sebastian Burckhardt bbbabbaeb1
update version to 1.4.1 (#328) 2023-10-24 13:22:45 -07:00
Sebastian Burckhardt 96b832eb5c
remove ? to prevent nullable warnings (#327) 2023-10-24 10:57:34 -07:00
Sebastian Burckhardt 7a6e760b59
Fix fencing of checkpoints (#326)
* fix fencing of checkpoints that caused occasional concurrent modification exceptions and can cause history corruption

* add more comments to explain sessions and fencing
2023-10-23 13:56:27 -07:00
Varshitha Bachu 76d01adabc
Copy ParentTraceContext when cloning ExecutionStartedEvent (#309)
* Update CreationRequestReceived.cs

* remove the faster-alternate data store and fix broken deserialization. (#308)

* fix and instrument commitlog serialization and deserialization. (#305)

* fix discard of packets so the next packet to expect is updated correctly (#302)

* New EventHubs performance tests (#178)

* rebase, and remove changes to scale monitor

* add hello cities test that prints the history of a nested orchestration

* implement a watchdog that terminates CompletePending quickly if it hangs (#318)

* add unique id to scale monitor constructor (#316)

* fix bugs from PR #302 (#315)

* fix cache size reporting (#321)

* added a comment

* update durable task package references to 2.15.1 and 2.12.0 (#317)

* update DT and DF package references

* update to latest

* sync dev w/ main (#324)

* Update GH automation (#303)

* initial commit (#290)

* Revert "initial commit (#290)" (#314)

This reverts commit 3a2d193e57.

* Bump Azure.Identity from 1.7.0 to 1.10.2 in /samples/TokenCredentialDTFx (#323)

Bumps [Azure.Identity](https://github.com/Azure/azure-sdk-for-net) from 1.7.0 to 1.10.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-net/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-net/compare/Azure.Identity_1.7.0...Azure.Identity_1.10.2)

---
updated-dependencies:
- dependency-name: Azure.Identity
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Terminate partition when FASTER refuses to checkpoint for over a minute (#301)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Varshitha Bachu <vabachu@microsoft.com>
Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com>
Co-authored-by: David Justo <david.justo.1996@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-23 13:55:55 -07:00
David Justo fb4aa617ff
sync dev w/ main (#324)
* Update GH automation (#303)

* initial commit (#290)

* Revert "initial commit (#290)" (#314)

This reverts commit 3a2d193e57.

* Bump Azure.Identity from 1.7.0 to 1.10.2 in /samples/TokenCredentialDTFx (#323)

Bumps [Azure.Identity](https://github.com/Azure/azure-sdk-for-net) from 1.7.0 to 1.10.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-net/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-net/compare/Azure.Identity_1.7.0...Azure.Identity_1.10.2)

---
updated-dependencies:
- dependency-name: Azure.Identity
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Terminate partition when FASTER refuses to checkpoint for over a minute (#301)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Varshitha Bachu <vabachu@microsoft.com>
Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-23 11:20:13 -07:00
Sebastian Burckhardt 08e760aeaf
update durable task package references to 2.15.1 and 2.12.0 (#317)
* update DT and DF package references

* update to latest
2023-10-23 11:19:37 -07:00
David Justo ba15eed262
Terminate partition when FASTER refuses to checkpoint for over a minute (#301) 2023-10-19 13:55:54 -07:00
dependabot[bot] 6effbf572c
Bump Azure.Identity from 1.7.0 to 1.10.2 in /samples/TokenCredentialDTFx (#323)
Bumps [Azure.Identity](https://github.com/Azure/azure-sdk-for-net) from 1.7.0 to 1.10.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-net/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-net/compare/Azure.Identity_1.7.0...Azure.Identity_1.10.2)

---
updated-dependencies:
- dependency-name: Azure.Identity
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-18 14:46:37 -07:00