зеркало из https://github.com/microsoft/FASTER.git
16 Коммитов
| Автор | SHA1 | Сообщение | Дата |
|---|---|---|---|
Ted Hart
|
32d04e6563
|
[C#] Replace LockingMode with ConcurrencyControlMode (#867)
* Replace LockingMode with ConcurrencyControlMode * Fix 'Remote' solution build |
|
|
Ted Hart
|
73fe128475
|
add Delete to YCSB Benchmark; change "-r read%" to "-rumd read% upsert% rmw% delete%" (#839) | |
|
Ted Hart
|
3c6dad76cf
|
[C#] Change locking to use FasterBase hash buckets (#799)
* Locking fixes
- CompletePending* must also check to lock non-auxiliary records
- Ephemeral locking must look for lock evictions as well
- InternalLock must check for SpinWaitUntilRecordIsClosed
- Replace oneMiss handling in InternalLock
- EphemeralSUnlock must follow unlock failures
- LockTable should have its own bufferPool
- Ensure OnPagesClosed does eviction scans in address order
* More locking fixes:
- Fix missing unlock of a Tombstoned record in InternalContinuePendingRead
- Add more Asserts where ManualLocking requires a locked record
- Update and remove obsolete comments
* - Remove key-present optimization for SpinWaitUntilRecordIsClosed
- Better checking on following records transferred from LockTable in InternalLock(unlock)
- Add some additional comments
* Fix hash bucket overflow pointer during recovery
Fix debug assert logic
* WIP on locktable implemented via mainhash overflow buckets
* unstaged files missed in previous push
* Updates to EphemeralOnly locking for transfer-from-RO/RC
* Change LockTableTests to OverflowBucketlockTableTests
* Remove RecordInfo.Tentative, .IsIntermediate*
* More locking WIP: session.NeedKeyLockCode, test cleanup
* WIP locking: Tests build
* WIP on mainhash locking:
- Add KeyCode creation and sorting and methods public and internal
- Remove unnecessary IsLocked (it does not make work with EphemeralOnly and SessionControlled may map to multiple keys)
- More tests run now
* Most UTs run
* More UTs succeed
* Still some UT failures
* - use XLocks as Tentative indicator for insertions/splices at log tail and readcache insertions
- improve readcache consistency checking
- clear HasInMemoryLock when HeadAddress goes above srcRecord address following BlockAllocate
- rename LockingMode.SessionControlled -> Mixed and other renaming for clarity
- remove some unused methods
- fix locking sequences in InternalContinuePendingR(ead|MW)
* More fixes
- Missing 'ref' on recordInfo var in FindAndUnlockTransferredRecord
- Improve record allocation checking for below-HA in-memory source and RecordInfo lock clearing if so
- Add LockCode to HashEntryInfo in DEBUG
* Remove SpinWaitUntilRecordIsClosed; it was only used in ReadCache and in the relevant case we just need to wait for ReadCacheEvict to complete.
* - MixedModeTentativeOrClosed is not needed
- Add/Improve some ToString()s
- Remove unused RecSrc.ClearSrc()
- Rename to HasRecordInfoLock
- Move ICPRead HandleImmediateRetryStatus outside the lock scope
- Consoldiate EphemeralSUnlock(AfterPendingIO)
* Streamline locking and FindInMemory calls in InternalXxx
* - Change names to pendingContext.Initial*Address and improve setting of these in InternalXxx and handling of these in CompletePending*() and TryFindRecordInMemory.
- Also consistentize related param names for readcache and log search to minAddress.
* Clean up pending RMW call to CreateNewRecordRMW, per prior changes that make the inputSrc unnecessary.
* - Fix stale readcache assignment to recSrc.Log
- make searches after pendingIO completion non-inclusive of minAddress
- Some ToString() and naming changes for consistency
- Remove RecordInfo.IsValidUpdateOrLockSource (superseded by direct call to .IsClosed)
* Allow length specification on KeyCode array operations
* - Fix check for lock having escaped to eviction zone following BlockAllocate
- Add some comments for SafeHeadAddress et al.
* - Make sure the readcache bit is set when needed on addresses put into stackCtx.newLogicalAddress.
- Rename to HandleNewRecordOnException for clarity
* Update remote project
* - Make sure FindAndUnlockTransferredRecord can go to SafeHeadAddress
- Add non-ref overload of GetLockCode
* Tweaks for benchmark with Mixed locking
* Add ILockableKey.CompareLockCodes
* Change LockCode to just be hashcode, since we may grow the hash table.
* - Make CPR consistency check work without conflict with the key latches in the HashBuckets.
- Change some names for clarity.
- Pending read completion should return values from any in-memory record, as they are the latest
* Remove duplicate opCtx param and removed sessionCtx (aka currentCtx, aka ctx) from calls where FasterSession was also passed (in favor of (new) FasterSession.SessionCtx), where the called method has a type constraint of "where FasterSession : IFasterSession<Key, Value, Input, Output, Context>". Not yet done: converting "whereFasterSession : IFasterSession" constraints to generic to remove the duplicate ctx parameter.
* Fix DoLockOp to compare bucket indexes, not KeyCodes
* Finish removal of duplicate sessionCtx param from calls where FasterSession was also passed (in favor of (new) FasterSession.SessionCtx), where the called method has a type constraint of "where FasterSession : IFasterSession" (non-generic). This cannot be done in Synchronization (the state machines) due to their use of NullFasterSession.
* Fix key duplication in readcache and log:
- VerifyInMemoryAddresses should not refresh hei if no splice point is present
- FindRecordInMemory checks pendingContext.InitialHash
- Move CompleteCopyToTail in ITCTT into the non-readcache section
- TryFindRecordInMemory needs to check pc.InitialEntryIndex's readcache bit
- Updaters need to scan down to hlog.ReadOnlyAddress to ensure the "key is not at or below LatestLogicalAddress" logic holds
- ReadCacheCheckTailAfterSplice needs to check hei.*Address readcache bit
* Replace use of SafeHeadAddress with SpinWaitUntilClosed in FindInReadCache
* Remove LockingMode.EphemeralOnly
add --hashpack arg to FASTER.benchmark
* Tweaks for the Standard locking change
* fix remote to use LockingMode.Standard
* Add DisposeTest comments and more Asserts
* Use calls on RecordSource to make it easier to verify correct matching of log and physicalAddress
* Temporarily disable LocalMemoryDevice tests for investigation
* test removal of DisposeTests
* - Fix for CTT to address SameKeyInsertAndCTTTest failure
- Add missing arg parsing for MemOnlyCache
- Re-enable DeviceType.LocalMemory in tests
* [C#] MemOnlyCache sample: improved size tracking, including read cache (#800)
* Improved cache size tracking, including read cache
* Rename MemOnlyCache to ResizableCacheStore.
* update doc
* fixes
* Updates
* ClearBitsForDiskImages should unseal record as well (Q: why was it sealed to begin with?)
* Correct clearing of index bits during recovery
* Remove RecordInfo sealing
* WIP Ephemeral locking at IFunctions-level only, part 1
* Remove unused SessionType enumeration; consolidate duplicate IFasterSession implementations
* - Add EphemeralLocking readcache detach/reattach logic
- Remove unused ReadFlags.ResetModifiedBit
* Rename a couple files
* re-add RecordInfo.Seal for Ephemeral locking; remove unused (RMW|Upsert)Action.NeedMoreSpace (this is the default 'false' behavior)
* Remove single-key overloads of Lock() and Unlock()
* - Update a too-aggressive assert.
- CreateLogDevice errors on Windows if path > MAX_PATH minus room for segment#.
* Improve previous MAX_PATH fix
* Chnge Read of an expired to just return NOTFOUND | Expired rather than tombstoning
* RETRY_LATER rather than trying to re-find splice point directly in VerifyInMemoryAddresses
---------
Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
|
|
Badrish Chandramouli
|
0b6eb1ffc0
|
[C#] Improved lock table implementation (#775)
* Locktable revision using InMemKV * remove AzureStorageDevice from TestCategory==Stress * Fix delta application in incr snapshot, if record sizes go down * More doc updates.. more to come * Check UseReadCache when testing for kReadCacheBit Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> |
|
|
Ted Hart
|
2f55e900e0
|
[C#] Convert *UnsafeContext to structs and acquire on demand (#750)
* Make *UnsafeContext a struct and acquire on demand (like LockableContext); change Resume/SuspendThread naming to Begin/EndUnsafe and add Begin/EndLockable * Add AcquireLockable check for in-prepare checkpoint |
|
|
Badrish Chandramouli
|
50a19c0ef3
|
[C#] FASTER v2 release (#563)
* various changes from indexing work (#562) * various changes from indexing work - Add BeginAddress and EndAddress to scan iterators - Allow null deltaFileDevice in HybridLogCheckpointInfo.Recover - Rename FoldOverSnapshot to useFoldOverCheckpoint - Make TakeHybridLogCheckpoint overload call through to overload to avoid duplication - Add default values to new scanDelta and recoverTo arguments - Add missing checkpointManager.OnRecovery call to IntenralRecoverAsync - Fix some comments * Fix shortened name in test to work on Linux * [C#] Record-expiration support in RMW (#545) * Add support for record-expiration management to RMW and I(Advanced)Functions * Update Remote to RMW changes for expiration support * moving remote to IAdvFun * LogDir semantics - minor update Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * [C#] Added PostCopyUpdater to IAdvancedFunctions (#561) * Add support for record-expiration management to RMW and I(Advanced)Functions * Update Remote to RMW changes for expiration support * Added PCU * Fixing testcase so that RMW does not expect old address. * Fix PCU bugs * Extend PostCopyUpdate to IFunctions; fix a few comments in remote Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> * [C#] Merge IAdvancedFunctions into IFunctions (#565) * Merge IAdvancedFunctions into IFunctions * Add SupportsPostOperations and Post(IU|CU|SD); expand "ref RecordInfo recordInfo, long address" on Functions * Add RecordMetadata to RMWCompletionCallback params; improve ReadAddressTests * Add tests of Upsert with Input * [C#] RecordInfo v2 (#577) * RecordInfo v2 * fix version jump in CPR. * Updated misc stuff * misc updates to SpanByte * Remove IntExclLocker * Add Try APIs to record locking * Remove FASTERLegacy (#581) * Add Output parameter to Upsert (#574) * Add overloads of Upsert that take "ref Output" and one that takes "out RecordMetadata"; Fix RMW setting pendingContext.RecordInfo/LA; add locking of PostCopyUpdater, PostInitalUpdater, PostSingleWriter; * Reorganize ClientSession IntenralFasterSession to group functions for a given area (Read, Upsert, RMW, etc.) together * Fix locking on Post* calls * Fixes for Post* locking * remove net461 and netcoreapp2.1; add net6.0 in <Choose> to work on later in VS2022 * Update to new RecordInfo locking calls * Update to new IFunctions Locking signatures and RecordInfo implementation * Add post operations to Faster.benchmark * [C#] Checkpoint, Recovery, RecordInfo updates (#588) * Removed WAIT_PENDING * * Use only 1 bit for newVersion in RecordInfo, for CPR. Semantics are that sessions in v+1 will create records in fuzzy region with this bit set. Recovery only scans fuzzy region to elide such records. * Add 1 Filler bit for future use with fillers in records * Increase read lock bits to 6 (64 parallel readers) * Use dirty bit for incremental checkpoints * fix ycsb to not delete data file if we are using --recover * Fix epoch re-entry violation * Move to long versions, for KV checkpoints. * Shuffle around stuff for better cache alignment in benchmark. * Updates based on review * [C#] FasterLog Commit Optimization (v2) (#587) * Checkpoint work * Add notion of manual commit * Recovery of commit cookie * alternative design of log-cookie * Implementation improvement * code review comments * Cleanup + testing * Add management utility for LogCommitManager * Handle race on manual commit code path where not the latest tail is written out * Add condition to only scan log in fastCommitMode * Rough draft to allow for 1. elided commit record when fastCommit turned off and 2. "loose" commits * checkpoint changes * fix bug with readonly log allocator initialization * disable fast commit temporarily for CI * fix MLSD and mem device fast commit behavior * fix AzureStorageDevice exception type and remove self-termination on failure * test case on concurrent inserts * Re-enable epoch protection on CommitInternal * Ensure that full flush list error callbacks do not break FlushedUntilAddress tracking * Fix unclean shutdown of stress test * update device log test to scan until tail instead * change error code used for full flush list to better distinguish from others * Minor updates. Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Remove legacy local log commit manager * [C#] FasterLog v2 - misc changes, removed OverWrite commits (#593) * Checkin misc changes, removed overWrite commits * simplify OnRecovery logic * Fix build after merge * fix alignment of device commit manager * [C#] v2 pending flush list (#596) * Make PendingFlushList never error out in stressful situations. Current implementation uses lock for ~5-10% perf degradation over prior structure. * [C#] FasterLog v2: Fix FasterLog missed final commit (#597) * Fix FasterLog missed final commit. * [C#] Log compaction v2 (#598) * initial checkin * New ShiftBeginAddress behavior -- only shifts BeginAddress variable and does not truncate physical log unless option is specified. Also added Log.Truncate API for physical log truncation. Truncate log until persisted BeginAddress after taking a checkpoint. * Fix epoch protection orderings. * minor improvement to TestUtils to avoid race exception. * update Program.cs * updates from master. * [C#] V2 Custom Commit Strategy (#599) * Add FasterLogCommitStrategy * Use Bakery algorithm for latch * Documentation + tests + improved threading performance * Fix initialization error * Revert to v2 synchronization scheme * Optimization around commit strategies * Refactor CommitPolicy only for normal commits * fix missing spin wait * [C#] Fix v2 CommitPolicy Warnings (#607) * fix warnings * fix broken test * update for code consistency * Merged PR 59053: Network Stack Refactoring into FASTER This pull request contains the necessary changes for the updated network stack interface used by FASTER. It adds INetworkSender.cs and NetworkSenderBase.cs that can be used to implement custom communication protocols. Currently only TcpNetworkSender is supported. * Create .dockerignore * Added TentativeHeadAddress - we first shift this, and when all threads agree, we can do some pre-closing work such as eviction. Then we can shift the actual HeadAddress, closing pages when all threads agree on this. (#608) * Cleanup and remove dead code + reuse base provider. * Code cleanup * [C#] minor cleanup of fasterlog v2 (#611) * minor cleanup of fasterlog * [C#] Remove CheckpointType from CheckpointSettings (v2) (#614) * Remove CheckpointType from CheckpointSettings. Users have to specify checkpoint type on a per checkpoint basis. * fix build break * Fix CTT to call PostSingleWriter. * Avoid double init of FasterLog * [C#] Fixing warnings (#615) * Fixing CI warnings * [C#] Updating dependent package versions in v2 (#616) * Updating dependent package versions * nit * [C#] Improved FasterKV and FasterLog settings API for v2 (#618) * Simpler config API for v2. Move FasterLog to its own folder. * Remove purgeEarlierCommits from reader. Instead: 1. writer retries file deletes (in device commit manager) if access denied, in a loop, until timeout 2. reader (user code) retries recover async if it fails due to file deleted, until timeout * [C#] FasterLog V2 Device error handling (#612) * prototype device * prototype failure handling * Add proposed error handling code * Basic error handling tasks * Fault tolerant devices * Change to failure handler based approach instead of just throwing an exception * Remove brittle error handling code * make sure to read pages from flaky device on recovery in test * fix warnings * code review comments * See if we can enforce clean shutdown * Add extensions to enqueue SpanByte to FasterLog, and unsafely iterate without memory copy. (#621) * [C#] FasterLog v2 Add ability to explicitly terminate a log (#620) * Add ability to explicitly terminate a log * remove special 0-length return on eof * Add LogCompleted in iterator * Change comment and test * fix async iteration logic * fixes Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Maybe fix SVS issue with atomic reads (#622) * [C#] FasterLog v2 iteration fix (#623) * First pass of fixes * undo local change * [C#] Updated docs for FasterLog v2 (#624) * some updates * Add missing documentation * Update 40-fasterlog-basics.md Co-authored-by: Badrish Chandramouli <badrish@gmail.com> * Fix logic for directory delete timeouts. * Speed up testcases * [C#] Lockable Unsafe Context (LUC) in FasterKV v2 (WIP) (#605) * WIP on ManualFasterOperations * fixes for v2 merge * Updates to locking (still in progress) - Remove User-defined locks - Move SupportsLocking up to FasterKV level - Skip Stub and Sealed in iterators - Change PendingContext.OperationFlags from byte to ushort - Add ReadFlags.SkipCopyToTail - Add IsLocked*, TranferLocks methods to RecordInfo - Add UnsafeCompletePending* to ManualOps - Track lock counts in ManualOps and throw if Dispose() is called with locks or epoch held - Add LockInfo for ManualOps.*Lock - Implementation and testing of Faster operations with ManualOps locks - Fix ReadAddressTests to reflect improved consistency of not returning readcache addresses * Locking docs * Update 30-fasterkv-manual-locking.md * Updated doc to GH review and discussions * WIP for LockableRawContext * Existing tests run * Rename to LockableRawContext, IFasterContext * Perf workaround for LockTable.IsActive * WIP on ReadCacheEvict and MemoryPageLockEvictionScan * Add ReadCache and LockTable-transfer tests; move ReadCache tests out to their own sub-hierarchy * Test MemoryPageLockEvictionScan * Add testing for two-phase upsert/copytotail * Remove manual locking bits in Recovery; minor refactorings in Recovery.cs to reduce duplication * Add manual locking to FASTER.benchmark * More LockTable testing and fixes, mostly around ocking nonexistent key while Upsert/RMW/Deleting that key * Add UnsafeContext and make it the default for FASTER.Benchmark * Updates from code review: - IsLocked returns shared lock count (added test) - Fix location of IsIntermediate checks in RecordInfo.Lock/Seal - Remove unused IEqualityComparer in LockTableEntry * Rename Take(Full|Index|HybridLog)Checkpoint to TryInitiate(Full|Index|HybridLog)Checkpoint, so it is obvious they need to complete it * - Remove SupportsPostOperations; this is now always done - Add SingleDeleter for symmetry with Upsert - Add DisposeKey and DisposeValue * Add CopyWriter, to distinguish "maintentance" copying (CopyToTail or copy to ReadCache) from API-parameter copies * merge fixes * Remove obsolete Functions-level locking and postOps specifications * Fix Remote build Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * fix semantic merge conflict in MemoryPageScanIterator (#632) * [C#] Add WriteReason parameter to SingleWriter (#630) * Add Functions.PostCopyWriter; update MemOnlyCache to use Post(Single|Copy)Writer * Remove (Post)CopyWriter and replace with WriteReason parameter to (Post)SingleWriter * Move SingleWriter "reason" parameter * Move ReadFlags and WriteReason to their own files; rename SupportsLocking -> DisableLocking with default 'false' * Update Remote to disableLocking name and default change * [C#] Various Fixes (#635) * Fixes: - Fixed WriteReason when CTT overrides ReadCache - make DisableLocking a readonly member - Call SetTentativeAtomic after Post* - Make readcache and comparer private with internal accessors for test - FIx intermittent failure of BasicHighLatencyDeviceTest * back out changes to move from internal -> private for FKV.readcache and .comparer * [C#] Remove StrictCPR (#636) * Removed WAIT_PENDING * StrictCPR removal WIP * Fix bug in state machine test, as it was not using unsafe context for fine grained control over the state machine execution. * Remove PendingContext.HeldLatch and heldOperation local * Revise FASTER.Benchmark to replace --noaff with --safectx and integrate this into Ycsb and SpanbyteYcsb * check epoch protection for complete pending * *UnsafeContext: AnyInstanceProtected -> ThisInstanceProtected Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> * [C#] Support int session ID for FasterKV (#641) * Support int session ID for FasterKV * fix remote solution, minor name updates * update FasterLog API for TryEnqueue of custom header * [C#] IEntry interface for FasterLog (#639) * enqueue with entries prototype * Add IEntry tests and method variants * Address code review comments * Rename + read entries * Add async versions of consumer * Make interfaces top-level * Correct header offset in FasterLog new API * [C#] Recover returns version we recovered to (#648) * add to FasterLog SpanByte API * fix breaking tests * Fix dispose loop * Update FasterServerBase.cs * do not remove ioPendingRequests in RMWAsync if there was no disk request (#653) * [C#] [WIP] Status return code v2 (#638) * Prototype of status as struct * improving status a bit * fix build break, expose underlying (raw) status code to user * misc * Update StatusCode * Update Remote to new Status(Code); fix test cases; fix StatusCode layout; fine-tune Status(Code) names; make StatusCode internal * Update StatusCode and Status to final spec * Update Remote to new Status spec * unrelated nit Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> * [C#] Change IFunctions 'address' parameter to UpdateInfo or ReadInfo (#652) * Add UpdateInfo and ReadInfo to IFunctions calls * Update Remote IFunctions implementations to UpdateInfo/ReadInfo changes Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Wire status into remote v2 (#657) Add NotFound API for status * Status naming changes and UpdateInfo structure split for v2 preview (#658) * [C#] Add Status.Record (#659) * Add Status.Record; update OperationStatus with all Advanced Status codes * Update Remote for status.Pending => .IsPending * Update nuspecs * more updates * Update azure-pipelines.yml * Update azure-pipelines.yml * revert windows-2022 in yml * Rename StatusCode.OK to Found (#662) * nit * update SpanByte with API to get pointer including metadata * Fixing ranges in OnPagesReadyToClose * [C#] Websocket bug-fixes (#663) * Fixed nits in websockets: * Clean disconnection * Support correction for large packets * Small nit for correcting message length * changed status to new FASTER status codes * Modified status codes Co-authored-by: Rohan Kadekodi <kadekodirohan@gmail.com> * Revert THA * nits * Improving cache tracker * fix NET5_0 to NET5_0_OR_GREATER * [C#] Add CompletePending[WithOutputs] to IFasterContext (#664) * Move CompletePending(WithOutputs) into IFasterContext * Skip LockEvictionObserver entirely if no LUCs active * Make Decrement interlocked; inline one accessor; revert CMakeLists.txt * [C#] Upgrade blob tests to Azurite, enable blob tests on Linux, use windows-2022 for CI (#666) * install azurite * Change testcase to make sure commit completed before dispose, it is up to user to ensure no ongoing commits at time of dispose. * [C#] UnsealPhysicalAddress fix (#665) * Fix unsealPA going below HA; return SUCCESS instead of NOTFOUND from normal reads that find an obsolete record in ITCTT * Ensure InternalLock calls require epoch protection * WIP: Spin wait on Pending Reads/RMWs until ClosedUntilAddress catches up. Still needs to handle partial page evictions. * Ensure partial ClosedUntilAddress shifts occur correctly. * [C#] Cancellation, ReadOptions, ReadFlags refactor (#668) * Add DeleteRecord and CancelOperation to *Info structures and IFunctions * Added ReadOptions * Move IFunctions methods' "ref RecordInfo" parameter to an *Info property, as it does not need to be updated * Update ReadFlags * ReadFlags fixes * Change XxxInfo.DeleteRecord/.CancelAction to XxxAction enum * Fixing Expiration tests * Add Phase to ExpirationTests * Add ReadOnly region to expiration tests * Delete and Upsert search down to HeadAddress if active LUC * [C#] Misc v2 fixes (#675) Fix misc bugs in v2 * update spanbyte * Yield thread when unable to reserve epoch entry, move iteration count outside table loop. * remove exception from overloaded lightepoch * Fix benchmark to run on .net 6 and linux. * fix another place * [C#] Improvements to LightEpoch * Use AllocateArray in LightEpoch * LightEpoch: Use two hash functions + linear probe, cache last successful probe per thread. Adjust table size based on number of machine processors. * [C#] Use GC.AllocateArray (#683) * Use GC.AllocateArray where possible * update benchmark as well. * Update buffer pool to use GC.AllocateArray * fix debug assert * Make DisposeSession protected in FasterServerBase. * Added SetBytesRead to IServerSession * - update sample .csprojs to target net6 and remove platform-related conditionals from all .csprojs now that we've moved to VS 2022 (#687) - add Status.Expired - update FASTER.client's copies of Status(Code).cs - Replace a TODO with an explanatory comment * Slight refactor of network stack. * Use GC.AllocateArray in SeaaBuffer * Another GC.AllocateArray substitution * [C#] [WIP] Wire disposal into IFunctions v2 (#685) * Add hook for failed CAS during SingleWriter. Still need to support other failed CAS scenarios. * Implement Dispose() functions - (MemoryLogCompactionTests fails at this time) * Fix MemoryLogCompactionTests; move DisposeDeserializedFromDisk up to ICPRFC Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> Co-authored-by: Ted Hart <15467143+TedHartMS@users.noreply.github.com> Co-authored-by: Tianyu Li <litianyu@mit.edu> Co-authored-by: Vasileios Zois <vazois@microsoft.com> Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com> Co-authored-by: rohankadekodi-msr <69916400+rohankadekodi-msr@users.noreply.github.com> Co-authored-by: Rohan Kadekodi <kadekodirohan@gmail.com> |
|
|
Ted Hart
|
8c0c6e1fd5
|
[C#] Update to C#9 (#486)
* Add <LangVersion>latest to csprojs to pick up C#9; remove "is {}" in favor of "is not null"; use "Type x = new()", which is particularly useful for field definitions
* Add net5.0 framework to core component builds; add "|| NET" to "#if NETSTANDARD" where necessary; switch samples to net5.0; add Debug builds to Linux
* Update azure-pipelines.yml
* Move C# Windows tasks to dotnet; add build/test tasks for Remote to C# pipelines
* search for all *.test.csproj in build/test tasks
* Restrict Linux to netcoreapp3.1 and net5.0
* remove duplicate langversion
Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
|
|
|
Ted Hart
|
09445f37b6
|
add FASTER.benchmark cmdline options: --sd --sm --noaff --chkptms --dumpdist (#485) | |
|
Badrish Chandramouli
|
e9671dc54e | [C#] minor cleanup | |
|
Badrish Chandramouli
|
ac06ea912d
|
YCSB improvements to test incr-snapshot (#454) | |
|
Badrish Chandramouli
|
ebb5e783df
|
[C#] Improve wait handling when tail insertion triggers a flush (#446)
* WIP on UpsertAsync and AllocateAsync * Fixed async code for upsert * Updated FasterLog to use new allocation API * removed NeedToWait * Added stress test based on Parallel.For * speed up stress test * remove concurrentdict for tasks * edit text * edit output text * add option to enable OS file buffering in MLSD * Add scripts for benchmark run/comparison; fix kUseSmallData Uniform subset loading; modify AsyncStress to call MLSD with buffering and take cmdline args for single-threading * Ensure last epoch suspender drains all pending actions. * Revert incidental change to LightEpoch, for clarity. * ProtectDrain in ephemeral loop to ensure trigger of flush action. * Add commit thread to testcase * Remove lock around filestream ops * Revert "Remove lock around filestream ops" * Added test to playground * Alternate commit strategy using LIFO work queue for FasterLog * FlushAsync * Fixing commit issue * Rename playground project to avoid clash on Test name. * Clean up AsyncStress: - Add commandline args for threading mode, task count, number of ops, option to enable OS read buffering, usage message - Use ValueTask instead of Task - Add chunked operations - Add option for sync vs. async lambdas for Parallel.For - Repoort TailAddress and pending upsert count Update .gitignore to ignore LaunchSettings.json anywhere Fix size setting in AsyncPool ctor * Add fast path to BlockAllocate. * Use deleteOnClose by default for hlog * Ensure FASTER.benchmark -k checkpoints are snapshot; use session pool in AsyncStress Chunk methods; make BlockAllocate fast path inlined * Add session.CompletePending overload that returns a CompletedOutputsd; use this in FasterWrapper; add UTs for it and modify TestTypes to add AdvancedFunctions and variations of this and Functions that take a context * Added try/catch around work invocation. * Change to CompletedOutputEnumerator for speed and move it outside of FasterKV to its own file * Add RecordInfo and address to CompletedOutput; add CloneAndBuild param comment in run_benchmark.ps1; update docs; add docs folder to .sln * add session.CompletePendingWithOutputsAsync(); * Replace SpinWait with semaphore wait in InternalCompletePending * Cleanup, make CompletePendingAsync similar to CompletePending. * Add CompletePendingWithOutputs() as separate call * FasterWrapper made generic Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> |
|
|
Ted Hart
|
7107010783
|
[C#] Add locking methods to Functions (#428)
* Add locking methods to Functions; add FASTER.benchmark: -z, --locking option to exercise * Revise SpanByteFunctions and MemoryFunctions to use IFunctions.Lock(); change Lock() param name to lockContext * Change Lock/Unlock implementations to remove "if (locking)" because FASTER will only call if SupportsLocking is true; use inheritdoc in FunctionsBase * Missed one ":if (locking)" * update docs to include information on Locking and ConcurrentDeleter * Make ConcurrentDeleter void and always do default handling * Update 20-fasterkv-basics.md Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> |
|
|
Ted Hart
|
8530a71c81
|
[C#] Performance benchmark refactor (#424)
* Refactor YCSB benchmark: - centralize loading and stats-line generation - move options out of program.cs and add parameter validation - move constants to a central file - output more detailed, parsable and readable stats - add synthetic zipf generation - move -k (backup) output to subdirs specific to distribution, ycsb vs. synthetic data, and sizes * YCSB: change --backup to --recover and remove BackupMode; add --runsec * ycsb commandline tweaks |
|
|
Ted Hart
|
a27f0b56c8
|
[C#] Clean up TargetFramework defines, enable IAsyncEnumerable on netstandard2.0/net461 (#377)
* Clean up targetframework stuff: - Change the order in the csproj so netstandard2.1 is first, which appears to be the one that's used by the IDE for #if <TargetFramework> resolution - Switch to the TargetFramework preprocessor symbols already defined by the build system (we use NETSTANDARD and NETSTANDARD2_1) - Add Microsoft.Bcl.AsyncInterfaces for pre-netstandard2.1 frameworks, which allows IAsyncEnumerable to work - Added tests for FasterLogIterator.GetAsyncEnumerable() - Enable netcore2.2 in FASTER.tests.csproj * Fix DeviceFasterLogTests hang with IAsyncEnumerator; add tests for FasterLogIterator.GetAsyncEnumerable(MemoryPool<T>) * Update azure-pipelines.yml Put UseDotNet@2 task in the right place * Drop FASTER.test netcoreapp v2.2 to 2.1 (2.2 isn't needed and 2.1 is on the pipeline by default) * Add VSTest2 pipeline task for Linux * use DotNetCoreCLI@2 task to run Linux tests * Specify framework version on C# Linux dotnetcorecli task * Replace directory separator in tests with front slash (/) for Linux compatibility. * Fixed SharedDirectoryTests to work on Linux * change !NET461 to NETCOREAPP; restore previous LocalStorageDevice GetSectorSize behavior Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> |
|
|
Badrish Chandramouli
|
897d6a82b9 | [C#] Benchmark corner case fixes | |
|
Badrish Chandramouli
|
a0209aaece
|
[C#] SpanByte - a safe wrapper for variable-length keys and values (#349)
* Initial checkin of SpanByte to make it easy to safely use inline variable length keys and values. * Improved equality * Added extensions * Update FasterKV.md * Support variable length Input type * Handle varlen inputs * Handle varlen outputs in sync and async code paths via new SpanByteAndMemory Output type * updates + testcase * Converted SpanByte to a proper stack type, similar to Span<byte>, but operating over fixed data + works with generics. * Added benchmark |