* Fix cache when calling EncodeToIds
* Make EnglishRoberta _mergeRanks thread safe
* Delete Trainer
* Remove the setters on the Bpe properties
* Remove Roberta and Tiktoken special casing in the Tokenizer and support the cases in the Model abstraction
* Support text-embedding-3-small/large embedding
* Remove redundant TokenToId abstraction and keep the one with the extra parameters
* Enable creating Tiktoken asynchronously or directly using the tokenizer data
* Add cancellationToken support in CreateAsync APIs
* Rename sequence to text and Tokenize to Encode
* Rename skipSpecialTokens to considerSpecialTokens
* Rename TokenizerResult to EncodingResult
* Make Token publicly immutable
* Change offset tuples from (Index, End) to (Index, Length)
* Rename NormalizedString method's parameters
* Rename Model's methods to start with verb
* Convert Model.GetVocab() method to a Vocab property
* Some method's parameters and variable renaming
* Remove Vocab and VocabSize from the abstraction
* Cleanup normalization support
* Minor Bpe cleanup
* Resolve rebase change
* Address the feedback
- Add a CancellationToken to CreateByModelNameAsync, allowing the download and parsing to be canceled.
- Use ReadLineAsync(cancellationToken), which not only allows it to be canceled, but avoids ~100K task allocations
- Fix Helpers.FromBase64String to support lines longer than 300 chars
* Add prIssueManagement.yml to onboard repo to GitOps.ResourceManagement as FabricBot replacement
Details on the replacement service and the syntax of the new yaml configuration file is available publicly at: https://microsoft.github.io/GitOps/policies/resource-management.html
Please review and merge this PR to complete the process of onboarding to the new service.
* Deleting fabricbot.json
* De-Dupe the Write permission check
* Add automation for adding/removing the untriaged label to issues
---------
Co-authored-by: dotnet-policy-service[bot] <123482357+dotnet-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
* Add new type to key-value
* Make it IDataView
* clean
* add dataview support in source generator
* Update type_converter_search_space.json
* Update search-space-schema.json
---------
Co-authored-by: XiaoYun Zhang <xiaoyuz@microsoft.com>
Co-authored-by: Xiaoyun Zhang <bigmiao.zhang@gmail.com>
* Split out non concurrent test collections.
* Fix Core Tests
* removed torchsharp and tensorflow tests when not x64
* fixes from pr comments
* removing unnecessary line from the proj file
* Update dependencies from https://github.com/dotnet/arcade build 20230519.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23269.2
* Add dotnet8 nuget feed
* Update dependencies from https://github.com/dotnet/arcade build 20230529.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23279.1
* Update dependencies from https://github.com/dotnet/arcade build 20230602.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23302.3
* Update dependencies from https://github.com/dotnet/arcade build 20230609.8
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23309.8
* Update dependencies from https://github.com/dotnet/arcade build 20230616.6
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23316.6
* Update dependencies from https://github.com/dotnet/arcade build 20230622.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23322.2
* Update dependencies from https://github.com/dotnet/arcade build 20230630.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23330.1
* Update dependencies from https://github.com/dotnet/arcade build 20230710.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23360.1
* Update dependencies from https://github.com/dotnet/arcade build 20230714.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23364.2
* Update dependencies from https://github.com/dotnet/arcade build 20230721.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23371.1
* Update dependencies from https://github.com/dotnet/arcade build 20230728.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23378.2
* Update dependencies from https://github.com/dotnet/arcade build 20230804.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23404.2
* Update dependencies from https://github.com/dotnet/arcade build 20230811.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23411.1
* Update dependencies from https://github.com/dotnet/arcade build 20230819.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23419.1
* Update dependencies from https://github.com/dotnet/arcade build 20230825.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23425.2
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20231008.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23508.1
* Update dependencies from https://github.com/dotnet/arcade build 20231010.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23510.4
* Update dependencies from https://github.com/dotnet/arcade build 20231018.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23518.2
* Update dependencies from https://github.com/dotnet/arcade build 20231028.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23528.2
* Update dependencies from https://github.com/dotnet/arcade build 20231103.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23553.1
* Update dependencies from https://github.com/dotnet/arcade build 20231110.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23560.1
* Update dependencies from https://github.com/dotnet/arcade build 20231117.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23567.1
* Fixed version update breaks.
* Update dependencies from https://github.com/dotnet/arcade build 20231122.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23572.2
* Update dependencies from https://github.com/dotnet/arcade build 20231201.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23601.1
* Update dependencies from https://github.com/dotnet/arcade build 20231207.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23607.2
* Update dependencies from https://github.com/dotnet/arcade build 20231215.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23615.2
* Update XUnitVersion
* Update MicrosoftMLOnnxRuntimeVersion to 1.16.3
* Rollback OnnxRuntime and suppress warning
* Update to Xunit with fix for https://github.com/xunit/xunit/issues/2821
* Ensure we pull down 8.0 runtime.
* Update Centos docker containers
* Fix packaging step
* Try including stdint.h to fix missing uint8_t on centos
* Update Centos test queue
* Attempt to use runtime centos-stream8-helix container for tests
* Use centos-stream8-mlnet-helix container for testing
* Undo changes to test data
* Make NETFRAMEWORK ifdef versionless
* Switch back to centos7 for testing
* Revert "Switch back to centos7 for testing"
This reverts commit ab0d41e4b7.
* Update dependencies from https://github.com/dotnet/arcade build 20231221.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23621.2
* Update dependencies from https://github.com/dotnet/arcade build 20231228.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23628.1
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Michael Sharp <misharp@microsoft.com>
Asserting on `_rowCount < Utils.Size(_valueBoundaries)` was catching a
case where `_rowCount`'s update was reordered before `_valueBoundaries`
This was unnecessary, since this method doesn't need to use `_rowCount`.
Instead, make the asserts use only `idx` which will be maintained
consistent with the waiter logic in this cache.
Ensure we only ever use `_rowCount` from the caching thread, so write
reordering won't matter.
* Don't include the SDK in our helix payload
I noticed that the tests included the latest SDK - including the host -
in our helix payloads.
This is a large amount of unnecessary downloads and it also makes it so
we use the latest host on the older frameworks which can fail when the
latest host drops support for distros.
Since our tests shouldn't need the full CLI, remove this from our helix
payloads.
We'll instead get just the runtime we need through `AdditionalDotNetPackage`
* Place Helix downloaded runtime on the PATH
Helix only sets the path when the CLI is included, however we don't
need the CLI.
* added in win-arm64
* fixed MKL arm64 cmake issue
* right helix queue
* added in win-arm64
* fixed MKL arm64 cmake issue
* right helix queue
* fixing arm tests
* makes x64 test detection better
* change test label
* fixed onnx files not being included
* added new win-arm baselines
* baseline changes
* fixed build issue
* fixed test
* one more basleine
* fixed pack for mkl redist
* .NET update
* updated lightGBM version
* finished LightGBM baseline update
* removed accidental enums
* Fixed LightGBM Test
* fixed lightgbm
* fixed version
* fixed test with LightGBM
* fixed test result
* using tensor primitives
* added missing files
* some with indexes changed
* Initial swap for TensorPrimitives done
* Rebased and cleaned code
* more minor cleanup
* added system.numerics.tensors version to props
* build fixes
* added net6 again
* updates from PR comments
* fixed sumabsu
* fixed baseline tests
* test fixes
* fixed test failure for kmeans
* changed decimal comparison
* updated more baselines
* Test fixes.
* template update
* Test Fixes.
* fixed performance test csproj
* added baselines for linux arm/64
* fixed linux arm baselines
* fixed arm baselines
* removed extra files
* arm32 baselines updated
* fixed arm baselines
Tarek Mahmoud Sayed
Stephen Toub
Michael Sharp
zewditu Hailemariam
dotnet-policy-service[bot]
dotnet-maestro[bot]
Eric StJohn
Xiaoyun Zhang
Aleksei Smirnov
The Algorithms Guy
JJ
taeerhebend
Jeff Handley
Akash Kundu