Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.24159.1 -> To Version 9.0.0-beta.24165.3
Will backport this to the 3.0 branch too.
This also updates Newtonsoft which is a product dependency - but only
updates to the latest servicing release which we can do as a servicing
release ourselves.
* Adding needed Tokenizer's APIs
* Address the feedback
* Small update to the newly exposed APIs
* fix comments
* Update the APIs signatures
* More feedback addressing
* Fix the comments
* Packaging cleanup
Originally I was just trying to remove mentions of snupkg, but then
things got a bit carried away. :)
This is trying to remove as much duplication and dead code related to
packaging that I can.
* Apply code review feedback
* Suppress copying indirect references
* Remove unwanted bundled files from AutoML
* Remove leading slash
* Refactor model download
* Correct the packaging path of native symbols
* Rename NoTargets projects from csproj to proj
* Fix build issues around model download and respond to feedback
* Remove NoTargets file extension enforcement
* Rename proj to CSProj, include in SLN
I'd like to ensure all our projects are included in the SLN and don't
rely on separate build steps.
VS prefers *.csproj in the sln so I renamed things back to csproj.
* Respond to PR feedback
* Fix cache when calling EncodeToIds
* Make EnglishRoberta _mergeRanks thread safe
* Delete Trainer
* Remove the setters on the Bpe properties
* Remove Roberta and Tiktoken special casing in the Tokenizer and support the cases in the Model abstraction
* Support text-embedding-3-small/large embedding
* Remove redundant TokenToId abstraction and keep the one with the extra parameters
* Enable creating Tiktoken asynchronously or directly using the tokenizer data
* Add cancellationToken support in CreateAsync APIs
* Rename sequence to text and Tokenize to Encode
* Rename skipSpecialTokens to considerSpecialTokens
* Rename TokenizerResult to EncodingResult
* Make Token publicly immutable
* Change offset tuples from (Index, End) to (Index, Length)
* Rename NormalizedString method's parameters
* Rename Model's methods to start with verb
* Convert Model.GetVocab() method to a Vocab property
* Some method's parameters and variable renaming
* Remove Vocab and VocabSize from the abstraction
* Cleanup normalization support
* Minor Bpe cleanup
* Resolve rebase change
* Address the feedback
- Add a CancellationToken to CreateByModelNameAsync, allowing the download and parsing to be canceled.
- Use ReadLineAsync(cancellationToken), which not only allows it to be canceled, but avoids ~100K task allocations
- Fix Helpers.FromBase64String to support lines longer than 300 chars
* Add prIssueManagement.yml to onboard repo to GitOps.ResourceManagement as FabricBot replacement
Details on the replacement service and the syntax of the new yaml configuration file is available publicly at: https://microsoft.github.io/GitOps/policies/resource-management.html
Please review and merge this PR to complete the process of onboarding to the new service.
* Deleting fabricbot.json
* De-Dupe the Write permission check
* Add automation for adding/removing the untriaged label to issues
---------
Co-authored-by: dotnet-policy-service[bot] <123482357+dotnet-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
* Add new type to key-value
* Make it IDataView
* clean
* add dataview support in source generator
* Update type_converter_search_space.json
* Update search-space-schema.json
---------
Co-authored-by: XiaoYun Zhang <xiaoyuz@microsoft.com>
Co-authored-by: Xiaoyun Zhang <bigmiao.zhang@gmail.com>
* Split out non concurrent test collections.
* Fix Core Tests
* removed torchsharp and tensorflow tests when not x64
* fixes from pr comments
* removing unnecessary line from the proj file
* Update dependencies from https://github.com/dotnet/arcade build 20230519.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23269.2
* Add dotnet8 nuget feed
* Update dependencies from https://github.com/dotnet/arcade build 20230529.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23279.1
* Update dependencies from https://github.com/dotnet/arcade build 20230602.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23302.3
* Update dependencies from https://github.com/dotnet/arcade build 20230609.8
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23309.8
* Update dependencies from https://github.com/dotnet/arcade build 20230616.6
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23316.6
* Update dependencies from https://github.com/dotnet/arcade build 20230622.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23322.2
* Update dependencies from https://github.com/dotnet/arcade build 20230630.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23330.1
* Update dependencies from https://github.com/dotnet/arcade build 20230710.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23360.1
* Update dependencies from https://github.com/dotnet/arcade build 20230714.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23364.2
* Update dependencies from https://github.com/dotnet/arcade build 20230721.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23371.1
* Update dependencies from https://github.com/dotnet/arcade build 20230728.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23378.2
* Update dependencies from https://github.com/dotnet/arcade build 20230804.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23404.2
* Update dependencies from https://github.com/dotnet/arcade build 20230811.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23411.1
* Update dependencies from https://github.com/dotnet/arcade build 20230819.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23419.1
* Update dependencies from https://github.com/dotnet/arcade build 20230825.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23425.2
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20231008.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23508.1
* Update dependencies from https://github.com/dotnet/arcade build 20231010.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23510.4
* Update dependencies from https://github.com/dotnet/arcade build 20231018.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23518.2
* Update dependencies from https://github.com/dotnet/arcade build 20231028.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23528.2
* Update dependencies from https://github.com/dotnet/arcade build 20231103.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23553.1
* Update dependencies from https://github.com/dotnet/arcade build 20231110.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23560.1
* Update dependencies from https://github.com/dotnet/arcade build 20231117.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23567.1
* Fixed version update breaks.
* Update dependencies from https://github.com/dotnet/arcade build 20231122.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23572.2
* Update dependencies from https://github.com/dotnet/arcade build 20231201.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23601.1
* Update dependencies from https://github.com/dotnet/arcade build 20231207.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23607.2
* Update dependencies from https://github.com/dotnet/arcade build 20231215.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23615.2
* Update XUnitVersion
* Update MicrosoftMLOnnxRuntimeVersion to 1.16.3
* Rollback OnnxRuntime and suppress warning
* Update to Xunit with fix for https://github.com/xunit/xunit/issues/2821
* Ensure we pull down 8.0 runtime.
* Update Centos docker containers
* Fix packaging step
* Try including stdint.h to fix missing uint8_t on centos
* Update Centos test queue
* Attempt to use runtime centos-stream8-helix container for tests
* Use centos-stream8-mlnet-helix container for testing
* Undo changes to test data
* Make NETFRAMEWORK ifdef versionless
* Switch back to centos7 for testing
* Revert "Switch back to centos7 for testing"
This reverts commit ab0d41e4b7.
* Update dependencies from https://github.com/dotnet/arcade build 20231221.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23621.2
* Update dependencies from https://github.com/dotnet/arcade build 20231228.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23628.1
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Michael Sharp <misharp@microsoft.com>
Asserting on `_rowCount < Utils.Size(_valueBoundaries)` was catching a
case where `_rowCount`'s update was reordered before `_valueBoundaries`
This was unnecessary, since this method doesn't need to use `_rowCount`.
Instead, make the asserts use only `idx` which will be maintained
consistent with the waiter logic in this cache.
Ensure we only ever use `_rowCount` from the caching thread, so write
reordering won't matter.
* Don't include the SDK in our helix payload
I noticed that the tests included the latest SDK - including the host -
in our helix payloads.
This is a large amount of unnecessary downloads and it also makes it so
we use the latest host on the older frameworks which can fail when the
latest host drops support for distros.
Since our tests shouldn't need the full CLI, remove this from our helix
payloads.
We'll instead get just the runtime we need through `AdditionalDotNetPackage`
* Place Helix downloaded runtime on the PATH
Helix only sets the path when the CLI is included, however we don't
need the CLI.
dotnet-maestro[bot]
Eric StJohn
Michael Sharp
dotnet-maestro[bot]
Jo Shields
Tarek Mahmoud Sayed
Stephen Toub
zewditu Hailemariam
dotnet-policy-service[bot]
Xiaoyun Zhang