Will backport this to the 3.0 branch too.
This also updates Newtonsoft which is a product dependency - but only
updates to the latest servicing release which we can do as a servicing
release ourselves.
* Adding needed Tokenizer's APIs
* Address the feedback
* Small update to the newly exposed APIs
* fix comments
* Update the APIs signatures
* More feedback addressing
* Fix the comments
* Packaging cleanup
Originally I was just trying to remove mentions of snupkg, but then
things got a bit carried away. :)
This is trying to remove as much duplication and dead code related to
packaging that I can.
* Apply code review feedback
* Suppress copying indirect references
* Remove unwanted bundled files from AutoML
* Remove leading slash
* Refactor model download
* Correct the packaging path of native symbols
* Rename NoTargets projects from csproj to proj
* Fix build issues around model download and respond to feedback
* Remove NoTargets file extension enforcement
* Rename proj to CSProj, include in SLN
I'd like to ensure all our projects are included in the SLN and don't
rely on separate build steps.
VS prefers *.csproj in the sln so I renamed things back to csproj.
* Respond to PR feedback
* Fix cache when calling EncodeToIds
* Make EnglishRoberta _mergeRanks thread safe
* Delete Trainer
* Remove the setters on the Bpe properties
* Remove Roberta and Tiktoken special casing in the Tokenizer and support the cases in the Model abstraction
* Support text-embedding-3-small/large embedding
* Remove redundant TokenToId abstraction and keep the one with the extra parameters
* Enable creating Tiktoken asynchronously or directly using the tokenizer data
* Add cancellationToken support in CreateAsync APIs
* Rename sequence to text and Tokenize to Encode
* Rename skipSpecialTokens to considerSpecialTokens
* Rename TokenizerResult to EncodingResult
* Make Token publicly immutable
* Change offset tuples from (Index, End) to (Index, Length)
* Rename NormalizedString method's parameters
* Rename Model's methods to start with verb
* Convert Model.GetVocab() method to a Vocab property
* Some method's parameters and variable renaming
* Remove Vocab and VocabSize from the abstraction
* Cleanup normalization support
* Minor Bpe cleanup
* Resolve rebase change
* Address the feedback
* Split out non concurrent test collections.
* Fix Core Tests
* removed torchsharp and tensorflow tests when not x64
* fixes from pr comments
* removing unnecessary line from the proj file
* Update dependencies from https://github.com/dotnet/arcade build 20230519.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23269.2
* Add dotnet8 nuget feed
* Update dependencies from https://github.com/dotnet/arcade build 20230529.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23279.1
* Update dependencies from https://github.com/dotnet/arcade build 20230602.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23302.3
* Update dependencies from https://github.com/dotnet/arcade build 20230609.8
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23309.8
* Update dependencies from https://github.com/dotnet/arcade build 20230616.6
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23316.6
* Update dependencies from https://github.com/dotnet/arcade build 20230622.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23322.2
* Update dependencies from https://github.com/dotnet/arcade build 20230630.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23330.1
* Update dependencies from https://github.com/dotnet/arcade build 20230710.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23360.1
* Update dependencies from https://github.com/dotnet/arcade build 20230714.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23364.2
* Update dependencies from https://github.com/dotnet/arcade build 20230721.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23371.1
* Update dependencies from https://github.com/dotnet/arcade build 20230728.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23378.2
* Update dependencies from https://github.com/dotnet/arcade build 20230804.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23404.2
* Update dependencies from https://github.com/dotnet/arcade build 20230811.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23411.1
* Update dependencies from https://github.com/dotnet/arcade build 20230819.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23419.1
* Update dependencies from https://github.com/dotnet/arcade build 20230825.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23425.2
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20231008.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23508.1
* Update dependencies from https://github.com/dotnet/arcade build 20231010.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23510.4
* Update dependencies from https://github.com/dotnet/arcade build 20231018.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23518.2
* Update dependencies from https://github.com/dotnet/arcade build 20231028.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23528.2
* Update dependencies from https://github.com/dotnet/arcade build 20231103.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23553.1
* Update dependencies from https://github.com/dotnet/arcade build 20231110.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23560.1
* Update dependencies from https://github.com/dotnet/arcade build 20231117.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23567.1
* Fixed version update breaks.
* Update dependencies from https://github.com/dotnet/arcade build 20231122.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23572.2
* Update dependencies from https://github.com/dotnet/arcade build 20231201.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23601.1
* Update dependencies from https://github.com/dotnet/arcade build 20231207.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23607.2
* Update dependencies from https://github.com/dotnet/arcade build 20231215.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23615.2
* Update XUnitVersion
* Update MicrosoftMLOnnxRuntimeVersion to 1.16.3
* Rollback OnnxRuntime and suppress warning
* Update to Xunit with fix for https://github.com/xunit/xunit/issues/2821
* Ensure we pull down 8.0 runtime.
* Update Centos docker containers
* Fix packaging step
* Try including stdint.h to fix missing uint8_t on centos
* Update Centos test queue
* Attempt to use runtime centos-stream8-helix container for tests
* Use centos-stream8-mlnet-helix container for testing
* Undo changes to test data
* Make NETFRAMEWORK ifdef versionless
* Switch back to centos7 for testing
* Revert "Switch back to centos7 for testing"
This reverts commit ab0d41e4b7.
* Update dependencies from https://github.com/dotnet/arcade build 20231221.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23621.2
* Update dependencies from https://github.com/dotnet/arcade build 20231228.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23628.1
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Michael Sharp <misharp@microsoft.com>
* added in win-arm64
* fixed MKL arm64 cmake issue
* right helix queue
* added in win-arm64
* fixed MKL arm64 cmake issue
* right helix queue
* fixing arm tests
* makes x64 test detection better
* change test label
* fixed onnx files not being included
* added new win-arm baselines
* baseline changes
* fixed build issue
* fixed test
* one more basleine
* fixed pack for mkl redist
* .NET update
* updated lightGBM version
* finished LightGBM baseline update
* removed accidental enums
* Fixed LightGBM Test
* fixed lightgbm
* fixed version
* fixed test with LightGBM
* fixed test result
* using tensor primitives
* added missing files
* some with indexes changed
* Initial swap for TensorPrimitives done
* Rebased and cleaned code
* more minor cleanup
* added system.numerics.tensors version to props
* build fixes
* added net6 again
* updates from PR comments
* fixed sumabsu
* fixed baseline tests
* test fixes
* fixed test failure for kmeans
* changed decimal comparison
* updated more baselines
* Test fixes.
* template update
* Test Fixes.
* fixed performance test csproj
* added baselines for linux arm/64
* fixed linux arm baselines
* fixed arm baselines
* removed extra files
* arm32 baselines updated
* fixed arm baselines
* Fixes incorrect work of DataFrame with VBufferColumn when number of elements is greater than Int.MaxValue
* Fix calculation of max capacity and amount of required buffers
* Fix unit test
* Run test allocating more than 2 Gb of memory on 64bit env only
* Fix StringDataFrameColumn same way as VBufferDataFrameColumn
* Fix wrong amount of buffers created in constructor of StringDataFrameColumn
* Fix code review findings
* Split Test for AppendMany into 4 different tests
* Block init of null validity buffer instead of setting individual bits
* Add unit tests for PrimitiveDataFrameColumn.Clone
* Fixes#6821
* Fix
* Fix bug with AppendMany values to not empty column
* Restart unit tests
* Add more unit tests
* Fix failing unit test
* Fix code review findings
* Add performance tests
* Add extra tests
* Fix
* Fix typo
* Fix Divide_Int16 and Divide_Int32_Int16 benchmarks
* Fix
* Change csproj file
* Update BenchmarkDotNetVersion to 0.13.5
* Fix
* Change to 0.13.1 because that is what is latest version in our nuget feeds.
---------
Co-authored-by: Jake Radzikowski <JakeRad@Microsoft.com>
* Fix wrong type conversion on PrimitiveDataFrameColumn
* Added tests for #6829
* Fix test
* Add file generated from tt template and fix unit tests
---------
Co-authored-by: Aleksei Smirnov <tlalok@inbox.ru>
* Fix inconsistent null handling in DataFrame Arithmetics
* Fix Null Count and division by zero issues
* Minor changes to restart build and rerun flaky tests
* Fix error with allocating more than MaxCapacity of Byte Memory Buffer
* Remove Unit test as it consumes too much memory
* Fix issue with increasing buffer capacity over limit when double it size
Michael Sharp
Eric StJohn
Tarek Mahmoud Sayed
dotnet-maestro[bot]
Aleksei Smirnov
Xiaoyun Zhang
Raffaello Fraboni
zewditu Hailemariam
Lehonti Ramos