* add llama
* add test for tokenizer
* make llama 3.1 working
* update
* add shape test for 70b and 405b
* clean up
* add tests
* update
* fix error
* calculate rotary embedding in model layer
* remove rotary_emb from attention
* update feed
* update .csproj
* Update NuGet.config
* fix test
* pass device
* fix test
* update constructor
* disable 405b test
* update
* disable 70b test
* use windows only fact
* revert change
* rename test to LLaMA3_1
* add genai.phi and tests
* formatter
* refactor Phi3Tokenizer
* update
* add configuration for phi-series
* add semantic kernel and autogen intergration
* update
* add Microsoft.ML.GenAI.Sample
* use tokenzier model from testTokenizer package
* use defaults
* add quantize linear
* use version string
* remove special token from CreatePhi2 API
* set up quantize sample
* initialize linear with zeros
* update sample
* add 6.0 to targetframework
* fix tests
* update
* remove Phi3Tokenizer and use LlamaTokenizer instead
* revert change in tokenizer package
* run test on x64
* fix tests
* check in approved file
* run test in net6.0
* use meta device
* copy approval tests to output folder
* set up approval test file location
* fix comment
* rename to AddGenAITextGeneration and AddGenAIChatCompletion
* Update job-template.yml
* add mit license
* add reference
* bump code coverage version
* add <PreserveCompilationContext>true</PreserveCompilationContext>
* add runtime package
* remove flag
* add flag
* fix build error
* update
* update
* minor formatting issues, use preview 5, and package readmes
* fixing async void test
* test fixes
* added in ReadExactly extension to stream
* fixes from PR comments
* added types to fast tree
* fixed overflow
* fixed package md file
* Allow developers to supply their own GuessType function
* Add a test for using a custom GuessType function.
* Fix typo in string resource identifier
* Convert 0-based line number to 1-based line number in error message.
* Add test that FormatException is thrown when one row in a data frame has less columns than the others
* Revert "Fix typo in string resource identifier"
This reverts commit 5a2ee30a63.
# Conflicts:
# src/Microsoft.Data.Analysis/DataFrame.IO.cs
* Add a column filled with null to the test data frame to improve test coverage.
* Make most Tokenizer abstract methods virtual
All of the functionality in all of the methods can be implemented in terms of just a single Decode and EncodeToTokens set of methods. Only those two need to be abstract; everything else that was abstract can instead be virtual and implemented in terms of those.
* Address feedback and clean up a few things
* Add dependency to the latest System.Numerics.Tensors library
* Initial implementation of NumericArithmetic for net8 and TensorPrimitives
* Use CompositeArithmetic
* Refcatoring, using .net8 features to improve performance and code readability
* Remove needless changes
* Fix build
* Update System.Numerics.Tensors lib
* Introducing CodeGen Tokenizer
* Mark a method as private. It was not intended to be public
* Init vocab atomically.
* Prevent returning tokens that are only partially mapped to a code point.
* Ensure Tiktoken precise token's count with IndexOf & LastIndexOf. Ensure accurate offsets too.
* Address the feedback
* Embed Tiktoken data files
* Address the feedback
* Address the feedback
* More feedback addressing
* More feedback addressing
* More feedback
* Feedback++
* Fix merge conflict
* Remove teh confusing specialTokens flag parameter from all APIs
* Normalize casing of the Tiktoken name
* Make Model.Encode work with span instead of string
* Support granular Last/IndexOf.
* Remove wrong assert.
* Address the feedback
* Update the package doc.
* Add Tiktoken Synchronous Creation Using Model Name
* Add RemoteExecutor to Tokenizers tests
* Address the feedback
* Add tests
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Will backport this to the 3.0 branch too.
This also updates Newtonsoft which is a product dependency - but only
updates to the latest servicing release which we can do as a servicing
release ourselves.
* Adding needed Tokenizer's APIs
* Address the feedback
* Small update to the newly exposed APIs
* fix comments
* Update the APIs signatures
* More feedback addressing
* Fix the comments
* Packaging cleanup
Originally I was just trying to remove mentions of snupkg, but then
things got a bit carried away. :)
This is trying to remove as much duplication and dead code related to
packaging that I can.
* Apply code review feedback
* Suppress copying indirect references
* Remove unwanted bundled files from AutoML
* Remove leading slash
* Refactor model download
* Correct the packaging path of native symbols
* Rename NoTargets projects from csproj to proj
* Fix build issues around model download and respond to feedback
* Remove NoTargets file extension enforcement
* Rename proj to CSProj, include in SLN
I'd like to ensure all our projects are included in the SLN and don't
rely on separate build steps.
VS prefers *.csproj in the sln so I renamed things back to csproj.
* Respond to PR feedback
* Fix cache when calling EncodeToIds
* Make EnglishRoberta _mergeRanks thread safe
* Delete Trainer
* Remove the setters on the Bpe properties
* Remove Roberta and Tiktoken special casing in the Tokenizer and support the cases in the Model abstraction
* Support text-embedding-3-small/large embedding
* Remove redundant TokenToId abstraction and keep the one with the extra parameters
* Enable creating Tiktoken asynchronously or directly using the tokenizer data
* Add cancellationToken support in CreateAsync APIs
* Rename sequence to text and Tokenize to Encode
* Rename skipSpecialTokens to considerSpecialTokens
* Rename TokenizerResult to EncodingResult
* Make Token publicly immutable
* Change offset tuples from (Index, End) to (Index, Length)
* Rename NormalizedString method's parameters
* Rename Model's methods to start with verb
* Convert Model.GetVocab() method to a Vocab property
* Some method's parameters and variable renaming
* Remove Vocab and VocabSize from the abstraction
* Cleanup normalization support
* Minor Bpe cleanup
* Resolve rebase change
* Address the feedback
* Split out non concurrent test collections.
* Fix Core Tests
* removed torchsharp and tensorflow tests when not x64
* fixes from pr comments
* removing unnecessary line from the proj file
* Update dependencies from https://github.com/dotnet/arcade build 20230519.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23269.2
* Add dotnet8 nuget feed
* Update dependencies from https://github.com/dotnet/arcade build 20230529.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23279.1
* Update dependencies from https://github.com/dotnet/arcade build 20230602.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23302.3
* Update dependencies from https://github.com/dotnet/arcade build 20230609.8
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23309.8
* Update dependencies from https://github.com/dotnet/arcade build 20230616.6
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23316.6
* Update dependencies from https://github.com/dotnet/arcade build 20230622.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23322.2
* Update dependencies from https://github.com/dotnet/arcade build 20230630.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23330.1
* Update dependencies from https://github.com/dotnet/arcade build 20230710.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23360.1
* Update dependencies from https://github.com/dotnet/arcade build 20230714.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23364.2
* Update dependencies from https://github.com/dotnet/arcade build 20230721.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23371.1
* Update dependencies from https://github.com/dotnet/arcade build 20230728.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23378.2
* Update dependencies from https://github.com/dotnet/arcade build 20230804.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23404.2
* Update dependencies from https://github.com/dotnet/arcade build 20230811.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23411.1
* Update dependencies from https://github.com/dotnet/arcade build 20230819.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23419.1
* Update dependencies from https://github.com/dotnet/arcade build 20230825.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23425.2
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230901.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23451.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20230913.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23463.1
* Update dependencies from https://github.com/dotnet/arcade build 20231008.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23508.1
* Update dependencies from https://github.com/dotnet/arcade build 20231010.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23510.4
* Update dependencies from https://github.com/dotnet/arcade build 20231018.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23518.2
* Update dependencies from https://github.com/dotnet/arcade build 20231028.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23528.2
* Update dependencies from https://github.com/dotnet/arcade build 20231103.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23553.1
* Update dependencies from https://github.com/dotnet/arcade build 20231110.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23560.1
* Update dependencies from https://github.com/dotnet/arcade build 20231117.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23567.1
* Fixed version update breaks.
* Update dependencies from https://github.com/dotnet/arcade build 20231122.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23572.2
* Update dependencies from https://github.com/dotnet/arcade build 20231201.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23601.1
* Update dependencies from https://github.com/dotnet/arcade build 20231207.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23607.2
* Update dependencies from https://github.com/dotnet/arcade build 20231215.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23615.2
* Update XUnitVersion
* Update MicrosoftMLOnnxRuntimeVersion to 1.16.3
* Rollback OnnxRuntime and suppress warning
* Update to Xunit with fix for https://github.com/xunit/xunit/issues/2821
* Ensure we pull down 8.0 runtime.
* Update Centos docker containers
* Fix packaging step
* Try including stdint.h to fix missing uint8_t on centos
* Update Centos test queue
* Attempt to use runtime centos-stream8-helix container for tests
* Use centos-stream8-mlnet-helix container for testing
* Undo changes to test data
* Make NETFRAMEWORK ifdef versionless
* Switch back to centos7 for testing
* Revert "Switch back to centos7 for testing"
This reverts commit ab0d41e4b7.
* Update dependencies from https://github.com/dotnet/arcade build 20231221.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23621.2
* Update dependencies from https://github.com/dotnet/arcade build 20231228.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 8.0.0-beta.23265.1 -> To Version 9.0.0-beta.23628.1
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Michael Sharp <misharp@microsoft.com>
Tarek Mahmoud Sayed
Xiaoyun Zhang
Michael Sharp
dotnet-maestro[bot]
Joshua Zierhut
Stephen Toub
Aleksei Smirnov
Tarek Mahmoud Sayed
feiyun0112
Eric StJohn