* Add step for emitting rate limits for a token
* add step to eng/common sync template
* Fix resource label output
* Use propery name instead of nested 'resource' property
* Add percent metric too
* Add divide by zero safety
---------
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Common cleanup with the docs docker images no longer being used
* Remove docker-pull-image.yml
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Update task_query_response.prompty
remove required keys
* Update task_simulate.prompty
* Update task_query_response.prompty
* Update task_simulate.prompty
* Fix the api_key needed
* Update for release
* Black fix for file
* Add original text in global context
* Update test
* Update the indirect attack simulator
* Black suggested fixes
* Update simulator prompty
* Update adversarial scenario enum to exclude XPIA
* Update changelog
* Black fixes
* Remove duplicate import
* Fix the mypy error
* Mypy please be happy
* Updates to non adv simulator
* accept context from assistant messages, exclude them when using them for conversation
* update changelog
* pylint fixes
* pylint fixes
* remove redundant quotes
* Fix typo
* pylint fix
* Update broken tests
* Include the grounding json in the manifest
* Fix typo
* Come on package
* Release 1.0.0b5
* Notice from Chang
* Remove adv_conv template parameters from the outputs
* Update chanagelog
* Experimental tags on adv scenarios
* Readme fix onbreaking change
* Add the category and both user and assistant context to the response of qr_json_lines
* Update changelog
* Rename _kwargs to _options
* _options as prefix
* update troubleshooting for simulator
* Rename according to suggestions
---------
Co-authored-by: Nagkumar Arkalgud <nagkumar@naarkalg-work-mac.local>
Co-authored-by: Nagkumar Arkalgud <nagkumar@naarkalgworkmac.lan>
Co-authored-by: Nagkumar Arkalgud <nagkumar@Mac.lan>
* Catch llm parsing errors due to llm not following prompt format or not returning score
* Add raise from to exception
* Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/utils.py
Co-authored-by: changliu2 <99364750+changliu2@users.noreply.github.com>
* Add model output to error message
---------
Co-authored-by: changliu2 <99364750+changliu2@users.noreply.github.com>
* Remove RetrievalEvaluator overrides - make it inherit fully from EvaluatorPromptyBase like other prompt-based evaluators
* Re-record test
* Address linting
* Retry container deletion
* Do not try to purge keyvaults with purge protection
* Delete all blobs when container has immutability
* Skip missing blob container properties
* Fix null container
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* make evaluate aggregation more resiliant against bad inputs
* cl
* remove accidental test file changes
* remove accidental test file changes2
* remove accidental test file changes3
* more useful errors
* change exception to warnings
* removing exception when instrumenting multiple times, adding content recording parameter
* fixing a typo
* Update version number and changelog
* Update model used in AOAI chat completions tests
* Update test recordings
* check tool related fix and readme code snippet update
* sphinx tool warning fix
---------
Co-authored-by: Marko Hietala <markhiet@microsoft.com>
Co-authored-by: Darren Cohen <39422044+dargilco@users.noreply.github.com>
* Remove Optional type annotation from get() methods
* Remove duplicate overloads
* Include xpia in handled_metrics for evaluation aggregation
* Update output.py
* Add samples to evaluate and evaluator docstrings
* Add samples to simulator directory docstrings
* Fix docs/linting errors
* Update evaluator descriptions to match registry descriptions, add groundedness pro example, address linting
* Update evaluator descriptions to match registry descriptions, add groundedness pro example, address linting
* Address linting and add simulator samples
* Update README
* Fix linting errors, build docs, update readme
* Update sample and fix link
* Prompty support within Azure AI Inference SDK
* Fix unit test
* Address PR feedback with copyright, merge PromptConfig to PromptTemplate
* Add comment and set model_name as optional
* Bug fixes
* Updated parameter names from PM feedbacks
* Improve sample code and unit tests
* Update readme and comments
* Rename files
* Address PR comment
* add Pydantic as dependency
* Fix type errors
* Fix spelling issues
* Address PR comments and fix linter issues
* Fix type import for "Self"
* Change to keyword-only constructor and fix linter issues
* Rename function `from_message` to `from_str`; `render` to `create_messages`
* Change from `from_str` to `from_string`
* Merge latest code from `microsoft/prompty` and resolve linter issues
* Fix PR comment
* Fix PR comments
Waqas Javed
Azure SDK Bot
David Wu
Xiang Yan
Zimu Li (MSFT)
Nagkumar Arkalgud
Diondra
Billy Hu
MilesHolland
Darren Cohen
Yuchao Yan
Jeremy Voss
Neehar Duvvuri
M-Hietala
Simon Moreno
Ankit Singhal
Annie Liang
James Suplizio
swathipil
Paul Van Eck
Scott Addie
Nethra C Sashikar