OLive/examples
shaahji 71d3522c3e
Add evaluation support for using lm-eval harness (#1349)
## Add evaluation support for using lm-eval harness

* Added a new Evaluator to use lm_eval to evaluate hf model
* Added a llama2 config to use the new evaluator

## Checklist before requesting a review
- [ ] Add unit tests for this change.
- [x] Make sure all tests can pass.
- [ ] Update documents if necessary.
- [x] Lint and apply fixes to your code by running `lintrunner -a`
- [ ] Is this a user-facing change? If yes, give a description of this
change to be included in the release notes.
- [ ] Is this PR including examples changes? If yes, please remember to
update [example
documentation](https://github.com/microsoft/Olive/blob/main/docs/source/examples.md)
in a follow-up PR.

## (Optional) Issue link
2024-09-11 10:39:00 -07:00
..
AST CI: Make compatible with `ort>=1.19.0` (#1320) 2024-08-15 19:35:45 -07:00
bert HfModelHandler: Use `optimum` for automatic `io_config` and `dummy_inputs` (#1317) 2024-08-15 22:44:41 -07:00
deberta Data: `label_cols` -> `label_col` (#1302) 2024-08-12 20:02:53 -07:00
directml SD: Make `CompVis/stable-diffusion-v1-4` the default model (#1339) 2024-08-30 11:01:46 -07:00
falcon
gptj
inception
llama2 Add evaluation support for using lm-eval harness (#1349) 2024-09-11 10:39:00 -07:00
mistral INC: Remove `diagnosis` config param (#1301) 2024-08-12 12:57:52 -07:00
mobilenet
open_llama HfModelHandler: Use `optimum` for automatic `io_config` and `dummy_inputs` (#1317) 2024-08-15 22:44:41 -07:00
opt_125m GptqQuantizer: Option to save in hf format (#1338) 2024-09-02 13:01:05 -07:00
phi2 LoRA: Remove `eval_dataset_size`, Generalize data collator, Remove redundant examples (#1308) 2024-08-13 17:35:21 -07:00
phi3 GptqQuantizer: Option to save in hf format (#1338) 2024-09-02 13:01:05 -07:00
red_pajama
resnet CI: Make compatible with `ort>=1.19.0` (#1320) 2024-08-15 19:35:45 -07:00
stable_diffusion SD: Make `CompVis/stable-diffusion-v1-4` the default model (#1339) 2024-08-30 11:01:46 -07:00
super_resolution Rename onnxruntime_extensions (#1276) 2024-08-02 16:43:52 -07:00
test Add llama2 example to pipeline (#1328) 2024-08-28 14:52:04 -07:00
utils Extract/Export Adapters: Support `safetensors`, `pt` formats (#1310) 2024-08-16 09:28:03 -07:00
vgg
whisper Whisper: pin `transformers` version (#1296) 2024-08-12 08:39:42 -07:00
README.md

README.md

Examples

This folder contains actively maintained examples of use of Olive with different models, optimization tools and hardware.

Each example is a self-contained folder with a README.md file that explains how to run it.

Important

To ensure that the latest versions of the examples can be run without issues, you have to install Olive from source. We also recommend using a new conda or virtual environment.

To install Olive from source, run the following command in a new conda or virtual environment:

git clone https://github.com/microsoft/Olive.git
cd Olive
python -m pip install .

Then cd into the desired example folder and follow the instructions in the README.md file.

For examples corresponding to a specific release of Olive, checkout the corresponding tag. For instance, to use the examples corresponding to the v0.2.0 release, run the following command:

git checkout tags/v0.2.0

Conda env

To create a new conda environment and activate it, run the following command:

conda create -n olive-env python=3.8
conda activate olive-env

You can replace olive-env with any name you want and python=3.8 with the version of python you want to use.

Please refer to the conda documentation for more information on how to create and manage conda environments.

Virtual env

To create a new virtual environment and activate it, run the following command:

On Linux:

python -m venv olive-env
source olive-env/bin/activate

On Windows (CMD):

python -m venv olive-env
olive-env\Scripts\activate.bat

On Windows (PowerShell):

python -m venv olive-env
olive-env\Scripts\Activate.ps1

You can replace olive-env with any path you want. A new folder will be created at the specified path to contain the virtual environment.

Please refer to the python documentation for more information on how to create and manage virtual environments.