Intelligent APIs

What are Intelligent APIs?

Intelligent APIs aim to make machine learning tasks easier for developers to leverage in their applications without needing ML expertise or creating a new model. By just importing a nuget package and calling a function, we want developers to be able to build intelligent app experiences without needing to deal with the complexities of inferencing machine learning models on Windows.

Each of these APIs employs WinML (Windows Machine Learning) to use the models on Windows. WinML helps abstract a lot of the model-specific code away and performs hardware optimizations to improve performance significantly on Windows. Learn more about WinML here.

Pre-requisites

Visual Studio 2017 Version 15.7.4 or Newer

Windows 10 - Build 17763 (RS5) or Newer

Windows SDK - Build 17763 (RS5) or Newer

Getting started with the nuget packages

We have 2 nuget packages ready for you to test and play around with. These nuget packages enable you to perform classic machine learning tasks like image classification and object detection in 1-2 lines of code.

Steps to import the nuget package into your project:

1. Add a new nuget source with the feed URL as https://pkgs.dev.azure.com/dotnet/CommunityToolkit/_packaging/CommunityToolkit-Labs/nuget/v3/index.json. If you have not done this before, follow the steps listed here to add a new nuget package source. Name the source as "WCT Labs" and set the URL as https://pkgs.dev.azure.com/dotnet/CommunityToolkit/_packaging/CommunityToolkit-Labs/nuget/v3/index.json.
2. Open a blank UWP app or an existing app and add one or both packages from the newly created source. Follow step 3 here to do so.
3. There should be two nuget packages listed, CommunityToolkit.Labs.Intelligent.ImageClassification and CommunityToolkit.Labs.Intelligent.ObjectDetection.

Using the packages

Image Classification

To perform image classification, import the CommunityToolkit.Labs.Intelligent.ImageClassification nuget package. To classify an image, you will need to pass a StorageFile object which is the image file itself, and the number of top results that you want (optional). In the following example, we pass an image of a Rottweiler and we want the top 3 results.

   using CommunityToolkit.Labs.Intelligent.ImageClassification;  
   ...
   List<ClassificationResult> imageClasses = await SqueezeNetImageClassifier.ClassifyImage(selectedStorageFile, 3);

This nuget package performs SqueezeNet model inferencing using WinML. SqueezeNet can detect 1000 different classes.

Object Detection

(Note: This currently only works with Windows 11. We are looking into this issue)

To perform object detection on your images/video, import the CommunityToolkit.Labs.Intelligent.ObjectDetection nuget package. To detect objects in the image, you can either pass an image file as a StorageFile object, VideoFrame or SoftwareBitmap.

   using CommunityToolkit.Labs.Intelligent.ObjectDetection;
   ...
   List<DetectionResult> listOfObjects = await YOLOObjectDetector.DetectObjects(selectedStorageFile);    

This nuget package performs object detection using YOLOv4 model inference on WinML and also return the co-ordinates of the bounding boxes around the detected objects. YOLOv4 can detect objects of 80 different classes.

NEW!!! Emotion Recognition

To perform emotion recognition, import the CommunityToolkit.Labs.Intelligent.EmotionRecognition nuget package. To detect emotion in an image, you can either pass a StorageFile object or a SoftwareBitmap of the image.

   using CommunityToolkit.Labs.Intelligent.EmotionRecognition;
   ...
   DetectedEmotion detectedEmotion = await EmotionRecognizer.DetectEmotion(selectedStorageFile);    

🧪 This project is under Community Toolkit Labs. What does that mean?

Community Toolkit Labs is a place for rapidly prototyping ideas and gathering community feedback. It is an incubation space where the developer community can come together to work on new ideas before thinking about final quality gates and ship cycles. Developers can focus on the scenarios and usage of their features before finalizing docs, samples, and tests required to ship a complete idea within the Toolkit.

Steps to clone the repo

1. git clone the repo
2. Open the .sln file in VS 2017 or newer
3. Set IntelligentLabsTest as the startup project.
4. Build the project and run!

Note: If you have the Windows machine learning code generator or mlgen extension on VS, please disable it for this project since all the code is already generated for you.