Azure Search Cognitive Skill to extract technical and business skills from text
Перейти к файлу
microsoft-github-policy-service[bot] 50006cd23e
Auto merge mandatory file pr
This pr is auto merged as it contains a mandatory file and is opened for more than 10 days.
2023-06-12 18:22:25 +00:00
data Updating documentation. Adding extra_skill_patterns.jsonl to allow users to simply add new skill patterns. 2019-04-23 12:29:00 -07:00
deployment Update resumesextractskills.json 2019-05-02 18:31:23 -07:00
docs Updating documentation. Adding extra_skill_patterns.jsonl to allow users to simply add new skill patterns. 2019-04-23 12:29:00 -07:00
extract_skills Updating README and comments for clarity 2019-04-25 10:39:45 -07:00
get_skill Updating Docs. Renaming for clarity. 2019-04-22 16:53:34 -07:00
services Updating documentation. Adding extra_skill_patterns.jsonl to allow users to simply add new skill patterns. 2019-04-23 12:29:00 -07:00
tests Updating Docs. Renaming for clarity. 2019-04-22 16:53:34 -07:00
.dockerignore Updating container image to mcr public repo 2019-04-18 11:01:17 -07:00
.funcignore Initial Commit 2019-04-05 10:30:59 -07:00
.gitignore Renaming some folders, fixing a bug with skill aliases. Adding deployment scripts for search and ARM 2019-04-18 10:48:16 -07:00
Dockerfile Initial Commit 2019-04-05 10:30:59 -07:00
LICENSE Updating Docs. Renaming for clarity. 2019-04-22 16:53:34 -07:00
README.md Update README.md 2020-03-11 13:52:57 -07:00
SECURITY.md Microsoft mandatory file 2023-06-02 17:46:09 +00:00
azure-pipelines.yml Move vars to env in azure-pipelines.yml 2019-04-23 11:01:46 -07:00
extensions.csproj Initial Commit 2019-04-05 10:30:59 -07:00
host.json Initial Commit 2019-04-05 10:30:59 -07:00
requirements.txt Updating README.md with Contributing section. Pushing to Microsoft Organization. 2019-04-17 12:24:10 -07:00

README.md

Deprecation Notice

We've launched a better version of this service with Azure Cognitive Serivces - Text Analytics in the new V3 of the Named Entity Recognition (NER) endpoint.

NER V3 Information

You can read more about this work and how to use it here:

(NEW) Custom Entity Lookup Skill

Azure Cognitive Search recently introduced a new built-in Cognitive Skill that does essentially what this repository does. You provide a dictionary of terms you want to match and it will extract those for you from any text field in your search index. You can read more about that here: https://docs.microsoft.com/en-us/azure/search/cognitive-search-skill-custom-entity-lookup

Custom Cognitive Skill with Containers

This repo is no longer supported but you're free to use the index and skill definitions provided to enable the personalized job recommendations scenario. If you are just looking to deploy a container as a custom skill, I highly recommend utilizing this more generic cookiecutter repository: https://github.com/microsoft/cookiecutter-spacy-fastapi. We will continue to support this project. For more information on deploying Containers on Azure see:


Introduction

The Skills Extractor is a Named Entity Recognition (NER) model that takes text as input, extracts skill entities from that text, then matches these skills to a knowledge base (in this sample a simple JSON file) containing metadata on each skill. It then returns a flat list of the skills identified.

Definitions

What is a Cognitive Skill?

A Cognitive Skill is a Feature of Azure Search designed to Augment data in a search index.

What is a Skill in terms of the Skills Extractor?

A Skill is a Technical Concept/Tool or a Business related/Personal attribute.

Example skills: Machine Learning, Artificial Intelligence, PyTorch, Business, Advertising

For the current goals of the service, we are focused on technical skills. Technical skills are the abilities and knowledge needed to perform specific tasks. They are practical, and often relate to mechanical, information technology, mathematical, or scientific tasks. The Taxonomies the API pulls from primarily consist of concepts and tools related to technology. For example, Programming Languages are considered a higher-level technical skill, and C# or Python are a sub of that larger skill.

P.S. Sorry for the confusing naming.

Skill Sources

We pull skills and technologies from many open online sources and build Record Linkage models to conflate skills and categories across each source into a single Knowledge Graph.

Here is a list of our sources:

Use Cases

The original idea stemmed from a few organizational needs. Here are a few:

  • Determine the skills required for a job opening at your company and match applicant resumes based on skills.

  • Extract skills from Learning Content that your company creates to improve search and recommendations.

  • Identify the technical and professional skills of your team or organization and work to close skill gaps.

Prerequisites

Before running this sample, you must have the following:

  • Install the Azure CLI. This article requires the Azure CLI version 2.0 or later. Run az --version to find the version you have.
    You can also use the Azure Cloud Shell.

Dependencies

Quickstart: Extract Skills for your data in Azure Search using a Custom Cognitive Skill

If you're unfamiliar with Azure Search Cognitive Skills you can read more about them here: https://docs.microsoft.com/en-us/azure/search/cognitive-search-concept-intro

Follow one of the scenarios below

  1. Extract Skills from an Existing Search Index
  2. Use the sample Search Scenario of extracting Skills from Jobs and Resumes

Create your own Custom Cognitive Skill

If you would like to create your own Custom Skill leveraging the NLP power of the Python Ecosystem you can use this cookiecutter project to bootstrap a containerized API to deploy in your own infrastructure.

https://github.com/Microsoft/cookiecutter-azure-search-cognitive-skill

Feedback & Support

Were eager to improve, so please take a couple of minutes to answer some questions about your experience https://aka.ms/AA4xoy5. For support, please contact: WWL_Skills_Service@microsoft.com

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.