First cut of TinyML lab
This commit is contained in:
Jim Bennett
Родитель
49d8f2f73a
Коммит
0236acd7ce
|
|
@ -140,4 +140,5 @@ obj
|
|||
# Node stuff
|
||||
node_modules
|
||||
|
||||
~$*.docx
|
||||
~$*.docx
|
||||
*.a
|
||||
|
|
|
|||
|
|
@ -22,6 +22,12 @@ These labs cover running AI workloads either in the cloud from the IoT device, o
|
|||
* [Assembly line QA](./ai-edge/vision/manufacturing-part-check/) - A prototype of an AI image classification based quality assurance tool for manufacturing showing how to detect broken parts on an assembly line using AI, controlled from another device
|
||||
* [Speech](./ai-edge/speech) - Speech to text, text to speech and speech translation using a Raspberry Pi and USB microphone/speaker
|
||||
|
||||
## TinyML
|
||||
|
||||
These labs cover training tiny machine learning workloads and running them on embedded hardware such as Adruino microcontrollers.
|
||||
|
||||
* [Audio classifier](./tiny-ml/audio-classifier/) - a TinyML lab to capture audio data, use it to train a model, then classify audio data using that model on an Arduino Nano 33 Sense BLE board.
|
||||
|
||||
## Digital Agriculture
|
||||
|
||||
These labs cover scenarios around digital agriculture.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2021 Jim Bennett
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
# TinyML Audio classifier
|
||||
|
||||
This folder contains a lab with multiple parts working towards an audio classifier running on an Arduino Nano 33 BLE Sense microcontroller, taking advantage of the built-in microphone.
|
||||
|
||||
> This lab was inspired by [Better word classification with Arduino Nano 33 BLE Sense and Machine Learning](https://eloquentarduino.github.io/2020/08/better-word-classification-with-arduino-33-ble-sense-and-machine-learning/) by [Eloquent Arduino](https://eloquentarduino.github.io/about-me/).
|
||||
|
||||
| Author | [Jim Bennett](https://github.com/JimBobBennett) |
|
||||
|:---|:---|
|
||||
| Target platform | <ul><li>Arduino Nano 33 BLE Sense</li></ul> |
|
||||
| Hardware required | <ul><li>Arduino Nano 33 BLE Sense</li><li>USB cable</li></ul> |
|
||||
| Software required | <ul><li>[Visual Studio Code](http://code.visualstudio.com?WT.mc_id=academic-7372-jabenn)</li><li>[PlatformIO](https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide&WT.mc_id=academic-7372-jabenn)</li></ul>|
|
||||
| Programming Language | <ul><li>C++</li><li>Python</li></ul> |
|
||||
| Prerequisites | Basic proficiency in using VS Code, C++ and Python.<br>If you want to learn Python, check out these free resources:<br><ul><li>[Python for beginners video series on Channel9](https://channel9.msdn.com/Series/Intro-to-Python-Development?WT.mc_id=academic-7372-jabenn)</li><li>[Take your first steps with Python learning path on Microsoft Learn](https://docs.microsoft.com/learn/paths/python-first-steps/?WT.mc_id=academic-7372-jabenn)</li></ul> |
|
||||
| Date | February 2021 |
|
||||
| Learning Objectives | <ul><li>Capture audio data suitable for TinyML using an Arduino device</li><li>Train a TinyML model using Python</li><li>Classify audio using TinyML on an Arduino device</li></ul> |
|
||||
| Time to complete | 1 hours |
|
||||
|
||||
## The lab parts
|
||||
|
||||
This lab has the following parts
|
||||
|
||||
1. Program the Arduino device for audio capture and capture training data
|
||||
1. Train a ML model using the training data
|
||||
1. Program the Arduino device to classify audio
|
||||
|
||||
## Audio classification
|
||||
|
||||
Audo classification is the process of classifying a sound based on labelled training data. For example - you could train a model by using multiple recordings of someone saying the word "Yes" labelled as `Yes`, and multiple recordings of someone saying the word "No" labelled as `No`. The model could then take a new sound recording and classify it as either `Yes` or `No`.
|
||||
|
||||
This lab starts by coding the Arduino to record multiple samples that are labelled, then these labelled samples are used to train a model, which is then add to device code that runs on the microcontroller to classify new audio data.
|
||||
|
||||
The classifier you will create here needs at least 2 labels in the model, and will pick the most probable one to classify the audio.
|
||||
|
||||
## TinyML
|
||||
|
||||
TinyML is the coming together of machine learning and embedded systems. It involves training ML models that are tiny - that is substantially smaller than the models historically created, and able to run on microcontrollers with limited memory and power. It was originally defined as ML models that can run using less than 1mW of power, but has become a general term for running ML on microcontrollers.
|
||||
|
||||
Microcontrollers have memory limits usually in the kilobytes, meaning traditional ML models that are many megabytes in size cannot even be installed on the device, let alone run. By using TinyML models you can bring the world of ML to a microcontroller. An audio classifier (a model that can distinguish between multiple sounds) or a wake word model (a model that can detect one specific sound, such as the command to wake a smart device up), for example, can be compressed to less than 20KB using TinyML.
|
||||
|
||||
There is a trade off - with smaller models you lose accuracy, but this is an acceptable tradeoff for the advantages of smaller, low powered models. For example, if you were creating a voice activated smart device you would want it to consume as little power as possible waiting for a wake word, and only then powering up to listen for more instructions and using bigger models, or even sending data to the cloud. If you are using audio to classify animal behavior on a smart collar, you want long battery life and the device to be as small and light-weight as possible to reduce the inconvenience to the animal being tracked.
|
||||
|
||||
In this lab, the device in question is an [Arduino Nano 33 BLE Sense microcontroller](https://store.arduino.cc/usa/nano-33-ble-sense) - a board that has a built in microphone that can be used to detect and classify audio signals. It has 256KB of memory available.
|
||||
|
||||
## Labs
|
||||
|
||||
1. [Create the audio capture tool](./steps/audio-capture.md)
|
||||
1. [Train the TinyML model](./steps/train-model.md)
|
||||
1. [Create the audio classifier](./steps/audio-classifier.md)
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
.pio
|
||||
.vscode/.browse.c_cpp.db*
|
||||
.vscode/c_cpp_properties.json
|
||||
.vscode/launch.json
|
||||
.vscode/ipch
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
// See http://go.microsoft.com/fwlink/?LinkId=827846
|
||||
// for the documentation about the extensions.json format
|
||||
"recommendations": [
|
||||
"platformio.platformio-ide"
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
# Audio Capture
|
||||
|
||||
This folder contains a [PlatformIO](https://platformio.org/platformio-ide) project to build an audio capture app that runs on an [Arduino Nano 33 BLE Sense](https://store.arduino.cc/usa/nano-33-ble-sense) board.
|
||||
|
||||
You can read more about this project in the [top level README.md file](../README.md).
|
||||
|
||||
To build and deploy this project, you will need to open this folder in [Visual Studio Code](https://code.visualstudio.com/?WT.mc_id=academic-7372-jabenn) with the [PlatformIO extension](https://platformio.org/platformio-ide) installed. You will then be able to build the project and upload to a Arduino Nano 33 BLE Sense board.
|
||||
|
||||
## Building this project
|
||||
|
||||
This project has a library dependency on the ARM CMSIS static library. You will need to download this from the ARM GitHub repo.
|
||||
|
||||
* Head to [the CMSIS GCC libs folder in the GitHub repo](https://github.com/ARM-software/CMSIS_5/tree/5.7.0/CMSIS/DSP/Lib/GCC)
|
||||
* Download `libarm_cortexM4l_math.a` from that repo page
|
||||
* Add the file to the root folder of this project
|
||||
* Build and upload as normal
|
||||
|
||||
## Running the project
|
||||
|
||||
Once the project is running on the board, it will listen for audio and output RMS values to the serial port.
|
||||
|
||||
* Connect to the serial monitor to view the audio values
|
||||
* Make the one of the relevant noises into the microphone of the board. Pause after each noise and you will see a line output to the serial monitor.
|
||||
* Repeat 15-30 times for that one noise
|
||||
* Copy the values from the serial monitor into a CSV file named using the name of the label (for eample if you were capturing numbers you would put the data for one into `one.csv`, for two into `two.csv` and so on).
|
||||
* Clear the serial output and repeat the above steps for all the noises you are interested in
|
||||
|
||||
## Processing the output
|
||||
|
||||
Refer to the [top level README.md file](../README.md) for instructions on how to process and use the output to classify noises.
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
|
||||
This directory is intended for project header files.
|
||||
|
||||
A header file is a file containing C declarations and macro definitions
|
||||
to be shared between several project source files. You request the use of a
|
||||
header file in your project source file (C, C++, etc) located in `src` folder
|
||||
by including it, with the C preprocessing directive `#include'.
|
||||
|
||||
```src/main.c
|
||||
|
||||
#include "header.h"
|
||||
|
||||
int main (void)
|
||||
{
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Including a header file produces the same results as copying the header file
|
||||
into each source file that needs it. Such copying would be time-consuming
|
||||
and error-prone. With a header file, the related declarations appear
|
||||
in only one place. If they need to be changed, they can be changed in one
|
||||
place, and programs that include the header file will automatically use the
|
||||
new version when next recompiled. The header file eliminates the labor of
|
||||
finding and changing all the copies as well as the risk that a failure to
|
||||
find one copy will result in inconsistencies within a program.
|
||||
|
||||
In C, the usual convention is to give header files names that end with `.h'.
|
||||
It is most portable to use only letters, digits, dashes, and underscores in
|
||||
header file names, and at most one dot.
|
||||
|
||||
Read more about using header files in official GCC documentation:
|
||||
|
||||
* Include Syntax
|
||||
* Include Operation
|
||||
* Once-Only Headers
|
||||
* Computed Includes
|
||||
|
||||
https://gcc.gnu.org/onlinedocs/cpp/Header-Files.html
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
; PlatformIO Project Configuration File
|
||||
;
|
||||
; Build options: build flags, source filter
|
||||
; Upload options: custom upload port, speed and extra flags
|
||||
; Library options: dependencies, extra library storages
|
||||
; Advanced options: extra scripting
|
||||
;
|
||||
; Please visit documentation for the other options and examples
|
||||
; https://docs.platformio.org/page/projectconf.html
|
||||
|
||||
[env:nano33ble]
|
||||
platform = nordicnrf52
|
||||
board = nano33ble
|
||||
framework = arduino
|
||||
build_flags =
|
||||
-L.
|
||||
-l arm_cortexM4l_math
|
||||
-w
|
||||
|
|
@ -0,0 +1,109 @@
|
|||
// Copyright (c) Microsoft Corporation.
|
||||
// Licensed under the MIT license.
|
||||
|
||||
/**
|
||||
* Audio Capture
|
||||
*
|
||||
* This program listens on the microphone capturing audio data.
|
||||
* Instead of capturing raw data, it captures root mean squared values allowing
|
||||
* audio to be reduced to a few values instead of thousands of values every second.
|
||||
* These values are then output to the serial port - one line per audio sample.
|
||||
*
|
||||
* This data can then be used to train a TinyML model to classify audio.
|
||||
*/
|
||||
|
||||
#include "mic.h"
|
||||
|
||||
// Settings for the audio
|
||||
// Try tuning these if you need different results
|
||||
// 128 samples is enough for 2 seconds of audio - it's captured at 64 samples per second
|
||||
#define SAMPLES 128
|
||||
#define GAIN (1.0f / 50)
|
||||
#define SOUND_THRESHOLD 1000
|
||||
|
||||
// An array of the audio samples
|
||||
float features[SAMPLES];
|
||||
|
||||
// A wrapper for the microphone
|
||||
Mic mic;
|
||||
|
||||
/**
|
||||
* @brief PDM callback to update the data in the mic object
|
||||
*/
|
||||
void onAudio()
|
||||
{
|
||||
mic.update();
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Read given number of samples from mic
|
||||
* @return True if there is enough data captured from the microphone,
|
||||
* otherwise False
|
||||
*/
|
||||
bool recordAudioSample()
|
||||
{
|
||||
// Check the microphone class as captured enough data
|
||||
if (mic.hasData() && mic.pop() > SOUND_THRESHOLD)
|
||||
{
|
||||
// Loop through the samples, waiting to capture data if needed
|
||||
for (int i = 0; i < SAMPLES; i++)
|
||||
{
|
||||
while (!mic.hasData())
|
||||
delay(1);
|
||||
|
||||
// Add the features to the array
|
||||
features[i] = mic.pop() * GAIN;
|
||||
}
|
||||
|
||||
// Return that we have features
|
||||
return true;
|
||||
}
|
||||
|
||||
// Return that we don't have enough data yet
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Sets up the serial port and the microphone
|
||||
*/
|
||||
void setup()
|
||||
{
|
||||
// Start the serial connection so the captured audio data can be output
|
||||
Serial.begin(115200);
|
||||
|
||||
// Set up the microphone callback
|
||||
PDM.onReceive(onAudio);
|
||||
|
||||
// Start listening
|
||||
mic.begin();
|
||||
|
||||
// Wait 3 seconds for everything to get started
|
||||
delay(3000);
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Runs continuously capturing audio data and writing it to
|
||||
* the serial port
|
||||
*/
|
||||
void loop()
|
||||
{
|
||||
// wait for audio data
|
||||
if (recordAudioSample())
|
||||
{
|
||||
// print the audio data to serial port
|
||||
for (int i = 0; i < SAMPLES; i++)
|
||||
{
|
||||
Serial.print(features[i], 6);
|
||||
|
||||
// Seperate the audio values with commas, at the last value
|
||||
// send a newline
|
||||
Serial.print(i == SAMPLES - 1 ? '\n' : ',');
|
||||
}
|
||||
|
||||
// Wait between samples
|
||||
delay(1000);
|
||||
}
|
||||
|
||||
// Sleep to allow background microphone processing
|
||||
delay(20);
|
||||
}
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
// Copyright (c) Microsoft Corporation.
|
||||
// Licensed under the MIT license.
|
||||
#include <PDM.h>
|
||||
#include <arm_math.h>
|
||||
|
||||
// The size of the data coming in from the microphone
|
||||
#define MICROPHONE_BUFFER_SIZE_IN_WORDS (256U)
|
||||
#define MICROPHONE_BUFFER_SIZE_IN_BYTES (MICROPHONE_BUFFER_SIZE_IN_WORDS * sizeof(int16_t))
|
||||
|
||||
/**
|
||||
* @brief A helper class for accessing the BLEs microphone. This reads
|
||||
* samples from the microphone at 16KHz and computes the root mean squared value.
|
||||
* This value is then retrieved which resets the value so it can be recalculated
|
||||
* from more incomng data.
|
||||
*
|
||||
* This allows for a small data set that represents a few seconds of sound data.
|
||||
* This is not enough to reproduce the audio, but is enough to train or use a
|
||||
* TinyML model.
|
||||
*/
|
||||
class Mic
|
||||
{
|
||||
public:
|
||||
/**
|
||||
* @brief Initializes the Mic class
|
||||
*/
|
||||
Mic() : _hasData(false)
|
||||
{
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Setup the PDM library to access the microphone at a sample rate of 16KHz
|
||||
*/
|
||||
void begin()
|
||||
{
|
||||
PDM.begin(1, 16000);
|
||||
PDM.setGain(20);
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Gets if the microphone has new data
|
||||
* @return true if the microphone has new data, otherwise false
|
||||
*/
|
||||
bool hasData()
|
||||
{
|
||||
return _hasData;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Get the root mean squared data value from the microphone
|
||||
* @return The root mean square value of the last data point from the microphone
|
||||
*/
|
||||
int16_t data()
|
||||
{
|
||||
return _rms;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Gets the last root mean squared value and resets the calculation
|
||||
* @return The last root mean square value
|
||||
*/
|
||||
int16_t pop()
|
||||
{
|
||||
int16_t rms = data();
|
||||
|
||||
reset();
|
||||
|
||||
return rms;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Reads the audio data from the PDM object and calculates the
|
||||
* root mean square value
|
||||
*/
|
||||
void update()
|
||||
{
|
||||
// Get the available bytes from the microphone
|
||||
int bytesAvailable = PDM.available();
|
||||
|
||||
// Check we have a full buffers worth
|
||||
if (bytesAvailable == MICROPHONE_BUFFER_SIZE_IN_BYTES)
|
||||
{
|
||||
// Read from the buffer
|
||||
int16_t _buffer[MICROPHONE_BUFFER_SIZE_IN_WORDS];
|
||||
|
||||
_hasData = true;
|
||||
PDM.read(_buffer, bytesAvailable);
|
||||
|
||||
// Calculate a running root mean square value
|
||||
arm_rms_q15((q15_t *)_buffer, MICROPHONE_BUFFER_SIZE_IN_WORDS, (q15_t *)&_rms);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Mark data as read
|
||||
*/
|
||||
void reset()
|
||||
{
|
||||
_hasData = false;
|
||||
}
|
||||
|
||||
private:
|
||||
int16_t _rms;
|
||||
bool _hasData;
|
||||
};
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
|
||||
This directory is intended for PlatformIO Unit Testing and project tests.
|
||||
|
||||
Unit Testing is a software testing method by which individual units of
|
||||
source code, sets of one or more MCU program modules together with associated
|
||||
control data, usage procedures, and operating procedures, are tested to
|
||||
determine whether they are fit for use. Unit testing finds problems early
|
||||
in the development cycle.
|
||||
|
||||
More information about PlatformIO Unit Testing:
|
||||
- https://docs.platformio.org/page/plus/unit-testing.html
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
.pio
|
||||
.vscode/.browse.c_cpp.db*
|
||||
.vscode/c_cpp_properties.json
|
||||
.vscode/launch.json
|
||||
.vscode/ipch
|
||||
7
labs/tiny-ml/audio-classifier/code/audio-classifier/.vscode/extensions.json
поставляемый
Normal file
7
labs/tiny-ml/audio-classifier/code/audio-classifier/.vscode/extensions.json
поставляемый
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
// See http://go.microsoft.com/fwlink/?LinkId=827846
|
||||
// for the documentation about the extensions.json format
|
||||
"recommendations": [
|
||||
"platformio.platformio-ide"
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
# Audio Classifier
|
||||
|
||||
This folder contains a [PlatformIO](https://platformio.org/platformio-ide) project to build an audio classifier app that runs on an [Arduino Nano 33 BLE Sense](https://store.arduino.cc/usa/nano-33-ble-sense) board.
|
||||
|
||||
You can read more about this project in the [top level README.md file](../README.md).
|
||||
|
||||
To build and deploy this project, you will need to open this folder in [Visual Studio Code](https://code.visualstudio.com/?WT.mc_id=academic-7372-jabenn) with the [PlatformIO extension](https://platformio.org/platformio-ide) installed. You will then be able to build the project and upload to a Arduino Nano 33 BLE Sense board.
|
||||
|
||||
## Building this project
|
||||
|
||||
This project has a library dependency on the ARM CMSIS static library. You will need to download this from the ARM GitHub repo.
|
||||
|
||||
* Head to [the CMSIS GCC libs folder in the GitHub repo](https://github.com/ARM-software/CMSIS_5/tree/5.7.0/CMSIS/DSP/Lib/GCC)
|
||||
* Download `libarm_cortexM4l_math.a` from that repo page
|
||||
* Add the file to the root folder of this project
|
||||
|
||||
You will also need to add code to the [`classifier.h`](./src/classifier.h) file. This code is generated by the model training utility. Refer to the [top level README.md file](../README.md) for more details on creating the contents of this file.
|
||||
|
||||
## Running the project
|
||||
|
||||
Once the project is running on the board, it will listen for audio, classify based on a trained model, then output the label of the predicted value to the serial port.
|
||||
|
||||
* Connect to the serial monitor to view the audio values
|
||||
* Make the one of the relevant noises into the microphone of the board. Pause after each noise and you will see the predicted label for that sound output to the serial monitor.
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
|
||||
This directory is intended for project header files.
|
||||
|
||||
A header file is a file containing C declarations and macro definitions
|
||||
to be shared between several project source files. You request the use of a
|
||||
header file in your project source file (C, C++, etc) located in `src` folder
|
||||
by including it, with the C preprocessing directive `#include'.
|
||||
|
||||
```src/main.c
|
||||
|
||||
#include "header.h"
|
||||
|
||||
int main (void)
|
||||
{
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Including a header file produces the same results as copying the header file
|
||||
into each source file that needs it. Such copying would be time-consuming
|
||||
and error-prone. With a header file, the related declarations appear
|
||||
in only one place. If they need to be changed, they can be changed in one
|
||||
place, and programs that include the header file will automatically use the
|
||||
new version when next recompiled. The header file eliminates the labor of
|
||||
finding and changing all the copies as well as the risk that a failure to
|
||||
find one copy will result in inconsistencies within a program.
|
||||
|
||||
In C, the usual convention is to give header files names that end with `.h'.
|
||||
It is most portable to use only letters, digits, dashes, and underscores in
|
||||
header file names, and at most one dot.
|
||||
|
||||
Read more about using header files in official GCC documentation:
|
||||
|
||||
* Include Syntax
|
||||
* Include Operation
|
||||
* Once-Only Headers
|
||||
* Computed Includes
|
||||
|
||||
https://gcc.gnu.org/onlinedocs/cpp/Header-Files.html
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
; PlatformIO Project Configuration File
|
||||
;
|
||||
; Build options: build flags, source filter
|
||||
; Upload options: custom upload port, speed and extra flags
|
||||
; Library options: dependencies, extra library storages
|
||||
; Advanced options: extra scripting
|
||||
;
|
||||
; Please visit documentation for the other options and examples
|
||||
; https://docs.platformio.org/page/projectconf.html
|
||||
|
||||
[env:nano33ble]
|
||||
platform = nordicnrf52
|
||||
board = nano33ble
|
||||
framework = arduino
|
||||
build_flags =
|
||||
-L.
|
||||
-l arm_cortexM4l_math
|
||||
-w
|
||||
lib_deps =
|
||||
# RECOMMENDED
|
||||
# Accept new functionality in a backwards compatible manner and patches
|
||||
eloquentarduino/EloquentTinyML @ ^0.0.3
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
// Copyright (c) Microsoft Corporation.
|
||||
// Licensed under the MIT license.
|
||||
|
||||
// Paste the output of the model trainer here
|
||||
|
|
@ -0,0 +1,108 @@
|
|||
// Copyright (c) Microsoft Corporation.
|
||||
// Licensed under the MIT license.
|
||||
|
||||
/**
|
||||
* Audio Classifier
|
||||
*
|
||||
* This program listens on the microphone capturing audio data.
|
||||
* Instead of capturing raw data, it captures root mean squared values allowing
|
||||
* audio to be reduced to a few values instead of thousands of values every second.
|
||||
* These values are then output to the serial port - one line per audio sample.
|
||||
*
|
||||
* This data is then run through a TinyML classifier to classify the sound based on
|
||||
* a trained model using code pasted into the classifier.h header file.
|
||||
*/
|
||||
|
||||
#include "mic.h"
|
||||
#include "classifier.h"
|
||||
|
||||
// Settings for the audio
|
||||
// These need to match the settings you used in the audio capture program
|
||||
// 128 samples is enough for 2 seconds of audio - it's captured at 64 samples per second
|
||||
#define SAMPLES 128
|
||||
#define GAIN (1.0f / 50)
|
||||
#define SOUND_THRESHOLD 1000
|
||||
|
||||
// An array of the audio samples
|
||||
float features[SAMPLES];
|
||||
|
||||
// A wrapper for the microphone
|
||||
Mic mic;
|
||||
|
||||
// The classifier
|
||||
Eloquent::ML::Port::SVM clf;
|
||||
|
||||
/**
|
||||
* @brief PDM callback to update the data in the mic object
|
||||
*/
|
||||
void onAudio()
|
||||
{
|
||||
mic.update();
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Read given number of samples from mic
|
||||
* @return True if there is enough data captured from the microphone,
|
||||
* otherwise False
|
||||
*/
|
||||
bool recordAudioSample()
|
||||
{
|
||||
// Check the microphone class as captured enough data
|
||||
if (mic.hasData() && mic.pop() > SOUND_THRESHOLD)
|
||||
{
|
||||
// Loop through the samples, waiting to capture data if needed
|
||||
for (int i = 0; i < SAMPLES; i++)
|
||||
{
|
||||
while (!mic.hasData())
|
||||
delay(1);
|
||||
|
||||
// Add the features to the array
|
||||
features[i] = mic.pop() * GAIN;
|
||||
}
|
||||
|
||||
// Return that we have features
|
||||
return true;
|
||||
}
|
||||
|
||||
// Return that we don't have enough data yet
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Sets up the serial port and the microphone
|
||||
*/
|
||||
void setup()
|
||||
{
|
||||
// Start the serial connection so the captured audio data can be output
|
||||
Serial.begin(115200);
|
||||
|
||||
// Set up the microphone callback
|
||||
PDM.onReceive(onAudio);
|
||||
|
||||
// Start listening
|
||||
mic.begin();
|
||||
|
||||
// Wait 3 seconds for everything to get started
|
||||
delay(3000);
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Runs continuously capturing audio data and classifying it,
|
||||
* writing the predicted label to the serial port
|
||||
*/
|
||||
void loop()
|
||||
{
|
||||
// wait for audio data
|
||||
if (recordAudioSample())
|
||||
{
|
||||
// Write out the classification to the serial port
|
||||
Serial.print("Sound number: ");
|
||||
Serial.println(clf.predictLabel(features));
|
||||
|
||||
// Wait between samples
|
||||
delay(1000);
|
||||
}
|
||||
|
||||
// Sleep to allow background microphone processing
|
||||
delay(20);
|
||||
}
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
// Copyright (c) Microsoft Corporation.
|
||||
// Licensed under the MIT license.
|
||||
#include <PDM.h>
|
||||
#include <arm_math.h>
|
||||
|
||||
#define MICROPHONE_BUFFER_SIZE_IN_WORDS (256U)
|
||||
#define MICROPHONE_BUFFER_SIZE_IN_BYTES (MICROPHONE_BUFFER_SIZE_IN_WORDS * sizeof(int16_t))
|
||||
|
||||
/**
|
||||
* @brief A helper class for accessing the BLEs microphone. This reads
|
||||
* samples from the microphone at 16KHz and computes the root mean squared value.
|
||||
* This value is then retrieved which resets the value so it can be recalculated
|
||||
* from more incomng data.
|
||||
*
|
||||
* This allows for a small data set that represents a few seconds of sound data.
|
||||
* This is not enough to reproduce the audio, but is enough to train or use a
|
||||
* TinyML model.
|
||||
*/
|
||||
class Mic
|
||||
{
|
||||
public:
|
||||
/**
|
||||
* @brief Initializes the Mic class
|
||||
*/
|
||||
Mic() : _hasData(false)
|
||||
{
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Setup the PDM library to access the microphone at a sample rate of 16KHz
|
||||
*/
|
||||
void begin()
|
||||
{
|
||||
PDM.begin(1, 16000);
|
||||
PDM.setGain(20);
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Gets if the microphone has new data
|
||||
* @return true if the microphone has new data, otherwise false
|
||||
*/
|
||||
bool hasData()
|
||||
{
|
||||
return _hasData;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Get the root mean squared data value from the microphone
|
||||
* @return The root mean square value of the last data point from the microphone
|
||||
*/
|
||||
int16_t data()
|
||||
{
|
||||
return _rms;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Gets the last root mean squared value and resets the calculation
|
||||
* @return The last root mean square value
|
||||
*/
|
||||
int16_t pop()
|
||||
{
|
||||
int16_t rms = data();
|
||||
|
||||
reset();
|
||||
|
||||
return rms;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Reads the audio data from the PDM object and calculates the
|
||||
* root mean square value
|
||||
*/
|
||||
void update()
|
||||
{
|
||||
// Get the available bytes from the microphone
|
||||
int bytesAvailable = PDM.available();
|
||||
|
||||
// Check we have a full buffers worth
|
||||
if (bytesAvailable == MICROPHONE_BUFFER_SIZE_IN_BYTES)
|
||||
{
|
||||
// Read from the buffer
|
||||
int16_t _buffer[MICROPHONE_BUFFER_SIZE_IN_WORDS];
|
||||
|
||||
_hasData = true;
|
||||
PDM.read(_buffer, bytesAvailable);
|
||||
|
||||
// Calculate a running root mean square value
|
||||
arm_rms_q15((q15_t *)_buffer, MICROPHONE_BUFFER_SIZE_IN_WORDS, (q15_t *)&_rms);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Mark data as read
|
||||
*/
|
||||
void reset()
|
||||
{
|
||||
_hasData = false;
|
||||
}
|
||||
|
||||
private:
|
||||
int16_t _rms;
|
||||
bool _hasData;
|
||||
};
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
|
||||
This directory is intended for PlatformIO Unit Testing and project tests.
|
||||
|
||||
Unit Testing is a software testing method by which individual units of
|
||||
source code, sets of one or more MCU program modules together with associated
|
||||
control data, usage procedures, and operating procedures, are tested to
|
||||
determine whether they are fit for use. Unit testing finds problems early
|
||||
in the development cycle.
|
||||
|
||||
More information about PlatformIO Unit Testing:
|
||||
- https://docs.platformio.org/page/plus/unit-testing.html
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# C extensions
|
||||
*.so
|
||||
|
||||
# Distribution / packaging
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
MANIFEST
|
||||
|
||||
# PyInstaller
|
||||
# Usually these files are written by a python script from a template
|
||||
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
||||
*.manifest
|
||||
*.spec
|
||||
|
||||
# Installer logs
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
|
||||
# Unit test / coverage reports
|
||||
htmlcov/
|
||||
.tox/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache
|
||||
nosetests.xml
|
||||
coverage.xml
|
||||
*.cover
|
||||
.hypothesis/
|
||||
.pytest_cache/
|
||||
|
||||
# Translations
|
||||
*.mo
|
||||
*.pot
|
||||
|
||||
# Django stuff:
|
||||
*.log
|
||||
local_settings.py
|
||||
db.sqlite3
|
||||
|
||||
# Flask stuff:
|
||||
instance/
|
||||
.webassets-cache
|
||||
|
||||
# Scrapy stuff:
|
||||
.scrapy
|
||||
|
||||
# Sphinx documentation
|
||||
docs/_build/
|
||||
|
||||
# PyBuilder
|
||||
target/
|
||||
|
||||
# Jupyter Notebook
|
||||
.ipynb_checkpoints
|
||||
|
||||
# pyenv
|
||||
.python-version
|
||||
|
||||
# celery beat schedule file
|
||||
celerybeat-schedule
|
||||
|
||||
# SageMath parsed files
|
||||
*.sage.py
|
||||
|
||||
# Environments
|
||||
.env
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Spyder project settings
|
||||
.spyderproject
|
||||
.spyproject
|
||||
|
||||
# Rope project settings
|
||||
.ropeproject
|
||||
|
||||
# mkdocs documentation
|
||||
/site
|
||||
|
||||
# mypy
|
||||
.mypy_cache/
|
||||
|
||||
# Junit test results (used by VSTS)
|
||||
junit
|
||||
|
||||
# Coverage results
|
||||
coverage/
|
||||
|
||||
# Pycharm
|
||||
.idea/
|
||||
|
||||
# vim
|
||||
*.swp
|
||||
|
||||
# Certificates
|
||||
*.pem
|
||||
demoCA/
|
||||
*.rnd
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
// See http://go.microsoft.com/fwlink/?LinkId=827846
|
||||
// for the documentation about the extensions.json format
|
||||
"recommendations": [
|
||||
"ms-python.vscode-pylance"
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# Audio model trainer
|
||||
|
||||
This Python program takes the output from the [audio capture program](../audio-capture) and uses it to train a TinyML model that can then be used in the the [audio classifier program](../audio-classifier).
|
||||
|
||||
## Setting up the Python environment
|
||||
|
||||
* Create a virtual environment for this folder using Python 3.8 or above
|
||||
* Install the pip packages in the `requirements.txt` file
|
||||
|
||||
> If you are using a new Apple Silicon based Mac, then the packages may not install as at the time of writing there is no supported Scikit-Learn package for the Mac. Instead you will need to use MiniForge. Refer to [this blog post](https://dev.to/jimbobbennett/installing-scikit-learn-on-an-apple-m1-114d) for instructions.
|
||||
* Copy the CSV files created from the [audio capture program](../audio-capture) into the [`data`](./data) folder. These files should be named for the relevant label for the audio data.
|
||||
* Run the `train_classifier.py` file
|
||||
|
||||
The model will be trained, and the output sent to the console.
|
||||
|
||||
```output
|
||||
(.venv) ➜ model-trainer git:(master) ✗ python train_classifier.py
|
||||
Accuracy 1.0
|
||||
Exported classifier to plain C
|
||||
#pragma once
|
||||
#include <cstdarg>
|
||||
namespace Eloquent {
|
||||
}
|
||||
```
|
||||
|
||||
Copy everything from `#pragma once` to the end of the output and paste it into the [`classifier.h`](../audio-classifier/src/classifier.h) header file in the audio classifier project.
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
scikit-learn
|
||||
micromlgen
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
import numpy as np
|
||||
from os.path import basename
|
||||
from glob import glob
|
||||
from sklearn.svm import SVC
|
||||
from micromlgen import port
|
||||
from sklearn.model_selection import train_test_split
|
||||
|
||||
def load_features(folder):
|
||||
dataset = None
|
||||
classmap = {}
|
||||
for class_idx, filename in enumerate(glob('%s/*.csv' % folder)):
|
||||
class_name = basename(filename)[:-4]
|
||||
classmap[class_idx] = class_name
|
||||
samples = np.loadtxt(filename, dtype=float, delimiter=',')
|
||||
labels = np.ones((len(samples), 1)) * class_idx
|
||||
samples = np.hstack((samples, labels))
|
||||
dataset = samples if dataset is None else np.vstack((dataset, samples))
|
||||
return dataset, classmap
|
||||
|
||||
np.random.seed(0)
|
||||
dataset, classmap = load_features('data')
|
||||
X, y = dataset[:, :-1], dataset[:, -1]
|
||||
|
||||
# this line is for testing your accuracy only: once you're satisfied with the results, set test_size to 1
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
clf = SVC(kernel='poly', degree=2, gamma=0.1, C=100)
|
||||
clf.fit(X_train, y_train)
|
||||
|
||||
print('Accuracy', clf.score(X_test, y_test))
|
||||
print('Exported classifier to plain C')
|
||||
print(port(clf, classmap=classmap))
|
||||
Двоичные данные
labs/tiny-ml/audio-classifier/images/vscode-platformio-new-capture-project.png
Normal file
Двоичные данные
labs/tiny-ml/audio-classifier/images/vscode-platformio-new-capture-project.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 59 KiB |
|
|
@ -0,0 +1,424 @@
|
|||
# Create the audio capture tool
|
||||
|
||||
In this step you will program the Arduino device to capture audio data ready to classify to train the model.
|
||||
|
||||
## Arduino Nano 33 Sense
|
||||
|
||||
The Arduino Nano 33 Sense is an Arduino board bwith a number of sensors, including a microphone which makes it ideal for audio classification. This board is programmed in C++, using Platform IO.
|
||||
|
||||
## Platform IO
|
||||
|
||||
[Visual Studio Code](http://code.visualstudio.com?WT.mc_id=academic-7372-jabenn) (VS Code) is a free developer text editor that can be expanded with extensions to support multiple features or programming languages.
|
||||
|
||||
[Platform IO](https://platformio.org) is an open source extension for VS Code that gives you the ability to program embedded devices without any additional dependencies.
|
||||
|
||||
### Install Platform IO
|
||||
|
||||
To install Platform IO:
|
||||
|
||||
* Make sure you have VS Code installed. If not, you can install it from the [Visual Studio Code Downloads page](https://code.visualstudio.com/Download?WT.mc_id=academic-7372-jabenn)
|
||||
* From VS Code, select the **Extensions** tab from the side menu
|
||||
|
||||
|
||||
|
||||
1. Search for `PlatformIO` and select the **PlatformIO IDE**. Select the **Install** button to install this extension.
|
||||
|
||||
|
||||
|
||||
> PlatformIO is a self-contained embedded development environment. If you have the Arduino extension installed, it can clash with PlatformIO, so make sure to uninstall the Arduino extension before using PlatformIO.
|
||||
|
||||
## Create the project
|
||||
|
||||
Once you have PlatformIO installed, it can be used to write the audio capture tool to gather data for classification.
|
||||
|
||||
### Create the PlatformIO project
|
||||
|
||||
1. From VS Code, select the **PlatformIO** tab from the side menu
|
||||
|
||||
|
||||
|
||||
1. The *PlatformIO: Quick Access* bar will appear. Select *PIO Home->Open*
|
||||
|
||||
|
||||
|
||||
1. From the home screen, select **+ New Project** from the *Quick Access* section
|
||||
|
||||
|
||||
|
||||
1. Name the project `audio-capture`
|
||||
|
||||
1. For the *Board*, type in `Arduinno Nano 33 BLE` and select the `Arduinno Nano 33 BLE` board
|
||||
|
||||
1. Set the *Framework* to **Arduino**
|
||||
|
||||
1. For the location, either leave the default option checked, or uncheck it and select a location for the project. If you hover over the question mark you can see the default project location.
|
||||
|
||||
1. Select the **Finish** button
|
||||
|
||||
|
||||
|
||||
PlatformIO will take some time to download the components it needs to use the board, and create the project. Once done, the project will be opened in the VS Code window.
|
||||
|
||||
### Navigate the project
|
||||
|
||||
The PlatformIO project contains a number of files and folders:
|
||||
|
||||
* `.pio` - this folder contains files used by PlatformIO such as the output of the build and other temporary files. You should never need to read or edit any files in this folder. This folder would not normally be checked in to source code control.
|
||||
* `.vscode` - this folder contains configuration information used by VS Code and may be specific to your PC or Mac, such as paths to library files. You should never need to read or edit any files in this folder.
|
||||
* `include` - this folder is for any header files for your project that you would want to expose to anyone using your code. This is normally when building libraries for consumption by other developers, and won't be used when writing code for a board.
|
||||
* `lib` - this folder is for any additional libraries you want to use in your project. PlatformIO has a library manager than can import third party libraries, but if you want to use an additional libraries you need to add them to this folder.
|
||||
* `src` - this folder is where the source code for your app lives. It will have a single file in it `main.cpp`. This file contains the code for an empty Arduino project.
|
||||
* `test` - this folder is where you would put any unit test you write against your code.
|
||||
* `.gitignore` - this is a [`gitignore`](https://git-scm.com/docs/gitignore) file that defines what will be ignored if you add this project to Git source code control.
|
||||
* `platform.ini` - this file defines the configuration for the PlatformIO project including the board types (PlatformIO projects can target multiple boards), any additional libraries to use, and any build flags to control the code compilation.
|
||||
|
||||
This project uses the [Arduino framework](https://www.arduino.cc), an open source framework for software and hardware. When writing arduino code, the core flow is around two methods that are defined in your main source code file. In a PlatformIO project, this file is called `main.cpp`.
|
||||
|
||||
* `setup()` - the setup method is called once and only once when the board starts up. This function is used to do one-time setup, such as configuring the microphone or seial port connection.
|
||||
* `loop()` - the loop method is called after setup, and once the method completes it is called again and again. All the time the board is running, this method will be called as soon as it completes.
|
||||
|
||||
|
||||
|
||||
## Write the code
|
||||
|
||||
The code needs to listen to the microphone and capture data, saving it out in a way that can be used to train a classifier.
|
||||
|
||||
Traditionally, audio data is sampled at a rate of many of thousands of values a second - for example low quality audio is sampled at 16KHz, or 16 thousand values every second. This would give a huge amount of inputs to our model, meaning we'd need a large model - something TinyML is trying to avoid. Instead the audio can be averaged out into smaller samples, only a few a second. This gives a less accurate representation of the audio data, but is good enough to distinguish between different sounds. This is done by calculating the root mean square (RMS) value of the sampled input for a period of time.
|
||||
|
||||
### Import a library
|
||||
|
||||
The code to calculate the RMS value is at the time of writing not shipped with the code to support this Arduino board in PlatformIO. You can get round this by downloading a static library with the implementation that can be linked into your code.
|
||||
|
||||
1. Head to the [ARM CMSIS GitHub repo compiled static libraries page](https://github.com/ARM-software/CMSIS_5/tree/5.7.0/CMSIS/DSP/Lib/GCC)
|
||||
1. Download the `libarm_cortexM4l_math.a` file and add it to the root folder of the project.
|
||||
> There are a number of libraries with similar names targetting different M chips, so make sure to download the correct one
|
||||
1. Open the `platformio.ini` file from the root of the project in VS Code
|
||||
1. Add the following to the end of the `[env:nano33ble]` section to tell the linker to look for libraries in the root and link this file when building the project:
|
||||
|
||||
```sh
|
||||
build_flags =
|
||||
-L.
|
||||
-l arm_cortexM4l_math
|
||||
-w
|
||||
```
|
||||
|
||||
The `-L.` flag tells the linker to look for libraries in the root folder.
|
||||
|
||||
The `-l arm_cortexM4l_math` flag tells the linker to link the `libarm_cortexM4l_math.a` library
|
||||
|
||||
The `-w` flag ignores warnings that come from the Arduino libraries. Although it is generally good practice to find and fix all warnings, in this case the Arduino libraries have warnings that can make it harder to catch errors in your code, so it is easier to ignore them
|
||||
|
||||
### Add code to read from the microphone
|
||||
|
||||
To keep the code cleaner, the code to read from the microphone can be added to a separate file, and this file re-used later when building the audio classifier.
|
||||
|
||||
1. In VS Code, create a new file in the `src` folder called `mic.h`. This file will contain code to read the data from the microphone using a standard Arduino library, and calculate the RMS value of periods of audio.
|
||||
|
||||
1. Add the following code to this file:
|
||||
|
||||
```cpp
|
||||
#include <PDM.h>
|
||||
#include <arm_math.h>
|
||||
|
||||
// The size of the data coming in from the microphone
|
||||
#define MICROPHONE_BUFFER_SIZE_IN_WORDS (256U)
|
||||
#define MICROPHONE_BUFFER_SIZE_IN_BYTES (MICROPHONE_BUFFER_SIZE_IN_WORDS * sizeof(int16_t))
|
||||
|
||||
/**
|
||||
* @brief A helper class for accessing the BLEs microphone. This reads
|
||||
* samples from the microphone at 16KHz and computes the root mean squared value.
|
||||
* This value is then retrieved which resets the value so it can be recalculated
|
||||
* from more incomng data.
|
||||
*
|
||||
* This allows for a small data set that represents a few seconds of sound data.
|
||||
* This is not enough to reproduce the audio, but is enough to train or use a
|
||||
* TinyML model.
|
||||
*/
|
||||
class Mic
|
||||
{
|
||||
public:
|
||||
/**
|
||||
* @brief Initializes the Mic class
|
||||
*/
|
||||
Mic() : _hasData(false)
|
||||
{
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Setup the PDM library to access the microphone at a sample rate of 16KHz
|
||||
*/
|
||||
void begin()
|
||||
{
|
||||
PDM.begin(1, 16000);
|
||||
PDM.setGain(20);
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Gets if the microphone has new data
|
||||
* @return true if the microphone has new data, otherwise false
|
||||
*/
|
||||
bool hasData()
|
||||
{
|
||||
return _hasData;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Get the root mean squared data value from the microphone
|
||||
* @return The root mean square value of the last data point from the microphone
|
||||
*/
|
||||
int16_t data()
|
||||
{
|
||||
return _rms;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Gets the last root mean squared value and resets the calculation
|
||||
* @return The last root mean square value
|
||||
*/
|
||||
int16_t pop()
|
||||
{
|
||||
int16_t rms = data();
|
||||
reset();
|
||||
return rms;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Reads the audio data from the PDM object and calculates the
|
||||
* root mean square value
|
||||
*/
|
||||
void update()
|
||||
{
|
||||
// Get the available bytes from the microphone
|
||||
int bytesAvailable = PDM.available();
|
||||
|
||||
// Check we have a full buffers worth
|
||||
if (bytesAvailable == MICROPHONE_BUFFER_SIZE_IN_BYTES)
|
||||
{
|
||||
// Read from the buffer
|
||||
int16_t _buffer[MICROPHONE_BUFFER_SIZE_IN_WORDS];
|
||||
|
||||
_hasData = true;
|
||||
PDM.read(_buffer, bytesAvailable);
|
||||
|
||||
// Calculate a running root mean square value
|
||||
arm_rms_q15((q15_t *)_buffer, MICROPHONE_BUFFER_SIZE_IN_WORDS, (q15_t *)&_rms);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Mark data as read
|
||||
*/
|
||||
void reset()
|
||||
{
|
||||
_hasData = false;
|
||||
}
|
||||
|
||||
private:
|
||||
int16_t _rms;
|
||||
bool _hasData;
|
||||
};
|
||||
```
|
||||
|
||||
This code defines a class called `Mic` that uses the [Arduino `PDM` library](https://www.arduino.cc/en/Reference/PDM) to get data from the microphone at a sample rate of 16KHz, then uses the `arm_math` library to calculate the RMS of the incoming data from the microphone buffer.
|
||||
|
||||
The PDM buffer is read in blocks of 256 2-byte values, with each full buffer converted to a single RMS value. Once the buffer has been converted to a single RMS value, that value can be read and the next full buffer converted to a new value.
|
||||
|
||||
A 256 value buffer at a sample rate of 16KHz is enough for 1/64th of a second, so 64 RMS value will be created for each second of audio.
|
||||
|
||||
You can read the code and the comments to understand how each call works.
|
||||
|
||||
### Add the program code
|
||||
|
||||
The main program now needs to be written to read data from the microphone at regular intervals, then output the RMS values to the serial port. The data that is captured needs to be stored somewhere so that it can be used to train the model, and the easiest way to do this is to output it from the Arduino device to the serial port, then monitor that serial port using PlatformIO.
|
||||
|
||||
1. Open the `main.cpp` file
|
||||
|
||||
1. Replace the existing code in this file with the following:
|
||||
|
||||
```cpp
|
||||
/**
|
||||
* Audio Capture
|
||||
*
|
||||
* This program listens on the microphone capturing audio data.
|
||||
* Instead of capturing raw data, it captures root mean squared values allowing
|
||||
* audio to be reduced to a few values instead of thousands of values every second.
|
||||
* These values are then output to the serial port - one line per audio sample.
|
||||
*
|
||||
* This data can then be used to train a TinyML model to classify audio.
|
||||
*/
|
||||
|
||||
#include "mic.h"
|
||||
|
||||
// Settings for the audio
|
||||
// Try tuning these if you need different results
|
||||
// 128 samples is enough for 2 seconds of audio - it's captured at 64 samples per second
|
||||
#define SAMPLES 128
|
||||
#define GAIN (1.0f / 50)
|
||||
#define SOUND_THRESHOLD 1000
|
||||
|
||||
// An array of the audio samples
|
||||
float features[SAMPLES];
|
||||
|
||||
// A wrapper for the microphone
|
||||
Mic mic;
|
||||
|
||||
/**
|
||||
* @brief PDM callback to update the data in the mic object
|
||||
*/
|
||||
void onAudio()
|
||||
{
|
||||
mic.update();
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Read given number of samples from mic
|
||||
* @return True if there is enough data captured from the microphone,
|
||||
* otherwise False
|
||||
*/
|
||||
bool recordAudioSample()
|
||||
{
|
||||
// Check the microphone class as captured enough data
|
||||
if (mic.hasData() && mic.pop() > SOUND_THRESHOLD)
|
||||
{
|
||||
// Loop through the samples, waiting to capture data if needed
|
||||
for (int i = 0; i < SAMPLES; i++)
|
||||
{
|
||||
while (!mic.hasData())
|
||||
delay(1);
|
||||
|
||||
// Add the features to the array
|
||||
features[i] = mic.pop() * GAIN;
|
||||
}
|
||||
|
||||
// Return that we have features
|
||||
return true;
|
||||
}
|
||||
|
||||
// Return that we don't have enough data yet
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Sets up the serial port and the microphone
|
||||
*/
|
||||
void setup()
|
||||
{
|
||||
// Start the serial connection so the captured audio data can be output
|
||||
Serial.begin(115200);
|
||||
|
||||
// Set up the microphone callback
|
||||
PDM.onReceive(onAudio);
|
||||
|
||||
// Start listening
|
||||
mic.begin();
|
||||
|
||||
// Wait 3 seconds for everything to get started
|
||||
delay(3000);
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Runs continuously capturing audio data and writing it to
|
||||
* the serial port
|
||||
*/
|
||||
void loop()
|
||||
{
|
||||
// wait for audio data
|
||||
if (recordAudioSample())
|
||||
{
|
||||
// print the audio data to serial port
|
||||
for (int i = 0; i < SAMPLES; i++)
|
||||
{
|
||||
Serial.print(features[i], 6);
|
||||
|
||||
// Seperate the audio values with commas, at the last value
|
||||
// send a newline
|
||||
Serial.print(i == SAMPLES - 1 ? '\n' : ',');
|
||||
}
|
||||
|
||||
// Wait between samples
|
||||
delay(1000);
|
||||
}
|
||||
|
||||
// Sleep to allow background microphone processing
|
||||
delay(20);
|
||||
}
|
||||
```
|
||||
|
||||
This code sets up the microphone in the `setup` function. It then registers for callbacks from the microphone when the 256 2-byte value buffer is full.
|
||||
|
||||
Each time a callback is fired, the code will check to see if the RMS value exceeds a threshold and if it does start gathering data. This is done to ignore silence.
|
||||
|
||||
Once sound is detected, 128 RMS values are read from the buffer - enough for 2 seconds of data. This is then written as comma-separated values to the serial port, with a new line after each 2-seconds of data. This means the output will be a valid CSV file, with one line per audio sample.
|
||||
|
||||
> If you need different length audio recordings, then change the value of `SAMPLES` to 64 x the number of seconds. For example, for 4 seconds of data set it to 256. Be warned that the longer the audio, the bigger the model - too long will exceed the available RAM when building the final project.
|
||||
|
||||
## Run the code
|
||||
|
||||
Now the code is written, it can be deployed to the Arduino device and run to capture data.
|
||||
|
||||
### Upload the code to the device
|
||||
|
||||
1. Connect your Arduino board to your PC or Mac using a USB cable
|
||||
|
||||
1. From Visual Studio Code, launch the Command Palette. This is a pop-up menu that allows you to run actions from VS Code as well as any extensions installed.
|
||||
|
||||
1. If you are using Windows or Linux, press `Ctrl+Shift+p`
|
||||
1. If you are using macOS, press `Command+Shift+p`
|
||||
|
||||
1. Search for `PlatformIO: Upload` by typing in the box, then select that option
|
||||
|
||||
|
||||
|
||||
The relevant tooling will be installed, and the code will be compiled for the device. Once built, it will be uploaded to the Arduino. You will see the progress in the VS Code terminal.
|
||||
|
||||
```output
|
||||
Linking .pio/build/nano33ble/firmware.elf
|
||||
Checking size .pio/build/nano33ble/firmware.elf
|
||||
Building .pio/build/nano33ble/firmware.bin
|
||||
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
|
||||
RAM: [== ] 16.8% (used 44080 bytes from 262144 bytes)
|
||||
Flash: [= ] 8.2% (used 80236 bytes from 983040 bytes)
|
||||
================================================ [SUCCESS] Took 12.28 seconds ================================================
|
||||
```
|
||||
|
||||
### Monitor the output
|
||||
|
||||
The serial output of the Arduino is the USB cable that is used to connect it to your PC or Mac. You can monitor the data being sent using the PlatformIO serial monitor.
|
||||
|
||||
1. Launch the serial monitor using one of these methods:
|
||||
* Open the VS Code Command Palette, search for `PlatformIO: Serial Monitor` and select this option
|
||||
|
||||
|
||||
|
||||
* Select the Serial Monitor button from the status bar
|
||||
|
||||
|
||||
|
||||
The serial monitor will listen for all messages from the device. You'll see nothing output to start with.
|
||||
|
||||
### Capture audio data
|
||||
|
||||
To train the model, you will need multiple audio samples for each label you want to classify - for example if you want to detect the difference between "Yes" and "No", you will need multiple samples for the word "Yes", and multiple for the word "No". Ideally you'd want between 15-30 samples per label.
|
||||
|
||||
1. Make the relevant sound into the microphone on the Arduino device, either by speaking in to it, or playing the relevant audio. You will see a line of samples appear on the serial monitor output.
|
||||
|
||||
1. Repeat this multiple times, slightly adjusting the position of the microphone relative to the audio. You want 15-30 samples. Each new sample will be a new line.
|
||||
|
||||
```output
|
||||
> Executing task: pio device monitor <
|
||||
|
||||
--- Available filters and text transformations: colorize, debug, default, direct, hexlify, log2file, nocontrol, printable, send_on_enter, time
|
||||
--- More details at http://bit.ly/pio-monitor-filters
|
||||
--- Miniterm on /dev/cu.usbmodem101 9600,8,N,1 ---
|
||||
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
|
||||
25.599998,42.820000,13.540000,31.559999,43.279999,45.919998,46.059998,44.919998,40.459999,29.840000,30.500000,27.799999,23.459999,19.139999,16.980000,9.559999,8.860000,6.260000,0.000000,3.620000,0.000000,3.620000,8.080000,11.440000,12.540000,15.339999,14.020000,14.020000,12.000000,10.860000,8.860000,6.260000,3.620000,0.000000,0.000000,0.000000,3.620000,3.620000,6.260000,8.860000,9.559999,10.860000,11.440000,12.540000,13.040000,13.540000,14.020000,14.920000,15.780000,16.180000,16.580000,16.580000,16.980000,16.980000,17.359999,16.980000,16.580000,16.180000,15.780000,14.920000,14.480000,13.540000,13.040000,12.540000,11.440000,10.860000,10.219999,9.559999,8.860000,8.080000,8.080000,7.240000,7.240000,6.260000,6.260000,5.100000,5.100000,5.100000,5.100000,3.620000,3.620000,3.620000,3.620000,3.620000,3.620000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
|
||||
38.299999,34.139999,31.139999,27.799999,19.480000,7.240000,7.240000,8.860000,17.719999,20.779999,23.740000,21.400000,21.100000,20.779999,18.459999,15.339999,10.860000,5.100000,3.620000,0.000000,0.000000,0.000000,0.000000,0.000000,3.620000,3.620000,5.100000,5.100000,6.260000,8.080000,9.559999,11.440000,12.540000,14.020000,14.480000,15.339999,15.780000,16.180000,16.980000,17.359999,17.719999,18.100000,17.719999,17.719999,16.980000,16.580000,15.780000,15.339999,14.480000,13.540000,12.540000,12.000000,10.860000,10.219999,8.860000,8.860000,8.080000,7.240000,7.240000,6.260000,5.100000,5.100000,5.100000,3.620000,3.620000,3.620000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
|
||||
```
|
||||
|
||||
1. Copy all the data from the serial monitor into a text file. Save this as `<label>.csv` - a comma-separated values file. Replace `<label>` with the label you want to use for this audio, for example if you captured saying the word "Yes" then save this as `Yes.csv`. You only want the audio samples, so ignore the monitor output before the samples.
|
||||
|
||||
1. Clear the serial monitor output by right-clicking and selecting *Clear* and repeat this for the other labels you want to classify.
|
||||
|
||||
## Next steps
|
||||
|
||||
In this step you programed the Arduino device to capture audio data ready to classify to train the model. In the [next step](./train-model.md) you will use the output from the audio capture to train a TinyML model.
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# Create the audio classifier
|
||||
|
||||
In the [previous step](./train-model.md), you used the output from the audio capture to train a TinyML model. In this step you will use this model to create an audio classifier.
|
||||
|
||||
##
|
||||
|
||||
## Next steps
|
||||
|
||||
In this step you used the model create an audio classifier.
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# Train the TinyML model
|
||||
|
||||
In the [previous step](./audio-capture.md) you programed the Arduino device to capture audio data ready to classify to train the model. In this step you will use the output from the audio capture to train a TinyML model.
|
||||
|
||||
##
|
||||
|
||||
## Next steps
|
||||
|
||||
In this step, you used the output from the audio capture to train a TinyML model. In the [next step](./audio-classifier.md) you will use this model to create an audio classifier.
|
||||
Загрузка…
Ссылка в новой задаче