diff options
Diffstat (limited to 'python/pyarmnn/examples/speech_recognition')
16 files changed, 177 insertions, 793 deletions
diff --git a/python/pyarmnn/examples/speech_recognition/README.md b/python/pyarmnn/examples/speech_recognition/README.md index c4096efcc5..c39959bfbc 100644 --- a/python/pyarmnn/examples/speech_recognition/README.md +++ b/python/pyarmnn/examples/speech_recognition/README.md @@ -29,31 +29,31 @@ Install the PortAudio package: $ sudo apt-get install libsndfile1 libportaudio2 ``` -Install the required Python modules: +Install the required Python modules: ```bash $ pip install -r requirements.txt ``` ### Model +The model we are using is the [Wav2Letter](https://github.com/ARM-software/ML-zoo/tree/master/models/speech_recognition/wav2letter/tflite_int8 ) which can be found in the [Arm Model Zoo repository]( +https://github.com/ARM-software/ML-zoo/tree/master/models). -The model for this can be found in the Arm Model Zoo repository: -https://github.com/ARM-software/ML-zoo/tree/master/models - -The model we're looking for: -https://github.com/ARM-software/ML-zoo/tree/master/models/speech_recognition/wav2letter/tflite_int8 +A small selection of suitable wav files containing human speech can be found [here](https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/sampledata/audiofiles). +Labels for this model are defined within run_audio_file.py. ## Performing Automatic Speech Recognition ### Processing Audio Files +Please ensure that your audio file has a sampling rate of 16000Hz. + To run ASR on an audio file, use the following command: ```bash -$ python run_audio_file.py --audio_file_path <path/to/your_audio> --model_file_path <path/to/your_model> --labels_file_path <path/to/your_labels> +$ python run_audio_file.py --audio_file_path <path/to/your_audio> --model_file_path <path/to/your_model> ``` -Please ensure that your audio file has a sampling rate of 16000Hz. You may also add the optional flags: @@ -79,15 +79,18 @@ You may also add the optional flags: ### Initialization -The application parses the supplied user arguments and loads the audio file into the `AudioCapture` class, which initialises the audio source and sets sampling parameters required by the model with `ModelParams` class. +The application parses the supplied user arguments and loads the audio file in chunks through the `capture_audio()` method which accepts sampling criteria as an `AudioCaptureParams` tuple. -`AudioCapture` helps us to capture chunks of audio data from the source. With ASR from an audio file, the application will create a generator object to yield blocks of audio data from the file with a minimum sample size. +With ASR from an audio file, the application will create a generator object to yield blocks of audio data from the file with a minimum sample size defined in AudioCaptureParams. -To interpret the inference result of the loaded network, the application must load the labels that are associated with the model. The `dict_labels()` function creates a dictionary that is keyed on the classification index at the output node of the model. The values of the dictionary are the corresponding characters. +MFCC features are extracted from each block based on criteria defined in the `MFCCParams` tuple. +these extracted features constitute the input tensors for the model. + +To interpret the inference result of the loaded network; the application passes the label dictionary defined in run_audio_file.py to a decoder and displays the result. ### Creating a network -A PyArmNN application must import a graph from file using an appropriate parser. Arm NN provides parsers for various model file types, including TFLite, TF, and ONNX. These parsers are libraries for loading neural networks of various formats into the Arm NN runtime. +A PyArmNN application must import a graph from file using an appropriate parser. Arm NN provides parsers for various model file types, including TFLite and ONNX. These parsers are libraries for loading neural networks of various formats into the Arm NN runtime. Arm NN supports optimized execution on multiple CPU, GPU, and Ethos-N devices. Before executing a graph, the application must select the appropriate device context by using `IRuntime()` to create a runtime context with default options. We can optimize the imported graph by specifying a list of backends in order of preference and implementing backend-specific optimizations, identified by a unique string, for example CpuAcc, GpuAcc, CpuRef represent the accelerated CPU and GPU backends and the CPU reference kernels respectively. @@ -120,13 +123,17 @@ self.output_binding_info = parser.GetNetworkOutputBindingInfo(graph_id, output_n ``` ### Automatic speech recognition pipeline +Mel-frequency Cepstral Coefficients (MFCCs, [see Wikipedia](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum)) are extracted based on criteria defined in the MFCCParams tuple and associated`MFCC Class`. +MFCCs are the result of computing the dot product of the Discrete Cosine Transform (DCT) Matrix and the log of the Mel energy. + +The `MFCC` class is used in conjunction with the `AudioPreProcessor` class to extract and process MFCC features from a given audio frame. + -The `MFCC` class is used to extract the Mel-frequency Cepstral Coefficients (MFCCs, [see Wikipedia](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum)) from a given audio frame to be used as features for the network. MFCCs are the result of computing the dot product of the Discrete Cosine Transform (DCT) Matrix and the log of the Mel energy. +After all the MFCCs needed for an inference have been extracted from the audio data we convolve them with 1-dimensional Savitzky-Golay filters to compute the first and second MFCC derivatives with respect to time. The MFCCs and the derivatives constitute the input tensors that will be classified by an `ArmnnNetworkExecutor`object. -After all the MFCCs needed for an inference have been extracted from the audio data, we convolve them with 1-dimensional Savitzky-Golay filters to compute the first and second MFCC derivatives with respect to time. The MFCCs and the derivatives are concatenated to make the input tensor for the model. ```python -# preprocess.py +# mfcc.py & wav2lettermfcc.py # Extract MFCC features log_mel_energy = np.maximum(log_mel_energy, log_mel_energy.max() - top_db) mfcc_feats = np.dot(self.__dct_matrix, log_mel_energy) @@ -165,4 +172,4 @@ Having now gained a solid understanding of performing automatic speech recogniti An important step to improving accuracy of the generated output sentences is by providing cleaner data to the network. This can be done by including additional preprocessing steps such as noise reduction of your audio data. -In this application, we had used a greedy decoder to decode the integer-encoded output however, better results can be achieved by implementing a beam search decoder. You may even try adding a language model at the end to aim to correct any spelling mistakes the model may produce. +In this application, we had used a greedy decoder to decode the integer-encoded output however, better results can be achieved by implementing a beam search decoder. You may even try adding a language model at the end to aim to correct any spelling mistakes the model may produce.
\ No newline at end of file diff --git a/python/pyarmnn/examples/speech_recognition/audio_capture.py b/python/pyarmnn/examples/speech_recognition/audio_capture.py deleted file mode 100644 index 0c899208a4..0000000000 --- a/python/pyarmnn/examples/speech_recognition/audio_capture.py +++ /dev/null @@ -1,56 +0,0 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. -# SPDX-License-Identifier: MIT - -"""Contains AudioCapture class for capturing chunks of audio data from file.""" - -from typing import Generator - -import numpy as np -import soundfile as sf - - -class ModelParams: - def __init__(self, model_file_path: str): - """Defines sampling parameters for model used. - - Args: - model_file_path: Path to ASR model to use. - """ - self.path = model_file_path - self.mono = True - self.dtype = np.float32 - self.samplerate = 16000 - self.min_samples = 47712 # (model_input_size-1)*stride + frame_len - - -class AudioCapture: - def __init__(self, model_params): - """Sampling parameters for model used.""" - self.model_params = model_params - - def from_audio_file(self, audio_file_path, overlap=31712) -> Generator[np.ndarray, None, None]: - """Creates a generator that yields audio data from a file. Data is padded with - zeros if necessary to make up minimum number of samples. - - Args: - audio_file_path: Path to audio file provided by user. - overlap: The overlap with previous buffer. We need the offset to be the same as the inner context - of the mfcc output, which is sized as 100 x 39. Each mfcc compute produces 1 x 39 vector, - and consumes 160 audio samples. The default overlap is then calculated to be 47712 - (160 x 100) - where 47712 is the min_samples needed for 1 inference of wav2letter. - - Yields: - Blocks of audio data of minimum sample size. - """ - with sf.SoundFile(audio_file_path) as audio_file: - for block in audio_file.blocks( - blocksize=self.model_params.min_samples, - dtype=self.model_params.dtype, - always_2d=True, - fill_value=0, - overlap=overlap - ): - # Convert to mono if specified - if self.model_params.mono and block.shape[0] > 1: - block = np.mean(block, axis=1) - yield block diff --git a/python/pyarmnn/examples/speech_recognition/audio_utils.py b/python/pyarmnn/examples/speech_recognition/audio_utils.py index f03d2e1290..1ac78e8074 100644 --- a/python/pyarmnn/examples/speech_recognition/audio_utils.py +++ b/python/pyarmnn/examples/speech_recognition/audio_utils.py @@ -1,10 +1,9 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. +# Copyright © 2021 Arm Ltd and Contributors. All rights reserved. # SPDX-License-Identifier: MIT """Utilities for speech recognition apps.""" import numpy as np -import pyarmnn as ann def decode(model_output: np.ndarray, labels: dict) -> str: @@ -50,33 +49,6 @@ def display_text(text: str): print(text, sep="", end="", flush=True) -def quantize_input(data, input_binding_info): - """Quantize the float input to (u)int8 ready for inputting to model.""" - if data.ndim != 2: - raise RuntimeError("Audio data must have 2 dimensions for quantization") - - quant_scale = input_binding_info[1].GetQuantizationScale() - quant_offset = input_binding_info[1].GetQuantizationOffset() - data_type = input_binding_info[1].GetDataType() - - if data_type == ann.DataType_QAsymmS8: - data_type = np.int8 - elif data_type == ann.DataType_QAsymmU8: - data_type = np.uint8 - else: - raise ValueError("Could not quantize data to required data type") - - d_min = np.iinfo(data_type).min - d_max = np.iinfo(data_type).max - - for row in range(data.shape[0]): - for col in range(data.shape[1]): - data[row, col] = (data[row, col] / quant_scale) + quant_offset - data[row, col] = np.clip(data[row, col], d_min, d_max) - data = data.astype(data_type) - return data - - def decode_text(is_first_window, labels, output_result): """ Slices the text appropriately depending on the window, and decodes for wav2letter output. @@ -88,7 +60,6 @@ def decode_text(is_first_window, labels, output_result): is_first_window: Boolean to show if it is the first window we are running inference on labels: the label set output_result: the output from the inference - text: the current text string, to be displayed at the end Returns: current_r_context: the current right context text: the current text string, with the latest output decoded and appended @@ -109,25 +80,3 @@ def decode_text(is_first_window, labels, output_result): # Store the right context, we will need it after the last inference current_r_context = decode(output_result[0][0][0][right_context_start:], labels) return current_r_context, text - - -def prepare_input_tensors(audio_data, input_binding_info, mfcc_preprocessor): - """ - Takes a block of audio data, extracts the MFCC features, quantizes the array, and uses ArmNN to create the - input tensors. - - Args: - audio_data: The audio data to process - mfcc_instance: the mfcc class instance - input_binding_info: the model input binding info - mfcc_preprocessor: the mfcc preprocessor instance - Returns: - input_tensors: the prepared input tensors, ready to be consumed by the ArmNN NetworkExecutor - """ - - data_type = input_binding_info[1].GetDataType() - input_tensor = mfcc_preprocessor.extract_features(audio_data) - if data_type != ann.DataType_Float32: - input_tensor = quantize_input(input_tensor, input_binding_info) - input_tensors = ann.make_input_tensors([input_binding_info], [input_tensor]) - return input_tensors diff --git a/python/pyarmnn/examples/speech_recognition/preprocess.py b/python/pyarmnn/examples/speech_recognition/preprocess.py deleted file mode 100644 index 553ddba5de..0000000000 --- a/python/pyarmnn/examples/speech_recognition/preprocess.py +++ /dev/null @@ -1,260 +0,0 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. -# SPDX-License-Identifier: MIT - -"""Class used to extract the Mel-frequency cepstral coefficients from a given audio frame.""" - -import numpy as np - - -class MFCCParams: - def __init__(self, sampling_freq, num_fbank_bins, - mel_lo_freq, mel_hi_freq, num_mfcc_feats, frame_len, use_htk_method, n_FFT): - self.sampling_freq = sampling_freq - self.num_fbank_bins = num_fbank_bins - self.mel_lo_freq = mel_lo_freq - self.mel_hi_freq = mel_hi_freq - self.num_mfcc_feats = num_mfcc_feats - self.frame_len = frame_len - self.use_htk_method = use_htk_method - self.n_FFT = n_FFT - - -class MFCC: - - def __init__(self, mfcc_params): - self.mfcc_params = mfcc_params - self.FREQ_STEP = 200.0 / 3 - self.MIN_LOG_HZ = 1000.0 - self.MIN_LOG_MEL = self.MIN_LOG_HZ / self.FREQ_STEP - self.LOG_STEP = 1.8562979903656 / 27.0 - self.__frame_len_padded = int(2 ** (np.ceil((np.log(self.mfcc_params.frame_len) / np.log(2.0))))) - self.__filter_bank_initialised = False - self.__frame = np.zeros(self.__frame_len_padded) - self.__buffer = np.zeros(self.__frame_len_padded) - self.__filter_bank_filter_first = np.zeros(self.mfcc_params.num_fbank_bins) - self.__filter_bank_filter_last = np.zeros(self.mfcc_params.num_fbank_bins) - self.__mel_energies = np.zeros(self.mfcc_params.num_fbank_bins) - self.__dct_matrix = self.create_dct_matrix(self.mfcc_params.num_fbank_bins, self.mfcc_params.num_mfcc_feats) - self.__mel_filter_bank = self.create_mel_filter_bank() - self.__np_mel_bank = np.zeros([self.mfcc_params.num_fbank_bins, int(self.mfcc_params.n_FFT / 2) + 1]) - - for i in range(self.mfcc_params.num_fbank_bins): - k = 0 - for j in range(int(self.__filter_bank_filter_first[i]), int(self.__filter_bank_filter_last[i]) + 1): - self.__np_mel_bank[i, j] = self.__mel_filter_bank[i][k] - k += 1 - - def mel_scale(self, freq, use_htk_method): - """ - Gets the mel scale for a particular sample frequency. - - Args: - freq: The sampling frequency. - use_htk_method: Boolean to set whether to use HTK method or not. - - Returns: - the mel scale - """ - if use_htk_method: - return 1127.0 * np.log(1.0 + freq / 700.0) - else: - mel = freq / self.FREQ_STEP - - if freq >= self.MIN_LOG_HZ: - mel = self.MIN_LOG_MEL + np.log(freq / self.MIN_LOG_HZ) / self.LOG_STEP - return mel - - def inv_mel_scale(self, mel_freq, use_htk_method): - """ - Gets the sample frequency for a particular mel. - - Args: - mel_freq: The mel frequency. - use_htk_method: Boolean to set whether to use HTK method or not. - - Returns: - the sample frequency - """ - if use_htk_method: - return 700.0 * (np.exp(mel_freq / 1127.0) - 1.0) - else: - freq = self.FREQ_STEP * mel_freq - - if mel_freq >= self.MIN_LOG_MEL: - freq = self.MIN_LOG_HZ * np.exp(self.LOG_STEP * (mel_freq - self.MIN_LOG_MEL)) - return freq - - def mfcc_compute(self, audio_data): - """ - Extracts the MFCC for a single frame. - - Args: - audio_data: The audio data to process. - - Returns: - the MFCC features - """ - if len(audio_data) != self.mfcc_params.frame_len: - raise ValueError( - f"audio_data buffer size {len(audio_data)} does not match the frame length {self.mfcc_params.frame_len}") - - audio_data = np.array(audio_data) - spec = np.abs(np.fft.rfft(np.hanning(self.mfcc_params.n_FFT + 1)[0:self.mfcc_params.n_FFT] * audio_data, - self.mfcc_params.n_FFT)) ** 2 - mel_energy = np.dot(self.__np_mel_bank.astype(np.float32), - np.transpose(spec).astype(np.float32)) - - mel_energy += 1e-10 - log_mel_energy = 10.0 * np.log10(mel_energy) - top_db = 80.0 - - log_mel_energy = np.maximum(log_mel_energy, log_mel_energy.max() - top_db) - - mfcc_feats = np.dot(self.__dct_matrix, log_mel_energy) - - return mfcc_feats - - def create_dct_matrix(self, num_fbank_bins, num_mfcc_feats): - """ - Creates the Discrete Cosine Transform matrix to be used in the compute function. - - Args: - num_fbank_bins: The number of filter bank bins - num_mfcc_feats: the number of MFCC features - - Returns: - the DCT matrix - """ - dct_m = np.zeros(num_fbank_bins * num_mfcc_feats) - for k in range(num_mfcc_feats): - for n in range(num_fbank_bins): - if k == 0: - dct_m[(k * num_fbank_bins) + n] = 2 * np.sqrt(1 / (4 * num_fbank_bins)) * np.cos( - (np.pi / num_fbank_bins) * (n + 0.5) * k) - else: - dct_m[(k * num_fbank_bins) + n] = 2 * np.sqrt(1 / (2 * num_fbank_bins)) * np.cos( - (np.pi / num_fbank_bins) * (n + 0.5) * k) - - dct_m = np.reshape(dct_m, [self.mfcc_params.num_mfcc_feats, self.mfcc_params.num_fbank_bins]) - return dct_m - - def create_mel_filter_bank(self): - """ - Creates the Mel filter bank. - - Returns: - the mel filter bank - """ - num_fft_bins = int(self.__frame_len_padded / 2) - fft_bin_width = self.mfcc_params.sampling_freq / self.__frame_len_padded - - mel_low_freq = self.mel_scale(self.mfcc_params.mel_lo_freq, False) - mel_high_freq = self.mel_scale(self.mfcc_params.mel_hi_freq, False) - mel_freq_delta = (mel_high_freq - mel_low_freq) / (self.mfcc_params.num_fbank_bins + 1) - - this_bin = np.zeros(num_fft_bins) - mel_fbank = [0] * self.mfcc_params.num_fbank_bins - - for bin_num in range(self.mfcc_params.num_fbank_bins): - left_mel = mel_low_freq + bin_num * mel_freq_delta - center_mel = mel_low_freq + (bin_num + 1) * mel_freq_delta - right_mel = mel_low_freq + (bin_num + 2) * mel_freq_delta - first_index = last_index = -1 - - for i in range(num_fft_bins): - freq = (fft_bin_width * i) - mel = self.mel_scale(freq, False) - this_bin[i] = 0.0 - - if (mel > left_mel) and (mel < right_mel): - if mel <= center_mel: - weight = (mel - left_mel) / (center_mel - left_mel) - else: - weight = (right_mel - mel) / (right_mel - center_mel) - - enorm = 2.0 / (self.inv_mel_scale(right_mel, False) - self.inv_mel_scale(left_mel, False)) - weight *= enorm - this_bin[i] = weight - - if first_index == -1: - first_index = i - last_index = i - - self.__filter_bank_filter_first[bin_num] = first_index - self.__filter_bank_filter_last[bin_num] = last_index - mel_fbank[bin_num] = np.zeros(last_index - first_index + 1) - j = 0 - - for i in range(first_index, last_index + 1): - mel_fbank[bin_num][j] = this_bin[i] - j += 1 - - return mel_fbank - - -class Preprocessor: - - def __init__(self, mfcc, model_input_size, stride): - self.model_input_size = model_input_size - self.stride = stride - - # Savitzky - Golay differential filters - self.__savgol_order1_coeffs = np.array([6.66666667e-02, 5.00000000e-02, 3.33333333e-02, - 1.66666667e-02, -3.46944695e-18, -1.66666667e-02, - -3.33333333e-02, -5.00000000e-02, -6.66666667e-02]) - - self.savgol_order2_coeffs = np.array([0.06060606, 0.01515152, -0.01731602, - -0.03679654, -0.04329004, -0.03679654, - -0.01731602, 0.01515152, 0.06060606]) - - self.__mfcc_calc = mfcc - - def __normalize(self, values): - """ - Normalize values to mean 0 and std 1 - """ - ret_val = (values - np.mean(values)) / np.std(values) - return ret_val - - def __get_features(self, features, mfcc_instance, audio_data): - idx = 0 - while len(features) < self.model_input_size * mfcc_instance.mfcc_params.num_mfcc_feats: - features.extend(mfcc_instance.mfcc_compute(audio_data[idx:idx + int(mfcc_instance.mfcc_params.frame_len)])) - idx += self.stride - - def extract_features(self, audio_data): - """ - Extracts the MFCC features, and calculates each features first and second order derivative. - The matrix returned should be sized appropriately for input to the model, based - on the model info specified in the MFCC instance. - - Args: - mfcc_instance: The instance of MFCC used for this calculation - audio_data: the audio data to be used for this calculation - Returns: - the derived MFCC feature vector, sized appropriately for inference - """ - - num_samples_per_inference = ((self.model_input_size - 1) - * self.stride) + self.__mfcc_calc.mfcc_params.frame_len - if len(audio_data) < num_samples_per_inference: - raise ValueError("audio_data size for feature extraction is smaller than " - "the expected number of samples needed for inference") - - features = [] - self.__get_features(features, self.__mfcc_calc, np.asarray(audio_data)) - features = np.reshape(np.array(features), (self.model_input_size, self.__mfcc_calc.mfcc_params.num_mfcc_feats)) - - mfcc_delta_np = np.zeros_like(features) - mfcc_delta2_np = np.zeros_like(features) - - for i in range(features.shape[1]): - idelta = np.convolve(features[:, i], self.__savgol_order1_coeffs, 'same') - mfcc_delta_np[:, i] = (idelta) - ideltadelta = np.convolve(features[:, i], self.savgol_order2_coeffs, 'same') - mfcc_delta2_np[:, i] = (ideltadelta) - - features = np.concatenate((self.__normalize(features), self.__normalize(mfcc_delta_np), - self.__normalize(mfcc_delta2_np)), axis=1) - - return np.float32(features) diff --git a/python/pyarmnn/examples/speech_recognition/requirements.txt b/python/pyarmnn/examples/speech_recognition/requirements.txt index 4b8f3e6d24..96782eafd0 100644 --- a/python/pyarmnn/examples/speech_recognition/requirements.txt +++ b/python/pyarmnn/examples/speech_recognition/requirements.txt @@ -1,2 +1,5 @@ numpy>=1.19.2 -soundfile>=0.10.3
\ No newline at end of file +soundfile>=0.10.3 +pytest==6.2.4 +pytest-allclose==1.0.0 +sounddevice==0.4.2 diff --git a/python/pyarmnn/examples/speech_recognition/run_audio_file.py b/python/pyarmnn/examples/speech_recognition/run_audio_file.py index 942de2081c..0430f68c16 100644 --- a/python/pyarmnn/examples/speech_recognition/run_audio_file.py +++ b/python/pyarmnn/examples/speech_recognition/run_audio_file.py @@ -1,20 +1,29 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. +# Copyright © 2021 Arm Ltd and Contributors. All rights reserved. # SPDX-License-Identifier: MIT """Automatic speech recognition with PyArmNN demo for processing audio clips to text.""" import sys import os -from argparse import ArgumentParser +import numpy as np script_dir = os.path.dirname(__file__) sys.path.insert(1, os.path.join(script_dir, '..', 'common')) +from argparse import ArgumentParser from network_executor import ArmnnNetworkExecutor -from utils import dict_labels -from preprocess import MFCCParams, Preprocessor, MFCC -from audio_capture import AudioCapture, ModelParams -from audio_utils import decode_text, prepare_input_tensors, display_text +from utils import prepare_input_tensors +from audio_capture import AudioCaptureParams, capture_audio +from audio_utils import decode_text, display_text +from wav2letter_mfcc import Wav2LetterMFCC, W2LAudioPreprocessor +from mfcc import MFCCParams + +# Model Specific Labels +labels = {0: 'a', 1: 'b', 2: 'c', 3: 'd', 4: 'e', 5: 'f', 6: 'g', 7: 'h', 8: 'i', 9: 'j', 10: 'k', 11: 'l', 12: 'm', + 13: 'n', + 14: 'o', 15: 'p', 16: 'q', 17: 'r', 18: 's', 19: 't', 20: 'u', 21: 'v', 22: 'w', 23: 'x', 24: 'y', + 25: 'z', + 26: "'", 27: ' ', 28: '$'} def parse_args(): @@ -32,12 +41,6 @@ def parse_args(): help="Path to ASR model to use", ) parser.add_argument( - "--labels_file_path", - required=True, - type=str, - help="Path to text file containing labels to map to model output", - ) - parser.add_argument( "--preferred_backends", type=str, nargs="+", @@ -52,22 +55,23 @@ def parse_args(): def main(args): # Read command line args audio_file = args.audio_file_path - model = ModelParams(args.model_file_path) - labels = dict_labels(args.labels_file_path) # Create the ArmNN inference runner - network = ArmnnNetworkExecutor(model.path, args.preferred_backends) + network = ArmnnNetworkExecutor(args.model_file_path, args.preferred_backends) + + # Specify model specific audio data requirements + audio_capture_params = AudioCaptureParams(dtype=np.float32, overlap=31712, min_samples=47712, sampling_freq=16000, + mono=True) + + buffer = capture_audio(audio_file, audio_capture_params) - audio_capture = AudioCapture(model) - buffer = audio_capture.from_audio_file(audio_file) + # Extract features and create the preprocessor - # Create the preprocessor mfcc_params = MFCCParams(sampling_freq=16000, num_fbank_bins=128, mel_lo_freq=0, mel_hi_freq=8000, - num_mfcc_feats=13, frame_len=512, use_htk_method=False, n_FFT=512) - mfcc = MFCC(mfcc_params) - preprocessor = Preprocessor(mfcc, model_input_size=296, stride=160) + num_mfcc_feats=13, frame_len=512, use_htk_method=False, n_fft=512) - text = "" + wmfcc = Wav2LetterMFCC(mfcc_params) + preprocessor = W2LAudioPreprocessor(wmfcc, model_input_size=296, stride=160) current_r_context = "" is_first_window = True diff --git a/python/pyarmnn/examples/speech_recognition/tests/conftest.py b/python/pyarmnn/examples/speech_recognition/tests/conftest.py index 730c291cfa..151816e919 100644 --- a/python/pyarmnn/examples/speech_recognition/tests/conftest.py +++ b/python/pyarmnn/examples/speech_recognition/tests/conftest.py @@ -1,34 +1,24 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
-# SPDX-License-Identifier: MIT
-
-import os
-import ntpath
-
-import urllib.request
-
-import pytest
-
-script_dir = os.path.dirname(__file__)
-
-@pytest.fixture(scope="session")
-def test_data_folder(request):
- """
- This fixture returns path to folder with shared test resources among all tests
- """
-
- data_dir = os.path.join(script_dir, "testdata")
-
- if not os.path.exists(data_dir):
- os.mkdir(data_dir)
-
- files_to_download = ["https://raw.githubusercontent.com/Azure-Samples/cognitive-services-speech-sdk/master"
- "/sampledata/audiofiles/myVoiceIsMyPassportVerifyMe04.wav"]
-
- for file in files_to_download:
- path, filename = ntpath.split(file)
- file_path = os.path.join(script_dir, "testdata", filename)
- if not os.path.exists(file_path):
- print("\nDownloading test file: " + file_path + "\n")
- urllib.request.urlretrieve(file, file_path)
-
- return data_dir
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. +# SPDX-License-Identifier: MIT + +import os +import ntpath + +import urllib.request + +import pytest + +script_dir = os.path.dirname(__file__) + +@pytest.fixture(scope="session") +def test_data_folder(request): + """ + This fixture returns path to folder with shared test resources among asr tests + """ + + data_dir = os.path.join(script_dir, "testdata") + + if not os.path.exists(data_dir): + os.mkdir(data_dir) + + return data_dir
\ No newline at end of file diff --git a/python/pyarmnn/examples/speech_recognition/tests/context.py b/python/pyarmnn/examples/speech_recognition/tests/context.py deleted file mode 100644 index a810010e9f..0000000000 --- a/python/pyarmnn/examples/speech_recognition/tests/context.py +++ /dev/null @@ -1,13 +0,0 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. -# SPDX-License-Identifier: MIT - -import os -import sys - -sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..', 'common')) -import utils as common_utils - -sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) -import audio_capture -import audio_utils -import preprocess diff --git a/python/pyarmnn/examples/speech_recognition/tests/test_audio_file.py b/python/pyarmnn/examples/speech_recognition/tests/test_audio_file.py deleted file mode 100644 index 281d0df587..0000000000 --- a/python/pyarmnn/examples/speech_recognition/tests/test_audio_file.py +++ /dev/null @@ -1,17 +0,0 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. -# SPDX-License-Identifier: MIT - -import os - -import numpy as np - -from context import audio_capture - - -def test_audio_file(test_data_folder): - audio_file = os.path.join(test_data_folder, "myVoiceIsMyPassportVerifyMe04.wav") - capture = audio_capture.AudioCapture(audio_capture.ModelParams("")) - buffer = capture.from_audio_file(audio_file) - audio_data = next(buffer) - assert audio_data.shape == (47712,) - assert audio_data.dtype == np.float32 diff --git a/python/pyarmnn/examples/speech_recognition/tests/test_decoder.py b/python/pyarmnn/examples/speech_recognition/tests/test_decoder.py index 1db71a47b8..14db7f2064 100644 --- a/python/pyarmnn/examples/speech_recognition/tests/test_decoder.py +++ b/python/pyarmnn/examples/speech_recognition/tests/test_decoder.py @@ -5,13 +5,16 @@ import os import numpy as np -from context import common_utils from context import audio_utils +labels = {0: 'a', 1: 'b', 2: 'c', 3: 'd', 4: 'e', 5: 'f', 6: 'g', 7: 'h', 8: 'i', 9: 'j', 10: 'k', 11: 'l', 12: 'm', + 13: 'n', + 14: 'o', 15: 'p', 16: 'q', 17: 'r', 18: 's', 19: 't', 20: 'u', 21: 'v', 22: 'w', 23: 'x', 24: 'y', + 25: 'z', + 26: "'", 27: ' ', 28: '$'} + def test_labels(test_data_folder): - labels_file = os.path.join(test_data_folder, "wav2letter_labels.txt") - labels = common_utils.dict_labels(labels_file) assert len(labels) == 29 assert labels[26] == "\'" assert labels[27] == r" " @@ -19,10 +22,8 @@ def test_labels(test_data_folder): def test_decoder(test_data_folder): - labels_file = os.path.join(test_data_folder, "wav2letter_labels.txt") - labels = common_utils.dict_labels(labels_file) - output_tensor = os.path.join(test_data_folder, "inf_out.npy") + output_tensor = os.path.join(test_data_folder, "inference_output.npy") encoded = np.load(output_tensor) decoded_text = audio_utils.decode(encoded, labels) - assert decoded_text == "and they walkd immediately out of the apartiment by anothe" + assert decoded_text == "my voice is my pass" diff --git a/python/pyarmnn/examples/speech_recognition/tests/test_mfcc.py b/python/pyarmnn/examples/speech_recognition/tests/test_mfcc.py deleted file mode 100644 index d692ab51c8..0000000000 --- a/python/pyarmnn/examples/speech_recognition/tests/test_mfcc.py +++ /dev/null @@ -1,286 +0,0 @@ -# Copyright © 2020 Arm Ltd and Contributors. All rights reserved. -# SPDX-License-Identifier: MIT - -import numpy as np - -from context import preprocess - -test_wav = [ - -3,0,1,-1,2,3,-2,2, - 1,-2,0,3,-1,8,3,2, - -1,-1,2,7,3,5,6,6, - 6,12,5,6,3,3,5,4, - 4,6,7,7,7,3,7,2, - 8,4,4,2,-4,-1,-1,-4, - 2,1,-1,-4,0,-7,-6,-2, - -5,1,-5,-1,-7,-3,-3,-7, - 0,-3,3,-5,0,1,-2,-2, - -3,-3,-7,-3,-2,-6,-5,-8, - -2,-8,4,-9,-4,-9,-5,-5, - -3,-9,-3,-9,-1,-7,-4,1, - -3,2,-8,-4,-4,-5,1,-3, - -1,0,-1,-2,-3,-2,-4,-1, - 1,-1,3,0,3,2,0,0, - 0,-3,1,1,0,8,3,4, - 1,5,6,4,7,3,3,0, - 3,6,7,6,4,5,9,9, - 5,5,8,1,6,9,6,6, - 7,1,8,1,5,0,5,5, - 0,3,2,7,2,-3,3,0, - 3,0,0,0,2,0,-1,-1, - -2,-3,-8,0,1,0,-3,-3, - -3,-2,-3,-3,-4,-6,-2,-8, - -9,-4,-1,-5,-3,-3,-4,-3, - -6,3,0,-1,-2,-9,-4,-2, - 2,-1,3,-5,-5,-2,0,-2, - 0,-1,-3,1,-2,9,4,5, - 2,2,1,0,-6,-2,0,0, - 0,-1,4,-4,3,-7,-1,5, - -6,-1,-5,4,3,9,-2,1, - 3,0,0,-2,1,2,1,1, - 0,3,2,-1,3,-3,7,0, - 0,3,2,2,-2,3,-2,2, - -3,4,-1,-1,-5,-1,-3,-2, - 1,-1,3,2,4,1,2,-2, - 0,2,7,0,8,-3,6,-3, - 6,1,2,-3,-1,-1,-1,1, - -2,2,1,2,0,-2,3,-2, - 3,-2,1,0,-3,-1,-2,-4, - -6,-5,-8,-1,-4,0,-3,-1, - -1,-1,0,-2,-3,-7,-1,0, - 1,5,0,5,1,1,-3,0, - -6,3,-8,4,-8,6,-6,1, - -6,-2,-5,-6,0,-5,4,-1, - 4,-2,1,2,1,0,-2,0, - 0,2,-2,2,-5,2,0,-2, - 1,-2,0,5,1,0,1,5, - 0,8,3,2,2,0,5,-2, - 3,1,0,1,0,-2,-1,-3, - 1,-1,3,0,3,0,-2,-1, - -4,-4,-4,-1,-4,-4,-3,-6, - -3,-7,-3,-1,-2,0,-5,-4, - -7,-3,-2,-2,1,2,2,8, - 5,4,2,4,3,5,0,3, - 3,6,4,2,2,-2,4,-2, - 3,3,2,1,1,4,-5,2, - -3,0,-1,1,-2,2,5,1, - 4,2,3,1,-1,1,0,6, - 0,-2,-1,1,-1,2,-5,-1, - -5,-1,-6,-3,-3,2,4,0, - -1,-5,3,-4,-1,-3,-4,1, - -4,1,-1,-1,0,-5,-4,-2, - -1,-1,-3,-7,-3,-3,4,4, -] - -def test_mel_scale_function_with_htk_true(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - - mfcc_inst = preprocess.MFCC(mfcc_params) - - mel = mfcc_inst.mel_scale(16, True) - - assert np.isclose(mel, 25.470010570730597) - - -def test_mel_scale_function_with_htk_false(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - - mfcc_inst = preprocess.MFCC(mfcc_params) - - mel = mfcc_inst.mel_scale(16, False) - - assert np.isclose(mel, 0.24) - - -def test_inverse_mel_scale_function_with_htk_true(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - - mfcc_inst = preprocess.MFCC(mfcc_params) - - mel = mfcc_inst.inv_mel_scale(16, True) - - assert np.isclose(mel, 10.008767240008943) - - -def test_inverse_mel_scale_function_with_htk_false(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - - mfcc_inst = preprocess.MFCC(mfcc_params) - - mel = mfcc_inst.inv_mel_scale(16, False) - - assert np.isclose(mel, 1071.170287494467) - - -def test_create_mel_filter_bank(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - - mfcc_inst = preprocess.MFCC(mfcc_params) - - mel_filter_bank = mfcc_inst.create_mel_filter_bank() - - assert len(mel_filter_bank) == 128 - - assert str(mel_filter_bank[0]) == "[0.02837754]" - assert str(mel_filter_bank[1]) == "[0.01438901 0.01398853]" - assert str(mel_filter_bank[2]) == "[0.02877802]" - assert str(mel_filter_bank[3]) == "[0.04236608]" - assert str(mel_filter_bank[4]) == "[0.00040047 0.02797707]" - assert str(mel_filter_bank[5]) == "[0.01478948 0.01358806]" - assert str(mel_filter_bank[50]) == "[0.03298853]" - assert str(mel_filter_bank[100]) == "[0.00260166 0.00588759 0.00914814 0.00798015 0.00476919 0.00158245]" - - -def test_mfcc_compute(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - audio_data = np.array(test_wav) / (2 ** 15) - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - mfcc_inst = preprocess.MFCC(mfcc_params) - mfcc_feats = mfcc_inst.mfcc_compute(audio_data) - - assert np.isclose((mfcc_feats[0]), -834.9656973095651) - assert np.isclose((mfcc_feats[1]), 21.026915475076322) - assert np.isclose((mfcc_feats[2]), 18.628541708201688) - assert np.isclose((mfcc_feats[3]), 7.341153529494758) - assert np.isclose((mfcc_feats[4]), 18.907974386153214) - assert np.isclose((mfcc_feats[5]), -5.360387487466194) - assert np.isclose((mfcc_feats[6]), 6.523572638527085) - assert np.isclose((mfcc_feats[7]), -11.270643644983316) - assert np.isclose((mfcc_feats[8]), 8.375177203773777) - assert np.isclose((mfcc_feats[9]), 12.06721844362991) - assert np.isclose((mfcc_feats[10]), 8.30815892468875) - assert np.isclose((mfcc_feats[11]), -13.499911910889917) - assert np.isclose((mfcc_feats[12]), -18.176121251436165) - - -def test_sliding_window_for_small_num_samples(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - mode_input_size = 9 - stride = 160 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - audio_data = np.array(test_wav) / (2 ** 15) - - full_audio_data = np.tile(audio_data, 9) - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - mfcc_inst = preprocess.MFCC(mfcc_params) - preprocessor = preprocess.Preprocessor(mfcc_inst, mode_input_size, stride) - - input_tensor = preprocessor.extract_features(full_audio_data) - - assert np.isclose(input_tensor[0][0], -3.4660944830426454) - assert np.isclose(input_tensor[0][1], 0.3587718932127629) - assert np.isclose(input_tensor[0][2], 0.3480551325669172) - assert np.isclose(input_tensor[0][3], 0.2976191917228921) - assert np.isclose(input_tensor[0][4], 0.3493037340849936) - assert np.isclose(input_tensor[0][5], 0.2408643285767937) - assert np.isclose(input_tensor[0][6], 0.2939659585037282) - assert np.isclose(input_tensor[0][7], 0.2144552669573928) - assert np.isclose(input_tensor[0][8], 0.302239565899944) - assert np.isclose(input_tensor[0][9], 0.3187368787077345) - assert np.isclose(input_tensor[0][10], 0.3019401051295793) - assert np.isclose(input_tensor[0][11], 0.20449412797602678) - - assert np.isclose(input_tensor[0][38], -0.18751440767749533) - - -def test_sliding_window_for_wav_2_letter_sized_input(): - samp_freq = 16000 - frame_len_ms = 32 - frame_len_samples = samp_freq * frame_len_ms * 0.001 - num_mfcc_feats = 13 - mode_input_size = 296 - stride = 160 - num_fbank_bins = 128 - mel_lo_freq = 0 - mil_hi_freq = 8000 - use_htk = False - n_FFT = 512 - - audio_data = np.zeros(47712, dtype=int) - - mfcc_params = preprocess.MFCCParams(samp_freq, num_fbank_bins, mel_lo_freq, mil_hi_freq, num_mfcc_feats, - frame_len_samples, use_htk, n_FFT) - - mfcc_inst = preprocess.MFCC(mfcc_params) - preprocessor = preprocess.Preprocessor(mfcc_inst, mode_input_size, stride) - - input_tensor = preprocessor.extract_features(audio_data) - - assert len(input_tensor[0]) == 39 - assert len(input_tensor) == 296 diff --git a/python/pyarmnn/examples/speech_recognition/tests/testdata/inf_out.npy b/python/pyarmnn/examples/speech_recognition/tests/testdata/inf_out.npy Binary files differdeleted file mode 100644 index a6f9ec0c70..0000000000 --- a/python/pyarmnn/examples/speech_recognition/tests/testdata/inf_out.npy +++ /dev/null diff --git a/python/pyarmnn/examples/speech_recognition/tests/testdata/inference_output.npy b/python/pyarmnn/examples/speech_recognition/tests/testdata/inference_output.npy Binary files differnew file mode 100644 index 0000000000..88c42e0b70 --- /dev/null +++ b/python/pyarmnn/examples/speech_recognition/tests/testdata/inference_output.npy diff --git a/python/pyarmnn/examples/speech_recognition/tests/testdata/quick_brown_fox_16000khz.wav b/python/pyarmnn/examples/speech_recognition/tests/testdata/quick_brown_fox_16000khz.wav Binary files differdeleted file mode 100644 index 761c36062e..0000000000 --- a/python/pyarmnn/examples/speech_recognition/tests/testdata/quick_brown_fox_16000khz.wav +++ /dev/null diff --git a/python/pyarmnn/examples/speech_recognition/tests/testdata/wav2letter_labels.txt b/python/pyarmnn/examples/speech_recognition/tests/testdata/wav2letter_labels.txt deleted file mode 100644 index d7485b7da2..0000000000 --- a/python/pyarmnn/examples/speech_recognition/tests/testdata/wav2letter_labels.txt +++ /dev/null @@ -1,29 +0,0 @@ -a
-b
-c
-d
-e
-f
-g
-h
-i
-j
-k
-l
-m
-n
-o
-p
-q
-r
-s
-t
-u
-v
-w
-x
-y
-z
-'
-
-$
\ No newline at end of file diff --git a/python/pyarmnn/examples/speech_recognition/wav2letter_mfcc.py b/python/pyarmnn/examples/speech_recognition/wav2letter_mfcc.py new file mode 100644 index 0000000000..1cac24d588 --- /dev/null +++ b/python/pyarmnn/examples/speech_recognition/wav2letter_mfcc.py @@ -0,0 +1,91 @@ +# Copyright © 2021 Arm Ltd and Contributors. All rights reserved. +# SPDX-License-Identifier: MIT + +import numpy as np +import os +import sys + +script_dir = os.path.dirname(__file__) +sys.path.insert(1, os.path.join(script_dir, '..', 'common')) + +from mfcc import MFCC, AudioPreprocessor + + +class Wav2LetterMFCC(MFCC): + """Extends base MFCC class to provide Wav2Letter-specific MFCC requirements.""" + + def __init__(self, mfcc_params): + super().__init__(mfcc_params) + + def spectrum_calc(self, audio_data): + return np.abs(np.fft.rfft(np.hanning(self.mfcc_params.frame_len + 1)[0:self.mfcc_params.frame_len] * audio_data, + self.mfcc_params.n_fft)) ** 2 + + def log_mel(self, mel_energy): + mel_energy += 1e-10 + log_mel_energy = 10.0 * np.log10(mel_energy) + top_db = 80.0 + return np.maximum(log_mel_energy, log_mel_energy.max() - top_db) + + def create_dct_matrix(self, num_fbank_bins, num_mfcc_feats): + """ + Creates the Discrete Cosine Transform matrix to be used in the compute function. + + Args: + num_fbank_bins: The number of filter bank bins + num_mfcc_feats: the number of MFCC features + + Returns: + the DCT matrix + """ + dct_m = np.zeros(num_fbank_bins * num_mfcc_feats) + for k in range(num_mfcc_feats): + for n in range(num_fbank_bins): + if k == 0: + dct_m[(k * num_fbank_bins) + n] = 2 * np.sqrt(1 / (4 * num_fbank_bins)) * np.cos( + (np.pi / num_fbank_bins) * (n + 0.5) * k) + else: + dct_m[(k * num_fbank_bins) + n] = 2 * np.sqrt(1 / (2 * num_fbank_bins)) * np.cos( + (np.pi / num_fbank_bins) * (n + 0.5) * k) + + dct_m = np.reshape(dct_m, [self.mfcc_params.num_mfcc_feats, self.mfcc_params.num_fbank_bins]) + return dct_m + + def mel_norm(self, weight, right_mel, left_mel): + """Over-riding parent class with ASR specific weight normalisation.""" + enorm = 2.0 / (self.inv_mel_scale(right_mel, False) - self.inv_mel_scale(left_mel, False)) + return weight * enorm + + +class W2LAudioPreprocessor(AudioPreprocessor): + + def __init__(self, mfcc, model_input_size, stride): + self.model_input_size = model_input_size + self.stride = stride + + super().__init__(self, model_input_size, stride) + # Savitzky - Golay differential filters + self.savgol_order1_coeffs = np.array([6.66666667e-02, 5.00000000e-02, 3.33333333e-02, + 1.66666667e-02, -3.46944695e-18, -1.66666667e-02, + -3.33333333e-02, -5.00000000e-02, -6.66666667e-02]) + + self.savgol_order2_coeffs = np.array([0.06060606, 0.01515152, -0.01731602, + -0.03679654, -0.04329004, -0.03679654, + -0.01731602, 0.01515152, 0.06060606]) + self._mfcc_calc = mfcc + + def mfcc_delta_calc(self, features): + """Over-riding parent class with ASR specific MFCC derivative features""" + mfcc_delta_np = np.zeros_like(features) + mfcc_delta2_np = np.zeros_like(features) + + for i in range(features.shape[1]): + idelta = np.convolve(features[:, i], self.savgol_order1_coeffs, 'same') + mfcc_delta_np[:, i] = idelta + ideltadelta = np.convolve(features[:, i], self.savgol_order2_coeffs, 'same') + mfcc_delta2_np[:, i] = ideltadelta + + features = np.concatenate((self._normalize(features), self._normalize(mfcc_delta_np), + self._normalize(mfcc_delta2_np)), axis=1) + + return features |