Mel-Frequency Cepstral Coefficients (MFCC) Feature Extraction Architecture

The Mel-Frequency Cepstral Coefficients (MFCC) subsystem in Sound Open Firmware provides real-time, psychoacoustically motivated audio feature extraction and embedded spectral analysis directly on digital signal processor (DSP) hardware.

In embedded speech recognition, keyword spotting, acoustic event detection, and generative speech-to-text models (such as OpenAI Whisper), feeding raw, uncompressed time-domain pulse-code modulation (PCM) audio samples directly into neural networks is computationally prohibitive. Raw audio waveforms exhibit immense data rates (e.g. 16,000 samples per second per channel), extreme temporal redundancy, sensitivity to room reverberation and phase shifts, and high computational dimensionality that overwhelms microcontrollers and DSP accelerators.

To solve this challenge, Sound Open Firmware integrates an optimized, fixed-point MFCC Feature Extraction Module. Designed as an in-line audio processing component conforming to the SOF Module Adapter framework, the MFCC engine transforms continuous, raw acoustic waveforms into compact, decorrelated spectro-temporal feature representations. Operating with a zero-heap memory architecture, the module executes high-pass pre-emphasis filtering, analysis windowing, Fast Fourier Transform (FFT) analysis, triangular auditory Mel filterbank integration, dynamic range clamping, logarithmic compression, Discrete Cosine Transform (DCT Type-II) decorrelation, and sinusoidal cepstral liftering.

Beyond classic cepstral coefficient extraction, the SOF MFCC subsystem incorporates modern deep learning features: an optimized Mel-only mode tailored for OpenAI Whisper and transformer speech models, an integrated Mel-domain Voice Activity Detector (VAD) utilizing IEC 61672-1 A-weighting formant curves, and Discontinuous Transmission (DTX) silence suppression that eliminates redundant DMA transfers to host CPUs and Neural Processing Units (NPUs).

This guide provides a comprehensive, high-level architectural walkthrough of the MFCC feature extraction subsystem in SOF, examining psychoacoustic foundations, homomorphic source-filter separation, five-stage transformation pipelines, Whisper dynamic max tracking, Mel-domain VAD mechanics, fixed-point scratch overlay memory designs, Cadence Tensilica HiFi SIMD vector acceleration, and ALSA Topology 2 / IPC streaming modes without delving into low-level C code.

—

1. Psychoacoustic Foundations & Auditory Representation

The design of the MFCC feature extraction pipeline is grounded in empirical principles of human psychoacoustics and the homomorphic deconvolution of speech production.

Biological Auditory Perception & The Mel Scale

The human auditory system does not perceive acoustic frequencies linearly. Sound waves entering the ear canal cause vibrations in the tympanic membrane and middle ear ossicles, which translate into traveling fluid waves within the cochlea. Along the length of the cochlear basilar membrane, different physical locations resonate at specific frequencies—a biological frequency decomposition known as tonotopic organization or the place theory of pitch.

High-frequency sounds stimulate hair cells near the stiff, narrow base of the cochlea, whereas low-frequency sounds travel to the flexible, wide apex. Crucially, the density of auditory sensory receptors is non-linear: humans possess extraordinary frequency resolution at low frequencies (below 1,000 Hz) to discern fundamental pitch and vowel formants, but significantly coarser resolution at high frequencies (above 1,000 Hz) where broadband fricative noises and consonant transients reside.

To model this non-linear perceptual sensitivity, Stevens, Volkmann, and Newman (1937) established the Mel Scale—a perceptual scale of pitches judged by human listeners to be equal in distance from one another. A frequency of 1,000 Hz at 40 dB above the listener’s threshold is defined as 1,000 Mel.

The conversion between physical acoustic frequency \(f\) (in Hertz) and perceptual pitch \(m\) (in Mel) is mathematically formulated using either the classic logarithmic approximation or the standard Auditory Toolbox formulation:

\[m = 2595 \cdot \log_{10}\left(1 + \frac{f}{700}\right) = 1127 \cdot \ln\left(1 + \frac{f}{700}\right)\]

Conversely, the inverse transformation from Mel space back to physical frequency in Hertz is expressed as:

\[f = 700 \cdot \left(10^{\frac{m}{2595}} - 1\right) = 700 \cdot \left(e^{\frac{m}{1127}} - 1\right)\]

In the Mel domain, equal distance corresponds to equal perceived musical interval and phonetic distinction. Below 1,000 Hz, the relationship between Hertz and Mel is approximately linear; above 1,000 Hz, the relationship becomes logarithmic, mirroring the human ear’s critical auditory bandwidths.

The Homomorphic Source-Filter Model of Speech

Human speech production is universally modeled as a linear time-invariant convolution of an acoustic excitation source \(e(t)\) with the acoustic resonance of the vocal tract filter \(h(t)\):

\[s(t) = e(t) * h(t)\]

• The Excitation Source \(e(t)\): Produced by airflow forced through the oscillating vocal cords (voiced speech, creating a periodic glottal pulse train with fundamental frequency \(F_0\)) or turbulent airflow forced through a narrow constriction in the vocal tract (unvoiced speech, creating broadband white noise).

• The Vocal Tract Filter \(h(t)\): Formed by the pharyngeal, oral, and nasal cavities. The geometric shape of the tongue, lips, jaw, and velum acts as an acoustic resonator, amplifying specific resonant frequencies called formants (\(F_1, F_2, F_3\)) that define phonetic vowels and consonants.

In automatic speech recognition (ASR) and keyword spotting, the identity of spoken words is dictated almost entirely by the vocal tract filter \(h(t)\) (the phonetic formants), whereas the excitation source \(e(t)\) conveys speaker-dependent pitch, gender, emotion, and vocal fry. To recognize words accurately across different speakers, an audio feature extractor must decouple the vocal tract resonance from the pitch excitation.

Because the source and filter are convolved in the time domain, their Fourier transforms are multiplied in the frequency domain:

\[S(f) = E(f) \cdot H(f)\]

Taking the complex magnitude and applying the natural logarithm transforms multiplication into addition:

\[\log |S(f)| = \log |E(f)| + \log |H(f)|\]

This mathematical operation is termed homomorphic filtering. In the log-magnitude spectrum, the slowly varying spectral envelope \(\log |H(f)|\) (the vocal tract formants) is linearly superimposed upon the rapidly fluctuating harmonic ripples \(\log |E(f)|\) (the glottal pitch harmonics).

The Cepstrum & Quefrency Domain

To separate these additive components, the logarithm of the power spectrum is treated as an ordinary time-domain signal, and its inverse Fourier transform or Discrete Cosine Transform (DCT) is computed. The resulting mathematical domain is called the Cepstrum (an anagram of spectrum), and its horizontal axis is defined as Quefrency (an anagram of frequency), measured in units of time (seconds or samples):

\[c[n] = \text{DCT}\Big( \log |S(f)| \Big) = \text{DCT}\Big( \log |H(f)| \Big) + \text{DCT}\Big( \log |E(f)| \Big)\]

• Low-Quefrency Coefficients (\(c_1\) to \(c_{12}\)): Represent the slowly varying spectral envelope, capturing the physical shape of the speaker’s vocal tract and formant locations. These coefficients are virtually invariant to pitch and fundamental frequency.

• High-Quefrency Coefficients: Represent the rapid spectral variations corresponding to the glottal pitch period \(T_0 = 1 / F_0\).

• Zero-th Coefficient (\(c_0\)): Represents the average log-energy of the entire frame.

By retaining only the low-quefrency cepstral coefficients (typically the first 13 to 40 values) and discarding high-quefrency bins, the MFCC feature extractor effectively strips away speaker pitch and acoustic excitation, producing an invariant, robust representation of human speech.

Figure 112 Psychoacoustic Auditory Representation and Homomorphic Source-Filter Separation

—

2. Five-Stage MFCC Feature Extraction Engine

The SOF MFCC subsystem structures feature extraction into a deterministic, five-stage mathematical processing pipeline executing within the module’s mfcc_process() audio loop.

Figure 113 Five-Stage MFCC Feature Extraction Pipeline in Sound Open Firmware

Stage 1: High-Pass Pre-Emphasis Filtering

During natural speech production, glottal airflow pulses radiating through the mouth opening experience acoustic impedance that causes a natural spectral roll-off of approximately -6 dB per octave across higher frequencies. Consequently, higher-frequency speech formants (above 1,000 Hz) possess significantly lower energy than low-frequency vowel fundamentals, despite conveying crucial phonetic information (such as dental and sibilant consonants like /s/, /t/, /f/).

The SOF MFCC module applies a first-order high-pass finite impulse response (FIR) pre-emphasis filter to flatten the speech spectrum and balance dynamic range:

\[y[n] = x[n] - \alpha \cdot x[n-1]\]

Where \(\alpha\) is the pre-emphasis coefficient, configured in fixed-point \(Q1.15\) format (typically \(\alpha = 0.97\), represented as 31785). This high-pass filter provides a \(+6\text{ dB/octave}\) boost, equalizing the dynamic range of formant peaks across the entire Nyquist bandwidth and improving the numerical conditioning of subsequent fixed-point FFT stages.

Stage 2: Overlapping Framing & Analysis Windowing

Speech is a non-stationary signal whose spectral characteristics evolve continuously over time. However, across short temporal intervals of 20 to 30 milliseconds, the vocal tract geometry remains physically quasi-stationary.

The MFCC module segments the continuous audio stream into overlapping frames:

• Frame Length (\(N\)): Duration of each analysis window, typically 25 ms (400 samples at 16,000 Hz).

• Frame Shift / Hop Size (\(H\)): Time interval between successive analysis frames, typically 10 ms (160 samples at 16,000 Hz).

• Frame Overlap: The consecutive frames overlap by \(N - H = 240\text{ samples}\) (15 ms), ensuring smooth temporal continuity and preventing data loss at frame boundaries.

To avoid abrupt truncation at frame edges (which introduces severe spectral leakage and artificial high-frequency sidelobes in the frequency domain), the module applies a tapered analysis window function \(w[n]\):

\[x_w[n] = x[n] \cdot w[n], \quad 0 \le n < N\]

The SOF MFCC engine supports five distinct window geometries:

1. Hann Window: \(w[n] = 0.5 - 0.5 \cos\left(\frac{2\pi n}{N - 1}\right)\), delivering -32 dB sidelobe suppression.

2. Hamming Window: \(w[n] = 0.54 - 0.46 \cos\left(\frac{2\pi n}{N - 1}\right)\), optimizing first sidelobe attenuation to -43 dB.

3. Blackman Window: Three-term cosine window with \(\alpha_0 = 0.42\), delivering -58 dB sidelobe suppression.

4. Povey Window: \(w[n] = \left(0.5 - 0.5 \cos\left(\frac{2\pi n}{N - 1}\right)\right)^{0.85}\), the standard window used in the Kaldi speech recognition toolkit.

5. Rectangular Window: Uniform weighting (\(w[n] = 1.0\)), used for baseline acoustic benchmarking.

Stage 3: Fast Fourier Transform (FFT) & Power Spectrum

To convert the windowed time-domain frames into the frequency domain, the module zero-pads the frame length \(N\) up to the next power of two (e.g. 400 samples zero-padded to 512 samples) and executes a 32-bit complex Fast Fourier Transform:

\[X[k] = \sum_{n=0}^{N_{FFT}-1} x_w[n] \cdot e^{-j \frac{2\pi k n}{N_{FFT}}}, \quad 0 \le k < N_{FFT}\]

Because the input audio is strictly real-valued, the resulting complex spectrum is conjugate-symmetric (\(X[N_{FFT} - k] = X^*[k]\)). The engine only needs to compute and retain the non-redundant positive frequencies:

\[K_{bins} = \frac{N_{FFT}}{2} + 1\]

For a 512-point FFT, exactly 257 complex frequency bins are produced. The power spectrum \(P[k]\) is subsequently computed as the squared magnitude of each bin:

\[P[k] = \frac{1}{N_{FFT}} |X[k]|^2 = \frac{1}{N_{FFT}} \Big( \text{Re}\{X[k]\}^2 + \text{Im}\{X[k]\}^2 \Big)\]

Stage 4: Auditory Mel Filterbank Integration

The linear frequency power spectrum \(P[k]\) is mapped into the auditory Mel domain by passing it through a bank of \(M\) overlapping triangular bandpass filters:

\[S_m = \sum_{k=0}^{K_{bins}-1} P[k] \cdot H_m[k], \quad 0 \le m < M\]

Each triangular filter \(H_m[k]\) is parameterized by three boundary frequencies in Hertz: lower edge \(f_{m-1}\), center peak \(f_m\), and upper edge \(f_{m+1}\):

\[\begin{split}H_m[k] = \begin{cases} 0 & f[k] < f_{m-1} \\ \frac{f[k] - f_{m-1}}{f_m - f_{m-1}} & f_{m-1} \le f[k] \le f_m \\ \frac{f_{m+1} - f[k]}{f_{m+1} - f_m} & f_m \le f[k] \le f_{m+1} \\ 0 & f[k] > f_{m+1} \end{cases}\end{split}\]

Following filterbank summation, logarithmic compression models human non-linear loudness perception (the Weber-Fechner law):

\[E_m = \log(S_m)\]

The engine supports three logarithmic bases via configuration: natural logarithm (MEL_LOG_IS_LOG), base-10 logarithm (MEL_LOG_IS_LOG10), and decibels (MEL_LOG_IS_DB where \(E_m = 10 \log_{10}(S_m)\)). A minimum power floor parameter (pmin, typically \(10^{-9}\)) prevents numerical underflow or infinite negative logarithms during absolute digital silence.

Stage 5: Discrete Cosine Transform (DCT-II) & Cepstral Liftering

In classic MFCC extraction (when num_ceps > 0), the log Mel filterbank energies \(E_m\) are highly correlated with one another due to spectral overlap between adjacent triangular filters. To compact the energy and produce uncorrelated features, the module applies an orthonormal Discrete Cosine Transform of Type II (DCT-II):

\[c_n = \sum_{m=0}^{M-1} E_m \cdot \cos\left( \frac{\pi n (m + 0.5)}{M} \right), \quad 0 \le n < N_{ceps}\]

Because the DCT decomposes the spectrum into orthogonal cosine basis functions, it acts as an optimal Karhunen-Loève transform (KLT) approximation for speech signals, concentrating the vast majority of phonetic information into the first 13 coefficients (\(c_0\) to \(c_{12}\)).

Finally, to equalize the numerical variance between lower-order coefficients (which have very high amplitudes) and higher-order coefficients (which have smaller amplitudes), the module applies a sinusoidal cepstral lifter:

\[\hat{c}_n = c_n \cdot w_{lifter}[n] = c_n \cdot \left( 1 + \frac{L}{2} \sin\left(\frac{\pi n}{L}\right) \right)\]

Where \(L\) is the cepstral lifter parameter (configured in \(Q7.9\) format, typically \(L = 22.0\)). This sinusoidal weighting scales up higher-order coefficients, balancing their contribution in downstream Euclidean distance metrics and neural network classifiers.

—

3. Auditory Mel Filterbank & Slaney Area Normalization

A critical architectural feature of the SOF MFCC filterbank is the distinction between standard triangular filters and Malcolm Slaney’s area-normalized filterbank (norm = MFCC_MEL_NORM_SLANEY), widely adopted in modern machine learning libraries (such as Librosa and Kaldi).

Figure 114 Auditory Mel Filterbank Construction: Linear Spacing in Mel vs Expanding Bandwidths in Hertz

The Problem with Unnormalized Triangular Filters

In a standard triangular filterbank, every triangular filter has a peak amplitude of \(1.0\). However, because the filters are uniformly spaced in the Mel domain, their physical bandwidth in Hertz expands dramatically as frequency increases:

• At 200 Hz, a filter may span a bandwidth of only 40 Hz.

• At 6,000 Hz, a filter spans a bandwidth exceeding 1,000 Hz.

If all filters have a peak height of 1.0, the area under each triangle is proportional to its bandwidth (\(\text{Area} = 0.5 \times \Delta f\)). Consequently, high-frequency filters integrate power over a much wider frequency span, artificially inflating high-frequency energies and tilting the spectral balance upward.

Slaney Area Normalization Principle

To ensure that flat white noise produces equal energy across all filterbank channels, Malcolm Slaney’s Auditory Toolbox normalizes each triangular filter by dividing its coefficients by the filter’s acoustic bandwidth in Hertz:

\[H_{m, \text{Slaney}}[k] = H_m[k] \cdot \left( \frac{2}{f_{m+1} - f_{m-1}} \right)\]

This normalization ensures that the integral of each triangular filter equals unity (\(\int H_{m,\text{Slaney}}(f) df = 1\)). In the SOF firmware, this calculation is executed during mfcc_setup() using high-precision integer division, scaling the filter weights so that each bin measures true power spectral density rather than total integrated bandwidth power.

—

4. OpenAI Whisper & Modern Deep Learning Feature Integration

While traditional speech pipelines require cepstral coefficients (\(c_1\) to \(c_{12}\)), modern deep neural networks—such as OpenAI Whisper, Conformer, and wav2vec 2.0—bypass the DCT entirely. These architectures ingest high-resolution 80-channel log Mel spectrograms directly, using multi-head self-attention mechanisms to learn optimal representations.

The SOF MFCC module provides first-class support for OpenAI Whisper front-end feature generation embedded entirely within the audio DSP.

Figure 115 OpenAI Whisper Preprocessing Data Path: Dynamic Max Tracking, Clamping, and Scaling

Whisper Architectural Requirements

OpenAI Whisper specifies precise mathematical constraints for audio ingestion:

1. 80 Mel Filterbank Channels: Spanning 0 Hz to 8,000 Hz on 16 kHz audio.

2. 25 ms Window with 10 ms Stride: Exactly 400 samples framing with 160 samples hop.

3. Dynamic Range Clamping: Clamping minimum Mel values to \(m_{max} - 8.0\) in natural log units (equivalent to -80 dB below the frame peak).

4. Affine Scaling and Normalization: Shifting by \(+4.0\) and scaling by \(0.25\) to map values into the normalized dynamic range \([-1.0, 1.0]\) expected by Whisper transformer encoders:

\[M_{\text{Whisper}}[j] = \frac{\max\Big(M[j], \; m_{max} - 8.0\Big) + 4.0}{4.0} = 0.25 \cdot \Big(\text{clamped}[j] + 4.0\Big)\]

On-DSP Dynamic Max Tracking & Clamping

In offline Python implementations, \(m_{max}\) is computed globally across an entire 30-second audio buffer. In real-time streaming DSP firmware, future audio is unknown. The SOF MFCC module solves this by implementing dynamic peak Mel tracking with asymmetric exponential decay:

• Instantaneous Attack: When the current frame peak exceeds the tracked maximum (\(\text{peak} > m_{max}\)), the tracker immediately snaps upward:

  \[m_{max} = \text{peak}\]

• Leaky Decay: During quieter intervals, \(m_{max}\) decays slowly according to an exponential decay coefficient \(\alpha_{mmax}\) (config->mmax_coef):

  \[m_{max} \gets m_{max} + \alpha_{mmax} \cdot (\text{peak} - m_{max})\]

This dynamic tracking ensures that speech signals remain perfectly clamped against local acoustic volume levels without clipping sudden loud utterances or dropping faint whispering.

Zero Host Wakeup & NPU Streaming

By executing pre-emphasis, windowing, FFT, Mel filterbanks, dynamic max tracking, and affine normalization directly on the low-power DSP, the host application processor remains in deep low-power sleep (D3 / S0ix). When speech occurs, the DSP streams normalized Mel frames directly to the Intel NPU or GPU via OpenVINO, bypassing all CPU-side feature extraction and eliminating host cache thrashing.

—

5. Integrated Mel-Domain VAD & Discontinuous Transmission

A major innovation of the SOF MFCC module is its integrated Mel-Domain Voice Activity Detector (VAD) and Discontinuous Transmission (DTX) engine. Rather than running a separate, computationally redundant time-domain VAD component, the MFCC module evaluates voice activity directly on the 80-channel Mel log spectrum already computed in SRAM.

Figure 116 Integrated Mel-Domain VAD and Discontinuous Transmission (DTX) State Machine

IEC 61672-1:2013 A-Weighting Formant Emphasis

Ambient acoustic environments contain significant low-frequency energy—HVAC rumble, vehicle road noise, wind buffeting—that possesses high physical energy but zero speech relevance. Conversely, human speech formants are concentrated in the 1,000 Hz to 4,000 Hz band.

To maximize detection sensitivity, the SOF VAD computes speech-frequency emphasis weights by linearly interpolating the international standard IEC 61672-1:2013 A-weighting table across the center frequencies of all Mel bins. The weights \(W[m]\) are normalized to sum to \(1.0\) in \(Q1.15\) format, prioritizing frequencies between 1 kHz and 4 kHz while heavily attenuating infrasound below 100 Hz.

Per-Bin Adaptive Noise Floor Tracking

Rather than applying a scalar noise floor, the VAD maintains an independent noise floor \(N[m]\) for every individual Mel bin:

1. Instant Downward Tracking: If the current Mel bin energy \(M[m]\) drops below the current noise floor estimate, speech is impossible. The noise floor instantly drops to match the new minimum:

   \[N[m] = M[m]\]

2. Slow Upward Rise: When the energy is higher than the floor, the noise floor rises very slowly with an exponential smoothing factor \(\alpha\):

   \[N[m] \gets N[m] + \alpha \cdot (M[m] - N[m])\]

To ensure rapid convergence upon stream initialization, the engine uses a dual-rate scheme:

• Fast Initialization Phase (`init_frames = 100`): During the first 100 frames (~1.0 second), \(\alpha_{fast} = 0.020\) (655 in \(Q1.15\)), rapidly acquiring the ambient room noise profile.

• Steady-State Tracking Phase: After 100 frames, \(\alpha_{slow} = 0.003\) (98 in \(Q1.15\)), preventing the noise floor from rising during sustained spoken sentences.

Energy Delta & Hangover Mechanics

The VAD computes the A-weighted total signal energy \(E_{signal}\) and noise floor energy \(E_{noise}\):

\[E_{signal} = \sum_{m=0}^{M-1} W[m] \cdot M[m], \quad E_{noise} = \sum_{m=0}^{M-1} W[m] \cdot N[m]\]

The energy delta is evaluated:

\[\Delta E = E_{signal} - E_{noise}\]

If \(\Delta E\) exceeds the detection threshold (MFCC_VAD_ENERGY_THRESHOLD = 2516582 in \(Q9.23\), corresponding to 0.30 natural log units or ~2.6 dB SNR), speech is declared (vad_flag = 1), and the hangover counter is reset to its maximum (hangover_max = 20 frames, or 200 ms).

During brief pauses between syllables or trailing stop-consonant releases, \(\Delta E\) may drop below the threshold. The hangover counter decrements frame by frame, keeping \(vad\_flag = 1\) active. This prevents stuttering or chopping at the ends of words.

Discontinuous Transmission (DTX) Silence Suppression

Streaming continuous silence frames across the host PCIe or SoundWire bus wastes power and memory bandwidth. When DTX is enabled (enable_dtx = true):

1. Trailing Silence Preservation: When speech concludes, the module continues transmitting a configurable number of trailing silence hops (dtx_trailing_silence_hops, e.g. 10 frames = 100 ms). This ensures downstream speech recognizers and VAD models observe natural sentence termination and acoustic decay.

2. Complete Frame Suppression: After trailing frames expire, the module completely suppresses output: zero bytes are committed to the sink, and the audio pipeline produces no DMA interrupts.

3. Periodic Keep-Alive Frames: During prolonged silence, the module optionally transmits a single silence frame every \(N\) hops (dtx_silence_hops_interval), maintaining pipeline keep-alive status and updating host ambient noise trackers without continuous streaming.

—

6. Fixed-Point Arithmetic & Scratch Overlay Memory Design

Audio DSPs operate under severe internal static RAM (SRAM) constraints. Allocating independent buffers for every mathematical stage (FFT input, FFT output, power spectrum, Mel log spectrum, DCT matrix) would exhaust memory and cause cache evictions.

The SOF MFCC subsystem solves this through an advanced SRAM Scratch Overlay Architecture combined with a decoupled output staging buffer.

Figure 117 Dual FFT Buffer Scratch Overlays and Decoupled Staging Memory Map

The Two-Buffer Scratch Overlay Map

The module allocates exactly two 32-bit complex buffers sized for the padded FFT (\(N_{FFT} = 512 \implies 4,096\text{ bytes}\) each):

1. Buffer 1 (`fft->fft_buf`, 4,096 bytes): * Phase 1: Receives windowed time-domain audio samples in the real part with zeroed imaginary components. * Phase 2 (Post-FFT Overlay): Overlaid to hold the computed 32-bit power spectrum (power_spectra, 257 bins = 1,028 bytes) and the 32-bit Mel log spectrum (mel_log_32, 80 bins = 320 bytes in \(Q9.23\)).

2. Buffer 2 (`fft->fft_out`, 4,096 bytes): * Phase 1: Receives raw complex frequency bins from the FFT engine. * Phase 2 (Post-FFT Overlay): Overlaid to hold 16-bit Mel log spectra (mel_spectra, 80 bins in \(Q9.7\) = 160 bytes) and 16-bit cepstral coefficients (cepstral_coef, 13 bins in \(Q9.7\) = 26 bytes) for DCT matrix multiplication.

Decoupled Output Staging Buffer

In multi-period audio pipelines, a sink buffer may not drain all features within a single 10 ms period. If output data were held directly inside the FFT scratch space, the next STFT hop could not execute without corrupting pending data.

To solve this, the SOF MFCC subsystem allocates a dedicated output staging buffer (`out_stage`):

• Upon completion of the STFT hop, the prepared features (Mel or widened cepstral coefficients) are immediately copied into out_stage.

• The STFT scratch buffers (fft_buf and fft_out) are instantly freed to process the next incoming audio frame.

• Sink drainage proceeds asynchronously across multiple periods using read pointer out_data_ptr and remaining count out_remain.

Fixed-Point Number Representations Across Stages

To maintain maximum numerical precision without floating-point emulation, data types transition systematically across fixed-point formats:

Table 4 Fixed-Point Number Format Transitions in SOF MFCC Pipeline

Processing Stage	Data Type	Fixed-Point Format	Dynamic Range / Description
Input PCM Audio	int16_t	\(Q1.15\)	Full scale audio waveform [-1.0, +0.999]
Windowed FFT Input	icomplex32	\(Q1.31\)	Windowed audio samples in real container
FFT Power Spectrum	int32_t	\(Q1.31\)	Normalized power spectral magnitude squared
Mel Log Energy (32-bit)	int32_t	\(Q9.23\)	9 integer bits, 23 fractional bits (Mel log values)
Mel Energy for DCT	int16_t	\(Q9.7\)	Scaled 16-bit Mel log values for matrix multiply
DCT Cepstral Output	int16_t	\(Q9.7\)	16-bit orthogonal cepstral coefficients
Output Stream Payload	int32_t	\(Q9.23\)	Widened 32-bit features committed to sink

—

7. SIMD Vector Acceleration Across DSP Architectures

The SOF MFCC module leverages Cadence Tensilica Xtensa HiFi 3 and HiFi 4/5 DSP instruction set architectures (ISAs) to execute overlap buffer shifting, windowing, and FFT execution with minimal cycle counts.

Cadence Tensilica HiFi 3 & HiFi 4 Optimization

On Intel and NXP audio DSP cores featuring Xtensa HiFi 3 or HiFi 4 engines (mfcc_hifi3.c, mfcc_hifi4.c), key operations are vectorized:

1. Hardware Circular Buffer Addressing (`AE_SETCBEGIN0` / `AE_SETCEND0`): When extracting overlap samples from the circular input buffer (mfcc_fill_prev_samples()), the DSP hardware circular addressing registers auto-wrap read pointers without software boundary comparison branches.

2. Vectorized 32-bit Load & Store: Samples are fetched using 32-bit circular loads (AE_L32_XC) and packed using auto-incrementing 32-bit stores (AE_S32_L_IP), halving memory bus transactions.

3. SIMD Fractional Windowing Multiplication: In mfcc_apply_window(), time-domain samples and window coefficients are multiplied using fractional multiply-with-rounding vector instructions:

   • sample = AE_SLAI32S(sample, 16): Shifts 16-bit audio into 32-bit \(Q1.31\) container.

   • temp = AE_MULFP32X16X2RS_L(sample, win): Multiplies 32-bit sample by 16-bit window with rounding and saturation.

   • temp = AE_SLAA32S(temp, input_shift): Applies dynamic scaling shift.

   • AE_S32_L_XP(temp, fft_in, fft_inc): Writes directly into FFT real scratch with auto-increment.

Generic Portable Fallback

On microcontrollers and processors lacking Tensilica HiFi extensions—such as ARM Cortex-M7 (PJRC Teensy 4.1) or RISC-V RV32 (Espressif ESP32-P4 / ESP32-C6)—the subsystem compiles clean, highly portable scalar C implementations (mfcc_generic.c), guaranteeing full bit-exact feature equivalence across simulation testbenches and silicon targets.

—

8. ALSA Topology 2, Host Streaming Modes & AI Pipeline Integration

The SOF MFCC module is declared declaratively in ALSA Topology 2.0 configuration files and integrated into end-to-end edge-to-host artificial intelligence capture pipelines.

ALSA Topology 2 Widget Declaration

In tools/topology/topology2/include/components/mfcc.conf, the module is defined with its unique cryptographic UUID and dedicated VAD notification control:

Class.Widget."mfcc" {
    uuid            "73:a7:10:db:a4:1a:ea:4c:a2:1f:2d:57:a5:c9:82:eb"
    type            "effect"
    no_pm           "true"
    num_input_pins  1
    num_output_pins 1

    # Switch control notifying user space of VAD state transitions
    Object.Control {
        mixer."1" {
            Object.Base.channel.1 { name "fc"; shift 0; }
            Object.Base.ops.1 { name "ctl"; info "volsw"; get 259; put 259; }
            max 1
        }
    }
}

Dual Output Streaming Modes

The module provides two distinct mechanisms for delivering feature data to downstream consumers:

1. Compress Output Mode (`compress_output = true`):

   • Designed for edge AI models (such as Whisper or TFLM).

   • Feature frames are packed contiguously into the sink without zero-padding, matching the exact byte length of the features.

   • Every frame is prepended with a 24-byte Data Header (`struct mfcc_data_header`):

     • magic: Fixed identifier 0x6d666363 (ASCII “mfcc”).

     • frame_number: Monotonically increasing hop counter.

     • energy: Speech-weighted signal energy (\(Q9.23\)).

     • noise_energy: Estimated background noise energy (\(Q9.23\)).

     • vad_flag: Voice activity decision (1 = speech, 0 = silence).

   • Integrates seamlessly with DTX to suppress frames during silence.

2. Legacy PCM Output Mode (`compress_output = false`):

   • Treats the sink buffer as an opaque byte container sized to match standard PCM period boundaries (e.g. S16_LE, S24_4LE, S32_LE).

   • Unfilled samples in the period are zero-padded, allowing standard ALSA tools (arecord) and host testbench scripts to capture feature streams over conventional PCM audio nodes.

End-to-End Edge Speech Pipeline Integration

In production edge-to-host AI systems, the MFCC module sits at the center of the acoustic capture graph:

Figure 118 Complete Edge-to-Host AI Speech Pipeline: From Microphone Array to Whisper and TFLM

—

9. Tuning Workflow & Upstream Source References

Sound Open Firmware provides a complete software ecosystem for tuning, simulating, and validating MFCC feature extraction.

MATLAB & GNU Octave Tuning Tools

Under src/audio/mfcc/tune/:

• setup_mfcc.m: Generates binary configuration blobs (sof_mfcc_config) from user-defined parameters (sample rate, frame length, hop size, window type, Mel bins, Slaney normalization, Whisper scaling).

• run_mfcc.sh: Shell script executing the SOF testbench (testbench) on raw audio files across S16, S24, and S32 bit depths, with optional Xtensa simulator (xt-run) execution.

• decode_all.m, decode_mel.m, decode_ceps.m: Decodes and plots generated binary feature files, visualizing 80-bin Mel spectrograms and 13-coefficient cepstral trajectories.

• sof_mel_to_text_live_dsp_vad.py: Live streaming Python application connecting the DSP audio features device (hw:0,48) directly to OpenVINO Whisper models running on the Intel NPU.

• sof_mel_spectrogram_compress.py: Real-time GTK 4 live spectrogram viewer displaying Mel energy waterfalls and VAD flags.

Upstream Source Code References

• src/audio/mfcc/README.md: Upstream module overview.

• src/include/user/mfcc.h: Configuration ABI, window types, and Whisper parameters.

• src/include/sof/audio/mfcc/mfcc_comp.h: Module private data, buffers, and STFT declarations.

• src/include/sof/audio/mfcc/mfcc_vad.h: Mel-domain VAD state and A-weighting tables.

• src/audio/mfcc/mfcc.c: Module lifecycle, init, prepare, and process callbacks.

• src/audio/mfcc/mfcc_setup.c: Parameter validation, FFT plan, Mel filterbank, and DCT initialization.

• src/audio/mfcc/mfcc_common.c: STFT pipeline execution, windowing, Mel calculation, and compress output.

• src/audio/mfcc/mfcc_vad.c: Mel-domain VAD update and noise floor tracking.

• src/audio/mfcc/mfcc_hifi3.c: Tensilica Xtensa HiFi 3 SIMD vectorization.

• src/audio/mfcc/mfcc_hifi4.c: Tensilica Xtensa HiFi 4/5 SIMD vectorization.

• tools/topology/topology2/include/components/mfcc.conf: ALSA Topology 2 widget definition.