STFT Process Architecture

The STFT Process (Short-Time Fourier Transform Process) subsystem in Sound Open Firmware (SOF) provides a high-performance, modular frequency-domain signal processing engine. It ingests continuous time-domain PCM audio streams, segments them into overlapping analysis frames, transforms them into discrete frequency bins via multi-channel 32-bit Fast Fourier Transforms (FFTs), executes frequency-domain filtering or polar transformations, and reconstructs continuous time-domain waveforms via Inverse Fast Fourier Transforms (iFFTs) and Overlap-Add (OLA) synthesis.

Frequency-domain processing is essential for modern embedded audio subsystems. While time-domain finite impulse response (FIR) and infinite impulse response (IIR) filters excel at static equalization, frequency-domain architectures allow per-bin spectral gain modification, acoustic echo suppression, non-stationary noise reduction, dynamic spectral shaping, psychoacoustic masking, and feature extraction for machine learning models (such as keyword detection and speech recognition).

The SOF STFT Process component is engineered specifically for hard real-time DSP constraints. It incorporates a single-allocation contiguous heap memory layout, zero-copy buffer sharing between Cartesian and polar representations, pre-computed window gain compensation, and Cadence Tensilica HiFi SIMD vector acceleration. It is fully integrated with the Intel IPC4 control plane, the Zephyr Loadable Linkable Extension (LLEXT) dynamic module loader, and ALSA Topology 2.

---

Architectural Overview & Frequency-Domain Processing Principles

Continuous acoustic signals are inherently non-stationary; their spectral properties vary continuously over time. The Short-Time Fourier Transform resolves this by partitioning the time-domain signal into short, overlapping quasi-stationary windows where the signal’s spectral properties can be assumed constant.

Theoretical Foundations of Frequency-Domain Processing

Processing audio signals in the frequency domain leverages fundamental mathematical properties:

1. The Convolution Theorem: Time-domain circular convolution corresponds to point-wise multiplication in the frequency domain:

   \[x[n] * h[n] \quad \Longleftrightarrow \quad X[k] \cdot H[k]\]

   For long impulse responses (such as room reverberation suppression or high-order linear-phase filters), executing a fast FFT, point-wise complex multiplication, and iFFT reduces computational complexity from \(\mathcal{O}(N^2)\) to \(\mathcal{O}(N \log_2 N)\).

2. Per-Bin Spectral Masking & Subtraction: Nonlinear and dynamic algorithms (such as spectral noise subtraction and speech enhancement) apply independent attenuation factors \(G[k]\) to individual frequency bins based on estimated Signal-to-Noise Ratios (SNR):

   \[Y[k] = G[k] \cdot X[k]\]

   Implementing equivalent sharp, dynamic, multi-band notch filters in the time domain would require hundreds of cascaded biquads with severe phase distortion and high computational overhead.

3. Decoupled Magnitude and Phase Control: By transforming Fourier coefficients into polar coordinates, algorithms can manipulate signal magnitude (gain, dynamic range, spectral shaping) while preserving or independently modeling phase trajectories.

Table 27 Architectural Comparison: Audio Processing Paradigms

Parameter	Time-Domain Filters (IIR/FIR)	SOF STFT Process	SOF Phase Vocoder
Processing Domain	Time domain (samples)	Frequency domain (FFT bins)	Frequency domain (Polar STFT)
Hop Geometry	Single sample (\(R = 1\))	Fixed hop (\(R_{\text{hop}} = N/2, N/4\))	Variable hop (\(R_s = R_a / \text{speed}\))
Primary Purpose	Static EQ, DC blocking, basic crossovers	Spectral filtering, noise gating, speech enhancement	Time-Scale Modification (0.5x to 2.0x playback)
Phase Behavior	Minimum or linear phase	Frame-dependent phase preservation	Active phase unwrapping & accumulation
Algorithmic Latency	Sub-millisecond (tap delay)	Frame hop size (1 ms to 32 ms)	Window hop size (2.7 ms to 5.3 ms)
Memory Footprint	Minimal (< 1 KB state)	Moderate (8 KB to 32 KB buffers)	Moderate (8 KB to 32 KB buffers)

Figure 196 SOF STFT Processing Architecture: Ingress, Framing, Analysis Windowing, Complex/Polar Transform, Synthesis Windowing & Overlap-Add Core

---

Circular Buffer Ingress, Hop Geometry & Overlap Assembly

The STFT Process component bridges the continuous, frame-by-frame streaming nature of the SOF pipeline with the block-based nature of Fourier transforms. Incoming audio frames typically arrive in scheduling intervals of 1 ms (e.g., 48 samples at 48 kHz or 16 samples at 16 kHz), whereas Fourier analysis frames are significantly larger (typically 192 to 1536 samples).

Ring Buffer Architecture & Pointer Arithmetic

To manage the rate difference between pipeline ticks and FFT block executions, the component maintains dedicated per-channel circular buffers for both ingress (stft_process_state.ibuf) and egress (stft_process_state.obuf).

Each buffer is managed by stft_process_buffer:

• stft_process_buffer.addr: Base memory address of the buffer.

• stft_process_buffer.end_addr: Pointer to the byte immediately following the buffer.

• stft_process_buffer.r_ptr: Read pointer advancing as samples are consumed.

• stft_process_buffer.w_ptr: Write pointer advancing as samples are produced.

• stft_process_buffer.s_avail: Count of available valid samples.

• stft_process_buffer.s_free: Count of free space remaining in samples.

• stft_process_buffer.s_length: Total capacity in samples.

When read or write pointers reach stft_process_buffer.end_addr, they wrap back to stft_process_buffer.addr via stft_process_buffer_wrap():

static inline int32_t *stft_process_buffer_wrap(struct stft_process_buffer *buffer, int32_t *ptr)
{
    if (ptr >= buffer->end_addr)
        ptr -= buffer->s_length;
    return ptr;
}

Hop Sizing & Overlap Geometry

The relationship between the analysis frame length \(N\) and the hop size \(R_{\text{hop}}\) defines the temporal and spectral characteristics of the STFT:

• Frame Length (\(N\)): Number of samples in each FFT analysis window (stft_process_fft.fft_size). Must be a multiple of 4 for Xtensa HiFi SIMD vector operations.

• Hop Size (\(R_{\text{hop}}\)): Number of new samples advanced between consecutive FFTs (stft_process_fft.fft_hop_size). Must be a multiple of 2.

• Overlap History Size (\(L_{\text{prev}}\)): Number of samples retained from the previous frame:

  \[L_{\text{prev}} = N - R_{\text{hop}}\]

  These samples are preserved in state->prev_data[ch].

Figure 197 STFT Framing, Overlap Buffer Geometry & Circular Ring Buffer Timeline (R_hop vs N_frame)

When stft_process_fill_fft_buffer() executes:

1. The first \(L_{\text{prev}}\) complex samples of stft_process_fft.fft_buf receive the overlap history from state->prev_data[ch], with imaginary components set to 0.

2. The next \(R_{\text{hop}}\) complex samples are dequeued from state->ibuf[ch], with imaginary components set to 0.

3. The trailing \(L_{\text{prev}}\) samples of the assembled buffer (indices \(R_{\text{hop}}\) to \(N-1\)) are copied back to state->prev_data[ch] to serve as the overlap history for the subsequent hop.

---

Window Functions, Spectral Leakage & Constant Overlap-Add (COLA)

Multiplying a finite-duration signal by an abrupt rectangular window causes sharp discontinuities at the boundaries, generating broad sidelobe energy across all Fourier bins (spectral leakage). To isolate narrow spectral peaks and prevent adjacent bin interference, the analysis frame is multiplied by a smooth taper window \(w[n]\).

Configurable Window Types

The SOF STFT Process component supports five distinct window functions defined in sof_stft_process_fft_window_type:

1. Rectangular Window (`STFT_RECTANGULAR_WINDOW = 0`): Uniform weighting (\(w[n] = 1\)). Yields the narrowest main lobe (\(\Delta \omega = 4\pi / N\)) for maximum frequency resolution, but severe first sidelobe leakage (\(-13\text{ dB}\) attenuation), causing high inter-bin spectral interference.

2. Blackman Window (`STFT_BLACKMAN_WINDOW = 1`): A three-term cosine window providing extreme sidelobe suppression (\(-58\text{ dB}\)), virtually eliminating cross-bin leakage at the expense of a wider main lobe (\(\Delta \omega = 12\pi / N\)):

   \[w[n] = a_0 - a_1 \cos\left(\frac{2\pi n}{N}\right) + a_2 \cos\left(\frac{4\pi n}{N}\right)\]

   Implemented with exact Q1.31 coefficients defined by WIN_BLACKMAN_A0_Q31.

3. Hamming Window (`STFT_HAMMING_WINDOW = 2`): Optimized raised cosine window designed to cancel the first sidelobe, achieving \(-43\text{ dB}\) first sidelobe attenuation:

   \[w[n] = 0.54 - 0.46 \cos\left(\frac{2\pi n}{N}\right)\]

4. Hann Window (`STFT_HANN_WINDOW = 3`, Standard Default): A standard raised cosine window tapering smoothly to zero at both boundaries:

   \[w[n] = 0.5 - 0.5 \cos\left(\frac{2\pi n}{N}\right), \quad 0 \le n < N\]

   Achieves \(-31.5\text{ dB}\) first sidelobe attenuation with a rapid asymptotic decay of \(-18\text{ dB/octave}\).

5. Povey Window (`STFT_POVEY_WINDOW = 4`): A specialized window widely adopted in automatic speech recognition (ASR) feature extraction pipelines (such as Kaldi):

   \[w[n] = \left(\frac{1 - \cos\left(\frac{2\pi n}{N}\right)}{2}\right)^{0.85}, \quad 0 \le n < N\]

   By raising the standard Hann raised-cosine curve to the power of \(0.85\), the Povey window broadens the effective analysis center while preserving smooth, zero-endpoint boundary transitions, optimizing the trade-off between spectral resolution and time localization for human speech formants.

Figure 198 Analysis & Synthesis Window Functions and Frequency Responses (Rectangular, Hann, Hamming, Blackman, Povey)

Constant Overlap-Add (COLA) Condition & Gain Compensation

When an audio signal is windowed during analysis and windowed again during synthesis, the cumulative gain across overlapping frames must sum to a constant across time to prevent amplitude modulation (tremolo) at the frame hop rate:

\[\sum_{m=-\infty}^{+\infty} w[n - m R_{\text{hop}}] = \text{constant} \quad \forall n\]

In a cascaded STFT / iSTFT processing system where both the forward FFT and the inverse FFT apply the window \(w[n]\), the effective overlap-add weight is the squared window \(w[n]^2\).

To guarantee exact unity gain (\(0\text{ dBFS}\)) reconstruction without amplitude ripple, SOF computes a 32-bit Q1.31 window gain compensation factor stft_process_state.gain_comp:

\[g_{\text{comp}} = \frac{R_{\text{hop}}}{\sum_{n=0}^{N-1} w[n]^2}\]

During synthesis overlap-add in stft_process_overlap_add_ifft_buffer(), each real sample from the inverse FFT is multiplied by \(g_{\text{comp}}\) before being accumulated into the output buffer:

sample = Q_MULTSR_32X32((int64_t)state->gain_comp, fft->fft_buf[idx].real, 31, 31, 31);
*w = sat_int32((int64_t)*w + sample);

---

Dual-Domain Processing: Cartesian Complex vs Polar Magnitude/Phase

The STFT Process component supports dual processing representations: Cartesian complex coordinates (\(\text{real} + j \cdot \text{imag}\)) and polar coordinates (magnitude \(M_k\) and phase angle \(\theta_k\)).

Cartesian Complex Pipeline (Default Fast Path)

In the default configuration, the 32-bit forward FFT generates complex coefficients directly into stft_process_fft.fft_out:

\[X[k] = \text{Re}\{X[k]\} + j \cdot \text{Im}\{X[k]\}, \quad k = 0 \dots N-1\]

For linear filtering and spectral convolution, algorithms operate directly on Cartesian complex numbers:

\[Y[k] = X[k] \cdot H[k] = \left(\text{Re}\{X\} \text{Re}\{H\} - \text{Im}\{X\} \text{Im}\{H\}\right) + j \left(\text{Re}\{X\} \text{Im}\{H\} + \text{Im}\{X\} \text{Re}\{H\}\right)\]

Polar Coordinate Pipeline (CONFIG_STFT_PROCESS_MAGNITUDE_PHASE)

When enabled via Kconfig, the component automatically transforms complex Fourier bins into polar coordinates:

1. Complex to Polar Conversion: Using sofm_icomplex32_to_polar(), the complex bin is converted to magnitude and phase:

   \[M_k = \sqrt{\text{Re}\{X[k]\}^2 + \text{Im}\{X[k]\}^2} \quad (\text{Q2.30})\]
   \[\theta_k = \text{atan2}\left(\text{Im}\{X[k]\}, \text{Re}\{X[k]\}\right) \quad (\text{Q5.27})\]

   Because the time-domain signal is strictly real, the frequency spectrum exhibits Hermitian symmetry (\(X[N-k] = X^*[k]\)). Therefore, polar conversion is only performed on the non-redundant lower half-spectrum:

   \[k = 0 \dots \left(\frac{N}{2}\right)\]

2. Algorithm Hook Execution: The application algorithm modifies magnitude \(M_k\) (e.g., applying Wiener gain masks, dynamic range compression, or noise reduction gains) and/or phase \(\theta_k\).

3. Polar to Complex Reconstruction: The modified polar coordinates are converted back to Cartesian form via sofm_ipolar32_to_complex():

   \[\text{Re}\{X[k]\} = M_k \cdot \cos(\theta_k), \quad \text{Im}\{X[k]\} = M_k \cdot \sin(\theta_k)\]

4. Conjugate Hermitian Symmetry Restoration: To guarantee that the subsequent inverse FFT yields a purely real time-domain signal (with zero imaginary component), the upper half-spectrum is reconstructed via stft_apply_fft_symmetry():

   \[\text{Re}\{X[N-k]\} = \text{Re}\{X[k]\}, \quad \text{Im}\{X[N-k]\} = -\text{Im}\{X[k]\}, \quad k = 1 \dots \frac{N}{2}-1\]

Figure 199 Dual-Domain Processing Pipeline: Cartesian Complex vs Polar Magnitude/Phase with Hermitian Symmetry Reconstruction

---

Single Contiguous Memory Block & Zero-Copy Polar Overlay

In hard real-time audio firmware, dynamic heap allocation during runtime is prohibited, and excessive memory fragmentation must be prevented. The STFT Process subsystem employs an optimized memory architecture that consolidates all circular sample buffers, overlap arrays, and window tables into a single contiguous allocation.

Single Contiguous Buffer Partitioning

During component initialization in stft_process_setup(), the total memory requirement across all channels is calculated and allocated as a single aligned buffer:

\[\text{Total Sample RAM} = \text{sizeof}(\text{int32\_t}) \times \left[ C \times (L_{\text{ibuf}} + L_{\text{obuf}} + L_{\text{prev}}) + N \right]\]

Where:

• \(C\) is the number of active audio channels (stft_comp_data.channels).

• \(L_{\text{ibuf}} = R_{\text{hop}} + \text{max\_frames}\) (input circular buffer length).

• \(L_{\text{obuf}} = N + \text{max\_frames}\) (output circular buffer length).

• \(L_{\text{prev}} = N - R_{\text{hop}}\) (overlap history length).

• \(N\) is the FFT frame size (window coefficient table length).

The component verifies that \(\text{Total Sample RAM} \le \text{STFT\_MAX\_ALLOC\_SIZE}\) (65,536 bytes / 64 KB) to protect against memory exhaustion. The buffer is allocated via mod_balloc_align() with 64-bit alignment (\(2 \times \text{sizeof}(\text{int32\_t})\)), satisfying Xtensa SIMD load requirements.

Zero-Copy Polar Buffer Overlay

To support polar magnitude and phase processing without allocating an additional 32-bit array, the component overlays the polar coordinate structure directly on top of the FFT output buffer:

/* Share the fft_out buffer for polar format */
fft->fft_polar = (struct ipolar32 *)fft->fft_out;

Because both icomplex32 (two 32-bit integers: real and imag) and ipolar32 (two 32-bit integers: magnitude and angle) have identical 64-bit sizes and alignments, in-place conversion from Cartesian to polar format consumes zero additional heap memory!

Figure 200 Single Contiguous Memory Block Partitioning & Zero-Copy Polar Overlay Architecture

---

Tensilica HiFi3 SIMD Vector Acceleration

The critical inner loops of the STFT Process subsystem—analysis windowing and synthesis overlap-add—are accelerated using Cadence Tensilica HiFi3 SIMD vector intrinsics.

Parallel Analysis Windowing (stft_process_apply_window)

The input FFT buffer stores interleaved 32-bit complex numbers (real and imag), but the analysis window applies only to the real audio samples (the imaginary component represents time-domain zero).

In stft_process_apply_window() on HiFi3:

1. Four complex samples are loaded per iteration using AE_L32_I.

2. The real components are unpacked and gathered into vector registers ae_f32x2 using AE_SEL32_HH.

3. Four Q1.31 window coefficients are loaded in parallel via AE_L32X2_IP.

4. Fractional vector multiplication with symmetric rounding is executed via AE_MULFP32X2RS:

   \[\text{real}' = \text{real} \times w[n] \quad (\text{Q1.31} \times \text{Q1.31} \to \text{Q1.31})\]

5. The updated real parts are stored back into the complex buffer using AE_S32_L_IP, leaving the imaginary parts unaltered.

Parallel Overlap-Add Synthesis (stft_process_overlap_add_ifft_buffer)

During synthesis overlap-add, the real output samples from the inverse FFT must be scaled by the window gain compensation factor \(g_{\text{comp}}\) and added with 32-bit saturation to the existing content of the circular output buffer:

1. The scalar gain compensation factor stft_process_state.gain_comp is broadcast into a dual 32-bit vector register via AE_MOVDA32.

2. Two IFFT real output samples are loaded and packed into ae_f32x2 via AE_L32_IP and AE_SEL32_HH.

3. Two samples from the circular output buffer are loaded via AE_L32X2_I.

4. A fused multiply-accumulate with saturation (AE_MULAFP32X2RS) computes:

   \[\text{obuf}[n] = \text{sat}_{32}\left( \text{obuf}[n] + (\text{ifft\_real}[n] \times g_{\text{comp}}) \right)\]

5. The accumulated samples are written back to the circular buffer via aligned vector store AE_S32X2_IP.

Figure 201 Cadence Tensilica HiFi3 SIMD Vector Execution: Parallel Analysis Windowing & Overlap-Add SIMD MACs

---

IPC4 Control Plane, LLEXT Packaging & ALSA Topology 2 Graph

The STFT Process component complies fully with the Intel IPC4 audio architecture and can be built either statically into the base firmware image or dynamically packaged as a Zephyr Loadable Linkable Extension (LLEXT).

Configuration Payload Structure

Runtime parameters are delivered via IPC4 large configuration messages as a 64-byte structured blob defined by sof_stft_process_config:

struct sof_stft_process_config {
    uint32_t size;                          /**< Size of this struct (64 bytes) */
    uint32_t reserved[8];
    int32_t sample_frequency;               /**< Sample rate in Hz (e.g., 16000, 48000) */
    int32_t window_gain_comp;               /**< Q1.31 gain compensation for iSTFT */
    int32_t reserved_32;
    int16_t channel;                        /**< Channel select (-1=all, 0=left, 1=right) */
    int16_t frame_length;                   /**< Frame length N (samples, e.g., 512, 1024) */
    int16_t frame_shift;                    /**< Hop size R_hop (samples, e.g., 128, 256) */
    int16_t reserved_16;
    enum sof_stft_process_fft_pad_type pad; /**< Padding type (PAD_END, PAD_CENTER) */
    enum sof_stft_process_fft_window_type window; /**< Window enum (RECT, BLACKMAN, HAMM, HANN, POVEY) */
} __attribute__((packed));

Zephyr LLEXT Dynamic Module Packaging

When built with CONFIG_COMP_STFT_PROCESS_MODULE=y, the component is compiled into a standalone relocatable ELF extension (stft_process.llext):

• Module Name: STFT_PROCESS (manifest alias STFTPROC)

• Component UUID: a6:6e:11:0d:50:91:de:46:98:b8:b2:b3:a7:91:da:29

• Topology GUID: a66e110d-5091-de46-98b8-b2b3a791da29

ALSA Topology 2 Pipeline Definition

In ALSA Topology 2, the widget is declared in tools/topology/topology2/include/components/stft_process.conf as type effect. Standard benchmark pipelines include pre-configured binary configurations:

• hann_192_48.conf: 4 ms window (192 samples), 1 ms hop (48 samples) @ 48 kHz

• hann_512_128.conf: 10.7 ms window (512 samples), 2.7 ms hop (128 samples) @ 48 kHz

• hann_768_120.conf: 16 ms window (768 samples), 2.5 ms hop (120 samples) @ 48 kHz

• hann_1024_256.conf: 21.3 ms window (1024 samples), 5.3 ms hop (256 samples) @ 48 kHz

• hann_1536_240.conf: 32 ms window (1536 samples), 5 ms hop (240 samples) @ 48 kHz

Figure 202 ALSA Topology 2 STFT Process Audio Pipeline Graph & IPC4 Configuration Dispatch

---

Factory Bringup, Verification & Test Runbook

This runbook outlines procedures to generate configuration blobs, compile topologies, verify standalone testbench processing, and confirm audio quality on physical targets.

1. Binary Blob Generation via GNU Octave

Generate standard pre-computed configuration blobs:

# Step 1: Navigate to the tuning directory
cd src/audio/stft_process/tune

# Step 2: Run Octave to produce topology configuration files
octave --no-gui setup_stft_process.m

# Output files generated in tools/topology/topology2/include/components/stft_process/:
# - hann_192_48.conf
# - hann_512_128.conf
# - hann_768_120.conf
# - hann_1024_256.conf
# - hann_1536_240.conf

2. Standalone Testbench Loopback Verification

Test the component within the SOF testbench without requiring target hardware:

# Define workspace root
export SOF_WORKSPACE=${HOME}/work

# Execute testbench with 16-bit PCM configuration
sof-testbench -p -i input_16k.wav -o output_stft_16k.wav \
    -t tools/topology/topology2/build/topology1/stft_process_s16.tplg

# Execute testbench with 32-bit PCM configuration
sof-testbench -p -i input_48k.wav -o output_stft_48k.wav \
    -t tools/topology/topology2/build/topology1/stft_process_s32.tplg

3. Real-Time Hardware Mixer Verification

On physical development hardware (such as Panther Lake, Arrow Lake, or Tiger Lake), verify runtime parameter delivery via SSH:

# Step 1: Query available mixer controls
ssh root@<dut-ip> "amixer -c0 controls | grep -i 'STFT_PROCESS'"

# Step 2: Push a new window configuration blob
ssh root@<dut-ip> "amixer -c0 cset name='Analog Playback STFT_PROCESS bytes' < hann_1024_256.blob"

4. Acoustic Reconstruction & Linearity Validation

When the STFT Process component runs in passthrough (zero spectral modification), the analysis and synthesis stages must achieve mathematically transparent reconstruction:

1. Sine Wave Invariance Test: Feed a pure 1000 Hz full-scale sine wave into the pipeline.

2. Harmonic Distortion (THD+N): Measure the reconstructed output using Audio Precision or Python FFT scripts. Total Harmonic Distortion must remain below \(-90\text{ dBFS}\), proving Constant Overlap-Add (COLA) compliance.

3. Transient Response Test: Feed a single-sample unit impulse (\(\delta[n]\)). The output waveform must reconstruct the exact impulse with zero pre-ringing, phase smearing, or amplitude attenuation beyond the inherent frame delay.