Phase Vocoder Architecture#
The Phase Vocoder subsystem in Sound Open Firmware (SOF) is an advanced frequency-domain audio processing component that performs real-time Time-Scale Modification (TSM) without altering pitch, and pitch shifting without altering duration. Operating in the Short-Time Fourier Transform (STFT) domain, the Phase Vocoder allows dynamic playback rate scaling from 0.5× (half speed) to 2.0× (double speed) via standard ALSA enum mixer controls or high-resolution Q3.29 fixed-point coefficients.
Embedded digital audio systems traditionally rely on sample rate conversion (resampling) or time-domain overlap-add algorithms (such as WSOLA) to modify stream tempo. However, resampling inherently alters pitch (the classic “chipmunk” or “slow-tape” effect), while time-domain slicing suffers from pitch-tracking errors, transient smearing, and metallic artifacts during polyphonic audio reproduction. The SOF Phase Vocoder overcomes these limitations by transforming time-domain PCM samples into complex frequency spectra, decoupling spectral magnitude from phase progression, tracking instantaneous bin frequencies across time, and re-synthesizing time-scaled waveforms via Overlap-Add (OLA) Inverse Fast Fourier Transforms.
The component incorporates an exact Greatest-Common-Divisor (GCD) frame counter normalization algorithm that prevents 32-bit integer overflow during indefinite streaming, an interactive transient-preserving phase re-anchoring state machine that eliminates phase smearing across live speed changes, a low-overhead mono downmix optimization that reduces memory and compute requirements by up to 75%, and full integration with the Intel IPC4 control plane and Zephyr Loadable Linkable Extension (LLEXT) dynamic module architecture.
Architectural Overview & Time-Scale Modification Principles#
Time-Scale Modification (TSM) is the process of altering the acoustic duration of an audio signal while strictly preserving its spectral envelope, pitch, and timbre. In automotive infotainment, podcast players, speech-to-text accessibility tools, and digital audio workstations (DAWs), users frequently accelerate or decelerate playback without wanting voices or musical instruments to shift pitch.
Theoretical Comparison of Time Modification Techniques#
Audio systems implement time-scale modification using three distinct paradigms:
Sample Rate Conversion (Resampling / Sample Rate Conversion Architecture (SRC & ASRC)): Alters the consumption rate of samples across time. Because frequency and duration are coupled in the time domain, doubling playback speed doubles all audio frequencies (\(f_{\text{out}} = 2 \cdot f_{\text{in}}\)), transposing pitch up by exactly one octave.
Time-Domain Overlap-Add (TD-PSOLA / WSOLA): Segments audio into pitch periods in the time domain and duplicates or drops periods based on cross-correlation matching. While computationally light, WSOLA depends on robust pitch-detection algorithms that fail on polyphonic music, percussion, noisy speech, or mixed multimedia streams.
Phase Vocoder (Frequency-Domain STFT): Deconstructs the audio into sinusoidal frequency bins using overlapping Fourier transforms. Spectral magnitudes and instantaneous phase trajectories are independently tracked, scaled along the synthesis timeline, and re-synthesized using Inverse FFTs. This works reliably across monophonic speech, polyphonic orchestrations, and percussive transients.
Parameter |
Sample Rate Converter (SRC) |
Time-Domain WSOLA |
SOF Phase Vocoder |
|---|---|---|---|
Processing Domain |
Time domain (polyphase FIR) |
Time domain (cross-correlation) |
Frequency domain (STFT / Polar) |
Pitch Invariance |
No (pitch shifts with tempo) |
Yes (monophonic signals only) |
Yes (full polyphonic & speech) |
Speed Range |
Continuous rational ratio (\(M/N\)) |
Typically 0.75x to 1.5x |
0.5x to 2.0x (16 discrete steps or Q3.29) |
Algorithmic Latency |
Sub-millisecond (FIR tap length) |
Moderate (20 to 40 ms) |
Window hop size (2.7 to 5.3 ms at 48 kHz) |
DSP Memory Footprint |
Small (< 2 KB coefficient RAM) |
Moderate (search windows) |
Medium (~8 KB to 32 KB depending on FFT size) |
Primary Use Case |
Clock domain bridging, resampling |
Low-power voice dictation |
High-fidelity multimedia, speech rate control |
Figure 189 SOF Phase Vocoder Architecture: STFT Analysis, Polar Coordinate Transformation, Spectral Modification & Synthesis Overlap-Add Core#
Short-Time Fourier Transform (STFT) Analysis & Overlap-Add Synthesis#
The foundation of the Phase Vocoder is the Short-Time Fourier Transform (STFT). Continuous audio signals are non-stationary; their spectral content changes dynamically over time. The STFT segments the incoming signal into short, overlapping quasi-stationary windows, transforms each window into the frequency domain, and subsequently recombines them using Overlap-Add (OLA) synthesis.
Window Selection & Spectral Leakage Control#
To prevent abrupt boundary truncation (which causes broadband spectral leakage across Fourier bins), each analysis frame is multiplied by a smooth window function \(w[n]\). The SOF Phase Vocoder supports four configurable window types declared in sof_phase_vocoder_fft_window_type:
Rectangular Window (`STFT_RECTANGULAR_WINDOW = 0`): Provides the narrowest main lobe (highest frequency resolution), but severe sidelobe leakage (-13 dB attenuation), causing audible inter-bin modulation distortion.
Blackman Window (`STFT_BLACKMAN_WINDOW = 1`): Provides extreme sidelobe attenuation (-58 dB) with exact coefficients defined by WIN_BLACKMAN_A0_Q31. Ideal for high-precision analytical inspection.
Hamming Window (`STFT_HAMMING_WINDOW = 2`): Attenuates first sidelobe to -43 dB, balancing main lobe width and spectral rolloff.
Hann Window (`STFT_HANN_WINDOW = 3`, Standard Default): Constructed from a raised cosine bell:
\[w[n] = 0.5 - 0.5 \cos\left(\frac{2\pi n}{N}\right), \quad 0 \le n < N\]The Hann window satisfies the Constant Overlap-Add (COLA) condition when the hop size is an integer submultiple of the window length (\(N/2\), \(N/4\)).
Frame Sizing & Hop Geometry#
The component operates on power-of-two frame lengths \(N\) and analysis hop sizes \(R_a\):
Standard Frame Lengths (\(N\)): 256 samples (5.33 ms @ 48 kHz), 512 samples (10.67 ms), or 1024 samples (21.33 ms).
Analysis Hop Size (\(R_a\)): Typically \(N/2\) (50% overlap) or \(N/4\) (75% overlap, e.g., 256-sample hop on 1024-sample window).
History Overlap Size: \(N - R_a\) samples, retained in
state->prev_data[ch]across ticks.
Reconstructive Window Gain Compensation#
During synthesis, the inverse-transformed time-domain samples are multiplied again by the synthesis window \(w[n]\) before being accumulated into the output buffer. Passing audio through two cascaded window stages scales the total signal power by the sum of the squared window coefficients. To guarantee exact unity gain (0 dBFS) reconstruction, SOF pre-calculates a 32-bit Q1.31 gain compensation factor:
This value is stored in config->window_gain_comp and multiplied into the synthesis overlap-add accumulator:
sample = Q_MULTSR_32X32((int64_t)state->gain_comp, fft->fft_buf[idx].real, 31, 31, 31);
*w = sat_int32((int64_t)*w + sample);
Figure 190 Short-Time Fourier Transform (STFT) Analysis & Overlap-Add (OLA) Synthesis Timeline with Window Gain Compensation#
Polar Domain Phase Unwrapping & Phase Accumulation Mechanics#
A naive time-stretching approach that merely duplicates or displaces STFT frames in time results in catastrophic acoustic artifacts: destructive comb filtering, rapid amplitude tremolo, and a hollow, reverberant “phasiness”. These distortions occur because the Fourier phase across successive analysis hops is non-stationary.
The Phase Discontinuity Problem#
When an input sinusoidal component of frequency \(\omega_0\) is analyzed at intervals of \(R_a\), its phase advances by \(\Delta \theta = \omega_0 R_a\). If the synthesis frames are re-positioned at a new synthesis hop interval \(R_s = R_a / \text{speed}\), the synthesis phase must advance by \(\Delta \phi = \omega_0 R_s\). If the original analysis phase is retained without modification, adjacent overlapping frames will destructively interfere at the synthesis boundary.
Polar Coordinate Conversion#
To manipulate magnitude and phase independently, the 32-bit complex Fourier output icomplex32 (\(X[k] = \text{real} + j \cdot \text{imag}\)) is converted to polar form ipolar32 using sofm_icomplex32_to_polar():
Spectral Magnitude (\(M_k\)): Represented in high-precision Q2.30 format.
Phase Angle (\(\theta_k\)): Converted from trigonometric Q3.29 to Q5.27 format using Q_SHIFT_RND(angle, 29, 27).
Phase Unwrapping Arithmetic#
Because phase angles are circular (\([-\pi, +\pi]\)), calculating the phase difference between consecutive frames introduces phase wrap-around ambiguity whenever the delta crosses \(\pm \pi\):
To recover the true instantaneous frequency deviation, SOF unwraps the phase difference into the fundamental interval \([-\pi, +\pi]\) via unwrap_angle_q27():
static int32_t unwrap_angle_q27(int32_t angle)
{
while (angle > PHASE_VOCODER_PI_Q27)
angle -= PHASE_VOCODER_TWO_PI_Q27;
while (angle < -PHASE_VOCODER_PI_Q27)
angle += PHASE_VOCODER_TWO_PI_Q27;
return angle;
}
where fixed-point radian constants are defined as:
PHASE_VOCODER_PI_Q27 = 421657428 (\(\pi \cdot 2^{27}\))
PHASE_VOCODER_TWO_PI_Q27 = 843314857 (\(2\pi \cdot 2^{27}\))
Synthesis Phase Accumulation#
The unwrapped phase difference \(\Delta \theta_k\) represents the authentic instantaneous phase progression of frequency bin \(k\). During synthesis, the output phase accumulator output_phase[k] continuously integrates these deltas:
This ensures that sinusoidal components remain strictly continuous across the time-scaled synthesis timeline, maintaining phase coherence and pristine transient clarity.
Figure 191 Polar-Domain Phase Unwrapping, Phase Difference Calculation & Synthesis Phase Accumulation Pipeline#
Variable Playback Speed & Fractional Interpolation Engine#
The SOF Phase Vocoder controls speed by adjusting the rate at which analysis frames are synthesized into output frames. Speed is represented internally as a 32-bit signed fixed-point integer in Q3.29 format:
Minimum Speed: PHASE_VOCODER_MIN_SPEED_Q29 = 0.5 * 2^29 = 0x10000000 (0.5x, half speed / slow motion).
Normal Speed: PHASE_VOCODER_SPEED_NORMAL = 1.0 * 2^29 = 0x20000000 (1.0x, unity passthrough).
Maximum Speed: PHASE_VOCODER_MAX_SPEED_Q29 = 2.0 * 2^29 = 0x40000000 (2.0x, double speed).
ALSA Discrete Enum Control Grid#
For intuitive integration with userspace players and ALSA mixers, the Phase Vocoder exposes an enum control with 16 uniform speed increments:
When an enum index \(E \in [0, 15]\) is written by the host, the driver calculates:
Yielding exact playback speeds of 0.5x, 0.6x, 0.7x, …, 1.9x, 2.0x.
Fractional Timeline Progression#
As synthesis frames (num_output_ifft) are produced, the corresponding virtual position on the input analysis timeline is calculated in 64-bit precision:
input_frame_num_frac = (int64_t)state->num_output_ifft * cd->state.speed; /* Q31.29 */
input_frame_num_floor = (int32_t)(input_frame_num_frac >> 29); /* Integer frame index */
state->num_input_fft_to_use = input_frame_num_floor + 1;
state->interpolate_fraction = input_frame_num_frac - ((int64_t)input_frame_num_floor << 29);
Dual-Domain Linear Interpolation#
Because the virtual analysis position falls between discrete FFT analysis frames, the vocoder performs linear interpolation across both spectral magnitude and phase delta:
The interpolated phase delta is then added to output_phase[k], and the resulting polar coordinate \((M_{\text{interp}}, \phi_{\text{out}})\) is converted back to Cartesian format for the Inverse FFT.
Greatest-Common-Divisor (GCD) Counter Normalization#
During extended playback (such as video streaming or days of continuous audio playback), the integer counters num_output_ifft and num_input_fft would eventually overflow a 32-bit signed integer (\(2^{31} - 1\)). However, arbitrarily resetting both counters to zero induces an instantaneous phase discontinuity, producing an audible pop.
To solve this, the SOF Phase Vocoder implements an exact Greatest-Common-Divisor (GCD) counter normalization algorithm in phase_vocoder_normalize_counters(). The fractional timeline repeats with a period defined by:
Whenever num_output_ifft exceeds \(2^{28}\), the component subtracts the largest integer multiple of \(P_{\text{output}}\):
Because \(\Delta_{\text{output}} \times \text{speed}\) is an exact multiple of \(2^{29}\) by construction, subtracting \(\Delta_{\text{output}}\) from num_output_ifft and \(\Delta_{\text{input}}\) from num_input_fft preserves the interpolation fraction identically down to the least significant bit, ensuring zero round-off error while keeping counters perpetually bounded!
Figure 192 Variable Time-Scale Modification (TSM) Engine: Output Timeline Interpolation & Greatest-Common-Divisor (GCD) Frame Counter Normalization#
Interactive Phase Re-Anchoring State Machine#
When a user adjusts the speed slider during live playback, changing cd->speed_ctrl triggers phase_vocoder_reset_for_new_speed(). Naive vocoder implementations re-initialize their analysis state from frame zero, causing an immediate volume drop and transient blurring.
The SOF Phase Vocoder implements a specialized phase re-anchoring mechanism:
Cold-Start Suppression: state->num_input_fft is set to 1 (never 0). This avoids re-entering the initial cold-start analysis path which copies absolute angles instead of deltas.
Phase Re-Anchoring Invariant: The synthesis accumulator is re-anchored so that the first post-reset IFFT lands exactly back on polar_prev.angle:
\[\text{output\_phase}[k] = \text{unwrap\_angle\_q27}\left(\text{polar\_prev}[k].\text{angle} - \text{angle\_delta\_prev}[k]\right)\]
This completely neutralizes phase drift accumulated across non-unity speed transitions, preventing transient smearing and ensuring the sound remains bright, focused, and crisp.
Figure 193 Dynamic Speed Transition & Interactive Phase Re-Anchoring State Machine#
Multi-Channel Processing & Mono Downmixing Optimization#
Processing high-resolution multi-channel audio through an STFT phase vocoder requires substantial memory and computational resources. For each audio channel, the DSP must maintain separate input ring buffers, overlap history buffers, output ring buffers, polar magnitude arrays, and phase accumulators.
Memory Footprint Optimization#
During component initialization in phase_vocoder_setup(), all buffer allocations are consolidated into single contiguous blocks to minimize heap fragmentation and allocator overhead:
Time-Domain Ring Buffers (`sample_buffers_size`): Calculated across all active processing channels:
\[\text{Bytes} = 4 \times \left[\text{channels} \cdot \left(L_{\text{ibuf}} + L_{\text{obuf}} + L_{\text{prev}}\right) + N\right]\]Subject to an upper safety bound of STFT_MAX_ALLOC_SIZE = 65536 bytes (64 KB).
Polar Domain Arrays (`phase_vocoder_polar_bytes`): Consolidates polar, polar_prev, angle_delta, angle_delta_prev, and output_phase:
\[\text{Bytes} = \text{channels} \times \frac{N}{2} \times \left(2 \cdot \text{sizeof(struct ipolar32)} + 3 \cdot \text{sizeof(int32\_t)}\right)\]
Mono Downmix Optimization Mode#
In many voice and multimedia pipelines, human speech is concentrated in the center channel, or the audio endpoint is bandwidth-constrained. The Phase Vocoder supports a specialized Mono Downmix Optimization (config->mono = 1):
Input Downmixing: When multi-channel audio enters the component, incoming channels are pre-summed into a single processing channel using a 32-bit Q1.31 gain coefficient:
\[\text{mono\_mix\_coef} = \left\lfloor \frac{2^{31}}{\text{stream\_channels}} \right\rfloor\]Single-Channel Core Execution: Only a single forward FFT, polar transformation, phase unwrapping, and Inverse FFT pipeline executes, slashing DSP CPU cycles and memory consumption by 50% for stereo streams and 75% for 4-channel streams.
Multi-Channel Replication: During output egress (
phase_vocoder_sink_s32()), the single processed channel is replicated across all output sink channels:for (i = 0; i < n; i++) { for (ch = 0; ch < stream_channels; ch++) *y++ = *obuf->r_ptr; obuf->r_ptr++; }
Universal PCM Frame Format Support#
The component implements specialized ingestion and egress functions for all standard SOF PCM formats:
16-Bit PCM (`SOF_IPC_FRAME_S16_LE`): Processes 16-bit audio via
phase_vocoder_s16(), converting to 32-bit internal representations for the FFT.24-Bit PCM in 32-Bit Container (`SOF_IPC_FRAME_S24_4LE`): Processes 24-bit audio via
phase_vocoder_s24()with sign extension.32-Bit PCM (`SOF_IPC_FRAME_S32_LE`): Operates on native 32-bit PCM via
phase_vocoder_s32().
Zero-Overhead Bypass Fast-Path#
When processing is disabled via ALSA mixer (cd->enable == false), the component completely bypasses FFT execution, polar transformations, and phase accumulation. It invokes source_to_sink_copy() directly:
if (!cd->enable) {
frames = MIN(source_frames, sink_frames);
source_to_sink_copy(source, sink, true, frames * cd->frame_bytes);
return 0;
}
This reduces active DSP cycles to a pure memory block transfer during passthrough.
IPC4 Control Plane, LLEXT Modular Packaging & ALSA Topology 2 Graph#
The Phase Vocoder complies with the Intel IPC4 modular audio architecture, allowing dynamic runtime configuration via standardized large configuration blobs, mixer switch controls, and enum speed selections.
IPC4 Runtime Configuration Handler#
Runtime control messages are dispatched to phase_vocoder_set_config():
Switch Control (`SOF_IPC4_SWITCH_CONTROL_PARAM_ID = 259`): Controls the enable/bypass state (cd->enable = ctl->chanv[0].value).
Enum Control (`SOF_IPC4_ENUM_CONTROL_PARAM_ID = 257`): Controls the playback speed enum index (0 .. 15), mapping directly to 0.5x .. 2.0x.
Large Configuration Blob (`struct sof_phase_vocoder_config`): Delivers the 64-byte structural configuration defining sample rate, window type, frame length, hop size, and mono mode:
struct sof_phase_vocoder_config {
uint32_t size;
uint32_t reserved[8];
int32_t sample_frequency;
int32_t window_gain_comp;
int32_t reserved_32;
int16_t mono;
int16_t frame_length;
int16_t frame_shift;
int16_t reserved_16;
int32_t reserved_pad;
enum sof_phase_vocoder_fft_window_type window;
} __attribute__((packed));
Zephyr LLEXT Dynamic Module Packaging#
When built as a loadable linkable extension (CONFIG_COMP_PHASE_VOCODER_MODULE=y), the component is linked into phase_vocoder.llext:
static const struct sof_man_module_manifest mod_manifest __section(".module") __used =
SOF_LLEXT_MODULE_MANIFEST("PHASEVOC", &phase_vocoder_interface, 1,
SOF_REG_UUID(phase_vocoder), 40);
Module Name:
"PHASEVOC"Component UUID:
7a:cb:fb:09:c5:a9:57:4a:84:34:44:40:e5:98:ab:24Topology GUID:
7acbfb09-c5a9-574a-8434-4440e598ab24
ALSA Topology 2 Widget Definition#
In ALSA Topology 2, the Phase Vocoder is declared in tools/topology/topology2/include/components/phase_vocoder.conf:
Class.Widget."phase_vocoder" {
DefineAttribute."index" { type "integer" }
DefineAttribute."instance" { type "integer" }
<include/components/widget-common.conf>
attributes {
!constructor [ "index" "instance" ]
!mandatory [
"num_input_pins"
"num_output_pins"
"num_input_audio_formats"
"num_output_audio_formats"
]
!immutable [ "uuid" "type" ]
unique "instance"
}
Object.Control {
# Switch controls (Bypass on/off)
mixer."1" {
Object.Base.ops.1 {
name "ctl"
info "volsw"
get 259
put 259
}
max 1
}
# Enum controls (Speed 0.5x to 2.0x)
enum."1" {
Object.Base {
text.0 {
name "phase_vocoder_speed_enum"
!values [
"0.5" "0.6" "0.7" "0.8" "0.9" "1.0"
"1.1" "1.2" "1.3" "1.4" "1.5" "1.6"
"1.7" "1.8" "1.9" "2.0"
]
}
ops.1 {
name "ctl"
info "enum"
get 257
put 257
}
}
}
}
uuid "7a:cb:fb:09:c5:a9:57:4a:84:34:44:40:e5:98:ab:24"
type "effect"
no_pm "true"
num_input_pins 1
num_output_pins 1
}
Figure 194 IPC4 Control Architecture, Switch & Enum Control Handlers & LLEXT Dynamic Module Binding#
Figure 195 ALSA Topology 2 Phase Vocoder Pipeline Graph with Benchmark Controls#
Factory Bringup, Acoustic Quality & Verification Runbook#
This runbook provides step-by-step procedures to build, deploy, tune, and test the Phase Vocoder subsystem using the SOF testbench and physical device under test (DUT).
1. Binary Blob Generation via GNU Octave#
Generate the pre-computed configuration blobs for standard Hann window configurations:
# Step 1: Navigate to the tuning directory
cd tools/tune/phase_vocoder
# Step 2: Run Octave to generate topology configurations
octave --no-gui setup_phase_vocoder.m
# Output files created in topology2/include/components/phase_vocoder/:
# - hann_256_128.conf (5.3 ms window, 2.7 ms hop)
# - hann_512_128.conf (10.7 ms window, 2.7 ms hop)
# - hann_512_256.conf (10.7 ms window, 5.3 ms hop)
# - hann_1024_256.conf (21.3 ms window, 5.3 ms hop - default stereo)
# - hann_1024_256_mono.conf (21.3 ms window, 5.3 ms hop - default mono)
2. Standalone Testbench Verification#
Execute the automated testbench runner scripts to verify time-stretching accuracy, pitch invariance, and memory safety without hardware dependencies:
# Define workspace root
export SOF_WORKSPACE=${HOME}/work
# Run 16-bit PCM testbench execution with automated speed sweep
tools/tune/phase_vocoder/phase_vocoder_s16.sh input_speech.wav output_s16.wav
# Run 32-bit PCM testbench execution
tools/tune/phase_vocoder/phase_vocoder_s32.sh input_music.wav output_s32.wav
# Verify output duration:
# Input: 10.0 seconds
# Output: Dynamically swept duration matching the control script schedule
3. Real-Time Hardware ALSA Mixer Verification#
On physical development hardware (such as Panther Lake, Arrow Lake, or Tiger Lake), verify runtime controls via SSH:
# Step 1: Query available mixer controls
ssh root@<dut-ip> "amixer -c0 controls | grep -i 'Phase Vocoder'"
# Expected controls:
# numid=10,iface=MIXER,name='Analog Playback Phase Vocoder enable'
# numid=11,iface=MIXER,name='Analog Playback Phase Vocoder speed'
# Step 2: Enable vocoder processing
ssh root@<dut-ip> "amixer -c0 cset name='Analog Playback Phase Vocoder enable' on"
# Step 3: Set slow-motion playback (0.5x speed)
ssh root@<dut-ip> "amixer -c0 cset name='Analog Playback Phase Vocoder speed' 0.5"
# Step 4: Sweep to accelerated playback (1.5x speed)
ssh root@<dut-ip> "amixer -c0 cset name='Analog Playback Phase Vocoder speed' 1.5"
# Step 5: Toggle zero-overhead bypass mode
ssh root@<dut-ip> "amixer -c0 cset name='Analog Playback Phase Vocoder enable' off"
4. Acoustic Quality & Pitch Invariance Verification#
To confirm that the Phase Vocoder alters tempo without modifying pitch:
Sine Wave Benchmark: Feed a pure 1000 Hz sine wave tone into the pipeline.
Frequency Domain Inspection: Record output at 0.5x, 1.0x, and 2.0x speeds.
FFT Verification: Compute the peak spectral bin using sox input.wav -n stat -freq or Python numpy.fft. The peak fundamental frequency must remain exactly at 1000 Hz (\(\pm 0.01\) Hz) across all speed settings, proving pitch invariance.
THD+N & Signal-to-Noise Verification: Verify that harmonic distortion products remain below -70 dBFS across the entire audible band.