Procedural audio (createAudio)

painter.createAudio synthesizes 16-bit PCM WAV Buffers — presets, custom multi-layer sounds, timelines, and mixes. Output is ready for mixAudio, videoPipeline().audio(), disk via save, or any API that accepts a WAV buffer.

Phase 9 hardens this surface with pre-allocation resource checks, operation-local seeded randomness, stricter DSP validation, and RIFF-safe PCM16 handling. createAudio remains a complete-buffer API: it returns a finished WAV Buffer, not a streaming audio object.

Hub: Audio (advanced) · Runtime limits: Runtime validation & resource governance · Mux on video: Video audio operations · Pipeline tracks: Video pipeline

#Entry points

Types export from apexify.js/types: SynthPresetName, SynthSoundOptions, SynthLayer, SynthSequenceOptions, SynthComposeOptions, …

#Deterministic seeded audio

Noise-based synthesis is nondeterministic by default. Supply seed when you need byte-stable procedural output for tests, cache keys, reproducible renders, or parallel jobs.

seed accepts a safe integer or a 1–256 character string on sound, sequence, compose, clip, and mix options where the type exposes it.

1const a = painter.createAudio.synth({2  seed: 'ui-impact-v1',3  layers: [4    { waveform: 'pink', duration: 0.18, gain: 0.35 },5  ],6});7 8const b = painter.createAudio.synth({9  seed: 'ui-impact-v1',10  layers: [11    { waveform: 'pink', duration: 0.18, gain: 0.35 },12  ],13});14 15// a and b are byte-identical WAV buffers.

Seeded layers/events/clips derive independent random streams. Concurrent seeded renders do not share RNG or filter state. Omitting seed intentionally keeps stochastic sounds nondeterministic.

#Presets

39 built-in names (game/UI oriented): laser, explosion, coin, jump, whoosh, menuSelect, footstep, thunder, …

1const pew = painter.createAudio.preset('laser', { volume: 0.8, transpose: 2 });2 3for (const info of painter.createAudio.listPresets()) {4  console.log(info.name, info.description, info.defaultDuration);5}

SynthPresetOverrides: volume, transpose, plus any SynthSoundOptions field layered on the preset definition. Preset definitions are cloned before use, so caller mutation does not alter the shared catalog.

#Custom sounds — synth / custom

Each SynthLayer can use waveforms sine, square, sawtooth, triangle, noise, pink:

1const uiTick = painter.createAudio.synth({2  seed: 'ui-tick',3  layers: [4    {5      waveform: 'square',6      frequency: 880,7      frequencyEnd: 440,8      duration: 0.06,9      gain: 0.35,10      adsr: { attack: 0.001, decay: 0.02, sustain: 0, release: 0.02 },11    },12  ],13  masterGain: 1,14  limiter: true,15});

sampleRate (default 44100), channels (1 or 2), limiter (default true) apply at the sound level.

Noise/pink layers reject tonal-only controls such as frequency sweeps, detune, vibrato, and harmonic partials instead of silently ignoring them. Filter cutoff must be below the selected sample rate's Nyquist frequency.

#Timelines — sequence

Events fire at at (seconds). Each event uses exactly one of preset or options (SynthSoundOptions), plus optional gain.

1const combo = painter.createAudio.sequence({2  seed: 'combo-v1',3  events: [4    { at: 0, preset: 'hit' },5    { at: 0.08, preset: 'coin', gain: 0.85 },6    { at: 0.2, preset: 'powerup' },7  ],8  tail: 0.2,9  masterGain: 0.95,10});

tail pads silence after the last event ends. Overlapping events sum in the same buffer (limiter applied). The renderer keeps the final output plus one event working buffer instead of retaining every rendered event in memory.

#Overlapping clips — compose

compose is for dense beds: multiple presets, custom sounds, or existing WAV buffers on one timeline with per-clip shaping.

Top-level SynthComposeOptions: duration, tail, masterGain, postHighpassHz (rumble cleanup), noiseGateThreshold (quiet overlap hiss), and optional seed.

1const bossIntro = painter.createAudio.compose({2  seed: 'boss-intro-v3',3  clips: [4    { at: 0, preset: 'rumble', duration: 1.2, gain: 0.5 },5    { at: 0.4, preset: 'alarm', gain: 0.4, fadeIn: 0.1 },6    { at: 0.9, preset: 'explosion', gain: 0.75 },7  ],8  postHighpassHz: 220,9  tail: 0.3,10});

For synthesized preset/custom sources, transpose, detune, and pitch modify oscillator pitch. Existing WAV clips do not accept those procedural pitch controls; use speed instead. speed is playback-rate resampling: it changes duration and pitch together and is not pitch-preserving time stretch.

The composer keeps the final output plus one processed clip working buffer rather than retaining every clip at once.

#mix

Combine one or more Buffers (or inline preset/sound definitions where supported):

1const layered = painter.createAudio.mix(2  [painter.createAudio.preset('beep'), painter.createAudio.preset('whoosh', { volume: 0.5 })],3  { masterGain: 0.9, seed: 'layered-v1' }4);

Use when you already have WAV buffers and need a single file without a full compose timeline. If any input uses timeline/clip controls, mix routes through the validated composition path.

#Resource limits and allocation behavior

Audio requests are validated before large sample allocations. Defaults are controlled by the shared Apexify runtime configuration:

maxAudioBytes is a peak working-memory guard, not just a final WAV-size limit. Apexify.js accounts for the Float32 render buffer, decoded/resampled sources where applicable, transient event/clip buffers, and the final PCM16 WAV allocation when those buffers coexist.

1configureApexifyRuntime({2  limits: {3    maxAudioDurationSeconds: 120,4    maxAudioBytes: 96 * 1024 * 1024,5  },6});

Lower limits for multi-tenant or memory-constrained services. Do not raise them merely to suppress ApexifyResourceLimitError.

Because the API returns a complete WAV Buffer, Phase 9 deliberately enforces bounded complete-buffer rendering rather than claiming streaming synthesis.

#WAV behavior

Generated output is mono/stereo PCM16 RIFF/WAVE. Internally, WAV inspection/decoding validates chunk boundaries and does not assume fmt and data occur at fixed offsets. Unsupported formats, bit depths, malformed chunk sizes, incomplete frames, inconsistent byte-rate/block-alignment metadata, and oversized decoded audio are rejected before unsafe allocation.

#Put audio on video

##Single operation — mixAudio

Pass any createAudio Buffer as MixAudioOverlayClip.source:

1const sfx = painter.createAudio.preset('laser');2 3await painter.createVideo({4  source: './clip.mp4',5  mixAudio: {6    outputPath: './clip-sfx.mp4',7    keepOriginalAudio: false,8    overlays: [{ source: sfx, startTime: 1.5, volume: 1 }],9  },10});

keepOriginalAudio: false → overlays only (no silent bed under SFX). Full field reference: Video audio operations.

##Editor stack — videoPipeline().audio()

Declare tracks without pre-building every buffer:

See Video pipeline — Audio layers.

#Standalone export

1await painter.createAudio.save(2  painter.createAudio.sequence({3    seed: 'menu-select',4    events: [{ at: 0, preset: 'menuSelect' }],5  }),6  './ui/select.wav'7);

#vs FFmpeg video audio

#Tips

• Add seed when output must be reproducible across repeated or concurrent renders.
• listPresets() before hard-coding names in editor UIs.
• Prefer sequence for linear SFX chains; compose when clips overlap with independent pitch/fades.
• For long gameplay videos, generate SFX once, then mixAudio or pipeline type: 'wav' — avoid re-synthesizing on every export pass.
• Keep deployment-specific audio limits below what the Node process can safely hold at peak.
• Pair with GIF / animate by muxing SFX after frame encode.