TanStack
AI
Media

Text-to-Speech

TanStack AI provides support for text-to-speech generation through dedicated TTS adapters. This guide covers how to convert text into spoken audio using OpenAI and Gemini providers.

Overview

Text-to-speech (TTS) is handled by TTS adapters that follow the same tree-shakeable architecture as other adapters in TanStack AI. The TTS adapters support:

  • OpenAI: TTS-1, TTS-1-HD, and audio-capable GPT-4o models
  • Gemini: Gemini 2.5 Flash TTS (experimental)
  • BytePlus: Seed Speech (seed-audio-1.0)
  • fal.ai: Kokoro, ElevenLabs, MiniMax, Chatterbox, Dia, Orpheus, F5-TTS, VibeVoice, and more

Most providers here ship a fixed catalog of voices. When none of them fit, create your own and pass the new voice ID as voice. On a provider whose catalog is per-account, listVoices() reads back what is available.

Basic Usage

OpenAI Text-to-Speech

ts
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

// Generate speech from text (uses OPENAI_API_KEY from environment)
const result = await generateSpeech({
  adapter: openaiSpeech('tts-1'),
  text: 'Hello, welcome to TanStack AI!',
  voice: 'alloy',
})

// result.audio contains base64-encoded audio data
console.log(result.format) // 'mp3'
console.log(result.contentType) // 'audio/mpeg'

Gemini Text-to-Speech (Experimental)

ts
import { generateSpeech } from '@tanstack/ai'
import { geminiSpeech } from '@tanstack/ai-gemini'

// Generate speech from text (uses GOOGLE_API_KEY or GEMINI_API_KEY from environment)
const result = await generateSpeech({
  adapter: geminiSpeech('gemini-3.1-flash-tts-preview'),
  text: 'Hello from Gemini TTS!',
})

console.log(result.audio) // Base64 encoded audio

fal.ai Text-to-Speech

fal.ai offers a broad selection of TTS models, Google's brand-new gemini-3.1-flash-tts, ElevenLabs v3, MiniMax 2.6 HD, Kokoro's multilingual voices, and more. Pass the model ID as a string literal for fully typed modelOptions.

ts
import { generateSpeech } from '@tanstack/ai'
import { falSpeech } from '@tanstack/ai-fal'

// Google Gemini 3.1 Flash TTS, 80+ languages, expressive audio tags
const result = await generateSpeech({
  adapter: falSpeech('fal-ai/gemini-3.1-flash-tts'),
  text: '[warm, enthusiastic] Welcome to TanStack AI!',
  voice: 'Kore',
})
ts
import { generateSpeech } from '@tanstack/ai'
import { falSpeech } from '@tanstack/ai-fal'

// Kokoro multilingual
const result = await generateSpeech({
  adapter: falSpeech('fal-ai/kokoro/american-english'),
  text: 'Hello from fal!',
  voice: 'af_heart',
  speed: 1.0,
})

console.log(result.audio) // Base64 encoded audio
console.log(result.format) // e.g. "wav"
ts
import { generateSpeech } from '@tanstack/ai'
import { falSpeech } from '@tanstack/ai-fal'

// ElevenLabs v3 with model-specific options
const result = await generateSpeech({
  adapter: falSpeech('fal-ai/elevenlabs/tts/eleven-v3'),
  text: 'Welcome to TanStack AI.',
  // The fal adapter maps top-level `voice`/`speed` into the model input;
  // `modelOptions` is reserved for model-specific keys.
  voice: 'Rachel',
  modelOptions: {
    stability: 0.5,
  },
})

BytePlus Seed Speech

Seed Speech is a separate BytePlus product from ModelArk, so it uses its own key (BYTEPLUS_VOICE_API_KEY) rather than the ARK_API_KEY the chat, image and video adapters read. Voice ids go on the wire as Seed Speech's speaker:

ts
import { generateSpeech } from '@tanstack/ai'
import { byteplusSpeech } from '@tanstack/ai-byteplus'

const result = await generateSpeech({
  adapter: byteplusSpeech('seed-audio-1.0'),
  text: 'Welcome to TanStack AI!',
  voice: 'en_female_stokie_uranus_bigtts',
  format: 'mp3',
})

console.log(result.contentType) // "audio/mpeg"

Formats are wav, mp3, pcm and ogg_opus, and synthesis is capped at 120 seconds of output.

Seed Audio also does multi-role dialogue and word-level timings. Use turns for dialogue and timestamps for timings, both covered below.

Seed Speech has no top-level speaker field: the adapter sends voice as references: [{ speaker }]. Because modelOptions.references replaces that array rather than merging into it, passing references for voice cloning silently drops voice — include a speaker member yourself if you still want a stock voice. See the BytePlus adapter for the voice-id naming conventions.

Options

Common Options

All TTS adapters support these common options:

OptionTypeDescription
textstringThe text to convert to speech. Required unless you pass turns
turnsArray<{ text, voice }>Dialogue lines, one per speaker turn. Use it instead of text
voicestringThe voice to use for generation
formatstringOutput audio format (e.g., "mp3", "wav")
timestampsbooleanAsk for alignment and segments on the result

OpenAI Voice Options

OpenAI provides several distinct voices:

VoiceDescription
alloyNeutral, balanced voice
echoWarm, conversational voice
fableExpressive, storytelling voice
onyxDeep, authoritative voice
novaFriendly, upbeat voice
shimmerClear, gentle voice
ashCalm, measured voice
balladMelodic, flowing voice
coralBright, energetic voice
sageWise, thoughtful voice
versePoetic, rhythmic voice

OpenAI Format Options

FormatDescription
mp3MP3 audio (default)
opusOpus audio (good for streaming)
aacAAC audio
flacFLAC audio (lossless)
wavWAV audio (uncompressed)
pcmRaw PCM audio

Dialogue With Two or More Voices

You want a two-person skit, and one text string with one voice cannot give you that. Prefixing speaker names into the string is a guess about the prompt format, and the provider never tells you which line landed where.

Pass turns instead. Each turn carries its own text and its own voice:

ts
import { generateSpeech } from '@tanstack/ai'
import { byteplusSpeech } from '@tanstack/ai-byteplus'

// Replace SECOND_VOICE with another voice id from the BytePlus voice list.
const SECOND_VOICE = 'your-second-voice-id'

const result = await generateSpeech({
  adapter: byteplusSpeech('seed-audio-1.0'),
  turns: [
    {
      text: 'Do you have a left-handed bass?',
      voice: 'en_female_stokie_uranus_bigtts',
    },
    { text: 'We do. Come and try it.', voice: SECOND_VOICE },
  ],
  format: 'mp3',
})

turns and text are mutually exclusive. Pass one or the other.

Not every provider can do dialogue, and the ones that can have different speaker limits. The adapter declares its limit, so a request that asks for too many voices fails before it reaches the provider:

AdapterDistinct voices per request
byteplusSpeech3
elevenlabsSpeech10
geminiSpeech2
Other TTS adaptersDialogue not supported

Read the limit at runtime from adapter.capabilities:

ts
import { byteplusSpeech } from '@tanstack/ai-byteplus'

const adapter = byteplusSpeech('seed-audio-1.0')

console.log(adapter.capabilities?.maxSpeakers) // 3
console.log(adapter.capabilities?.timestamps) // true

Timings: Where Each Word and Turn Lands

result.duration is the length of the file. It does not tell you where speech stops, and it does not tell you which turn is where. Trimming trailing silence or captioning a clip needs both.

Set timestamps: true and the result carries the timings:

ts
import { generateSpeech } from '@tanstack/ai'
import { byteplusSpeech } from '@tanstack/ai-byteplus'

const result = await generateSpeech({
  adapter: byteplusSpeech('seed-audio-1.0'),
  text: 'Welcome to the guitar store.',
  timestamps: true,
})

// Where speech ends, which is earlier than the end of the file.
const speechEnds = result.alignment?.endSeconds.at(-1)

for (const segment of result.segments ?? []) {
  console.log(segment.startSeconds, segment.endSeconds, segment.text)
}

Two fields come back, and they answer different questions:

  • alignment: one entry per character or per word, with start and end times. alignment.unit says which granularity the provider reported.
  • segments: one entry per turn (dialogue) or per sentence (single voice). A dialogue segment also carries turnIndex and voice.

All times are seconds. Providers that report milliseconds convert in the adapter, so you never mix units.

timestamps is rejected on an adapter that cannot return timings, rather than silently giving you a result with nothing in it. Check adapter.capabilities?.timestamps before you ask.

Playing Audio in the Browser

ts
// Convert base64 to audio and play
function playAudio(result: TTSResult) {
  const audioData = atob(result.audio)
  const bytes = new Uint8Array(audioData.length)
  for (let i = 0; i < audioData.length; i++) {
    bytes[i] = audioData.charCodeAt(i)
  }
  
  const blob = new Blob([bytes], { type: result.contentType })
  const url = URL.createObjectURL(blob)
  
  const audio = new Audio(url)
  audio.play()
  
  // Clean up when done
  audio.onended = () => URL.revokeObjectURL(url)
}

Saving Audio to File (Node.js)

ts
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'
import { writeFile } from 'fs/promises'

async function saveAudio(result: TTSResult, filename: string) {
  const audioBuffer = Buffer.from(result.audio, 'base64')
  await writeFile(filename, audioBuffer)
  console.log(`Saved to ${filename}`)
}

// Usage
const result = await generateSpeech({
  adapter: openaiSpeech('tts-1'),
  text: 'Hello world!',
})

await saveAudio(result, 'output.mp3')

Full-Stack Usage

TanStack AI provides React hooks and server-side streaming helpers to build full-stack text-to-speech with minimal boilerplate.

Streaming Mode (Server Route + Client Hook)

Server, Create an API route that wraps generateSpeech as a streaming response:

ts
// routes/api/generate/speech.ts
import { generateSpeech, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'
import { createFileRoute } from '@tanstack/react-router'

export const Route = createFileRoute('/api/generate/speech')({
  server: {
    handlers: {
      POST: async ({ request }) => {
        const body = await request.json()
        const { text, voice, format, model } = body.data

        const stream = generateSpeech({
          adapter: openaiSpeech(model ?? 'tts-1'),
          text,
          voice,
          format,
          stream: true,
        })

        return toServerSentEventsResponse(stream)
      },
    },
  },
})

Client, Use the useGenerateSpeech hook with a connection adapter:

tsx
import { useGenerateSpeech, fetchServerSentEvents } from '@tanstack/ai-react'

function SpeechGenerator() {
  const { generate, result, isLoading, error } = useGenerateSpeech({
    connection: fetchServerSentEvents('/api/generate/speech'),
  })

  const playAudio = () => {
    if (!result) return
    const audioData = atob(result.audio)
    const bytes = new Uint8Array(audioData.length)
    for (let i = 0; i < audioData.length; i++) {
      bytes[i] = audioData.charCodeAt(i)
    }
    const blob = new Blob([bytes], { type: result.contentType })
    const url = URL.createObjectURL(blob)
    const audio = new Audio(url)
    audio.play()
    audio.onended = () => URL.revokeObjectURL(url)
  }

  return (
    <div>
      <button
        onClick={() => generate({ text: 'Hello, welcome to TanStack AI!' })}
        disabled={isLoading}
      >
        {isLoading ? 'Generating...' : 'Generate Speech'}
      </button>
      {error && <p>Error: {error.message}</p>}
      {result && <button onClick={playAudio}>Play Audio</button>}
    </div>
  )
}

The other two transports (a server function returning JSON, or one returning an SSE Response) work the same way here. They are in Advanced: other transports, and explained once in Generations.

Hook API

The useGenerateSpeech hook accepts:

OptionTypeDescription
connectionConnectionAdapterStreaming transport (SSE, HTTP stream, custom)
fetcher(input) => Promise<TTSResult | Response>Direct async function, or server function returning an SSE Response
onResult(result) => TOutput | null | voidCallback when audio is generated. Optionally return a transformed value (see Result Transform)
onError(error) => voidCallback on error
onProgress(progress, message?) => voidProgress updates (0-100)

And returns:

PropertyTypeDescription
generate(input: SpeechGenerateInput) => Promise<void>Trigger generation
resultTOutput | nullThe result (or transformed result), or null
isLoadingbooleanWhether generation is in progress
errorError | undefinedCurrent error, if any
statusGenerationClientState'idle' | 'generating' | 'success' | 'error'
stop() => voidAbort the current generation
reset() => voidClear result, error, and return to idle

Result Transform

The onResult callback can optionally return a transformed value that replaces the stored result. This is useful for converting raw API responses into a more convenient format for your components.

Transform behavior:

  • Return a non-null value to replace the stored result with the transformed value
  • Return null to keep the previous result unchanged (useful for filtering)
  • Return nothing (void) to store the raw result as-is (backward compatible)

Example: Convert base64 audio to a playable Audio element

tsx
import { useGenerateSpeech, fetchServerSentEvents } from '@tanstack/ai-react'
import type { TTSResult } from '@tanstack/ai'

function SpeechPlayer() {
  const { generate, result, isLoading } = useGenerateSpeech({
    connection: fetchServerSentEvents('/api/generate/speech'),
    onResult: (raw: TTSResult) => {
      const audioData = atob(raw.audio)
      const bytes = new Uint8Array(audioData.length)
      for (let i = 0; i < audioData.length; i++) {
        bytes[i] = audioData.charCodeAt(i)
      }
      const blob = new Blob([bytes], { type: raw.contentType })
      const url = URL.createObjectURL(blob)
      return {
        audio: new Audio(url),
        duration: raw.duration,
      }
    },
  })

  return (
    <div>
      <button
        onClick={() => generate({ text: 'Hello world!', voice: 'alloy' })}
        disabled={isLoading}
      >
        Generate
      </button>
      {result && (
        <button onClick={() => result.audio.play()}>
          Play Audio
        </button>
      )}
    </div>
  )
}

TypeScript automatically infers the result type from your onResult return value, no explicit generic parameter needed. In this example, result is inferred as { audio: HTMLAudioElement; duration?: number } | null, so result.audio.play() is fully type-safe.

Advanced

Reference detail you do not need to get this working.

Other transports

Direct Mode (Server Function + Fetcher)

For non-streaming usage with TanStack Start server functions:

ts
// lib/server-functions.ts
import { createServerFn } from '@tanstack/react-start'
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

export const generateSpeechFn = createServerFn({ method: 'POST' })
  .inputValidator((data: { text: string; voice?: string }) => data)
  .handler(async ({ data }) => {
    return generateSpeech({
      adapter: openaiSpeech('tts-1'),
      text: data.text,
      voice: data.voice,
    })
  })
tsx
import { useGenerateSpeech } from '@tanstack/ai-react'
import { generateSpeechFn } from '../lib/server-functions'

function SpeechGenerator() {
  const { generate, result, isLoading } = useGenerateSpeech({
    fetcher: (input) => generateSpeechFn({ data: input }),
  })
  // ... same UI as above
}

Server Function Streaming (Fetcher + Response)

For TanStack Start server functions that stream results. The fetcher receives type-safe input and returns an SSE Response, the client parses it automatically:

ts
// lib/server-functions.ts
import { createServerFn } from '@tanstack/react-start'
import { generateSpeech, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

export const generateSpeechStreamFn = createServerFn({ method: 'POST' })
  .inputValidator((data: { text: string; voice?: string }) => data)
  .handler(({ data }) => {
    return toServerSentEventsResponse(
      generateSpeech({
        adapter: openaiSpeech('tts-1'),
        text: data.text,
        voice: data.voice,
        stream: true,
      }),
    )
  })
tsx
import { useGenerateSpeech } from '@tanstack/ai-react'
import { generateSpeechStreamFn } from '../lib/server-functions'

function SpeechGenerator() {
  const { generate, result, isLoading } = useGenerateSpeech({
    fetcher: (input) => generateSpeechStreamFn({ data: input }),
  })
  // ... same UI as above
}

Model Options

OpenAI Model Options

ts
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

const result = await generateSpeech({
  adapter: openaiSpeech('tts-1-hd'),
  text: 'High quality speech synthesis',
  voice: 'nova',
  format: 'mp3',
  speed: 1.0, // top-level option, 0.25 to 4.0
  modelOptions: {
    instructions: 'Speak in a calm, measured tone', // GPT-4o audio models only
  },
})

Note: voice, format, and speed are top-level generateSpeech options, not modelOptions keys.

OptionTypeDescription
instructionsstringVoice style instructions (GPT-4o audio models only)

Note: The instructions and stream_format options are only available with the gpt-4o-audio-preview model, not with tts-1 or tts-1-hd.

Response Format

The TTS result includes:

ts
interface TTSResult {
  id: string        // Unique identifier for this generation
  model: string     // The model used
  audio: string     // Base64-encoded audio data
  format: string    // Audio format (e.g., "mp3")
  contentType: string // MIME type (e.g., "audio/mpeg")
  duration?: number // Duration in seconds (if available)
}

Model Availability

OpenAI Models

ModelQualitySpeedUse Case
tts-1StandardFastReal-time applications
tts-1-hdHighSlowerProduction audio
gpt-4o-audio-previewHighestVariableAdvanced voice control

Gemini Models

ModelStatusNotes
gemini-2.5-flash-preview-ttsExperimentalMay require Live API for full features

Error Handling

ts
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

try {
  const result = await generateSpeech({
    adapter: openaiSpeech('tts-1'),
    text: 'Hello!',
  })
} catch (error) {
  if (error instanceof Error) {
    if (error.message.includes('exceeds maximum length')) {
      console.error('Text is too long (max 4096 characters)')
    } else if (error.message.includes('Speed must be between')) {
      console.error('Invalid speed value')
    } else {
      console.error('TTS error:', error.message)
    }
  }
}

Tip: To trigger speech generation from your frontend with loading states, see Generation Hooks.

Debugging: When a TTS request fails or produces unexpected output, pass debug: true on generateSpeech({...}) to log the outgoing request, every raw provider chunk, and any caught error. See Debug Logging.

Environment Variables

The TTS adapters use the same environment variables as other adapters:

  • OpenAI: OPENAI_API_KEY
  • Gemini: GOOGLE_API_KEY or GEMINI_API_KEY
  • BytePlus: BYTEPLUS_VOICE_API_KEY (Seed Speech is a different product from ModelArk — the ARK_API_KEY used by the chat/image/video adapters is not accepted here)

Explicit API Keys

For production use or when you need explicit control:

ts
import { createOpenaiSpeech } from '@tanstack/ai-openai'
import { createGeminiSpeech } from '@tanstack/ai-gemini'

// OpenAI
const openaiAdapter = createOpenaiSpeech('tts-1', 'your-openai-api-key')

// Gemini
const geminiAdapter = createGeminiSpeech('gemini-3.1-flash-tts-preview', 'your-google-api-key')

Best Practices

  1. Text Length: OpenAI TTS supports up to 4096 characters per request. For longer content, split into chunks.

  2. Voice Selection: Choose voices appropriate for your content—use onyx for authoritative content, nova for friendly interactions.

  3. Format Selection: Use mp3 for general use, opus for streaming, wav for further processing.

  4. Caching: Cache generated audio to avoid regenerating the same content.

  5. Error Handling: Always handle errors gracefully, especially for user-facing applications.

Edit on GitHub
Realtime Voice Chat
Transcription