Voice Module

Add real-time AI voice calls to your app. Talk to an AI agent, get live transcription, and control the microphone — all through WebRTC.

Overview

XpectrumVoice lets users have a real-time voice conversation with an AI agent. Under the hood it uses LiveKit (WebRTC) for low-latency audio streaming.

ℹ How it works

Your app calls voice.connect()
SDK requests a LiveKit token from your voice server
SDK connects to a LiveKit room via WebRTC
User's microphone is enabled (browser permission prompt)
Agent audio plays automatically, transcriptions stream in real-time
On disconnect, SDK notifies the server to end the call

1. Setup

typescript

import { XpectrumVoice } from '@xpectrum/sdk';

const voice = new XpectrumVoice({
  baseUrl: 'https://api.xpectrum.co/',   // Your voice server URL
  apiKey: 'xpectrum_ai_sk_...',                    // API key (x-api-key header)
  agentName: '6b00c03f-f848-47ce-...',             // Agent UUID to connect to
});

Property	Type	Required	Default	Description
baseUrl	string	Yes	—	Your FastAPI voice server URL.
apiKey	string	Yes	—	API key for voice auth. Sent as x-api-key header.
agentName	string	Yes	—	Agent UUID or name to connect to for voice calls.

2. Start a Voice Call

Call voice.connect() with callbacks to handle events:

typescript

await voice.connect({
  // Called when connection is established
  onConnected: (roomName) => {
    console.log('Call started! Room:', roomName);
    showCallUI();
  },

  // Called for every transcription update
  onTranscription: (segment) => {
    // segment.id      — unique ID (interim results share same ID)
    // segment.text    — transcribed text
    // segment.isFinal — true when transcription is finalized
    // segment.speaker — 'user' or 'agent'
    console.log(`${segment.speaker}: ${segment.text}`);
  },

  // Called when agent starts/stops speaking
  onAgentSpeaking: (isSpeaking) => {
    updateStatus(isSpeaking ? 'Agent speaking...' : 'Listening...');
  },

  // Called when call ends
  onDisconnected: (reason) => {
    console.log('Call ended:', reason);
    hideCallUI();
  },

  // Called when connection state changes
  onConnectionStateChanged: (state) => {
    // 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed'
    console.log('State:', state);
  },

  // Network reconnection events
  onReconnecting: () => showStatus('Reconnecting...'),
  onReconnected: () => showStatus('Reconnected!'),

  // Error handler
  onError: (err) => {
    console.error('Voice error:', err.message);
  },
});

⚠ Browser permission

The browser will prompt the user for microphone permission when connect() is called. This must be triggered by a user interaction (e.g. button click) — it can't happen automatically on page load.

3. Understanding Transcription

The onTranscription callback receives a TranscriptionSegment:

Property	Type	Required	Default	Description
id	string	Yes	—	Segment ID. Interim results update the same ID, so you can replace in-place.
text	string	Yes	—	The transcribed text.
isFinal	boolean	Yes	—	true when the transcription is finalized and won't change.
speaker	'user' \| 'agent'	Yes	—	Who said this — the user or the AI agent.

How interim results work

text

onTranscription: { id: "seg-1", text: "What",           isFinal: false, speaker: "user" }
onTranscription: { id: "seg-1", text: "What are your",   isFinal: false, speaker: "user" }
onTranscription: { id: "seg-1", text: "What are your hours?", isFinal: true, speaker: "user" }

Notice the same id is reused for interim updates. When building your UI, find the existing segment by ID and replace it — don't append duplicates.

React pattern for transcription

typescript

const [transcripts, setTranscripts] = useState<TranscriptionSegment[]>([]);

// In your connect callbacks:
onTranscription: (segment) => {
  setTranscripts(prev => {
    const idx = prev.findIndex(t => t.id === segment.id);
    if (idx >= 0) {
      // Update existing segment (interim → final)
      const updated = [...prev];
      updated[idx] = segment;
      return updated;
    }
    // New segment
    return [...prev, segment];
  });
},

4. Microphone Control

typescript

// Mute the mic
await voice.setMicEnabled(false);

// Unmute the mic
await voice.setMicEnabled(true);

// Check current state
const isMuted = !voice.isMicEnabled();

5. End a Call

typescript

await voice.disconnect();
// This:
//  1. Disconnects from the LiveKit room
//  2. Stops the microphone
//  3. Cleans up audio elements
//  4. Notifies the server via POST /call-control/end-call

6. Connection State

typescript

// Get current state
const state = voice.getConnectionState();
// 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed'

// Quick check
if (voice.isConnected()) {
  console.log('Currently in a call');
}

// Get room name
const roomName = voice.getRoomName(); // string | null

State	Meaning
disconnected	No active call
connecting	Requesting token and joining LiveKit room
connected	In an active voice call
reconnecting	Temporary network issue, trying to reconnect
failed	Connection failed permanently

7. Event Emitter (Alternative to Callbacks)

Besides passing callbacks to connect(), you can subscribe to events directly:

typescript

// Subscribe — returns an unsubscribe function
const unsub = voice.on('transcription', (segment) => {
  console.log(`${segment.speaker}: ${segment.text}`);
});

// Later: unsubscribe
unsub();

// Or use off()
const handler = (segment) => { ... };
voice.on('transcription', handler);
voice.off('transcription', handler);

// Remove all listeners
voice.removeAllListeners();

All events

Event	Payload	Description
connected	{ roomName: string }	Call connected
disconnected	{ reason: string }	Call ended
transcription	TranscriptionSegment	Speech transcribed
agentSpeaking	{ isSpeaking: boolean }	Agent started/stopped speaking
connectionStateChanged	{ state: VoiceConnectionState }	State transition
reconnecting	{}	Network reconnecting
reconnected	{}	Network reconnected
error	{ message: string; code?: string }	Error occurred
microphoneChanged	{ enabled: boolean }	Mic toggled

8. Cleanup

typescript

// Disconnect + remove all event listeners
voice.destroy();

// In React / Next.js:
useEffect(() => {
  const voice = new XpectrumVoice({ ... });
  voiceRef.current = voice;
  return () => voice.destroy();
}, []);

Full Working Example

VoiceCall.tsx

'use client';

import { useEffect, useRef, useState } from 'react';
import { XpectrumVoice } from '@xpectrum/sdk';
import type { TranscriptionSegment } from '@xpectrum/sdk';

export default function VoiceCall() {
  const voiceRef = useRef<XpectrumVoice | null>(null);
  const [state, setState] = useState<string>('disconnected');
  const [micOn, setMicOn] = useState(true);
  const [transcripts, setTranscripts] = useState<TranscriptionSegment[]>([]);

  useEffect(() => {
    voiceRef.current = new XpectrumVoice({
      baseUrl: process.env.NEXT_PUBLIC_VOICE_BASE_URL!,
      apiKey: process.env.NEXT_PUBLIC_VOICE_API_KEY!,
      agentName: process.env.NEXT_PUBLIC_VOICE_AGENT_NAME!,
    });
    return () => voiceRef.current?.destroy();
  }, []);

  const startCall = async () => {
    setTranscripts([]);
    await voiceRef.current?.connect({
      onConnected: () => setState('connected'),
      onDisconnected: () => setState('disconnected'),
      onConnectionStateChanged: (s) => setState(s),
      onTranscription: (seg) => {
        setTranscripts(prev => {
          const idx = prev.findIndex(t => t.id === seg.id);
          if (idx >= 0) {
            const updated = [...prev];
            updated[idx] = seg;
            return updated;
          }
          return [...prev, seg];
        });
      },
      onError: (err) => console.error(err.message),
    });
  };

  const endCall = async () => {
    await voiceRef.current?.disconnect();
  };

  const toggleMic = async () => {
    const next = !micOn;
    await voiceRef.current?.setMicEnabled(next);
    setMicOn(next);
  };

  const isConnected = state === 'connected';

  return (
    <div style={{ maxWidth: 500, margin: '0 auto', padding: 20 }}>
      <h1>Voice Call</h1>
      <p>Status: <strong>{state}</strong></p>

      <div style={{ display: 'flex', gap: 8, marginBottom: 20 }}>
        {!isConnected ? (
          <button onClick={startCall}>Start Call</button>
        ) : (
          <>
            <button onClick={endCall}>End Call</button>
            <button onClick={toggleMic}>
              {micOn ? 'Mute' : 'Unmute'}
            </button>
          </>
        )}
      </div>

      <div>
        {transcripts.map((t) => (
          <p key={t.id} style={{ opacity: t.isFinal ? 1 : 0.6 }}>
            <strong>{t.speaker}:</strong> {t.text}
          </p>
        ))}
      </div>
    </div>
  );
}