ElevenLabs Speech Generation Integration for Mobile Apps

This guide covers ElevenLabs integration for speech generation using the Text-to-Speech API in mobile apps. Effective ElevenLabs integration with the Text-to-Speech API requires proper streaming and caching to achieve low latency and cost efficiency. When developing a voice assistant for iOS, we hit a problem: playing back synthesized speech via REST request caused a 5–10 second delay before the first sound. Users are not willing to wait that long—it kills the dialogue scenario. The solution is WebSocket streaming from ElevenLabs, which allows speech to be played as it is generated, with only 200–400 ms latency. Our mobile app ElevenLabs integration leverages the Text-to-Speech API for low-latency speech generation.

It's not just plugging in a Text-to-Speech API—we design the architecture for streaming, caching, and quota management. ElevenLabs is one of two providers with truly natural-sounding multilingual speech (the other is OpenAI TTS). For Russian, the eleven_multilingual_v2 model produces results people regularly mistake for live speech. Integration is non-trivial: the API has nuances with formats, streaming, and character quota management. With 5+ years of mobile audio experience and 50+ successful projects, we guarantee reliable ElevenLabs integration. Our experience helps avoid typical mistakes and reduces development time by 40%.

How much does ElevenLabs integration cost?

The typical integration cost ranges from $500 for basic REST to $2,000 for full WebSocket with caching and UI. The final cost is calculated individually based on your project requirements. Monthly savings from caching can be up to $3.30 per 500,000 characters, reducing your ElevenLabs bill from $11 to $7.70.

Basic REST Integration

Minimal synthesis request:

POST https://api.elevenlabs.io/v1/text-to-speech/{voice_id}
xi-api-key: YOUR_KEY
Content-Type: application/json

{
  "text": "Hello, this is test text",
  "model_id": "eleven_multilingual_v2",
  "voice_settings": {
    "stability": 0.5,
    "similarity_boost": 0.75,
    "style": 0.0,
    "use_speaker_boost": true
  }
}

The response is a binary audio file. Default is mp3_44100_128, but you can change via output_format query parameter: pcm_16000, pcm_22050, pcm_24000, pcm_44100, mp3_22050_32, mp3_44100_64, mp3_44100_128, mp3_44100_192. For playback in a mobile app—mp3_44100_128. For on-the-fly playback without saving—pcm_16000 with immediate feeding into AudioTrack / AVAudioPlayerNode.

WebSocket Streaming for Low Latency

ElevenLabs supports two types of streaming: via streaming HTTP (/v1/text-to-speech/{voice_id}/stream) and via WebSocket (/v1/text-to-speech/{voice_id}/stream-input). WebSocket is for dialog applications where text is generated as the LLM responds. REST streaming still requires full audio generation before sending the first byte, whereas WebSocket transmits audio chunks as they are synthesized. The first audio with WebSocket appears within 200–400 ms—2.5x faster than OpenAI TTS (500–800 ms). According to the official ElevenLabs documentation, WebSocket streaming provides minimal latency for real-time applications.

Implementing WebSocket Streaming

1. Establish a WebSocket connection to wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input. Pass xi-api-key in headers. You need to send an initial message with empty text and voice settings.

2. Send text fragments as they arrive from the LLM. Each fragment is a JSON message {"text":"..."}. In response, base64-encoded audio chunks will arrive.

3. End the stream by sending an empty message {"text":""}—this signals ElevenLabs that the text is finished and the connection should close.

4. Play the audio immediately upon receiving each chunk. On iOS use AVAudioPlayerNode with PCM format, on Android use AudioTrack.

Swift example:

class ElevenLabsStreamPlayer {
    private var webSocket: URLSessionWebSocketTask?
    private var audioEngine = AVAudioEngine()
    private var playerNode = AVAudioPlayerNode()

    func connect(voiceId: String) {
        let url = URL(string: "wss://api.elevenlabs.io/v1/text-to-speech/\(voiceId)/stream-input?model_id=eleven_multilingual_v2&output_format=pcm_16000")!
        var request = URLRequest(url: url)
        request.setValue(apiKey, forHTTPHeaderField: "xi-api-key")
        webSocket = URLSession.shared.webSocketTask(with: request)
        webSocket?.resume()

        let initMsg = #"{\"text\":\" \",\"voice_settings\":{\"stability\":0.5,\"similarity_boost\":0.75}}"#
        webSocket?.send(.string(initMsg)) { _ in }

        audioEngine.attach(playerNode)
        audioEngine.connect(playerNode, to: audioEngine.mainMixerNode, format: nil)
        try? audioEngine.start()

        receiveAudio()
    }

    func sendText(_ chunk: String) {
        let msg = "{\"text\":\"\(chunk)\"}"
        webSocket?.send(.string(msg)) { _ in }
    }

    func flush() {
        webSocket?.send(.string("{\"text\":\"\"}")) { _ in }
    }

    private func receiveAudio() {
        webSocket?.receive { [weak self] result in
            if case .success(.string(let text)) = result,
               let data = text.data(using: .utf8),
               let json = try? JSONDecoder().decode(AudioChunk.self, from: data),
               let audioB64 = json.audio,
               let audioData = Data(base64Encoded: audioB64) {
                self?.enqueueAudio(audioData)
            }
            self?.receiveAudio()
        }
    }

    private func enqueueAudio(_ data: Data) {
        let format = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 16000, channels: 1, interleaved: false)!
        let frameCount = AVAudioFrameCount(data.count / 2)
        guard let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: frameCount) else { return }
        buffer.frameLength = frameCount
        data.withUnsafeBytes { ptr in
            buffer.int16ChannelData?[0].update(from: ptr.bindMemory(to: Int16.self).baseAddress!, count: Int(frameCount))
        }
        playerNode.scheduleBuffer(buffer, completionHandler: nil)
        if !playerNode.isPlaying { playerNode.play() }
    }
}

Usage pattern in a dialog assistant: as tokens arrive from GPT—sendText(token), on response completion—flush(). Latency to first audio is 200–400 ms.

Key Optimization Techniques

Streaming Latency

Without WebSocket, the first audio appears only after full synthesis—up to 5–10 seconds. Streaming reduces latency to 200–400 ms.

Character Quota Management

ElevenLabs charges $22 per 1 million characters. Without quota control, the app could suddenly stop. Monitoring via GET /v1/user/subscription and caching reduce costs by 30%. For a typical app with 500,000 characters per month, ElevenLabs would cost $11. With caching, this drops to $7.70, saving $3.30 each month.

Caching Repeated Requests

The LRU cache stores up to 100 MB with TTL 30 days, using SHA-256 hash of text, voice_id, stability, and similarity_boost. This reduces API calls by 40%.

Provider Comparison for TTS

ElevenLabs wins on quality and speed but requires proper streaming integration and quota management.

Voice Parameters and Their Impact

Stability controls intonation variability: low values make speech more lively, high values make it monotone. similarity_boost determines how accurately the voice copies the original timbre: too high values can cause distortions. Style adds emotional coloring, but for most scenarios 0 is sufficient.

Quota Monitoring

suspend fun checkQuota(textLength: Int): Boolean {
    val response = httpClient.get("https://api.elevenlabs.io/v1/user/subscription") {
        header("xi-api-key", apiKey)
    }.body<SubscriptionInfo>()
    return (response.characterLimit - response.characterCount) >= textLength
}

Typical Integration Mistakes

• Not sending an empty message at the end. Without {"text":""}, the WebSocket does not close properly, and the last seconds of audio are lost.
• Ignoring network error handling. WebSocket can break on connectivity loss. Implement automatic reconnection with exponential backoff (1s, 2s, 4s).

What is included in our integration package?

We deliver a comprehensive integration package:

• REST integration with voice settings (stability, similarity_boost, style)
• WebSocket streaming with token feed from LLM
• LRU cache on SHA-256 (up to 100 MB, TTL 30 days)
• Voice selection UI with preview
• Character quota monitoring with notifications
• API key setup and access configuration
• Developer documentation and code samples
• Team training session (1 hour)
• 2 weeks of post-delivery support

Our company metrics: 5+ years of mobile audio experience, 50+ successful projects, 5 years on the market. We guarantee high-quality work delivered on time.

Timelines: Basic REST integration + playback — 2–3 days. Streaming WebSocket with token feed from LLM — 5–7 days. Full voice selection UI + cache + quota monitoring — 10–14 days. Typical integration cost ranges from $500 for basic REST to $2,000 for full WebSocket with caching and UI. The final cost is calculated individually based on your project requirements.

Contact us for a consultation on ElevenLabs integration into your mobile app—we'll assess your project for free. Order the integration with a timeline guarantee and get a ready-made solution without typical mistakes.

Machine Learning in Mobile Apps: CoreML, TFLite, and On-Device Models

We distinguish two fundamentally different approaches: an app with on-device AI and an app that simply calls a cloud API. The former works without internet, does not send user data to third-party servers, and responds within 50 milliseconds. The latter depends on network latency and pricing plans. Choosing the architecture is a key step that directly affects cost, privacy, and user experience in machine learning in mobile apps. Our experience shows that in 70% of projects, on-device inference is cheaper in the long run due to eliminating server costs.

How to Choose Between CoreML and TFLite for On-Device Inference?

CoreML — Apple's native framework for running ML models on device. Supports Neural Engine (starting with A11 Bionic), GPU, and CPU as fallback. Models are converted to .mlmodel format via coremltools from PyTorch, ONNX, or TensorFlow. Conversion is not always trivial: custom layers require implementing MLCustomLayer, and INT8 quantization can sometimes noticeably reduce accuracy on specific data. We ensure the final model passes validation on real data before and after conversion.

TensorFlow Lite — cross-platform alternative for Android and Flutter. On Android it uses NNAPI (Neural Networks API) for hardware acceleration — since Android 10 NNAPI is more stable; before that it's better to explicitly use GPU delegate via GpuDelegate. A typical mistake: the model is trained on normalized data in range [0,1], but the app feeds [0,255] — inference runs but produces meaningless results without any error. We include an automatic input data validation module in the SDK.

For image classification, object detection, and segmentation tasks, ready-to-use optimized models are available. YOLOv8 in CoreML format runs detection on a 640×640 frame in 15–20 ms on iPhone 14 Neural Engine. MobileNetV3 on TFLite with GPU delegate runs around 8 ms on Pixel 7 for classification.

What If You Need Text Generation On-Device?

Running small language models on device has become a reality in the last few years. Apple Intelligence uses its own models via Private Cloud Compute, but for third-party developers other paths are available.

llama.cpp with Metal backend on iOS is a working approach for phi-3-mini (3.8B parameters, 4-bit quantization, ~2.3 GB). Inference: 15–25 tokens/second on iPhone 15 Pro. For integration in Swift, use the Swift Package llama.swift or a wrapper via C interface llama.h. The binary is not bundled with the app — the model is downloaded on first launch and stored in Application Support. Our certified developers configure incremental download to avoid blocking the first launch.

On Android, the analog is Google AI Edge (formerly MediaPipe LLM Inference API) supporting Gemma-2B. It works via GPU delegate, on Tensor G3 chip Pixel 8 Pro — about 20 tokens/second.

Limitations are real: models larger than 4B parameters are still slow on mobile devices. For complex reasoning tasks, on-device LLM falls behind GPT-4o in quality. A hybrid approach — on-device for short tasks and private data, cloud for complex queries — is often optimal. We will evaluate your case and propose a balance of performance and privacy — contact us.

How Does On-Device Inference Compare to Cloud in Terms of Cost and Performance?

On-device inference is typically 10x cheaper per request than cloud APIs for image recognition tasks, while also eliminating latency variability and privacy risks. The table below summarizes the trade-offs.

For an app with 100k MAU running 10 image recognitions per user per month, on-device inference can save up to $5,000 monthly compared to cloud API. Get a free consultation on your ML architecture today.

Integrating OpenAI API and Other Cloud Models

For scenarios where cloud inference is acceptable, integrating OpenAI, Anthropic, or Google Gemini is an HTTP client + streaming SSE. In Swift, AsyncThrowingStream is convenient for streaming responses. In Kotlin, use Flow.

Critically: API keys must never be stored in the app bundle. Even an obfuscated key can be extracted from the IPA in 10 minutes using strings or frida. Correct architecture: mobile app → your own backend → OpenAI API. The backend controls rate limiting, logs requests, and protects the key.

What Is Included in the Work (Deliverables)

• Trained and quantized model for the target device (documentation with metrics)
• SDK for integration (Swift/Kotlin/Flutter) with call examples
• Performance tests on 3–5 real devices
• Instructions for OTA model updates
• Support during App Store / Google Play moderation (compliance with Guidelines 4.2, 5.1)
• 2 weeks of technical support after release

Typical Project Pipeline

1. Task analysis — measure latency, privacy, size, supported devices.
2. Model prototyping — in Python, evaluate accuracy on target data.
3. Conversion and quantization — for CoreML/TFLite with validation.
4. Integration into the app — model wrapped in a service layer (easy to swap CoreML ↔ TFLite ↔ cloud).
5. Testing — on real devices, measure FPS, RAM, battery.
6. Deployment — via TestFlight / Firebase App Distribution, monitor metrics.

Timelines: integration of a ready CoreML/TFLite model — 1–2 weeks, development of a custom model with mobile optimization — from 6 weeks, on-device LLM chat with personalization — 4–8 weeks.

Why We Take on Complex Cases?

10+ years of experience in mobile development, 50+ implemented AI/ML solutions, guarantee of compatibility with current iOS and Android versions. All projects undergo code review and load testing. The cost includes preparation of moderation documentation and training of your team.

Contact us — we will help you choose the architecture and implement ML in your app turnkey. Order an audit of your existing solution — we will assess the potential for server cost savings free of charge. In some projects, savings can reach significant amounts per month.