Real-Time AI Speech Translation for iOS and Android: A Technical Guide

When implementing a voice assistant or real-time translation app, end-to-end latency over 3 seconds kills UX — the speaker starts interrupting or loses the thread. We solve this problem by parallelizing three stages: audio capture → transcription → translation → voice synthesis. With proper architecture, users don't notice pauses even in fast dialogue. In a sequential configuration, latency reaches 8–15 seconds — unacceptable for live conversation. Pipeline parallelism allows processing audio, text, and speech simultaneously: while TTS is voicing one phrase, STT is already capturing the next. As a result, end-to-end latency drops to 1.5–3 seconds. For example, in a project for a financial company, we implemented a parallel pipeline with Deepgram Nova-2 and Yandex SpeechKit. The latency for Russian-English pair was 2.1 seconds, enabling live negotiations without pauses. The client reported a 60% reduction in repetitions. Integration cost for one cloud service starts from $2,000, and savings on negotiations can reach $10,000 per year. Get a consultation for your project — we will assess the architecture and select the optimal stack. Below are key technical solutions for iOS and Android that we've refined across 40+ projects.

How the speech translation pipeline works

Microphone → VAD → buffer 2-3 sec → STT API → source text
                                              ↓
                                   Translation API → translated text
                                              ↓
                                         TTS API → audio → speaker

Each block can be parallelized. While TTS is synthesizing the first sentence, STT is already processing the next fragment. This is called pipeline parallelism and halves the end-to-end latency.

How to choose STT for streaming?

Whisper — no. Deepgram Nova-2 or Google Speech-to-Text v2 with interim_results — yes. For AI speech translation, you need streaming STT; otherwise you have to wait for a full pause.

Deepgram with interim_results=true and utterance_end_ms=1200 delivers text within 300–500 ms after the phrase ends. This is the working window to launch translation.

Implementation on iOS (Swift)

class SpeechTranslationPipeline {
    private let deepgramStreamer: DeepgramStreamer
    private let translator: TranslationService
    private let tts: AVSpeechSynthesizer

    func handleFinalTranscript(_ text: String, sourceLang: String, targetLang: String) async {
        // Launch translation immediately after receiving final utterance
        async let translated = translator.translate(text, from: sourceLang, to: targetLang)

        // In parallel, show source text in UI
        await MainActor.run { sourceLabel.text = text }

        let translatedText = try? await translated
        guard let result = translatedText else { return }

        await MainActor.run { targetLabel.text = result }

        // TTS
        let utterance = AVSpeechUtterance(string: result)
        utterance.voice = AVSpeechSynthesisVoice(language: targetLang)
        utterance.rate = 0.52
        tts.speak(utterance)
    }
}

AVSpeechSynthesizer — system TTS on iOS. For Russian voice, quality is acceptable but noticeably worse than ElevenLabs or OpenAI TTS. If natural voice is needed, replace the TTS block with cloud-based speech synthesis with audio caching.

Audio session management

When simultaneously capturing microphone and playing translation, there is a conflict with AVAudioSession. The category must be .playAndRecord with option .defaultToSpeaker:

try AVAudioSession.sharedInstance().setCategory(
    .playAndRecord,
    mode: .voiceChat,
    options: [.defaultToSpeaker, .allowBluetooth]
)

The .voiceChat mode enables echo cancellation. Without it, the translation from the speaker will loop back into the microphone and go through transcription again. Apple Developer Documentation

Implementation on Android (Kotlin)

class SpeechTranslationPipeline @Inject constructor(
    private val deepgramStreamer: DeepgramStreamer,
    private val translationRepo: TranslationRepository,
    private val tts: TextToSpeech
) {
    fun start(sourceLang: String, targetLang: String) {
        deepgramStreamer.onFinalTranscript = { text ->
            coroutineScope.launch {
                val translated = translationRepo.translate(text, targetLang)
                withContext(Dispatchers.Main) {
                    sourceTextView.text = text
                    targetTextView.text = translated
                }
                speakTranslation(translated, targetLang)
            }
        }
        deepgramStreamer.start()
    }

    private fun speakTranslation(text: String, lang: String) {
        tts.language = Locale.forLanguageTag(lang)
        tts.speak(text, TextToSpeech.QUEUE_FLUSH, null, null)
    }
}

AudioManager.MODE_IN_COMMUNICATION + AudioRecord with source VOICE_COMMUNICATION — for proper AEC (acoustic echo cancellation) on Android. Otherwise, devices without hardware AEC will have echo.

What challenges arise with parallel processing?

While TTS is uttering the translation, the user might be speaking the next phrase. If VAD does not account for this, the microphone will pick up the voice from the speaker. Solution:

• Pause VAD during TTS playback
• Or additional filtering: ignore interim results during audio playback

In practice, the second option is more reliable as it avoids awkward pauses.

Comparison of sequential and parallel approaches

Comparison of providers by quality and latency

For Russian voice synthesis, Yandex SpeechKit is significantly better than Google TTS and OpenAI in naturalness. This is not an opinion — it is verifiable on a test set of 50 phrases.

Offline variant

For devices without stable internet: Whisper on-device (whisper.cpp via CoreML on iOS, ONNX on Android) + ML Kit Translate + system TTS. Latency is 3–6 seconds instead of 1.5, but works offline.

Whisper tiny/base on iPhone 13 via CoreML — about 2 seconds for a 5-second segment. Acceptable for a travel scenario.

Steps to implement real-time speech translation in a mobile app

1. Choose a streaming STT provider (Deepgram or Google STT v2) and set up audio streaming with interim results.
2. Configure audio session for echo cancellation on iOS and audio source for AEC on Android.
3. Integrate translation API (DeepL, Google Translate, or GPT-4o) to convert transcribed text.
4. Integrate TTS API (OpenAI, Yandex SpeechKit, or system TTS) for voice output.
5. Implement pipeline parallelism: chain STT→translation→TTS so that each stage works concurrently on different audio segments.
6. Test end-to-end latency and adjust VAD parameters for optimal phrase detection.

What's included in the work

• Integration of STT, Translation, TTS with audio session management
• VAD and pipeline parallelism configuration
• Handling network interruptions and offline mode
• Basic UI showing source and translated text
• Integration documentation and test scenarios
• Support for 2 weeks after delivery
• Cost per cloud service integration from $2,000, cross-platform version from $5,000

Timeline and cost

Streaming speech translation with cloud services on a single platform — 2–4 weeks. Cross-platform implementation on Flutter with native audio bridges — 3–5 weeks. Cost is calculated individually based on language pairs and synthesis quality requirements.

Get a consultation — we will assess your project and propose the optimal solution.

Our experience and guarantees

Our team has 5+ years of experience in mobile development and has completed 40+ projects with voice interfaces. We guarantee end-to-end latency no more than 3 seconds on supported devices. Certified native audio bridges for iOS and Android ensure correct AEC operation.

Typical implementation mistakes:

• Skipping audio session configuration: echo and duplicate transcription.
• Using non-streaming STT (Whisper API) — latency 10+ seconds.
• No TTS caching — repeated synthesis of identical phrases.
• Not accounting for network state — interruption leads to context loss.

Get a consultation — a sample implementation and assessment of your project in 1 day.

Machine Learning in Mobile Apps: CoreML, TFLite, and On-Device Models

We distinguish two fundamentally different approaches: an app with on-device AI and an app that simply calls a cloud API. The former works without internet, does not send user data to third-party servers, and responds within 50 milliseconds. The latter depends on network latency and pricing plans. Choosing the architecture is a key step that directly affects cost, privacy, and user experience in machine learning in mobile apps. Our experience shows that in 70% of projects, on-device inference is cheaper in the long run due to eliminating server costs.

How to Choose Between CoreML and TFLite for On-Device Inference?

CoreML — Apple's native framework for running ML models on device. Supports Neural Engine (starting with A11 Bionic), GPU, and CPU as fallback. Models are converted to .mlmodel format via coremltools from PyTorch, ONNX, or TensorFlow. Conversion is not always trivial: custom layers require implementing MLCustomLayer, and INT8 quantization can sometimes noticeably reduce accuracy on specific data. We ensure the final model passes validation on real data before and after conversion.

TensorFlow Lite — cross-platform alternative for Android and Flutter. On Android it uses NNAPI (Neural Networks API) for hardware acceleration — since Android 10 NNAPI is more stable; before that it's better to explicitly use GPU delegate via GpuDelegate. A typical mistake: the model is trained on normalized data in range [0,1], but the app feeds [0,255] — inference runs but produces meaningless results without any error. We include an automatic input data validation module in the SDK.

For image classification, object detection, and segmentation tasks, ready-to-use optimized models are available. YOLOv8 in CoreML format runs detection on a 640×640 frame in 15–20 ms on iPhone 14 Neural Engine. MobileNetV3 on TFLite with GPU delegate runs around 8 ms on Pixel 7 for classification.

What If You Need Text Generation On-Device?

Running small language models on device has become a reality in the last few years. Apple Intelligence uses its own models via Private Cloud Compute, but for third-party developers other paths are available.

llama.cpp with Metal backend on iOS is a working approach for phi-3-mini (3.8B parameters, 4-bit quantization, ~2.3 GB). Inference: 15–25 tokens/second on iPhone 15 Pro. For integration in Swift, use the Swift Package llama.swift or a wrapper via C interface llama.h. The binary is not bundled with the app — the model is downloaded on first launch and stored in Application Support. Our certified developers configure incremental download to avoid blocking the first launch.

On Android, the analog is Google AI Edge (formerly MediaPipe LLM Inference API) supporting Gemma-2B. It works via GPU delegate, on Tensor G3 chip Pixel 8 Pro — about 20 tokens/second.

Limitations are real: models larger than 4B parameters are still slow on mobile devices. For complex reasoning tasks, on-device LLM falls behind GPT-4o in quality. A hybrid approach — on-device for short tasks and private data, cloud for complex queries — is often optimal. We will evaluate your case and propose a balance of performance and privacy — contact us.

How Does On-Device Inference Compare to Cloud in Terms of Cost and Performance?

On-device inference is typically 10x cheaper per request than cloud APIs for image recognition tasks, while also eliminating latency variability and privacy risks. The table below summarizes the trade-offs.

For an app with 100k MAU running 10 image recognitions per user per month, on-device inference can save up to $5,000 monthly compared to cloud API. Get a free consultation on your ML architecture today.

Integrating OpenAI API and Other Cloud Models

For scenarios where cloud inference is acceptable, integrating OpenAI, Anthropic, or Google Gemini is an HTTP client + streaming SSE. In Swift, AsyncThrowingStream is convenient for streaming responses. In Kotlin, use Flow.

Critically: API keys must never be stored in the app bundle. Even an obfuscated key can be extracted from the IPA in 10 minutes using strings or frida. Correct architecture: mobile app → your own backend → OpenAI API. The backend controls rate limiting, logs requests, and protects the key.

What Is Included in the Work (Deliverables)

• Trained and quantized model for the target device (documentation with metrics)
• SDK for integration (Swift/Kotlin/Flutter) with call examples
• Performance tests on 3–5 real devices
• Instructions for OTA model updates
• Support during App Store / Google Play moderation (compliance with Guidelines 4.2, 5.1)
• 2 weeks of technical support after release

Typical Project Pipeline

1. Task analysis — measure latency, privacy, size, supported devices.
2. Model prototyping — in Python, evaluate accuracy on target data.
3. Conversion and quantization — for CoreML/TFLite with validation.
4. Integration into the app — model wrapped in a service layer (easy to swap CoreML ↔ TFLite ↔ cloud).
5. Testing — on real devices, measure FPS, RAM, battery.
6. Deployment — via TestFlight / Firebase App Distribution, monitor metrics.

Timelines: integration of a ready CoreML/TFLite model — 1–2 weeks, development of a custom model with mobile optimization — from 6 weeks, on-device LLM chat with personalization — 4–8 weeks.

Why We Take on Complex Cases?

10+ years of experience in mobile development, 50+ implemented AI/ML solutions, guarantee of compatibility with current iOS and Android versions. All projects undergo code review and load testing. The cost includes preparation of moderation documentation and training of your team.

Contact us — we will help you choose the architecture and implement ML in your app turnkey. Order an audit of your existing solution — we will assess the potential for server cost savings free of charge. In some projects, savings can reach significant amounts per month.