When developing voice control for IoT devices in a mobile app, we often see clients limiting themselves to built-in assistants. But that's just the tip of the iceberg. A full solution includes speech recognition, intent extraction (NLU), mapping to device commands, and feedback—each stage can break without proper architecture. Order voice control development for your IoT project—we will audit and propose an architecture within 1 day.
How voice control for IoT works on mobile devices
Typical solution architecture: user speaks a command → microphone → speech recognition engine (local or cloud) → NLU (intent and entity extraction) → mapping to device commands → send via MQTT/HTTP/BLE to device → feedback via TTS. Each stage can be implemented differently, and the choice determines latency, autonomy, and accuracy.
Two fundamentally different approaches
Built-in voice assistants (Siri Shortcuts, Google Assistant Actions) work via the cloud and require explicit user permission. Siri Shortcuts on iOS are available via INPlayMediaIntent and INSendMessageIntent, but for arbitrary IoT commands you need AppIntent (iOS 16+)—a Swift framework for describing intents. Example: "Hey Siri, turn off the kitchen light" → Siri calls TurnOffLightIntent in your app, which sends an MQTT command. Latency is 2–4 seconds via Apple's cloud, no guarantees when offline.
Local recognition is another level. On iOS it's SFSpeechRecognizer with SFSpeechAudioBufferRecognitionRequest. Since iOS 13 it supports on-device mode (requiresOnDeviceRecognition = true) without sending audio to the cloud. On Android—SpeechRecognizer API (via Google cloud) or Vosk / Whisper.cpp for fully offline recognition.
For IoT apps where local network operation without internet is important, the choice is clear—local recognition plus offline NLU.
Why local recognition is more effective than cloud
Local processing offers three key advantages:
- Latency: 300–800 ms vs 1.5–3 seconds for cloud solutions.
- Offline operation: full autonomy when internet is disconnected.
- Privacy: audio data never leaves the device.
Compare both approaches:
| Parameter | Built-in assistants (cloud) | Local on-device recognition |
|---|---|---|
| Latency from tap to response | 2–4 s | 0.3–0.8 s |
| Works without internet | No | Yes |
| Accuracy on Russian | Good (Google) / medium (Apple) | 94% after training (fastText) |
| Integration complexity | Low (via SDK) | Medium (models, training) |
| Ownership cost | Pay per request | One-time development cost |
As can be seen, local recognition is 3–5 times faster and saves up to 30% on cloud services for large command volumes. Get a consultation for your project—we will assess possibilities and timelines in 1 day.
NLU: from text to device command
Recognized "turn on the kitchen light and raise the temperature to twenty-two"—now we need to extract:
- intent:
turn_on,set_temperature - entities:
device_type=light,location=kitchen,device_type=thermostat,value=22
For simple cases, a rule-based approach suffices: a dictionary of verb-intents + a dictionary of devices and rooms from the user's database. We build regexes or a simple intent matcher on the same device list already in the system.
For complex scenarios—Rasa NLU (self-hosted) or Duckling for numeric values. On Flutter we integrate via HTTP requests to a local server on the home network or via dart:ffi for an embedded model.
Real example: a smart apartment project with 35 devices, Russian language. We trained a simple model on fastText with ~500 command examples, converted to .tflite, ran via tflite_flutter. Accuracy on household commands—94% (from internal testing). Misses were on compound commands (two actions in one phrase)—solved via preprocessing by splitting on conjunctions "and", "then", "after that".
What stages does voice interface development include?
The process consists of six steps:
- Analysis—determine the list of devices, commands, languages, offline requirements.
- Architecture selection—cloud vs local, NLU engine choice.
- Design—command mapping, error handling, dialog scenario.
- Implementation—code, MQTT integration, model training (if needed).
- Testing—verify on real devices, stress-test for noise and accents.
- Deployment—publish to App Store / Google Play, set up CI/CD.
For comparison, here are NLU engines:
| NLU Engine | Type | Offline | Accuracy (Russian) | Complexity |
|---|---|---|---|---|
| Rule-based | Custom code | Yes | 70–80% | Low |
| Rasa NLU | Self-hosted | Yes | 85–90% | Medium |
| fastText + tflite | In-app model | Yes | 90–95% | High |
| Duckling | Numeric entities | Yes | >95% | Low |
Detailed description of testing stages
Testing includes verification on real devices, stress tests for noise and accents, and evaluation of wake word performance at 60 dB noise level.Feedback and edge cases
Push to talk vs always-on. Always-on on mobile is a battery killer. We recommend a push-to-talk button in the app plus optional wake word via Porcupine SDK (PicoVoice). Porcupine runs locally, consumes <5% CPU on idle.
What if the device is not recognized?
Don't stay silent. Return a voice response via AVSpeechSynthesizer (iOS) / TextToSpeech (Android), list what was understood, ask for clarification. The user doesn't see the screen—they need audio feedback.
On Flutter we use flutter_tts for synthesis and speech_to_text as a unified API over platform engines. Important: on Android 11+ SpeechRecognizer requires RECORD_AUDIO permission with explicit explanation in onRequestPermissionsResult. Without a clear rationale, Google Play Console flags it as a policy violation.
MQTT integration
Voice command → NLU → device command → publish to MQTT topic. Latency from button press to device response: recognition on device ~300–800ms, NLU ~50ms, MQTT publish <50ms with local broker. Total—feels instant.
With cloud recognition add 1.5–3 seconds. On Russian, cloud Google Speech-to-Text works well; Apple Speech is worse on specific IoT terms like "dimmer", "receiver", "relay".
Example MQTT publish in Swift:
let client = CocoaMQTT(clientID: "iPhone", host: "192.168.1.100", port: 1883) client.connect() client.publish("home/kitchen/light", withString: "on", qos: .qos1) What's included
- Development of recognition module (iOS/Android/Flutter) with chosen approach.
- NLU model training for your commands and devices (up to 500+ examples).
- Integration with MQTT broker and existing IoT infrastructure.
- Wake word setup (optional) and TTS feedback.
- Architecture documentation and instructions for adding new commands.
- Support for 30 days after deployment.
Get a consultation for your project—we will assess possibilities and timelines in 1 day.
Timelines
Push-to-talk with cloud recognition and simple command mapping—2–3 weeks. Offline recognition + NLU + wake word + TTS feedback—6–10 weeks. Cost depends on number of languages, platforms, and offline requirements. Contact us to evaluate your project—we will prepare a proposal in 1 day. Our team has 5+ years of experience in mobile IoT app development and has completed over 30 projects with voice control.
Apple Developer Documentation, Google Speech API