Building a Mobile Image Recognition Bot: A Practical Guide

User takes a photo — bot answers. Sounds simple, but between "attach photo" and "get useful answer" lies choosing a model, managing request size, and handling cases where the image does not contain what is expected. Our experience shows that proper Vision API selection and data flow optimization cut processing cost by 2–3 times, and on-device recognition reduces cost to zero per request. We offer turnkey bot development tailored to your scenarios — from retail to medical consultations. We'll explain which technologies to use and how to avoid common pitfalls.

The first step is to define the task: free-form photo description, document OCR, product recognition, or content moderation. This determines the model selection and server architecture. We use GPT-4o Vision (OpenAI), ML Kit (Google) on-device, CoreML + Vision (iOS), and other APIs — combining them for the best cost-quality ratio. In 3–5 days you get a working prototype; in 3–6 weeks a ready solution with a custom model and training.

Choosing the Right Vision API

The choice depends on task, budget, and latency requirements. We have tested all major solutions:

• GPT-4o Vision (OpenAI). Pass an image as base64 or URL in the request, get a text response. Understands complex scenes, documents, handwriting, diagrams. Cost depends on image size (tile-based pricing). High-resolution detailed analysis is more expensive.
• Claude 3.5 Sonnet / Haiku. Similar capability via Anthropic Messages API. Works well with documents and tables.
• Google Cloud Vision API. Specialized functions: OCR (TEXT_DETECTION), object detection (OBJECT_LOCALIZATION), face detection, logo detection, content safety (SAFE_SEARCH_DETECTION). Cheaper than LLMs for repetitive tasks.
• ML Kit (Google) on-device. Fully on-device: text recognition, barcode scanning, face detection, object detection. No network latency and no cost per request. Accuracy is lower than cloud LLMs for complex scenes, but sufficient for structured tasks (QR, barcodes, document text).
• CoreML + Vision (iOS). MobileNetV3, EfficientNet for classification. VNRecognizeTextRequest for OCR, VNDetectBarcodeRequest for QR/barcodes.

Apple CoreML Vision documentation recommends using on-device models for tasks where speed and privacy are critical.

Sending Images from the Mobile App

Images cannot be sent directly to a Vision API from a mobile client because the API key must not be stored in the app.

Data flow:

Mobile Client → Resize/Compress → Upload to S3/GCS → URL → Your Server → Vision API 

The image is compressed on the device to the required size before upload. GPT-4o with detail: "auto" determines the needed resolution itself, but sending a 12-megapixel photo without compression is wasteful and expensive. Compressing to 1024px on the longest side with 85% JPEG quality reduces file size by 95%, from 3 MB to 150 KB, saving $0.008 per request. With 1 million requests per month, that's $8,000 savings in API costs alone. Response time drops from 1500ms to 400ms on 3G.

// Android: compress image before upload fun compressForBot(uri: Uri, maxSizePx: Int = 1024): ByteArray { val bitmap = MediaStore.Images.Media.getBitmap(contentResolver, uri) val scale = maxSizePx.toFloat() / maxOf(bitmap.width, bitmap.height) val scaled = if (scale < 1f) { Bitmap.createScaledBitmap( bitmap, (bitmap.width * scale).toInt(), (bitmap.height * scale).toInt(), true ) } else bitmap val output = ByteArrayOutputStream() scaled.compress(Bitmap.CompressFormat.JPEG, 85, output) return output.toByteArray() } 

Why Compression Matters

Without compression, each GPT-4o Vision request can cost 3–5 times more. Moreover, upload time for a large file increases by 2–4 seconds on slow connections. Compressing to 1024px on the longest side with 85% JPEG quality reduces file size by 95%, saving $0.008 per request. With 1 million requests per month, that's $8,000 savings in API costs alone. Response time drops from 1500ms to 400ms on 3G.

Application Scenarios

• Retail bots. User photographs a product — bot finds it in the catalog, shows price and availability. Visual embedding search (CLIP + Qdrant) is more accurate than text from OCR.
• Medical bots. Photo of a symptom, prescription, or test result — bot explains (does not diagnose). The system prompt must explicitly limit the response scope and include a disclaimer.
• Document bots. Photo of an invoice, bill, passport — extract structured data. GPT-4o Vision with structured output via JSON Schema provides high accuracy on typical documents.
• Inspection bots. Builder photographs a defect — bot classifies defect type and creates a task in the management system.

Real-World Case: Retail Bot with On-Device Classification

For a retail chain, we implemented a bot that identifies products from user photos. We used on-device ML Kit for barcode scanning and a custom MobileNet model for product classification. This eliminated API costs for barcode queries (60% of all requests) and reduced latency from 800ms to under 50ms. The per-request cost dropped by 60% while maintaining 99% accuracy on barcode reads. The bot saved the company over $12,000 annually in API fees.

Handling Bad Photos

Mandatory test cases:

• Blurry image
• Poor lighting
• Irrelevant photo (user sends a cat instead of a receipt)
• Image with prohibited content

For the last case — moderation before sending to the main model. OpenAI Moderation API or Google Safe Search as a first filter.

How We Work

1. Scenario analysis — define recognition tasks, choose Vision API, evaluate budget.
2. Server architecture — design and develop backend for image upload, API integration.
3. Mobile development — implement camera/gallery module, compression, upload.
4. UI/UX — chat interface with preview, loading indicator.
5. Testing — validate on real data, bad photos, edge cases.
6. Documentation and training — deployment instructions, team support.

What's Included

Timeline Estimates

Bot with basic Vision API (Google Vision or GPT-4o) — 3–5 days from $2,000. With custom classification model, on-device inference, and complex scenarios — 3–6 weeks from $8,000. On-device processing can save up to $0.002 per request compared to cloud APIs. For high-volume applications (e.g., 2 million requests monthly), that's $4,000 savings per month.

Contact us to evaluate your project. Get a consultation on model selection and budget optimization. Our experience: 5+ years in mobile app development, 50+ projects with Vision API. We guarantee support and solution updates.