AI-Powered Document and Text Input for Mobile Apps

Task: Extract Data from a Document and Pass to an LLM

A user opens a mobile app, attaches a PDF contract, and asks: "What is the termination period of the agreement?" At first glance, a typical scenario. But between file_picker and a meaningful model response, there are a dozen non-trivial decisions: from rendering each page of a scan to chunking a 100-page file that doesn't fit into the context. We implemented such functionality for several projects—financial, legal, medical—and know every pitfall. A mistake in one link and the model's response becomes meaningless, or you risk getting a rejection for non-compliance with App Store Review Guidelines regarding user content. To avoid this, let's break down the key stages: choosing the transfer method, handling scans, and organizing RAG for large volumes.

How to Pass a Document to an LLM?

Most LLMs accept text, not PDF. Conversion is needed. We consider two main approaches.

Direct Upload via Files API

OpenAI Assistants API and Gemini Files API accept PDF, DOCX, TXT directly. For a mobile app, this is the cleanest path: upload file, get file_id, insert into messages[]. However, there are limitations—OpenAI has a 512 MB per file limit and 100 files per assistant, and Files API is tied to Assistants/Batch, not Chat Completions.

Client-Side Text Extraction

For PDF on Android—PdfRenderer (built-in since API 21) for rendering pages to Bitmap + OCR via ML Kit TextRecognizer, or an Apache PDFBox port. On iOS—PDFKit + PDFPage.string for typewritten PDF; for scans—Vision framework with VNRecognizeTextRequest. Text goes into content[] as a string. PDFKit documentation

Problem with Scanned Documents

PDFKit.string returns an empty string for PDFs consisting of scanned pages—there is no text layer. ML Kit TextRecognizer handles it, but you need to render each page to Bitmap/CGImage and run OCR. For a 50-page document, this takes 2–5 seconds on device.

What to Do with Scans and Large Files?

Text Extraction: Pitfalls

On Android, PdfRenderer requires a ParcelFileDescriptor with the MODE_READ_ONLY flag. If the file arrives via a content:// URI from FileProvider, you need contentResolver.openFileDescriptor(). A direct File() from content:// throws FileNotFoundException—a common mistake for those unfamiliar with SAF (Storage Access Framework).

Multi-page documents must be processed page by page without loading everything into memory at once. PdfRenderer.Page must be closed after each page—page.close() is mandatory, otherwise an IllegalStateException occurs on the next iteration.

On iOS, PDFDocument(url:) can return nil for encrypted PDFs. Handle isEncrypted and request the password via UI rather than crashing silently.

Architectural Solution for Large Documents

The full text of a 100-page contract won't fit into the context window of most models—or it will fit, but at a high cost. The right path for large documents is RAG: split into chunks of 500–1000 tokens with an overlap of 50–100 tokens, index in a vector DB, retrieve the top 5 relevant chunks on query, and pass only those into context. Token savings of up to 40% compared to passing the full text directly. For documents up to 10 pages, client-side extraction works 3 times faster than uploading via Files API with response waiting.

For a mobile app, this usually means server-side processing: the client uploads the file to the backend, which handles chunking and embeddings. Only the query UI and response rendering remain on the client. Implementing vector search directly on the phone makes sense only for offline scenarios.

Comparison of Approaches

Formats and Limits

DOCX on iOS without third-party libraries is painful. Either server-side conversion (LibreOffice headless) or limit format support to PDF + TXT for the mobile client.

What Is Included in Turnkey Work

• Audit of document formats in your product
• Selection of optimal strategy (Files API vs client extraction vs RAG)
• Implementation of file upload (file_picker, SAF, UIDocumentPickerViewController)
• Text conversion and cleaning (OCR for scans)
• Integration with LLM (OpenAI / Gemini / Anthropic)
• Progress indicators for long operations
• Testing on real documents of varying quality
• Documentation and team training

Why Choose Us

We are a team of mobile developers with 5+ years of experience in creating AI solutions for iOS and Android. We have implemented over 20 integrations of multimodal input for financial, legal, and medical projects. We guarantee code quality and adherence to deadlines. Contact us to assess your project—we will select the optimal solution for your budget.

Timelines: basic support for PDF + TXT with direct transfer — 1–2 weeks. Full pipeline with OCR, multiple formats, and RAG for large documents — 4–6 weeks. We will evaluate your project for free—reach out.

Machine Learning in Mobile Apps: CoreML, TFLite, and On-Device Models

We distinguish two fundamentally different approaches: an app with on-device AI and an app that simply calls a cloud API. The former works without internet, does not send user data to third-party servers, and responds within 50 milliseconds. The latter depends on network latency and pricing plans. Choosing the architecture is a key step that directly affects cost, privacy, and user experience in machine learning in mobile apps. Our experience shows that in 70% of projects, on-device inference is cheaper in the long run due to eliminating server costs.

How to Choose Between CoreML and TFLite for On-Device Inference?

CoreML — Apple's native framework for running ML models on device. Supports Neural Engine (starting with A11 Bionic), GPU, and CPU as fallback. Models are converted to .mlmodel format via coremltools from PyTorch, ONNX, or TensorFlow. Conversion is not always trivial: custom layers require implementing MLCustomLayer, and INT8 quantization can sometimes noticeably reduce accuracy on specific data. We ensure the final model passes validation on real data before and after conversion.

TensorFlow Lite — cross-platform alternative for Android and Flutter. On Android it uses NNAPI (Neural Networks API) for hardware acceleration — since Android 10 NNAPI is more stable; before that it's better to explicitly use GPU delegate via GpuDelegate. A typical mistake: the model is trained on normalized data in range [0,1], but the app feeds [0,255] — inference runs but produces meaningless results without any error. We include an automatic input data validation module in the SDK.

For image classification, object detection, and segmentation tasks, ready-to-use optimized models are available. YOLOv8 in CoreML format runs detection on a 640×640 frame in 15–20 ms on iPhone 14 Neural Engine. MobileNetV3 on TFLite with GPU delegate runs around 8 ms on Pixel 7 for classification.

What If You Need Text Generation On-Device?

Running small language models on device has become a reality in the last few years. Apple Intelligence uses its own models via Private Cloud Compute, but for third-party developers other paths are available.

llama.cpp with Metal backend on iOS is a working approach for phi-3-mini (3.8B parameters, 4-bit quantization, ~2.3 GB). Inference: 15–25 tokens/second on iPhone 15 Pro. For integration in Swift, use the Swift Package llama.swift or a wrapper via C interface llama.h. The binary is not bundled with the app — the model is downloaded on first launch and stored in Application Support. Our certified developers configure incremental download to avoid blocking the first launch.

On Android, the analog is Google AI Edge (formerly MediaPipe LLM Inference API) supporting Gemma-2B. It works via GPU delegate, on Tensor G3 chip Pixel 8 Pro — about 20 tokens/second.

Limitations are real: models larger than 4B parameters are still slow on mobile devices. For complex reasoning tasks, on-device LLM falls behind GPT-4o in quality. A hybrid approach — on-device for short tasks and private data, cloud for complex queries — is often optimal. We will evaluate your case and propose a balance of performance and privacy — contact us.

How Does On-Device Inference Compare to Cloud in Terms of Cost and Performance?

On-device inference is typically 10x cheaper per request than cloud APIs for image recognition tasks, while also eliminating latency variability and privacy risks. The table below summarizes the trade-offs.

For an app with 100k MAU running 10 image recognitions per user per month, on-device inference can save up to $5,000 monthly compared to cloud API. Get a free consultation on your ML architecture today.

Integrating OpenAI API and Other Cloud Models

For scenarios where cloud inference is acceptable, integrating OpenAI, Anthropic, or Google Gemini is an HTTP client + streaming SSE. In Swift, AsyncThrowingStream is convenient for streaming responses. In Kotlin, use Flow.

Critically: API keys must never be stored in the app bundle. Even an obfuscated key can be extracted from the IPA in 10 minutes using strings or frida. Correct architecture: mobile app → your own backend → OpenAI API. The backend controls rate limiting, logs requests, and protects the key.

What Is Included in the Work (Deliverables)

• Trained and quantized model for the target device (documentation with metrics)
• SDK for integration (Swift/Kotlin/Flutter) with call examples
• Performance tests on 3–5 real devices
• Instructions for OTA model updates
• Support during App Store / Google Play moderation (compliance with Guidelines 4.2, 5.1)
• 2 weeks of technical support after release

Typical Project Pipeline

1. Task analysis — measure latency, privacy, size, supported devices.
2. Model prototyping — in Python, evaluate accuracy on target data.
3. Conversion and quantization — for CoreML/TFLite with validation.
4. Integration into the app — model wrapped in a service layer (easy to swap CoreML ↔ TFLite ↔ cloud).
5. Testing — on real devices, measure FPS, RAM, battery.
6. Deployment — via TestFlight / Firebase App Distribution, monitor metrics.

Timelines: integration of a ready CoreML/TFLite model — 1–2 weeks, development of a custom model with mobile optimization — from 6 weeks, on-device LLM chat with personalization — 4–8 weeks.

Why We Take on Complex Cases?

10+ years of experience in mobile development, 50+ implemented AI/ML solutions, guarantee of compatibility with current iOS and Android versions. All projects undergo code review and load testing. The cost includes preparation of moderation documentation and training of your team.

Contact us — we will help you choose the architecture and implement ML in your app turnkey. Order an audit of your existing solution — we will assess the potential for server cost savings free of charge. In some projects, savings can reach significant amounts per month.