Imagine a user types "dog on the beach" and the app instantly shows all photos of a dog on the beach—no tags, no manual sorting. That's semantic image search powered by CLIP (Contrastive Language-Image Pre-training) from OpenAI. We implement it entirely on-device: images and texts are converted into 512-dimensional vectors, and cosine similarity between them determines relevance. No data leaves the device—user privacy is preserved. Our experience with MobileCLIP-S0 and vector databases allows us to deploy such a solution in 1–4 weeks.
Evaluate the potential for your app. Savings compared to cloud-based search reach 80% due to zero inference costs. On-device CLIP search is 5x faster than cloud thanks to local inference. The ROI for on-device AI typically ranges from 6 to 12 months. Typical project cost: from $5,000 for basic integration to $15,000 for a full solution. Get a consultation to discuss details.
Architecture and Benefits of On-Device AI
The pipeline consists of two independent stages. Indexing: one-time for the whole gallery, then incremental. For each photo, compute a CLIP Image Embedding (512-dimensional vector) using a ViT-B/32 backbone, normalized to unit L2 norm, and store it in a local vector database. Search: on every user query, convert the user query into a CLIP Text Embedding (same 512-dimensional vector), perform ANN search (e.g., HNSW) for nearest vectors in the database, return photos sorted by descending cosine similarity.
| Comparison | On-device | Cloud |
|---|---|---|
| Latency | 15–20 ms | 200–500 ms |
| Cost per 10k queries | $0 | $100–$500 |
| Privacy | Data on device | Data on server |
On-device AI outperforms cloud: no network latency, zero inference cost, full user privacy. For an app with 10,000 daily active users, cloud costs can exceed $5,000 per month, while on-device costs nothing—up to 80% savings compared to cloud solutions, quick ROI.
Integrating CLIP via CoreML
Apple hasn't included CLIP in the standard Vision framework, but Apple ML Research released ml-mobileclip—a distilled version specifically for mobile devices. MobileCLIP-S0: 18 MB, 3–5 ms per image inference on iPhone 14. Source: Apple ML Research, MobileCLIP.
MobileCLIP on GitHub (official repository with code).
CLIP uses a contrastive learning objective to align image and text embeddings in a shared multimodal space, enabling zero-shot classification and retrieval. The model employs a contrastive loss that pulls matching image-text pairs together and pushes non-matching ones apart in the embedding space.
import CoreML
class MobileCLIPEmbedder {
private let imageEncoder: MobileCLIPImageEncoder
private let textEncoder: MobileCLIPTextEncoder
func embedImage(_ cgImage: CGImage) throws -> [Float] {
let resized = resize(cgImage, to: CGSize(width: 256, height: 256))
let input = MobileCLIPImageInput(image: MLMultiArray(from: resized))
let output = try imageEncoder.prediction(input: input)
return l2Normalize(output.embedding.toFloatArray())
}
func embedText(_ query: String) throws -> [Float] {
let tokens = tokenize(query) // BPE tokenizer
let input = MobileCLIPTextInput(tokens: MLMultiArray(from: tokens))
let output = try textEncoder.prediction(input: input)
return l2Normalize(output.embedding.toFloatArray())
}
}
The tokenizer for CLIP is BPE (Byte Pair Encoding). A Swift implementation is available in the ml-mobileclip repository. On Android: ONNX Runtime with MobileCLIP—less straightforward but works.
How to Integrate CLIP into an iOS App?
Step-by-step guide for implementing semantic search:
- Prepare the model. Download MobileCLIP-S0 from the Apple ML repository. Convert to CoreML using coremltools.
- Integrate the encoders. Create classes for processing images and text, as in the example above. Ensure normalization and tokenization match.
- Set up indexing. Use BGProcessingTask for background indexing of the gallery. Save embeddings to a vector database (e.g., sqlite-vss).
- Implement search. On receiving a query, compute the text embedding and perform ANN search. Return sorted results.
Example of background indexing:
class GalleryIndexer {
private var lastIndexedDate: Date {
get { UserDefaults.standard.object(forKey: "lastIndexedDate") as? Date ?? .distantPast }
set { UserDefaults.standard.set(newValue, forKey: "lastIndexedDate") }
}
func indexNewPhotos() async {
let fetchOptions = PHFetchOptions()
fetchOptions.predicate = NSPredicate(format: "creationDate > %@", lastIndexedDate as CVarArg)
let newPhotos = PHAsset.fetchAssets(with: .image, options: fetchOptions)
newPhotos.enumerateObjects { [weak self] asset, _, _ in
guard let self else { return }
if let embedding = self.computeEmbedding(for: asset) {
self.vectorDB.insert(assetId: asset.localIdentifier, embedding: embedding)
}
}
lastIndexedDate = Date()
}
}
Search completes in ~20 ms: text embedding (5 ms) + ANN search (15 ms). Results are instant.
func search(query: String, topK: Int = 30) async throws -> [PHAsset] {
let textEmbedding = try mobileCLIP.embedText(query)
let results = vectorDB.search(vector: textEmbedding, limit: topK)
let fetchOptions = PHFetchOptions()
fetchOptions.predicate = NSPredicate(
format: "localIdentifier IN %@",
results.map { $0.assetId }
)
let assets = PHAsset.fetchAssets(with: fetchOptions)
let idToScore = Dictionary(uniqueKeysWithValues: results.map { ($0.assetId, $0.score) })
return assets.objects(at: IndexSet(0..<assets.count))
.sorted { idToScore[$0.localIdentifier, default: 0] > idToScore[$1.localIdentifier, default: 0] }
}
How We Choose a Vector Database on Device
For searching among 50,000 vectors, an ANN index is needed. ANN indexes like sqlite-vss use quantization and hierarchical navigable small world (HNSW) graphs to accelerate search. Consider three options with concrete characteristics:
| Technology | Performance | Integration Complexity | Database Size |
|---|---|---|---|
| SQLite + sqlite-vss | 15–20 ms per search | Medium (SQL extension) | 10k–50k |
| FAISS (C++ via JNI/Swift) | 5–10 ms per search | High (platform-specific build) | 50k–500k |
| Flat L2 via Accelerate | 15 ms per 10k vectors | Low (standard library) | up to 10k |
SQLite with the sqlite-vss extension adds virtual tables for vector search. Compact, works in embedded mode:
CREATE VIRTUAL TABLE photo_embeddings USING vss0(embedding(512));
INSERT INTO photo_embeddings(rowid, embedding) VALUES (42, json('[0.1, -0.3, ...]'));
SELECT rowid, distance FROM photo_embeddings WHERE vss_search(embedding, json('[0.2, -0.1, ...]')) LIMIT 20;
Simple flat L2/cosine via Accelerate for galleries up to 10k photos is sufficient without a specialized index:
func cosineSimilarity(_ a: [Float], _ b: [Float]) -> Float {
var dotProduct: Float = 0
vDSP_dotpr(a, 1, b, 1, &dotProduct, vDSP_Length(a.count))
return dotProduct // After L2-normalization, equals cosine similarity
}
Brute-forcing 10,000 512-dimensional vectors on iPhone 14 via vDSP_dotpr takes ~15 ms. Acceptable for galleries up to 20k.
Multilingual Search
CLIP is trained predominantly on English. For a Russian query "собака на пляже", quality is worse than for "dog on beach". Solution: translate the query using a simple dictionary of common words or Google Translate API before embedding. In practice, translating 100–200 frequent queries offline is sufficient.
What's Included and How Long Does It Take?
| Task | Timeline |
|---|---|
| Basic CLIP search with flat index for galleries up to 10k | 1–1.5 weeks |
| Scalable implementation with ANN index, incremental updates, multilingual support, and visual search by reference photo | 3–4 weeks |
Cost is calculated individually based on integration complexity and target devices. Get a consultation for an accurate estimate.
Why Trust Us with This Task?
Our team has 6+ years of experience in mobile ML and has delivered 15+ on-device AI projects, establishing a strong track record since 2018. We guarantee compliance with App Store Review Guidelines (sections 4.2 and 5.1) and user data security. We provide full documentation and post-deployment support.