Furniture Search by Photo: AI Recognition in Mobile Apps
A client wants users to snap a sofa and get similar models with prices. But furniture isn't clothing: visually identical chairs can differ in style and dimensions. A mistake in image classification leads to false recommendations. We solve this with a combination of transfer learning on convolutional networks and vector search. Our experience shows that a properly tuned model saves up to 40% of catalog manual processing time, and catalog maintenance costs drop by 25% through automation.
Why Furniture Is Harder Than Clothing
Furniture has rigid geometry and recognizable shapes — that simplifies classification. But searching for "similar for less money" requires not just visual match but also understanding style (Scandinavian minimalism, loft, classic) and scale. A sofa photo without context doesn't reveal whether it's a three-seater or a two-seater. We address this with additional furniture style classifiers and attribute filtering.
Source: recommendations from open furniture image datasets (e.g., Furniture-180).
Problems We Solve
- Categorization: Most off-the-shelf models (Google Cloud Vision, AWS Rekognition) are trained on ImageNet, which includes plenty of furniture. But accuracy drops for rare forms. We fine-tune EfficientNet or MobileNetV3 on the store's catalog: 50,000 images (200–300 per category) yield reliable classification of main categories: sofa, armchair, table, chair, wardrobe, bed, nightstand.
- Style detection: Category is just the first step. For similarity search, style matters. We use CLIP, which understands text descriptions: "Scandinavian minimalist", "loft", "classic". CLIP compares the image embedding with text style embeddings and returns the most likely.
- Filtered search: The architecture is the same as for clothing: embedding → vector search. But for furniture, filtering by material (wood, metal, fabric), color, and size group is crucial. Size class without AR is approximated by aspect ratio and typical category proportions.
Comparison of Classification Approaches
| Approach | Accuracy | Training Time | Dataset Size |
|---|---|---|---|
| Ready API (Google Cloud Vision) | 75-85% | 0 | Not required |
| Transfer Learning (MobileNetV3) | 92-96% | 2-4 days | 500–1000 per category |
| Training from scratch (ResNet) | 96%+ | 2-4 weeks | 10,000+ per category |
Transfer learning is the optimal choice for most retailers: 92% accuracy with minimal time investment. It outperforms ready APIs by 15% on average in our tests. And it requires 2-3 times less data than training from scratch at comparable accuracy. We use MobileNetV3 or EfficientNet-Lite — they run directly on the device.
How to Determine Furniture Style?
| Method | Style Accuracy | Labeling Required |
|---|---|---|
| Visual features only (CNN) | 70-80% | 500+ images per style |
| CLIP (text+image) | 85-95% | Text descriptions of styles |
| Hybrid (CLIP + custom classifier) | 90-97% | 200 images per style |
CLIP allows quick adaptation to a new style without retraining — just add a text description. More about CLIP in the official repository.
How We Do It: Stack and Examples
For Android we use TFLite with XNNPACK optimization. Example classifier:
// Android: TFLite furniture classifier
class FurnitureClassifier(context: Context) {
private val interpreter: Interpreter by lazy {
val model = FileUtil.loadMappedFile(context, "furniture_classifier_v2.tflite")
Interpreter(model, Interpreter.Options().apply {
numThreads = 4
useXNNPACK = true
})
}
fun classify(bitmap: Bitmap): List<FurnitureClassification> {
val resized = Bitmap.createScaledBitmap(bitmap, 224, 224, true)
val input = TensorImage.fromBitmap(resized)
val output = TensorBuffer.createFixedSize(intArrayOf(1, NUM_CLASSES), DataType.FLOAT32)
interpreter.run(input.buffer, output.buffer)
return output.floatArray
.mapIndexed { index, score -> FurnitureClassification(LABELS[index], score) }
.filter { it.score > 0.1f }
.sortedByDescending { it.score }
}
}
For iOS — Core ML. CLIP enables style detection via text prompts:
// iOS: CLIP-based style detection via Core ML
// CLIP model converted to .mlpackage
func detectStyle(_ image: UIImage) async throws -> [StyleScore] {
let styleDescriptions = [
"scandinavian minimalist furniture",
"industrial loft style furniture",
"classic traditional furniture",
"mid-century modern furniture",
"boho eclectic furniture"
]
// CLIP compares image embedding with text embeddings of styles
let imageEmbedding = try await clipEncoder.encodeImage(image)
return styleDescriptions.enumerated().map { i, desc in
let textEmbedding = clipEncoder.encodeText(desc)
let similarity = cosineSimilarity(imageEmbedding, textEmbedding)
return StyleScore(style: desc, score: similarity)
}.sorted { $0.score > $1.score }
}
How Does Similar Item Search Work in the Catalog?
Product embeddings are stored in a vector database (e.g., FAISS or Pinecone). Search involves several steps:
- Extract embedding: the user's image goes through the same model as the catalog.
- Vector search: cosine distance to all catalog embeddings.
- Apply filters: filter by category, style, material, color, size, and price.
- Post-processing: sort by similarity and return top-10 results.
struct FurnitureSearchFilters {
let category: FurnitureCategory
let style: StyleTag?
let colorFamily: ColorFamily? // warm, cool, neutral
let material: MaterialType? // wood, metal, upholstered
let maxDimensionClass: SizeClass? // compact, standard, large
let priceRange: ClosedRange<Int>?
let inStockOnly: Bool
}
Integration Process
Implementing recognition and similar-item search goes through several stages:
More details on stages
- Catalog analysis: estimate image volume, categories, styles.
- Data preparation: label 200–500 images per category/style (if fine-tuning is needed).
- Model training: transfer learning on TensorFlow/Keras or Create ML.
- App integration: connect TFLite/Core ML, set up vector storage.
- Testing: A/B test recognition accuracy and search speed.
- Deployment: release via App Store/Google Play, monitor.
A minimum viable product (MVP) can be launched in 1 week using a ready-made recognition API and cloud vector storage. A full solution with a custom model, CLIP, and AR measures takes 1–2 months. For example, for a client with 10,000 catalog items, we fine-tuned MobileNetV3 on 500 images per category. Classification accuracy rose from 72% to 94%.
Scope of Work
- MVP: integration with a ready API (Google Cloud Vision + vector search) — from 1 week.
- Full solution: fine-tuned TFLite/CoreML model, CLIP style classifier, vector store with filters, AR dimensions (LiDAR) — 1–2 months.
- Model and API documentation.
- Client team training.
- Post-release support for 2 weeks.
Our team has 7+ years of experience in mobile development and machine learning. We have completed 30+ projects in image recognition. We guarantee quality: the model will work offline without delays. If you want to implement a similar feature, contact us — we'll find the optimal solution. Get a free consultation on integration — we'll assess your project at no cost.