Furniture Search by Photo: AI Recognition in Mobile Apps

Furniture Search by Photo: AI Recognition in Mobile Apps

A client wants users to snap a sofa and get similar models with prices. But furniture isn't clothing: visually identical chairs can differ in style and dimensions. A mistake in image classification leads to false recommendations. We solve this with a combination of transfer learning on convolutional networks and vector search. Our experience shows that a properly tuned model saves up to 40% of catalog manual processing time, and catalog maintenance costs drop by 25% through automation.

Why Furniture Is Harder Than Clothing

Furniture has rigid geometry and recognizable shapes — that simplifies classification. But searching for "similar for less money" requires not just visual match but also understanding style (Scandinavian minimalism, loft, classic) and scale. A sofa photo without context doesn't reveal whether it's a three-seater or a two-seater. We address this with additional furniture style classifiers and attribute filtering.

Source: recommendations from open furniture image datasets (e.g., Furniture-180).

Problems We Solve

1. Categorization: Most off-the-shelf models (Google Cloud Vision, AWS Rekognition) are trained on ImageNet, which includes plenty of furniture. But accuracy drops for rare forms. We fine-tune EfficientNet or MobileNetV3 on the store's catalog: 50,000 images (200–300 per category) yield reliable classification of main categories: sofa, armchair, table, chair, wardrobe, bed, nightstand.
2. Style detection: Category is just the first step. For similarity search, style matters. We use CLIP, which understands text descriptions: "Scandinavian minimalist", "loft", "classic". CLIP compares the image embedding with text style embeddings and returns the most likely.
3. Filtered search: The architecture is the same as for clothing: embedding → vector search. But for furniture, filtering by material (wood, metal, fabric), color, and size group is crucial. Size class without AR is approximated by aspect ratio and typical category proportions.

Comparison of Classification Approaches

Transfer learning is the optimal choice for most retailers: 92% accuracy with minimal time investment. It outperforms ready APIs by 15% on average in our tests. And it requires 2-3 times less data than training from scratch at comparable accuracy. We use MobileNetV3 or EfficientNet-Lite — they run directly on the device.

How to Determine Furniture Style?

CLIP allows quick adaptation to a new style without retraining — just add a text description. More about CLIP in the official repository.

How We Do It: Stack and Examples

For Android we use TFLite with XNNPACK optimization. Example classifier:

// Android: TFLite furniture classifier class FurnitureClassifier(context: Context) { private val interpreter: Interpreter by lazy { val model = FileUtil.loadMappedFile(context, "furniture_classifier_v2.tflite") Interpreter(model, Interpreter.Options().apply { numThreads = 4 useXNNPACK = true }) } fun classify(bitmap: Bitmap): List<FurnitureClassification> { val resized = Bitmap.createScaledBitmap(bitmap, 224, 224, true) val input = TensorImage.fromBitmap(resized) val output = TensorBuffer.createFixedSize(intArrayOf(1, NUM_CLASSES), DataType.FLOAT32) interpreter.run(input.buffer, output.buffer) return output.floatArray .mapIndexed { index, score -> FurnitureClassification(LABELS[index], score) } .filter { it.score > 0.1f } .sortedByDescending { it.score } } } 

For iOS — Core ML. CLIP enables style detection via text prompts:

// iOS: CLIP-based style detection via Core ML // CLIP model converted to .mlpackage func detectStyle(_ image: UIImage) async throws -> [StyleScore] { let styleDescriptions = [ "scandinavian minimalist furniture", "industrial loft style furniture", "classic traditional furniture", "mid-century modern furniture", "boho eclectic furniture" ] // CLIP compares image embedding with text embeddings of styles let imageEmbedding = try await clipEncoder.encodeImage(image) return styleDescriptions.enumerated().map { i, desc in let textEmbedding = clipEncoder.encodeText(desc) let similarity = cosineSimilarity(imageEmbedding, textEmbedding) return StyleScore(style: desc, score: similarity) }.sorted { $0.score > $1.score } } 

How Does Similar Item Search Work in the Catalog?

Product embeddings are stored in a vector database (e.g., FAISS or Pinecone). Search involves several steps:

1. Extract embedding: the user's image goes through the same model as the catalog.
2. Vector search: cosine distance to all catalog embeddings.
3. Apply filters: filter by category, style, material, color, size, and price.
4. Post-processing: sort by similarity and return top-10 results.

struct FurnitureSearchFilters { let category: FurnitureCategory let style: StyleTag? let colorFamily: ColorFamily? // warm, cool, neutral let material: MaterialType? // wood, metal, upholstered let maxDimensionClass: SizeClass? // compact, standard, large let priceRange: ClosedRange<Int>? let inStockOnly: Bool } 

Integration Process

Implementing recognition and similar-item search goes through several stages: