AI Clothing Recognition and Similar Products Search in Mobile Applications
User sees a jacket on the street they want to buy—photographs it and instantly finds it in online stores or analogues. This is visual search in fashion: Pinterest, ASOS, and Zalando solved internally, but available via APIs or SDKs through several services.
Technical Stack
Two independent blocks: attribute recognition of clothing and similar product search by image vector.
Attribute recognition—category (jacket, dress, sneakers), color, pattern (stripe, check, solid), style (casual, formal). DeepFashion2 dataset, Fashionpedia annotations work well. Ready APIs: Google Vision AI (clothing detection), Clarifai Fashion Model, Snap ML Kit.
Similar search—similarity search task: image → embedding vector → nearest neighbors in product base. Backbone—ViT (Vision Transformer) or ResNet50, fine-tuned on fashion dataset. For vector search: Pinecone, Weaviate, or pgvector for catalogs up to 1–2 million items.
// iOS: pipeline from photo to search results
class FashionSearchService {
func searchSimilar(image: UIImage) async throws -> FashionSearchResult {
// 1. Clothing detection and crop
let detectedItems = try await detectFashionItems(image: image)
guard let primaryItem = detectedItems.first else {
throw FashionError.noClothingDetected
}
// 2. Crop by bounding box
let croppedImage = image.cropped(to: primaryItem.boundingBox)
// 3. Parallel: attributes + embedding
async let attributes = extractAttributes(croppedImage)
async let embedding = generateEmbedding(croppedImage)
// 4. Vector search via backend
let (attrs, vec) = try await (attributes, embedding)
let similarProducts = try await vectorSearch(
embedding: vec,
filters: SearchFilters(
category: attrs.category,
priceRange: nil // price filter optional
)
)
return FashionSearchResult(
detectedItem: primaryItem,
attributes: attrs,
similarProducts: similarProducts
)
}
}
Segmentation with Multiple Clothing Items
Often full outfits appear: jacket, jeans, sneakers. Segmentation searches each element separately.
struct DetectedFashionItem {
let category: FashionCategory // .top, .bottom, .shoes, .accessory
let boundingBox: CGRect
let confidence: Float
let attributes: FashionAttributes?
struct FashionAttributes {
let color: [String] // ["navy blue", "dark"]
let pattern: PatternType // .solid, .stripes, .plaid
let style: StyleTag // .casual, .sport, .formal
}
}
User selects what to search—tap detected outfit element. Better than auto-selecting "biggest object."
Catalog Indexing
For own shop catalog search—need pre-indexing. Per product card: image → embedding → vector store entry with metadata (SKU, price, category, color, availability).
# Backend: catalog product indexing
async def index_product(product: Product, image_url: str):
# Download and preprocess
image = await download_and_preprocess(image_url)
# Embedding via fashion-specific model
embedding = fashion_encoder.encode(image) # numpy array [512]
# Write to Pinecone
await pinecone_index.upsert(vectors=[{
"id": str(product.sku),
"values": embedding.tolist(),
"metadata": {
"category": product.category,
"color": product.color,
"brand": product.brand,
"price": product.price,
"in_stock": product.in_stock,
"image_url": product.thumbnail_url,
"product_url": product.url
}
}])
Metadata filtering on search (in_stock: true) is critical—showing "similar" without availability is pointless.
Timeline Estimates
Integrating ready API (Google Vision + external marketplace) with basic search UI—1 week. Full implementation with own vector store for catalog indexing, segmentation of multiple items, attribute filtering, and iOS + Android support—1–2 months including catalog indexing.